TL;DR: Kafka is being positioned as the durable commit log for agentic AI because replay, auditability, ordering, and schema governance matter more than transient event streams, according to Kong. The governance lesson is that agent memory is now an identity and control-plane problem, not just a data plumbing choice.
At a glance
What this is: NHI Mgmt Group's analysis, drawing on Kong's blog post: Kafka is the durable memory layer for agentic AI, with replay, auditability, ordering and schema governance as the central requirements.
Why it matters: IAM, PAM and NHI teams should care because the memory layer now carries governance evidence, access lineage and control decisions for agent workflows, not just application events.
Context
Agentic AI memory is the durable record of what an agent did, in what order, and with what context. Kong argues that this record must behave like a governed commit log rather than a disposable event stream, because replay, auditability and causality all depend on retention and ordering.
That shifts the problem from infrastructure convenience to identity governance. When an agent can take actions over time, the memory layer becomes part of the control plane for NHI governance, access review evidence and post-incident reconstruction, especially where compliance teams need a trustworthy history of tool use and context changes.
Key questions
Q: How should teams govern an agent memory layer built on Kafka?
A: Teams should govern it as a durable system of record, not as a messaging sidecar. That means enforcing topic access, schema compatibility, retention rules and audit lineage at the gateway or control plane, while preserving ordered records that can be replayed consistently for review, testing and compliance evidence.
Q: What breaks when agent memory is treated like ordinary event streaming?
A: Replay, auditability and causality break first. Ordinary event streams are usually optimised for throughput and short retention, but agent memory needs long-lived records, strict ordering and stable schemas so a past session can be reconstructed exactly and trusted by downstream consumers.
Q: When should security teams prioritise durable logs over ephemeral observability for AI agents?
A: Prioritise durable logs whenever agent actions can affect regulated workflows, model evaluation, investigations or access decisions. If the record must survive beyond the runtime session, observability is not enough because it is not designed to preserve the exact sequence of decisions as evidence.
Q: How do schema changes affect auditability in agent memory systems?
A: Schema changes can silently destroy auditability if older records can no longer be read or interpreted by downstream consumers. The fix is not just better storage. It is compatibility governance, so each event remains intelligible as the agent format evolves over time.
Technical breakdown
Why an agent memory layer needs more than event streaming
A memory layer for agentic systems is not the same as telemetry or queueing. It has to preserve the full sequence of agent decisions, tool calls and context shifts so the system can be replayed exactly as it ran. That means retention over long periods, strict ordering and schema governance that does not drift as the agent stack evolves. If any of those properties are weak, the log stops being evidence and becomes just another data feed. In governance terms, the memory layer has to be durable enough for audit and deterministic enough for reconstruction.
Practical implication: Design the agent memory layer as governed evidence storage, not disposable observability data.
How Kafka supports replay, ordering and retention for agents
Kafka’s value in this pattern comes from its log-first model. Topics retain data by default, partition ordering is explicit, and offsets let teams replay from a known point without reconstructing state from scattered logs. Tiered storage extends that model by making long retention economically feasible, which matters when agent sessions may need to be revisited months later. The architectural point is not that Kafka is the only possible log, but that its semantics align cleanly with the needs of a stable, replayable agent history.
Practical implication: Use offset-based replay and long-retention topics where agents need auditable, reconstructable histories.
Why governance belongs in the gateway, not the broker
A Kafka cluster alone does not solve agent governance. Kong’s article places the real control question at the gateway layer, where topic access, schema validation, PII redaction, retention policy and audit lineage can be enforced consistently before data reaches the broker. That separation matters because agent memory now sits on the boundary between identity, data governance and operational control. The deeper issue is not transport, but who is allowed to write which facts into the system of record for an agent’s behaviour.
Practical implication: Place policy enforcement in front of the log so memory writes are governed at ingestion, not after the fact.
Breaches seen in the wild
- CoPhish OAuth phishing via Copilot Studio: Datadog showed Copilot Studio agents on a Microsoft domain can front OAuth consent phishing and forward stolen tokens; no victims reported.
- Langflow Flodrix botnet 2025: Attackers used Langflow's unauthenticated code endpoint to dump AI servers' environment variables and enlist them in the Flodrix botnet.
Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.
NHI Mgmt Group analysis
Agent memory has become a governance surface, not a storage choice: once agent actions are durable, the memory layer starts carrying accountability, not just context. That means IAM, NHI and compliance teams need to treat log design as part of the control plane. The practical conclusion is that agent memory must be governed as evidence.
Durable replay changes the meaning of least privilege for agents: if an agent can be replayed, audited and reconstructed from a historical log, privilege scope must be intelligible over time, not only at execution. Kafka-style offsets and schema control make the history legible, but the governance model still has to explain what the agent was authorised to do at each step. Practitioners should align memory retention with authorisation traceability.
Schema governance is the difference between usable memory and expensive noise: without versioned schemas and compatibility rules, a long-lived agent log degrades into a collection of incompatible records that no consumer can trust. That creates a silent governance failure because the evidence exists but cannot be interpreted consistently. The practitioner takeaway is that schema discipline is part of audit readiness, not just data engineering hygiene.
Agent memory should be treated as a control-plane dependency across synchronous and asynchronous traffic: Kong’s framing usefully connects tool calls, model traffic and event logging into one governed path. That matters because identity boundaries do not stop at the prompt-response layer. The implication for teams is to model agent memory, gateway policy and downstream consumers as one lifecycle, not three separate systems.
Ordered memory creates a sharper accountability model for autonomous workflows: when every tool call and context shift is time-sequenced, post-incident review becomes more precise and harder to hand-wave. That benefits defenders, but it also raises the bar for policy clarity because the record will expose where governance failed. Teams should expect agent logs to become the primary source of truth for control validation.
From our research library:
- More than 80% of enterprises will have used generative AI APIs or deployed GenAI applications by 2026, up from 5% in 2023, according to Gartner.
- Read next: AI Agents vs Agentic AI
What this signals
Durable agent memory will force identity teams to think in lifecycle terms: once sessions are retained for audit and replay, the relevant question is no longer whether a tool call happened, but whether the record can still be trusted months later. That makes retention, schema compatibility and access lineage part of IAM and NHI governance, not a downstream data concern.
Agent memory is becoming the evidence layer for governance decisions: when action logs are durable, the strongest control is not after-the-fact review, but governed issuance of what may be written in the first place. Teams should prepare for logs to carry operational proof of authorisation, not just operational history.
For practitioners
- Define the agent memory boundary Separate transient observability data from the durable event log that will serve as the agent’s source of truth for replay and audit.
- Enforce schema compatibility at write time Require versioned schemas and compatibility checks before events are accepted into the log so downstream consumers do not break as agent formats evolve.
- Map agent sessions to ordered partitions Use a stable session_id-to-partition strategy so each agent session preserves causal order across tool calls, decisions and context shifts.
- Push access policy into the gateway layer Apply topic-level authorisation, retention policy, PII redaction and audit lineage before records reach Kafka, not after they are stored.
- Align replay capability with governance evidence Keep the historical record long enough to support audit, model evaluation and incident reconstruction without rebuilding state from separate systems.
Key takeaways
- Kafka is being positioned as the memory backbone for agentic systems because replay and auditability require a durable, ordered record rather than transient telemetry.
- The governance risk is not just retention. It is whether schema drift, uncontrolled writes or weak gateway policy make the memory layer unreliable as evidence.
- Identity and access teams should treat agent memory as a control-plane dependency, with policy enforced before records are accepted and replay kept available for review.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 and MITRE ATT&CK address the attack and risk surface, while NIST AI RMF and NIST CSF 2.0 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | ASI03 — Identity & Privilege Abuse | Agent memory governs how privileges and actions are recorded across sessions. |
| ASI06 — Memory & Context Poisoning | The article centres on durable memory, replay and schema integrity for agents. | |
| Recommendation — Treat memory writes as governed identity events and restrict agent privileges to approved paths. Protect agent memory from malformed or untrusted context by enforcing schema and write validation. | ||
| NIST AI RMF | GOVERN — AI Governance and Accountability | The article frames agent memory as an accountability and control-plane issue. |
| Recommendation — Define governance ownership for agent memory retention, replay and audit evidence. | ||
| NIST CSF 2.0 | PR.DS-01 — Data-at-rest is protected | Long-retained agent logs are governed data assets that need protection over time. |
| Recommendation — Protect stored agent records and preserve their integrity across the full retention period. | ||
| MITRE ATT&CK | TA0009 — Collection | Durable logs create a high-value collection target because they contain agent activity history. |
| Recommendation — Harden collection paths and monitor for unauthorised access to agent history records. | ||
Key terms
- Agent Memory: Agent memory is the stored context an AI agent uses across sessions or tasks. In governance terms, it is controlled state, because the memories an agent retains can influence future actions, permissions use, and the safety of subsequent decisions.
- Durable Commit Log: A durable commit log is an append-only record that preserves the sequence of actions in a form that can be replayed later. For agentic systems, it becomes the evidence layer for behaviour, not just a transport mechanism for events. Its value depends on retention, ordering, and schema stability.
- Schema Compatibility: Schema compatibility is the rule that newer event formats can still be understood by systems built for older versions. For agent memory, it prevents silent breakage in replay, audit and downstream consumers as the log evolves over time.
- Audit Lineage: The record that connects an identity to a session and then to the action it performed. In AI and NHI security, lineage is what makes attribution possible when multiple actors, tokens, and automations can use the same interfaces.
Deepen your knowledge
NHI governance, agentic AI identity, and machine identity lifecycle are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building or maturing an IAM programme, it is worth exploring.
Published by the NHIMG editorial team on June 25, 2026.
Updated on October 11, 2026.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org