Identity-aware Kafka ACLs expose the limits of static policies

At a glance

What this is: This is a Kong engineering post about replacing static Kafka ACLs with identity-aware policies driven by OAuth or JWT claims.

Why it matters: It matters because IAM, NHI, and platform teams increasingly need access decisions that reflect who the caller is, what they are entitled to, and how quickly those entitlements can change.

By the numbers:

• Only 5.7% of organisations have full visibility into their service accounts.

👉 Read Kong's engineering post on identity-aware Kafka ACLs

Context

Static Kafka ACLs define access at the broker level, which makes them hard to align with the identity context that modern platforms already know about. In practice, teams end up maintaining large topic and user lists that drift as environments, teams, and applications change. This is an identity governance problem as much as an infrastructure one, because the authorization model does not follow the caller's actual scope.

The post argues for moving access decisions into the identity layer, where token claims can carry entitlement data and the gateway can enforce it at request time. That shift is most relevant for NHI and workload identity programmes, where service principals, API clients, and automated producers need precise topic-level access without constant broker reconfiguration.

Key questions

Q: How should teams govern Kafka access when permissions depend on identity claims?

A: Treat the identity provider as the source of entitlement truth and the gateway as the enforcement point. That means defining claims carefully, validating token integrity, and making sure revocation follows lifecycle events. If teams keep Kafka ACLs as the primary policy store, identity-based access will stay brittle and slow to change.

Q: Why do static Kafka ACLs become risky in multi-team environments?

A: They create broad permissions, manual exceptions, and configuration drift because they cannot adapt to changing team scope or workload context. As the number of topics and clients grows, administrators often compensate with wildcards and over-permissioning. The result is a policy model that scales operationally, but not securely.

Q: What should security teams measure when using claim-based topic access?

A: Track how quickly entitlements change in the identity system, how often token claims are reviewed, and whether revoked access still works before token expiry. Those signals show whether the control is truly dynamic or just an old ACL model with a newer enforcement layer.

Q: Who should own Kafka authorization when access is driven by workload identity?

A: Ownership should sit with the IAM or identity platform team, with platform engineering responsible for enforcement plumbing and application teams responsible for declaring least-privilege needs. That separation keeps policy governance aligned with identity lifecycle, rather than scattered across broker administrators and application owners.

Technical breakdown

Why static Kafka ACLs fail in dynamic environments

Traditional Kafka ACLs are cluster-scoped and context-poor. They tell the broker who can reach which topic, but they do not understand team membership, request purpose, or environment-specific constraints. That makes them brittle when access needs to vary by client, topic pattern, or deployment stage. Once administrators start compensating with broad wildcards and manual exceptions, policy drift follows. The real architectural issue is that authorization is being expressed in the wrong layer, too far from the identity source that actually knows the caller's intent and scope.

Practical implication: Treat static broker ACLs as a control boundary that may need to be replaced or abstracted for identity-driven workloads.

How token claims can drive identity-aware ACLs

The Kong pattern moves authorization data into the JWT or OAuth token, then has the gateway extract claims and apply them to the ACL decision. In this model, the token becomes the portable entitlement container and the gateway becomes the enforcement point. That works because the gateway can validate the token via JWKS, read claims such as topic names or role scope, and resolve them at request time. This is a familiar identity pattern: authenticate once, encode policy in claims, and enforce it where the request is intercepted.

Practical implication: Use claim-based policy only where the token issuer, gateway, and entitlement model are tightly governed end to end.

Why dynamic policy changes alter the operating model

When access changes are made in the identity provider rather than in Kafka, the operational burden shifts from infrastructure change management to identity lifecycle management. Teams no longer need to update broker permissions for every topic adjustment, which reduces friction but also raises the stakes for claim design, scope naming, and token issuance controls. In other words, the access model becomes faster but also more dependent on trustworthy identity data. That is a better fit for modern NHI governance, but only if entitlement ownership and token freshness are tightly controlled.

Practical implication: Move entitlement approvals, reviews, and revocation logic into the identity process rather than treating Kafka as the source of truth.

NHI Mgmt Group analysis

Static ACLs create an identity blind spot when the caller is known at runtime. Kong's core point is not simply that ACLs are hard to maintain. It is that broker-level permission lists cannot express the identity context already available in tokens, so access policy lags behind the caller's actual state. For NHI governance, that means the control plane is making decisions with stale or incomplete identity information. Practitioners should treat that as a structural mismatch between authorization and identity source of truth.

Claim-driven authorization is a strong pattern for machine identities, but only when token governance is mature. The post shows how topic access can be derived from JWT claims and enforced at the gateway. That pattern is attractive because it aligns access with identity lifecycle changes, but it also concentrates trust in the token issuer, claim schema, and JWKS validation path. If those are weak, the policy becomes fast but fragile. The implication is that NHI teams must govern the claim structure as carefully as they govern the Kafka policy.

Identity-aware Kafka policy is really a workload identity programme decision. This is not just about streaming platforms. It is about deciding whether service accounts and API clients should inherit static entitlements or receive request-time access based on signed identity claims. The broader lesson is that modern NHI programmes need authorization models that can follow workload churn without manual ACL sprawl. Practitioners should evaluate streaming access as part of the wider workload identity architecture, not as an isolated broker issue.

Dynamic topic access exposes the governance gap between entitlement review and runtime enforcement. ACLs that are only updated when administrators remember to change them create a standing-privilege problem in a different form. The article's model reduces that lag by binding access to the identity provider, but it also makes the quality of reviews, scopes, and revocation timeliness more important. The practitioner conclusion is simple: if entitlements are not lifecycle-managed, dynamic enforcement only moves the failure point upstream.

Identity context should define Kafka access, but not at the expense of auditability. A token-based model is more expressive than static ACLs, yet it can become opaque if teams cannot explain why a given claim resulted in topic access. That makes governance, logging, and entitlement traceability non-negotiable. The field should interpret this pattern as a shift from permission lists to policy evidence. Practitioners should be able to reconstruct every access decision from the token, the claim, and the enforcement rule.

From our research:

• Only 5.7% of organisations have full visibility into their service accounts, according to the Ultimate Guide to NHIs.
• Only 97% of NHIs carry excessive privileges, increasing unauthorised access and broadening the attack surface, according to the Ultimate Guide to NHIs.
• For the governance angle behind this pattern, see Top 10 NHI Issues for the controls teams most often miss.

What this signals

Identity-aware policy will keep expanding beyond Kafka because teams want access decisions to follow the caller, not the cluster. The practical question for programmes is whether token claims are governed as first-class entitlements or treated as implementation detail. Where claims become the policy source, access reviews, revocation, and audit trails must move into the identity operating model. For a broader control baseline, compare the approach with the Ultimate Guide to NHIs.

Claim-based authorization creates a new governance asset: the entitlement schema itself. If team, topic, and environment claims are inconsistent, access decisions will be inconsistent too. That makes schema design, ownership, and validation as important as broker configuration. Teams that already struggle with service account visibility should expect similar pressure here, especially given our research showing that only 5.7% of organisations have full visibility into their service accounts.

Topic access is becoming a workload identity problem, not just an event-streaming problem. Security leaders should watch for the same pattern in other systems where request-time claims can replace static permissions. The governance payoff is clearer entitlement boundaries, but only if logging can explain each decision and revocation can outpace token validity. For policy design guidance, the Top 10 NHI Issues resource is the right companion reference.

For practitioners

• Map Kafka entitlements to identity source of truth Document which identity provider owns topic-level claims, which team approves them, and how those claims are revoked when roles or projects change.
• Replace hardcoded topic lists with scoped claims Use short-lived tokens carrying explicit topic or pattern claims, then validate that the gateway resolves them consistently across environments and client types.
• Audit wildcard access and policy drift Look for topic-* permissions, manual broker exceptions, and duplicated ACLs that indicate access is being managed outside the identity layer.
• Tie revocation to lifecycle events Ensure topic access is removed when a workload, application, or team no longer needs it, and verify that token expiry supports the intended revocation window.

Key takeaways

• Static Kafka ACLs fail because they cannot express the identity context that modern workloads already carry in tokens.
• Identity-driven policy reduces ACL sprawl, but it shifts governance pressure onto claim design, revocation, and auditability.
• Teams should treat Kafka topic authorization as part of the broader workload identity programme, not as an isolated broker setting.

Key terms

• Identity-aware policy: An authorization model that uses identity context, such as claims, roles, or scopes, to decide what a caller may access. In this pattern, the policy follows the identity at request time instead of relying only on static infrastructure rules, which improves precision but raises governance requirements.
• Kafka ACL: A Kafka access control list is a broker-level rule that allows or denies actions on topics, groups, or clusters. It is effective for simple environments, but it becomes difficult to govern when access must vary by team, environment, or workload context.
• Token claim: A token claim is a statement embedded in an authentication token that describes something about the caller, such as scope, role, or permitted resources. For NHI and workload identity use cases, claims can become the basis for authorization decisions if they are tightly governed and validated.
• Workload identity: Workload identity is the identity assigned to a service, application, API client, or other non-human runtime. It provides a way to authenticate and authorize machine actors without relying on human credentials, but it still requires lifecycle, scope, and revocation controls.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity security are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are responsible for identity security strategy or NHI governance in your organisation, it is worth exploring.

This post draws on content published by Kong: Dynamic Kafka ACLs: Implementing Identity-Aware Policies with Kong Event Gateway. Read the original.