Kong Event Gateway for Kafka: cutting cost without losing control

At a glance

What this is: This is Kong’s guidance on simplifying Kafka operations by centralising isolation, access control, mediation, and discovery in an event gateway.

Why it matters: It matters because Kafka governance is now an identity and access problem as much as an infrastructure problem, affecting NHI, human, and platform access patterns.

👉 Read Kong's guide to reducing Kafka cost and complexity with Event Gateway

Context

Kafka complexity is often framed as an infrastructure scaling issue, but the deeper problem is governance drift: isolated clusters, duplicate topics, and custom access paths quickly become hard to audit and harder to standardise. For identity teams, that turns event streaming into an access control and lifecycle management problem, not just a platform engineering one.

Kong’s article argues that event gateways can absorb several of those control points by centralising isolation, filtering, protocol mediation, and policy enforcement. The useful question for practitioners is whether this reduces operational sprawl without creating a new abstraction layer that obscures who or what is actually authorised to reach Kafka data.

For teams already working on API governance, workload identity, or internal developer platforms, the article fits into a broader pattern: event access is becoming another identity surface that needs consistent policy, observability, and review. That is typical of modern Kafka estates, especially once multiple teams and external consumers are involved.

Key questions

Q: How should teams govern Kafka access across humans and workloads?

A: Teams should treat Kafka as an identity-governed service, not a raw transport. The right model separates human, workload, and partner access, applies policy at a common enforcement point, and keeps the audit trail tied to the consumer identity, the topic, and the policy decision. That is the only way to preserve least privilege at scale.

Q: When does Kafka topic duplication become a security problem?

A: Topic duplication becomes a security problem when it is the default way to express access boundaries. At that point, the organisation is encoding entitlement complexity into infrastructure, which makes auditing, revocation, and review harder. If a policy layer can express the same boundary, duplicate topics are usually a governance liability rather than a control.

Q: What do security teams get wrong about centralising Kafka controls?

A: The common mistake is assuming that one gateway automatically improves governance. Centralisation only helps if the organisation can still prove who accessed what, under which identity, and through which policy path. Without that visibility, the team has reduced operational sprawl while concentrating uncertainty in a single layer.

Q: What should IAM and platform teams review before exposing Kafka through a gateway?

A: They should review how identities are represented, how filtering rules are maintained, and whether access reviews include event-stream consumers. They should also verify that the gateway does not flatten workload and human access into the same entitlement model. If it does, least privilege becomes difficult to defend in practice.

Technical breakdown

Virtual clusters and logical isolation in shared Kafka estates

Virtual clusters are an abstraction layer that lets teams present isolated environments without provisioning a separate Kafka cluster for every tenant or use case. The mechanism matters because many Kafka estates multiply infrastructure simply to satisfy separation requirements that are really about access boundaries, not broker independence. Logical isolation can reduce cost, but it also shifts trust into the gateway layer, where policy enforcement and tenancy separation must be consistent. The architectural trade-off is that isolation becomes policy-driven rather than physically enforced, so governance quality depends on how well the gateway expresses and audits those boundaries.

Practical implication: treat virtual isolation as a governance control and validate that tenancy boundaries are enforced, logged, and reviewable at the gateway layer.

Policy-based filtering and topic sprawl control

Topic sprawl appears when teams create parallel topics or partitions to satisfy different consumer access needs. Kong’s model instead exposes subsets of data from a single topic using virtual topics and policy-based filtering. Technically, that moves entitlement decisions from data duplication into request-time policy evaluation. This can simplify architecture, but it also makes filtering logic part of the security model. If the policy layer is weak, inconsistent, or poorly audited, the simplification becomes a hidden risk because the original data stream is now serving multiple access patterns under one control plane.

Practical implication: keep filtering policy versioned, tested, and auditable so access patterns do not become untracked security exceptions.

Kafka access control through the API layer

The article proposes shifting Kafka security controls upward into the API layer, where OAuth2, JWT, API keys, and RBAC can be applied consistently across APIs and event streams. Mechanically, this gives teams a common enforcement point rather than distributing access logic across brokers, clients, and custom tooling. That helps with observability and compliance, but it also means the API layer becomes part of the identity boundary for Kafka. In practice, the important question is whether the gateway preserves enough context to distinguish human consumers, workloads, and service identities without flattening them into a single access model.

Practical implication: align Kafka access policies with identity type so workload, human, and partner access are not governed as if they were the same subject.

NHI Mgmt Group analysis

Kafka governance is becoming an identity control problem, not just a streaming problem. Once teams rely on shared topics, gateway mediation, and layered access policy, the real question is who or what is authorised to consume which event data. That moves Kafka into the same governance conversation as APIs, service accounts, and internal platforms. Practitioners should stop treating event access as a separate technical silo.

Topic sprawl is the symptom of fragmented entitlement design. The article shows how duplicate topics often emerge because teams lack a clean way to express consumer-specific access without rebuilding the data plane. That is a governance failure as much as an operational one. The practical conclusion is that access models should be designed before topic proliferation becomes normalised.

Identity context matters at the event layer. A policy layer that can enforce OAuth2, JWT, API keys, and RBAC across Kafka and APIs only helps if the organisation still distinguishes workload access from human access. When every consumer is handled the same way, auditability and least privilege degrade quickly. Practitioners should align event access governance with the identity type consuming the stream.

Centralising control can reduce complexity, but it also concentrates risk. A gateway-based model can simplify enforcement and reporting, yet it creates a higher-value control plane whose correctness becomes foundational. If that layer misclassifies access or obscures lineage, teams lose visibility while believing they have improved governance. The implication is to verify policy semantics, not just architecture diagrams.

Named concept: event access abstraction debt. The article points to the accumulation of duplicated topics, custom proxies, and inconsistent ACLs that appears when organisations solve each Kafka access problem separately. That debt is paid later in audit failure, onboarding friction, and brittle security exceptions. Practitioners should recognise it early as a structural governance problem, not a tuning issue.

From our research:

• 92% agree governing AI agents is critical to enterprise security, yet only 44% have implemented any policies to do so, according to AI Agents: The New Attack Surface report.
• Only 52% of companies can track and audit the data their AI agents access, leaving 48% with a complete blind spot for compliance and breach investigation.
• That gap is a warning for Kafka governance too, so review NHI Lifecycle Management Guide for lifecycle and access-review patterns that can be applied to event consumers.

What this signals

Event gateways are becoming the control plane where identity, policy, and data distribution meet. For practitioners, that means Kafka governance should be reviewed alongside API governance, workload identity, and access review processes rather than treated as a separate infrastructure topic. The next maturity step is not more custom isolation, but tighter policy consistency across consumption paths.

The most common failure mode in large event estates is not throughput loss, but unmanaged entitlement growth. When duplicate topics, ad hoc connectors, and inconsistent ACLs accumulate, the organisation inherits what can be called event access abstraction debt: a system that appears simpler to developers but becomes harder to govern over time. Teams should plan for that debt early and document the identity model behind every mediated stream.

With only 52% of companies able to track and audit the data their AI agents access, per AI Agents: The New Attack Surface report, identity visibility is already the bottleneck in adjacent control planes. Kafka teams should expect the same pressure once event access is exposed through shared gateways and self-service portals.

For practitioners

• Map Kafka consumers by identity type Separate human users, service accounts, applications, and external partners before applying gateway policy. This prevents one-size-fits-all entitlements from hiding where least privilege is actually breaking down.
• Replace duplicate topics with policy-defined access paths Review whether topic duplication exists only to satisfy consumer segmentation. Where it does, evaluate whether a policy layer can enforce the same boundary without creating parallel data pipelines.
• Audit gateway policy semantics and logs Confirm that the gateway records who requested access, what was exposed, and which policy decision was applied. If the audit trail cannot reconstruct those three elements, the control is not complete enough for governance.
• Tie access reviews to event-stream exposure Include Kafka topics and mediated event paths in regular access reviews, especially where external consumers or shared platform teams are involved. Review should cover standing access, not just initial provisioning.
• Standardise protocol mediation for non-Kafka consumers Use a single mediation approach for REST or SSE exposure so teams do not build one-off connectors that bypass policy and monitoring. This reduces custom glue code while preserving a consistent control point.

Key takeaways

• Kafka complexity often shows up first as governance drift, not just operational cost, because duplicated topics and custom access paths are hard to audit.
• Moving isolation, filtering, and access control into a gateway can simplify the estate, but only if identity context and policy semantics remain visible.
• Practitioners should review Kafka consumption through the same lens as APIs and workload identity, because event access is now part of the identity surface.

Key terms

• Virtual Cluster: A virtual cluster is a logical isolation layer that presents separate Kafka environments without requiring a separate physical cluster for every use case. It shifts separation into policy and tenancy enforcement, which can reduce cost but also raises the importance of correct gateway controls and auditability.
• Topic Sprawl: Topic sprawl is the uncontrolled growth of Kafka topics created to satisfy one-off consumer needs, security boundaries, or data filtering patterns. It usually signals that the organisation lacks a scalable entitlement model and is using infrastructure duplication to solve an access-governance problem.
• Protocol Mediation: Protocol mediation is the practice of exposing Kafka data through alternate interfaces such as REST or Server-Sent Events while preserving central governance. It simplifies integration for non-Kafka consumers, but the control point must still preserve identity, policy, and audit context.
• Event Access Abstraction Debt: Event access abstraction debt is the accumulation of custom connectors, duplicated topics, and inconsistent policy layers created when each Kafka access problem is solved separately. It reduces short-term friction but creates long-term audit, review, and revocation complexity.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity security are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are responsible for identity security strategy or NHI governance in your organisation, it is worth exploring.

This post draws on content published by Kong: 5 Steps to Immediately Reduce Kafka Cost and Complexity. Read the original.