Kong Event Gateway for Kafka: cutting cost without losing control

At a glance

What this is: NHI Mgmt Group's analysis, drawing on Kong's blog post: Kafka teams can cut infrastructure sprawl and operational overhead by moving isolation, filtering, mediation, access control, and discovery into a gateway layer.

Why it matters: For IAM and platform teams, the governance question is whether Kafka access is controlled at the broker, the API layer, or both, because that choice shapes auditability, reuse, and administrative load.

Context

Kafka often becomes expensive because control planes fragment before throughput does. As teams create more clusters, duplicate topics, custom connectors, and broker-level permissions, the operating model grows harder to govern than the stream itself. This article is about the governance layer around Kafka, not Kafka performance.

For IAM and platform owners, the practical problem is not whether Kafka can move data. It is whether access, isolation, and discovery can be standardised well enough to avoid duplicated infrastructure and inconsistent policy enforcement across teams and consumers.

Key questions

Q: How should teams reduce Kafka cost without creating more governance sprawl?

A: Teams should reduce Kafka cost by treating isolation, filtering, access control, and discovery as governance problems rather than infrastructure defaults. The practical test is whether a new cluster, topic, or connector is being created to solve policy fragmentation. If so, consolidate control first and add infrastructure only where technical separation is truly required.

Q: Why do duplicate Kafka topics and clusters become an operational risk?

A: Duplicate topics and clusters increase risk because every copy creates another place where ownership, permissions, documentation, and retirement must be managed. The result is not just higher cost, but more entitlement drift, more inconsistent policy enforcement, and more difficulty proving who can access what.

Q: What breaks when Kafka access control is enforced only through broker-level configuration?

A: Broker-only control often breaks down when organizations need consistent governance across many teams, topics, and consumers. It increases configuration drift, duplicates policy logic, and makes observability harder. It can also force client-side changes for every access decision, which slows delivery and creates gaps when applications cannot be modified quickly or uniformly.

Q: What is the difference between Kafka ACLs and API-layer access control?

A: Kafka ACLs protect broker-native access, while API-layer access control can centralise identity policy, auditing, and reuse across Kafka and non-Kafka consumers. The difference matters when the goal is to govern multiple access patterns through one control plane instead of maintaining a separate permission model for every broker interaction.

Technical breakdown

Logical isolation on shared Kafka infrastructure

Kafka isolation is often implemented by creating a separate cluster for each team or use case, but that turns access boundaries into infrastructure sprawl. A virtual cluster model creates a logical boundary on shared infrastructure, which changes the governance unit from physical deployment to policy-enforced tenancy. That matters because many Kafka costs are operational, not computational, and they come from duplicating control planes rather than from message throughput itself. The architectural question is whether separation must be physical to remain trustworthy, or whether governance can be enforced at a shared control layer.

Practical implication: model isolation as a governed tenancy decision before allocating a new cluster.

Policy-based filtering and topic sprawl

Topic sprawl happens when teams duplicate streams just to present different slices of the same data to different consumers. That pattern is operationally convenient but weak from a lifecycle perspective because every parallel topic needs ownership, permissioning, documentation, and retirement handling. Policy-based filtering changes the unit of control from whole-topic duplication to data subset exposure. In governance terms, this reduces the number of identities and entitlements that have to be maintained just to deliver differentiated access patterns. The architectural win is not only fewer topics, but fewer places where entitlement drift can hide.

Practical implication: prefer filtered exposure over topic duplication wherever consumer-specific views are the real requirement.

Protocol mediation and access control at the API layer

Protocol mediation lets Kafka data be exposed through REST APIs or Server-Sent Events, which makes Kafka consumable by systems that do not speak Kafka natively. That is useful, but the governance impact is larger than protocol translation: moving access control to the API layer creates a single place to enforce OAuth2, JWT, API key policies, RBAC, observability, and audit trails across both event and API access. This is effectively an access abstraction layer for streaming. The key technical trade-off is that mediation simplifies integration while also concentrating policy enforcement, so design decisions about identity and auditing become architecture decisions, not broker settings.

Practical implication: treat protocol mediation as a control point and align identity policy there, not only at the broker.

NHI Mgmt Group analysis

Kafka governance fails when isolation is treated as an infrastructure problem instead of an access problem. The article shows a familiar pattern: teams duplicate clusters because they do not trust shared governance to preserve separation. That assumption drives cost and operational overhead more than Kafka itself does. The better framing is that tenancy, audit, and entitlement boundaries are the real control plane, and practitioners should design around that reality.

Centralised event exposure creates a cleaner governance boundary than broker-level sprawl. When filtering, discovery, and permissioning are scattered across teams, entitlement review becomes a reconciliation exercise rather than a control. The article points toward a model where access decisions and documentation sit together, which makes reuse easier and duplicate streams less likely. Practitioners should see this as a governance consolidation pattern, not just a platform simplification.

Kafka access looks increasingly like API governance, and that convergence matters for identity teams. Once event streams are exposed through standard policies, the relevant controls become familiar identity controls: OAuth2, JWT, RBAC, auditability, and consistent onboarding. That does not turn Kafka into an API, but it does mean the same policy discipline has to govern both. Teams should stop managing event access as a separate exception class.

Named concept: event access consolidation. The article describes a shift from distributed Kafka-specific controls to a central policy layer that governs isolation, filtering, mediation, and discovery together. This is not a product feature so much as a governance pattern. Practitioners should recognise it as a way to reduce entropy in event platforms by shrinking the number of places where access policy can diverge.

What this signals

Kafka platforms often fail economically before they fail technically. Once teams start duplicating clusters, topics, and connectors to express policy that the platform cannot otherwise enforce cleanly, the governance layer becomes the bottleneck. Practitioners should look for control patterns that reduce the number of places where access decisions are made.

Event access consolidation: the practical goal is not to make Kafka smaller, but to make entitlement management legible. When filtering, discovery, and access control sit in one policy layer, teams can review who can consume what without reconstructing the path across several broker-level exceptions.

For practitioners

• Map Kafka tenancy to logical isolation first Review whether new Kafka clusters are being created for governance reasons that a shared policy layer could absorb. Use logical isolation only where workload separation truly requires dedicated infrastructure, and document the entitlement model before provisioning.
• Replace duplicate topics with filtered views Identify topics that exist only to serve different consumer subsets and evaluate whether policy-based filtering can expose those slices from a single source topic. This reduces topic sprawl and lowers the number of entitlements to administer.
• Move access enforcement to one policy plane Standardise OAuth2, JWT, API key policy, and RBAC enforcement where Kafka and API access converge so audit trails and permissions are reviewed in one place. Keep broker-level permissions aligned with that control model rather than treating them as the primary control surface.
• Document Kafka discovery and reuse workflows Publish event inventories, usage guidance, and access request paths together so developers do not create new streams simply because they cannot find or understand existing ones. Treat discovery as a governance control, not just a developer experience issue.

Key takeaways

• The article frames Kafka cost as a governance problem driven by duplicated controls, not as a throughput problem.
• Centralising isolation, filtering, mediation, access control, and discovery can reduce both infrastructure sprawl and administrative overhead.
• For practitioners, the main decision is whether Kafka access should remain broker-centric or move into a shared policy layer.

Key terms

• Kafka Governance: Kafka governance is the set of controls used to manage who can create, read, filter, and expose event streams. In practice it covers entitlements, documentation, auditing, and lifecycle handling across clusters, topics, and consumers.
• Logical Isolation: Logical isolation is the separation of customer environments through software controls rather than physical infrastructure. It ensures that identities, policies, and data for one tenant cannot be accessed by another tenant through the shared platform. For MSSPs, it is the foundation of safe centralized management.
• Protocol Mediation: Protocol mediation is the use of an access layer that terminates one side of a session and relays approved actions to the target system. In industrial environments, it allows security policy to sit between the user and the asset, but it also creates a control point that must be audited and protected.
• Scope Sprawl: Scope sprawl is the accumulation of excessive, duplicated, or stale OAuth permissions across many applications and users. It usually grows when teams approve broad access for convenience and never remove it, leaving a large and poorly understood delegated-access surface.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity lifecycle are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building or maturing an IAM programme, it is worth exploring.