TL;DR: APIs are now the connective tissue for GenAI and agentic workflows, and Kong’s benchmark reports that its AI Gateway outperformed Portkey and LiteLLM on throughput and latency in a controlled EKS test, according to Kong. Performance matters, but identity governance and policy enforcement remain the deciding factors as AI usage moves into production.
At a glance
What this is: NHI Mgmt Group's analysis, drawing on Kong's blog post: API governance is now central to agentic workloads, alongside AI gateway throughput and latency benchmarks.
Why it matters: It matters because IAM, platform, and security teams now need to govern AI traffic as a first-class access path, not as a side effect of model integration.
Context
AI gateway governance is the practice of applying policy, visibility, and access controls to traffic moving between applications, models, agents, and tools. In this article, the governance problem is not just performance under load, but how to keep agentic AI requests controlled when APIs become the operational path into production systems.
Kong positions its AI Gateway as infrastructure for GenAI and agentic traffic, then uses benchmark testing to compare throughput and latency against Portkey and LiteLLM. The important issue for practitioners is that gateway speed alone does not solve authorisation, token control, observability, or policy enforcement for AI-driven access paths.
Key questions
Q: Where do AI gateways fail when agent traffic scales quickly?
A: They fail when teams treat them as a transport layer instead of an enforcement point. Once agents and tools start generating high-volume requests, weak policy design turns a fast gateway into a fast path for overbroad access, excessive spend, and poor traceability.
Q: Why do agentic workloads change API governance requirements?
A: Agentic workflows chain multiple requests, tools, and model calls together, so governance must track the whole request path rather than one application at a time. That makes authentication, quota enforcement, and auditability part of the same control problem.
Q: What should teams measure besides gateway latency in AI production?
A: They should measure whether identity context, quota enforcement, and request auditing still work under load. A gateway can look performant while still failing at the controls that determine who can access what and whether the activity can be investigated later.
Q: How should organisations govern LLM, agent, and MCP access together?
A: By using a single policy layer that binds consumer identity, request limits, and observability to every AI interaction path. That approach reduces fragmentation across models, agents, and tools, and it makes revocation and review more practical.
Technical breakdown
Why AI gateway throughput matters for agentic traffic
AI gateways sit on the request path, so latency and throughput determine whether policy enforcement becomes a bottleneck. When agents call models, tools, and MCP servers repeatedly, the gateway must handle high request volume without adding enough delay to break user experience or orchestration timing. In practice, that makes the gateway part of the control plane, not just a routing layer. The benchmark setup in the article isolates gateway behaviour precisely because production teams need to know whether their policy layer can scale with AI demand.
Practical implication: size gateway capacity and policy logic together, not as separate concerns.
What centralised API governance changes for AI agents and MCP servers
Agentic workloads do not just consume model endpoints. They chain API calls, tool calls, and context retrieval across systems, which means governance has to follow the request path end to end. A universal API layer can enforce authentication, token quotas, routing, and observability consistently across LLMs, AI agents, and MCP servers. That is materially different from managing each integration as a one-off. The control question is whether the gateway can act as the policy checkpoint for every machine-originated request that participates in an agent workflow.
Practical implication: treat the AI gateway as a policy enforcement point for every agent tool path.
Why benchmark results do not replace identity and policy design
A faster gateway can improve user experience, but it does not answer who or what is allowed to call which service, under what conditions, and with what limits. Identity governance for AI traffic still needs entitlement design, per-consumer quotas, auditability, and revocation logic. Without those, a performant gateway simply moves unauthorised or overbroad access faster. The article’s emphasis on token limits, observability, and self-service access shows that gateway architecture must be paired with explicit governance decisions.
Practical implication: use benchmark data to inform architecture, then validate access policy, quota, and audit controls separately.
Breaches seen in the wild
- LiteLLM MCP auth bypass 2026: An exploited LiteLLM MCP auth bypass and default sk-1234 master keys let attackers steal AI gateway master and provider API keys.
- LiteLLM PyPI package breach: LiteLLM PyPI supply chain attack, credentials stolen from users.
Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.
NHI Mgmt Group analysis
API governance is becoming the control surface for agentic AI. When agents, LLMs, and MCP servers all depend on the same traffic path, the gateway becomes the point where identity, policy, and cost controls converge. That makes AI infrastructure governance a cousin of NHI governance, because the practical question is who or what is permitted to invoke downstream services. Practitioners should now assess AI gateways as enforcement infrastructure, not as simple performance middleware.
Performance benchmarking without policy context is incomplete for production AI. The article benchmarks proxy behaviour under load, but production risk is created by how requests are authorised, constrained, observed, and revoked. A gateway that is fast but policy-light can still become the fastest path to excessive tool use or uncontrolled model spend. The governance signal is clear: throughput is necessary, but it is not a substitute for access design.
Agentic workloads expose a governance gap that traditional API management only partly covers. Classic API programmes were built around service-to-service access and application consumers. Agentic AI adds dynamic prompt-driven requests, tool chaining, and model selection, which changes the operational meaning of consumer identity and usage limits. That is why token quotas, routing policy, and observability now belong in the same conversation as IAM and PAM. Practitioners need a governance model that treats AI traffic as a distinct class of access.
Identity-aware policy will matter more than gateway branding as AI adoption scales. The market is moving toward unified control of APIs, models, and agents because the hard problem is not merely connectivity. It is deciding which identities can invoke which capabilities, at what rate, with what audit trail, and under which organisational controls. That will favour architectures that can express policy across human, machine, and agentic consumers without fragmenting governance into separate silos.
AI gateway governance now has a named failure mode: request-path sprawl. As AI use expands, every new model, tool, and agent integration adds another path that must be governed consistently. If teams do not centralise policy enforcement, access reviews and usage controls become fragmented across point solutions. The practical result is not just operational sprawl, but governance drift that is hard to detect once agentic traffic is in production.
What this signals
Request-path sprawl is the real governance problem behind agentic AI adoption. Every additional model, tool, or MCP server creates another control point, and teams that govern each one separately will lose consistency faster than they can document it. Central policy enforcement is becoming the only scalable way to preserve auditability and limit blast radius as AI use moves into production.
Gateway performance only matters when it is matched with policy fidelity. A low-latency gateway that cannot express authentication, quota, and observability decisions does not solve production risk. Practitioners should evaluate AI infrastructure on whether the policy layer survives scale, not just whether requests complete quickly.
For practitioners
- Centralise policy enforcement at the AI gateway Apply authentication, token quotas, routing policy, and observability in one request path so model, agent, and tool traffic is governed consistently.
- Classify agent and model consumers explicitly Define which workloads, agents, and downstream services are allowed to transact through the gateway, then bind those decisions to durable identities and scopes.
- Separate performance testing from policy testing Benchmark gateway latency and throughput independently from authorisation, quota, and audit behaviour so load results do not hide governance gaps.
- Map agentic traffic to an audit trail Ensure requests from AI agents and tool chains carry enough identity context to support investigation, chargeback, and access review.
- Review AI usage controls before production scale-up Check whether token limits, caching rules, and model routing policies still hold when usage moves from sandbox experimentation to business-critical workflows.
Key takeaways
- AI gateway design is now an identity and governance issue as much as an infrastructure issue, because agentic traffic depends on the same request path for access and control.
- Benchmarking throughput and latency tells only part of the story, since production readiness also depends on authentication, quotas, routing, and auditability.
- Teams moving AI into production should centralise request-path governance before they expand agent, model, and MCP usage across the environment.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10, OWASP API Security Top 10 and OWASP Non-Human Identity Top 10 address the attack and risk surface, while NIST CSF 2.0 and CSA Cloud Controls Matrix set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | ASI03 — Identity & Privilege Abuse | Agentic traffic is governed through identities, scopes, and tool access decisions. |
| Recommendation — Bind agent permissions to explicit scopes and review every identity-to-tool relationship. | ||
| OWASP API Security Top 10 | API2 — Broken Authentication | The gateway controls authentication for model and tool traffic at the API layer. |
| Recommendation — Enforce strong authentication on every AI-facing API and reject unauthenticated access paths. | ||
| NIST CSF 2.0 | PR.AA-05 — Access Permissions, Entitlements and Authorizations | AI gateway policy decisions are fundamentally about permissions and entitlements. |
| Recommendation — Map AI consumers to explicit permissions and validate authorisations continuously. | ||
| CSA Cloud Controls Matrix | IAM — Identity and Access Management | The article centres on centrally governing access to AI services and model traffic. |
| Recommendation — Apply IAM controls to AI request paths so access, quotas, and audit trails stay governed. | ||
| OWASP Non-Human Identity Top 10 | NHI-05 — Overprivileged NHI | Agentic and machine consumers can accumulate excessive rights through gateway pathways. |
| Recommendation — Reduce overprivileged AI consumers by narrowing scopes and removing unused access paths. | ||
Key terms
- AI Gateway: A control point that sits between AI applications and the models, tools, or data they call. In practice, it can authenticate requests, enforce policy, inspect runtime behaviour, and stop unsafe actions before they spread into connected systems.
- Agentic AI Workload: An agentic AI workload is an AI system that can reason, choose tools, and execute multi-step actions against enterprise systems. Unlike a simple model call, it may read data, invoke APIs, and trigger workflows. That autonomy creates higher governance requirements for permissions, logging, and containment.
- Token Quotas: Token quotas are usage limits measured by the amount of text an AI model processes or produces. They are used to control spend, prevent runaway consumption, and separate usage by user, team, model, or project. In enterprise AI, token-based governance is often more precise than request counts alone.
- Request-path Governance: Request-path governance is the set of policies that apply before an AI request executes. It covers who can call a model, which tools can be reached, what data can move, and whether the request should be blocked, throttled, or routed differently.
Deepen your knowledge
NHI governance, agentic AI identity, and machine identity lifecycle are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building or maturing an IAM programme, it is worth exploring.
Published by the NHIMG editorial team on June 23, 2026.
Updated on October 11, 2026.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org