Notifications
Clear all
Topic starter
24/06/2026 9:09 pm
TL;DR: APIs are now the connective tissue for GenAI and agentic workflows, and Kong’s benchmark reports that its AI Gateway outperformed Portkey and LiteLLM on throughput and latency in a controlled EKS test, according to Kong. Performance matters, but identity governance and policy enforcement remain the deciding factors as AI usage moves into production.
NHIMG editorial — based on content published by Kong: AI Gateway Benchmark: Kong AI Gateway, Portkey, and LiteLLM
By the numbers:
- Kong Konnect Data Planes showed a performance increase of over 200% when compared to Portkey, and over 800% against LiteLLM.
- At the same 12 CPU allocation, Kong had 65% lower latency compared to Portkey and 86% lower latency than LiteLLM.
- WireMock sustained 29,005.51 RPS with a P95 of 24.07ms and a P99 of 30.35ms in the baseline run.
Questions worth separating out
Q: How should security teams govern AI gateway traffic in production?
A: Security teams should govern AI gateway traffic by treating the gateway as a runtime control point for identity, quota, observability, and routing.
Q: When does an AI gateway become a governance control rather than just a proxy?
A: An AI gateway becomes a governance control when it consistently enforces authentication, usage limits, and visibility across model, agent, and MCP traffic.
Q: What do teams get wrong about performance testing AI gateways?
A: Teams often test only latency and throughput and ignore whether policy remains enforceable at scale.
Practitioner guidance
- Test gateway control-plane performance under production-like load Validate that the AI gateway can sustain enterprise traffic while keeping authentication, token enforcement, and observability inline.
- Map AI gateway policy to identity controls Define how consumer identity, quota policy, and routing rules interact before AI requests reach models or MCP services.
- Measure whether centralised AI access is still enforceable Track the points where teams bypass the gateway because it is too slow, too hard to use, or too limited in policy depth.
What's in the full article
Kong's full blog covers the operational detail this post intentionally leaves for the source:
- Exact benchmark architecture, including the AWS EKS layout and load-generation setup used for the tests
- Per-gateway deployment details for Kong, Portkey, and LiteLLM, including the resource settings used in each run
- Latency, throughput, and CPU charts that show how each gateway behaved under the same 12 CPU ceiling
- The article's own interpretation of why the Kong runtime stayed stable when the other gateways did not
👉 Read Kong's AI gateway benchmark comparing Kong, Portkey, and LiteLLM →
AI gateway benchmarks and the governance gap for agentic workloads?
Explore further
25/06/2026 6:11 am
AI gateway performance is now an identity governance issue, not just an infrastructure metric. When AI tools become a production access layer for models, agents, and MCP-connected services, the gateway influences who or what can act, how often, and under what policy. That pushes gateway selection into the same decision space as IAM, PAM, and workload governance. Practitioners should evaluate AI gateways as control points, not just proxies.
A few things that frame the scale:
- 92% agree governing AI agents is critical to enterprise security, yet only 44% have implemented any policies to do so, according to AI Agents: The New Attack Surface report.
- Only 52% of companies can track and audit the data their AI agents access, leaving 48% with a complete blind spot for compliance and breach investigation.
A question worth separating out:
Q: How can organisations reduce shadow AI when they add gateway controls?
A: Organisations reduce shadow AI by making the approved gateway path easier to use than bypass routes. That requires reliable performance, clear policy, and good developer experience, because users and agents will route around controls that are slow, opaque, or hard to integrate.
👉 Read our full editorial: AI gateway benchmarks show why API governance now matters for agents
