An API gateway that already fronts your services can route a call to a model provider, but it cannot tell a 40-token request from a 40,000-token one, read what the prompt contains or keep a provider key out of application hands. That gap is why AI gateways (also called LLM gateways) exist, and why teams keep asking whether they need one or can extend what they have. This guide compares the two control points on the five things LLM traffic changes: token-based quotas, prompt and response inspection, provider key management, model routing and streaming. For each it sets out the threat model, the architecture in words, the failure modes and the controls, then gives decision criteria and a worked scenario. It is written for platform engineers, API owners and security architects deciding where LLM controls should live. Products named in examples are listed alphabetically; order is never a ranking.
Key takeaways
- The unit of cost and abuse changes from requests to tokens. A request-rate limit that suits a REST API does not bound spend or capacity for LLM calls, which is the core of OWASP LLM10:2025 Unbounded Consumption.
- Token limits are enforced after the fact. Gateway documentation is consistent that usage is known only when the response finishes, so the request that crosses a limit completes and the next one is refused. Budget for that overshoot.
- The real security gain is key custody. An AI gateway lets you hold provider keys in one place and hand callers scoped, revocable gateway keys. Without it, provider keys spread into every application that calls a model.
- Converged products blur the line. Several API management platforms now ship token-aware policies, and several AI gateways are built on API gateway engines. Decide by capability, not by product label.
- Either gateway only governs traffic that passes through it. Egress controls that block direct calls to provider endpoints are what make the gateway the only path. See the Agent Egress Control and Network Isolation Guide.
The short answer
Keep your API gateway for what it does well: terminating TLS, authenticating callers, enforcing request-rate limits and routing to backends. Add AI gateway capability, either as a separate service or as token-aware policies on your API management platform, as soon as one of these is true: more than one team calls a model provider, provider spend needs a per-team cap, prompts may carry regulated data, or you use more than one model provider or deployment. A single application calling one provider can live behind a well-configured API gateway plus provider-side spend limits for a while, if designed so a gateway can be inserted later.
For how an AI gateway works in full see the AI Gateway and LLM Gateway Guide, for hardening the gateway itself see Securing the AI Gateway, and for the content-inspection layer see AI Guardrails vs AI Gateways.
AI gateway and API gateway side by side
| Dimension | API gateway | AI gateway |
|---|---|---|
| Traffic it is built for | Any HTTP or gRPC API, mostly short request and response pairs | Model API calls (chat completions, responses, embeddings) and increasingly MCP tool calls |
| Unit of limiting | Requests per second with a burst allowance | Input, output, cached and total tokens, or currency cost per caller and per model |
| Payload awareness | Headers, path, method; body schema validation where configured | Parses prompts, messages and usage fields; hosts prompt and response policy checks |
| Upstream credential | Usually passes the caller's token through, or uses one backend credential | Holds provider keys or federated credentials and issues its own virtual keys to callers |
| Routing logic | Path, host, header and weight based | Model alias to provider deployment, fallback on errors or content refusals, cost or latency based |
| Streaming | Passes long-lived responses if timeouts and buffering allow | Must count tokens and apply policy across a server-sent event stream |
| Caching | Exact-match response caching | Exact and semantic caching of model responses |
| Telemetry | Latency, status codes, request counts | Tokens, cost, model, prompt and response logs, policy verdicts |
| Main threats addressed | OWASP API Top 10 2023 risks such as API2 Broken Authentication and API4 Unrestricted Resource Consumption | OWASP LLM Top 10 2025 risks such as LLM10 Unbounded Consumption and, with guardrails, LLM01 Prompt Injection and LLM02 Sensitive Information Disclosure |
The two overlap more each year. Azure API Management documents a family of LLM policies, and Kong AI Gateway and Agent Router (formerly Envoy AI Gateway) are built on API gateway engines. Check which column a given product actually implements.
Token-based quotas
Threat model: a leaked key, a runaway agent loop or a single caller sending very long prompts can exhaust provider quota for everyone and run up a bill in hours. OWASP LLM10:2025 lists Denial of Wallet and variable-length input floods as examples. In ATT&CK terms the abuse of stolen keys is T1496.004 Cloud Service Hijacking; MITRE ATLAS tracks the cost side as AML.T0034 Cost Harvesting.
API gateway in words: request throttling uses a token bucket where, as the Amazon API Gateway documentation puts it, a token counts for a request. Usage plans tie a rate, burst and quota to an API key used as a client identifier, and AWS describes these limits as best-effort targets rather than guaranteed ceilings. A caller allowed 10 requests a second can send 10 maximum-context prompts a second, so the limit bounds neither spend nor model capacity.
AI gateway in words: the gateway reads the usage block the provider returns and charges it against a counter keyed to the caller, team or tenant, sometimes weighted by model price.
| Product (alphabetical) | Documented mechanism | Behaviour worth knowing |
|---|---|---|
| Agent Router | llmRequestCosts on an AIGatewayRoute with types InputToken, CachedInputToken, OutputToken, TotalToken or CEL, enforced by a global BackendTrafficPolicy backed by Redis | Usage is charged after the response; an admitted stream is never cut off and later requests get 429 until the window clears |
| Amazon API Gateway | Account, stage, method and usage plan throttling (rate and burst) with quotas per API key | Counts requests, applied in order from usage plan down to Regional limits, best-effort |
| Azure API Management | llm-token-limit with counter-key, tokens-per-minute (429 when exceeded) and token-quota with token-quota-period (403 when exceeded) | estimate-prompt-tokens can count before sending; streamed requests are always estimated; counters are per gateway and not aggregated across regional gateways |
| Kong AI Gateway | AI Rate Limiting Advanced with tokens_count_strategy of total_tokens, prompt_tokens, completion_tokens or cost; local, cluster or redis counters | The request that reaches the limit completes and the next is blocked; if Redis is unreachable nodes fall back to local counters |
| LiteLLM Proxy | Virtual keys with tpm_limit, rpm_limit, max_budget and budget_duration | Spend tracked in USD per key, user and team in a PostgreSQL database |
Failure modes: limits set only on requests; one shared counter key so a single caller starves everyone; regional gateways each enforcing the full quota so the real ceiling is the quota times the number of regions; a counter store outage silently dropping to per-node limits; and no provider-side spend cap as the last backstop.
Controls: key counters to an authenticated identity (team, application or agent), not an IP address; set both a per-minute rate and a period budget; cap max_tokens on requests; alert at 50, 80 and 100 percent of budget; and keep provider-side limits as the outer fence. See AI Agent Rate Limits and the Denial-of-Wallet and AI Agent Cost Runaway Guide.
Prompt and response inspection
Threat model: sensitive data leaving in prompts (LLM02 Sensitive Information Disclosure), injected instructions arriving in user input or retrieved content (LLM01 Prompt Injection) and unsafe output returning to an application that trusts it (LLM05 Improper Output Handling).
API gateway in words: it can validate a request body against a JSON schema and reject oversized payloads, which is useful hygiene, but it treats the messages array as opaque text. A WAF rule set written for SQL injection signatures is not a prompt injection control.
AI gateway in words: it understands the message structure, so it can run redaction, classification or a guardrail model on the prompt before forwarding and on the completion before returning, and log the verdict. Azure API Management, for example, documents content safety checks on LLM requests and semantic caching through llm-semantic-cache-store and llm-semantic-cache-lookup.
Failure modes: inspecting the prompt but not the response; inspecting non-streamed calls only; a guardrail that fails open on timeout without anyone noticing; and full prompt logging that turns the gateway's log store into the largest copy of sensitive data in the estate. See the Sensitive Data in Logs Guide.
Controls: decide per route whether checks fail open or closed and alert on fail-open events; redact before logging; restrict who can read prompt logs; and treat semantic caches as data stores that can return one caller's answer to another unless keyed by tenant. See the Generative AI Data Loss Prevention Guide.
Provider key management
Threat model: provider API keys are bearer credentials with direct financial value. The LLMjacking 2024 to 2026 campaigns show stolen cloud and AI keys being resold through reverse proxies while victims paid the inference bill (T1078.004 Cloud Accounts, T1552.001 Credentials In Files).
API gateway in words: it authenticates callers to itself, then either passes their credential through or attaches one backend credential. When applications call providers through it with their own provider keys, the API gateway adds routing but does nothing to reduce key sprawl.
AI gateway in words: it separates the two relationships. Callers hold gateway-issued keys scoped to models, budgets and an expiry; only the gateway holds the upstream credential, ideally a short-lived federated one. LiteLLM, for example, issues virtual keys through /key/generate using a master key, and Azure API Management can authenticate to Azure AI services with a managed identity instead of a key.
Failure modes, including gateway key sprawl: the problem moves rather than disappears. Common forms are one virtual key per team copied into dozens of services; keys with no expiry; the admin or master key used by an application; the same provider key loaded into several gateways; and gateway hosts that hold every provider key becoming a single high-value target, as the LiteLLM PyPI Package Breach 2026 illustrated when poisoned package versions harvested LLM provider keys from machines that installed them.
Controls: one gateway key per workload or agent with an owner and expiry; upper bounds on what any key can be issued; master and admin keys vaulted and never used by applications; upstream keys split by environment and readable only by the gateway; and scanning for provider key formats in repositories so direct keys are found and retired. See the API Keys vs OAuth Tokens Guide and the LLMjacking Response Playbook.
Model routing and failover
Threat model: routing decisions are data-handling decisions. A fallback from a regional, zero-retention deployment to a public endpoint in another jurisdiction is an unapproved transfer; an unpinned model alias can switch a production feature to an unevaluated model.
API gateway in words: weighted and health-based routing between backends works for model endpoints, and Azure API Management documents round-robin, weighted, priority-based and session-aware load balancing with a circuit breaker that honours the backend's Retry-After header. What a generic gateway does not do is translate one request format into another provider's API.
AI gateway in words: it exposes stable model aliases to applications, translates to each provider's API and falls back on errors, rate limits or content refusals.
Failure modes: fallback lists that ignore data residency; retries that multiply cost during a provider incident; and aliases changed without the change process. Controls: require every fallback target to carry the same data classification approval as the primary; cap retries; and treat alias changes as changes. See the AI Model Change Management vs Traditional Change Management comparison.
Streaming
Threat model: most chat interfaces stream tokens as server-sent events, a text/event-stream response made of data: lines separated by blank lines. Controls that wait for a complete response either break the user experience or are skipped for streamed calls.
API gateway in words: it can pass a stream if idle and total timeouts are long enough and response buffering is off, but it sees an opaque byte stream and cannot count tokens in it.
AI gateway in words: it parses the stream to extract usage at the end and may run response checks on chunks. The documented limits matter: Agent Router never cuts off an admitted stream and charges the usage when it ends; Azure API Management estimates prompt and completion tokens for streamed requests.
Failure modes: streamed routes exempted from inspection; a gateway that buffers whole responses and so gets switched off by product teams; and limits that let one long stream overshoot a budget. Controls: set max_tokens ceilings per route, decide whether chunk-level response checks are needed or whether redaction after completion is acceptable, and test inspection on streamed and non-streamed calls separately.
Which control handles which threat
| Threat | API gateway alone | AI gateway | Still needed elsewhere |
|---|---|---|---|
| Unauthenticated callers | Strong: OAuth, mTLS, API keys | Strong: virtual keys, OIDC or workload identity | Identity provider and key lifecycle |
| Request floods (API4:2023) | Strong: rate and burst | Present | Upstream DoS protection |
| Denial of Wallet (LLM10:2025) | Weak: request counts only | Strong: token and cost budgets | Provider spend caps, cost anomaly alerts |
| Provider key theft | Weak: keys stay in apps | Strong: central custody, scoped virtual keys | Secrets manager, egress blocking, scanning |
| Sensitive data in prompts (LLM02) | Weak | Medium to strong with redaction or guardrail plugins | DLP, data classification |
| Prompt injection (LLM01) | Weak | Medium with a guardrail model | Application design, least-privilege tools |
| Unapproved model or region | Medium with route rules | Strong: model allow lists per key | AI inventory and approval |
Where identity fits
Both gateways are identity enforcement points for non-human callers, and the AI gateway adds a second identity relationship upstream. Treat every gateway key as a non-human identity with an owner, scope, expiry and rotation, and prefer workload identity (OIDC or SPIFFE tokens) for agents so one agent's budget and models can be revoked without touching others. Upstream, managed identities or federated short-lived credentials remove the static provider key altogether where the provider supports it. See the API Gateway Security Guide for the caller-authentication side.
Decision guidance
- Count the callers. One application and one provider: API gateway plus provider spend caps is defensible. Several teams or agents: you need per-caller token budgets and key custody, which means AI gateway capability.
- Check what your API platform already does. If it documents token limits, token metrics and model-aware routing, extending it keeps one control plane and one audit trail. If it only counts requests, add an AI gateway behind it.
- Classify the data. If prompts may carry personal or regulated data, you need inspection and redaction before egress, and log handling to match.
- Decide where keys live. If any application holds a provider key today, the AI gateway's main value is taking it away.
- Size for scale. Small organisations usually run one gateway behind the existing API edge. Large organisations run federated gateways with a shared policy baseline and must aggregate token counters, since some products count per gateway instance.
Common layering: keep the API gateway at the edge for client authentication and coarse request limits, and route model calls to an AI gateway for token budgets, key custody, routing and inspection.
Worked scenario
A retailer exposes a customer service assistant through its API gateway with a 20 requests per second limit per partner key. Four internal teams also call two model providers directly with keys stored in their own pipelines. In one month a test agent loops overnight and a provider key appears in a public repository.
- What the API gateway saw: the agent's loop stayed under 20 requests a second, so nothing was throttled, while each request carried a long context. The leaked key never touched the gateway.
- What changes with an AI gateway: each team and agent receives its own gateway key with an allowed model list, a tokens-per-minute limit and a monthly budget; the provider keys move into the gateway's secrets store and are rotated; egress rules block direct calls to provider endpoints from build agents and workloads.
- What stays on the API gateway: partner authentication and request-rate limits at the edge.
- Residual risk: the gateway now concentrates keys, so its admin plane goes on a management network, the master key is vaulted and its hosts follow Securing the AI Gateway.
Detection and measurement
- Log: caller identity, model, provider, input and output tokens, cost, policy verdicts, fallback events and fail-open events for every model call; key creation, scope change and revocation on the admin plane.
- Bypass detection: DNS and egress logs showing workloads reaching provider endpoints without the gateway, and provider billing that exceeds gateway-recorded spend.
- Metric 1: share of model traffic (by tokens) that passes through the gateway; target close to 100 percent.
- Metric 2: number of provider keys held outside the gateway and secrets manager; target zero.
- Metric 3: percentage of gateway keys with an owner, expiry and token budget.
- Metric 4: gap between provider-invoiced spend and gateway-recorded spend per month.
- Metric 5: streamed routes covered by the same inspection policy as non-streamed routes.
See the AI gateway health check for a scored review.
Framework mapping
| Framework | Reference | How it applies |
|---|---|---|
| OWASP Top 10 for LLM Applications 2025 | LLM10 Unbounded Consumption; LLM02 Sensitive Information Disclosure; LLM01 Prompt Injection | Token budgets; redaction and inspection; guardrail checks |
| OWASP API Security Top 10 2023 | API4 Unrestricted Resource Consumption; API2 Broken Authentication | Request limits and caller authentication on both gateways |
| NIST SP 800-53 Rev 5 | SC-5, SC-7, AC-4, IA-5, AU-12, SI-4 | Resource limits, boundary and flow control, key management, logging and monitoring |
| NIST CSF 2.0 | PR.AA-05, PR.DS-02, DE.CM-09, ID.AM-02 | Least-privilege keys, data in transit, monitoring, inventory of model services |
| ISO/IEC 27001:2022 Annex A | 5.23, 8.15, 8.16, 8.20 | Cloud service use, logging, monitoring, network security |
| MITRE ATT&CK v20 and ATLAS | T1496.004, T1078.004, T1552.001; AML.T0034 | Stolen key abuse and cost harvesting the gateway helps detect and limit |
Common failure patterns
- Request limits mistaken for spend control: a familiar rate-limit configuration is copied to model routes and nobody notices the bill until month end.
- The gateway is optional: teams keep direct provider keys "for testing" and production traffic follows, which is the route the LLMjacking campaigns exploit when those keys leak.
- Virtual key sprawl: gateway keys without owners or expiry recreate the provider key problem one layer up.
- Streams exempted: inspection only on non-streamed calls, while the user-facing chat streams everything.
- Per-instance counters: quotas enforced separately by each regional gateway, so the effective budget multiplies with the number of regions.
Practitioner checklist
- Inventory every application and agent calling a model provider and how it authenticates.
- Decide whether to extend your API platform or add an AI gateway, based on documented token, routing and inspection features.
- Move provider keys into gateway custody or replace them with managed or federated identity.
- Issue one gateway key per workload or agent with model allow list, token rate, budget and expiry.
- Set provider-side spend caps as a backstop and alert on gateway budget thresholds.
- Block direct egress to provider endpoints from workloads and build agents.
- Apply inspection consistently to streamed and non-streamed routes and decide fail-open or fail-closed per route.
- Require fallback targets to match the primary's data residency and approval.
- Reconcile provider invoices with gateway-recorded spend monthly.
Standards and references
- OWASP: LLM10:2025 Unbounded Consumption
- OWASP: API4:2023 Unrestricted Resource Consumption
- Microsoft Learn: Limit large language model API token usage
- Kong: AI Rate Limiting Advanced plugin
- Agent Router: Usage-based rate limiting
- LiteLLM: Virtual keys
- AWS: Throttle requests to your REST APIs in API Gateway
- MDN: Using server-sent events
Related NHI Mgmt Group resources: AI Gateway and LLM Gateway Guide · AI Gateway Buyer's Guide · WAF vs API Gateway · API Security Guide · AI Gateway vs CASB/SSE for Controlling Shadow AI