AI Gateway and LLM Gateway Guide: Routing, Policy and Cost Control

When every team calls model providers directly, nobody can say which applications use which models, who holds the provider keys, what data leaves the organisation or what the monthly bill will be. An AI gateway (also called an LLM gateway) is a reverse proxy that sits between applications or agents and model providers, giving one place to authenticate callers, route requests, enforce policy, cap spend and record telemetry. This guide covers the gateway pattern itself: architecture choices, the credential model, routing and failover, token budgets, caching, policy enforcement and how to operate the gateway safely, since it becomes one of the most sensitive services you run. It is written for platform engineers, security architects and AI platform teams.

Key takeaways

• The gateway is a credential concentrator. It holds every upstream provider key and issues its own keys to callers, so a gateway compromise is a compromise of all your AI spend and much of your prompt data.
• Issue one credential per caller, never one shared key. Per-application, per-team or per-agent virtual keys with model allow lists, budgets and expiry are what make attribution, rate limiting and revocation possible.
• Count tokens, not just requests. Cost and abuse scale with tokens; gateways that only limit requests per minute leave Denial of Wallet (OWASP LLM10:2025) wide open.
• Never expose the admin plane. Default master keys and internet-facing admin APIs have already led to real compromises of AI gateways in 2026.
• Policy only works if the gateway is the only path. Pair the gateway with egress controls so applications cannot bypass it by calling provider endpoints directly with a personal key.

What an AI gateway does, and what it is not

An AI gateway terminates the client request, applies policy, translates the request to the target provider's API, attaches the upstream credential and returns the response, usually exposing one OpenAI-compatible API to clients regardless of the provider behind it. Many now also proxy Model Context Protocol (MCP) traffic.

See the Secure Web Gateway and Web Filtering Guide for the user-traffic side, the AI Guardrails and AI Firewalls Guide for content inspection and the MCP Gateway and MCP Server Governance Guide for tool traffic.

Architecture and deployment choices

Gateways come as open-source proxies you run (for example LiteLLM Proxy, Envoy AI Gateway, now renamed Agent Router, and Kong AI Gateway), as cloud provider services, and as SaaS gateways where the vendor operates the proxy and therefore sees your prompts and holds your provider keys.

• Two-tier pattern: the Envoy AI Gateway documentation describes a Tier One gateway as the central entry point for authentication, top-level routing and global rate limiting, and a Tier Two gateway in front of a self-hosted model serving cluster for fine-grained control and inference-aware endpoint picking. It is configured through Kubernetes resources such as AIGatewayRoute, AIServiceBackend and BackendSecurityPolicy.
• Central versus federated: a single central gateway gives one policy and one audit trail but is a shared failure domain; federated gateways per business unit or region reduce latency and blast radius but need a common policy baseline and telemetry schema so they do not drift.
• Self-hosted versus SaaS: self-hosting keeps prompts and keys inside your boundary but makes patching your problem. A SaaS gateway is a third-party processor of prompt data and a holder of credentials, so treat it under supplier risk (NIST CSF 2.0 GV.SC-07, SP 800-53 SA-9) and check where logs are stored. See the Third-Party AI Risk Guide.

Where identity fits: gateway keys, virtual keys and per-agent credentials

A gateway has two credential relationships, and both are non-human identities. The Envoy AI Gateway upstream authentication documentation describes them as client-to-gateway authentication and gateway-to-upstream authentication.

Upstream: gateway to provider

• Prefer short-lived federation: Envoy AI Gateway can obtain temporary AWS credentials through OIDC and AWS STS for Amazon Bedrock and short-lived Entra ID tokens for Azure OpenAI, instead of static keys.
• Static provider keys: where providers only issue API keys, store them in a secrets manager or Kubernetes secret readable only by the gateway, split keys by environment and business unit, set provider-side spend caps and rotate them on a schedule and on any suspicion. See the Secrets Management Guide.
• Per-tenant upstream credentials: Envoy AI Gateway's credentialOverride lets a trusted external authorisation filter supply the provider credential per request, so each tenant is billed to its own provider account through shared routes.

Downstream: caller to gateway

• Admin or master key: LiteLLM's proxy uses a master key (general_settings: master_key or LITELLM_MASTER_KEY) to administer the proxy and generate keys. It is a privileged credential: generate it randomly, vault it, never use it from applications and restrict the admin routes to a management network.
• Virtual keys: keys generated through /key/generate carry their own controls in LiteLLM: models (allow list), max_budget and budget_duration, tpm_limit, rpm_limit, max_parallel_requests, duration (expiry) and team_id or user_id. Kong's equivalent grouping is AI Consumer Groups, which scope model access and token budgets per team.
• Per-agent credentials: give each agent its own key or, better, have it present a workload identity token (OIDC, SPIFFE) that the gateway validates, so an agent's spend, models and tools can be capped and revoked without affecting others. See the Identity Governance for AI Agents Guide.

The scope boundary with the LLMjacking Guide is simple: that guide covers stolen provider keys; the gateway is how you stop handing provider keys to applications in the first place. See also the API Key Management Guide.

Routing, failover and continuity

• Routing strategies: LiteLLM documents simple-shuffle (its default and recommended production setting, weighted by rpm, tpm or weight), usage-based-routing-v2, latency-based-routing, least-busy and cost-based-routing. Kong and Envoy offer equivalent load balancing across providers and deployments.
• Fallbacks: configure an ordered list of alternative model groups or providers. LiteLLM separates general fallbacks from context_window_fallbacks and content_policy_fallbacks; Envoy AI Gateway falls back on network errors and 5xx responses, with retries controlled through a BackendTrafficPolicy.
• Cooldowns and retries: LiteLLM cools a deployment down after allowed_fails (default 3 per minute) for cooldown_time (default 5 seconds), and retries according to num_retries or a per-error RetryPolicy. Cap retries: an aggressive retry loop multiplies cost during a provider incident.
• Security rules for fallback: a fallback target must meet the same data residency, contractual and model approval requirements as the primary. A silent fallback from a regional, zero-retention deployment to a public endpoint in another jurisdiction is a data transfer you did not approve.
• Model aliases: expose stable logical names (for example chat-default) to applications and map them to real models at the gateway, so you can retire a model or respond to a provider incident without code changes, and record every alias change as a configuration change.
• Gateway continuity: the gateway itself becomes a single point of failure. Run it highly available, decide in advance whether it fails open or closed for each route (fail closed for regulated data), and test provider outage and gateway outage separately.

Token budgets, rate limits and cost control

OWASP LLM10:2025 Unbounded Consumption lists Denial of Wallet, variable-length input floods and model extraction through the API among its examples. The relevant MITRE ATLAS techniques are AML.T0034 Cost Harvesting and AML.T0040 AI Model Inference API Access, and stolen keys used for free inference map to ATT&CK T1496.004 Cloud Service Hijacking.

Set limits at three levels: per caller (key or agent), per team (budget) and per provider (upstream quota), and keep a provider-side hard cap as the last line of defence in case the gateway is bypassed.

Caching

• Exact and semantic caching: exact caching returns a stored response for an identical request; semantic caching (Kong AI Semantic Cache, LiteLLM semantic caching backends) returns a stored response for a prompt that is similar enough by embedding distance.
• Cross-tenant leakage: a shared cache can return one user's answer, including personal or confidential data drawn from their context, to another user who asks a similar question. Partition cache keys by tenant, application and, where responses are personalised, by user, and exclude routes that carry regulated data.
• Cache poisoning: anyone who can write to the cache can shape later answers. Protect the cache store (Redis, vector database) like the gateway, with authentication and network isolation.
• Freshness and correctness: set TTLs, and keep semantic similarity thresholds strict for anything where a slightly different question needs a different answer.

Policy and guardrails at the gateway

The gateway is the natural enforcement point because it sees every request and response in a normalised form. Kong's AI Gateway illustrates the range of plugins: AI Prompt Guard (topic and keyword allow and deny lists), AI Semantic Prompt Guard (natural-language jailbreak and prompt injection detection), AI Sanitizer (PII redaction before the request reaches the provider), AI Semantic Response Guard (response checks), AI Prompt Template (fixed prompts and default instructions) and connectors to cloud safety services such as AI AWS Guardrails, AI Azure Content Safety and AI GCP Model Armor.

Treat guardrail verdicts as signals with a measured false positive rate, run new rules in log-only mode first, and record which policy version produced each block. See the Generative AI Data Loss Prevention Guide and the Prompt Injection Guide.

Egress control and shadow AI

• Make the gateway the only route: block direct egress from workloads to provider API hostnames at the firewall, proxy or service mesh, and allow only the gateway's egress identity (NIST SP 800-53 SC-7 Boundary Protection, AC-4 Information Flow Enforcement).
• Remove direct provider keys: scan code and secrets stores for provider keys issued outside the gateway and replace them with virtual keys.
• Find bypasses: egress logs that show provider hostnames from anything other than the gateway are your shadow AI list for service traffic; browser use of public AI tools is found by the SWG. See the Shadow AI Discovery Guide.

Securing the gateway itself

• Admin plane: keep admin UIs and key-management APIs off the internet, behind private networking or an identity-aware proxy with SSO and MFA for human administrators.
• Patching: gateways are fast-moving open-source projects that now include MCP servers, test endpoints and pass-through routes. Track their advisories and treat critical fixes as urgent: an internet-facing gateway is a T1190 Exploit Public-Facing Application target.
• Host identity: run the gateway with a least-privilege cloud role and block or harden instance metadata access, because pass-through routes and server-side request features can be turned against it.
• Supply chain: pin gateway packages and images to verified versions and keep CI publishing credentials away from steps that run third-party code. The LiteLLM PyPI package breach in March 2026 shipped malicious versions that harvested LLM API keys, cloud tokens and kubeconfig files from any host that installed them.
• Logs: decide deliberately whether prompts and responses are logged. LiteLLM offers turn_off_message_logging and redact_user_api_key_info; Envoy AI Gateway's OpenInference tracing records full request and response content unless OPENINFERENCE_HIDE_INPUTS and OPENINFERENCE_HIDE_OUTPUTS are set.

Implementation roadmap

1. Inventory: list applications, agents and pipelines that call model providers, the keys they use and current monthly spend per key.
2. Deploy the gateway privately: generate a random admin key, store provider credentials in a vault or use workload federation, put the admin plane on a management network and enable telemetry export.
3. Pilot one or two applications: issue virtual keys with model allow lists, budgets and expiry; run token limits and guardrails in log-only mode for two to four weeks to set realistic thresholds.
4. Migrate in rings: move applications team by team, revoking each team's direct provider keys once its traffic is on the gateway.
5. Enforce: switch limits and guardrails to blocking, then block direct egress to provider endpoints except from the gateway.
6. Operate: review budgets and key inventory monthly, rotate upstream keys, and run outage drills for provider and gateway failure.

Rollback: keep the previous gateway configuration version and route map in version control so a bad policy can be reverted in minutes; during migration keep the old provider key for each team disabled rather than deleted until the team has run cleanly through the gateway for a full billing cycle, then delete it.

Detection and measurement

• Log per request: caller key or agent identity, end-user identifier, route and model alias, resolved provider and model, input and output tokens, cost, latency, policy verdicts and error type. The OpenTelemetry GenAI conventions (gen_ai.provider.name, gen_ai.request.model, gen_ai.usage.input_tokens) give a common schema, and Kong and Envoy both emit them. See the LLM Observability Guide.
• Alert on: spend or token spikes per key, a key used from a new network or workload, admin API calls, key creation outside change windows, repeated authentication failures, guardrail block surges and provider egress that does not originate from the gateway.

Framework mapping

Common failure patterns

• Default or shared admin keys on an exposed gateway. The LiteLLM MCP auth bypass 2026 (CVE-2026-59822, fixed in 1.84.0) let any bearer token reach the MCP endpoint, and a Wiz scan reported in the same write-up found that 294 of 3,074 internet-facing gateways accepted the documented example master key sk-1234 and 191 had no authentication at all.
• One gateway key shared by every application. Spend cannot be attributed, a leak cannot be contained without breaking everyone, and rate limits become meaningless.
• Request limits without token limits. A single agent loop with large contexts burns through a monthly budget in hours while staying under requests per minute.
• The gateway is optional. Teams keep direct provider keys "for testing", so the policy covers only the compliant traffic.
• Unpartitioned semantic cache. A shared cache serves one customer's personalised answer to another.
• Prompt logs nobody planned for. Full prompt and response capture is switched on by default in some tracing integrations, creating an unclassified store of personal and confidential data.

Lessons from incidents

The LiteLLM MCP case shows the full chain a gateway enables once compromised: in honeypots, attackers combined the bypass with a command injection bug (CVE-2026-42271), read the master key from process memory, harvested upstream provider keys and deployed cryptominers. Because the master key also signs session tokens, anyone holding it can forge sessions. After any exposure, rotate the master key, every upstream provider key and connected MCP credentials, not just the one that was seen.

Practitioner checklist

• Inventory every application and agent calling model providers, with its key and spend.
• Run the gateway with a randomly generated, vaulted admin key and the admin plane on a private network.
• Use workload federation for upstream providers where supported; vault and split static provider keys.
• Issue a virtual key or workload identity per application and agent, each with owner, expiry, model allow list, token limit and budget.
• Configure token-based limits and budgets at caller, team and provider level, plus provider-side hard caps.
• Restrict fallbacks to models that meet the same residency and data handling approval.
• Partition caches by tenant and exclude regulated routes.
• Decide prompt logging deliberately and enable redaction before export.
• Block direct egress to provider endpoints except from the gateway.
• Subscribe to the gateway project's security advisories and patch critical issues as emergency changes.

Standards and references

• OWASP: Top 10 for LLM Applications 2025
• OpenTelemetry: Semantic Conventions for Generative AI
• MITRE ATLAS: Cost Harvesting
• NIST: SP 800-53 Rev. 5
• Envoy AI Gateway (Agent Router): Project documentation and upstream authentication
• LiteLLM: Virtual Keys
• Kong: AI Gateway documentation
• MITRE ATT&CK: Cloud Service Hijacking

Related NHI Mgmt Group resources: AI Security Platform Buyer's Guide · AI Agent Observability and Incident Response Guide · LLM Security Guide · API Security Guide · Zero Trust for AI Agents Guide