An AI gateway ends up holding every model provider key, seeing every prompt and deciding every routing and spend limit, so choosing one is a security and identity decision as much as a platform one. Products sold as AI gateways or LLM gateways range from open-source proxies you run yourself to managed services and features bolted onto API gateways, and their security defaults differ widely. This vendor-neutral buyer's guide sets out how to define requirements, which capability areas to score, the questions that separate products, red flags and a proof of concept that tests real failure modes. It is for platform, security and procurement teams preparing a shortlist or an RFP. The gateway pattern itself, including architecture and credential design, is explained in the AI Gateway and LLM Gateway Guide; this page helps you buy one.
Key takeaways
- Buy for the credential model first. Per-caller virtual keys or federated identity with model allowlists, budgets and expiry matter more than the number of supported providers.
- Insist on fail-closed authentication. AI gateways have already been exploited through authentication that failed open and documented example admin keys left in place.
- Count tokens, not requests. Spend and abuse scale with tokens, which OWASP LLM10:2025 Unbounded Consumption ties to Denial of Wallet.
- Decide where prompts may live. Deployment model (self-hosted, private cloud, SaaS) decides who processes prompt data and holds provider keys.
- Prove bypass is impossible. A gateway only controls traffic forced through it, so test egress controls alongside the product.
Define requirements
Write down the use cases, callers and constraints before talking to vendors. These questions decide which deployment model and feature tier you need.
- Callers: internal applications, AI agents, developer tools, or all three? Agents need distinct identities so their usage and actions can be attributed. See the AI Agent Identity Security Buyer's Guide.
- Providers and models: which commercial APIs, cloud-hosted models and self-hosted models must be reachable, and whether some data classes may only go to specific providers or regions.
- Data handling: may prompts and responses leave your boundary? Must logs be redacted? How long must records be retained for audit?
- Spend control: budgets per team, application or agent, hard stops or alerts, and chargeback reporting.
- Tool traffic: will the gateway also front Model Context Protocol (MCP) servers? If so, read the MCP Gateway Buyers Guide alongside this page.
- Scale and resilience: expected requests and tokens per second, acceptable added latency, and failover between providers.
| Deployment model | Fits when | What you take on |
|---|---|---|
| Self-hosted open-source proxy | Strong engineering team, prompts must stay in your boundary, cost sensitivity | Patching, hardening defaults, high availability, your own support |
| Commercial software in your cloud | Prompts must stay in your boundary but you want vendor support and enterprise features | Operations and upgrades; licence terms |
| Cloud provider gateway service | Most models consumed through one cloud, existing identity and logging there | Coupling to that cloud; check multi-provider support |
| SaaS gateway | Fast start, small team, mixed providers | The vendor processes prompts and holds provider keys: supplier risk under NIST CSF 2.0 GV.SC-07 and SP 800-53 SA-9 |
Small organisations with one or two providers can often start with a managed gateway, per-application keys and provider-side spend caps as a backstop. Large organisations usually need a self-hosted or private deployment, federation with the corporate identity provider, central logging and a policy baseline shared across business units.
Capability areas
- Caller identity and credentials: per-caller virtual keys or OAuth and workload identity federation; model allowlists, budgets, rate limits and expiry per key; individual revocation. Upstream provider keys held in your secrets manager or encrypted at rest (SP 800-53 IA-5, SC-28).
- Admin plane security: single sign-on and role-based access for administrators, separation of admin and data planes, no default or example master key, and audit of admin changes.
- Routing and resilience: provider abstraction, fallbacks, retries, load balancing and region pinning for data residency.
- Consumption controls: token-based quotas and budgets, maximum output tokens, request size limits and alerts, covering the rate limiting and quota controls in OWASP LLM10.
- Content policy hooks: pre- and post-call checks for sensitive data, prompt injection and output policy, either built in or through callouts to guardrail services. See the AI Guardrails and AI Firewalls Guide.
- Telemetry and audit: per-request records of caller, model, tokens, cost, latency and policy decisions, exported to your SIEM and observability tools; support for the OpenTelemetry GenAI semantic conventions eases integration. See OpenTelemetry GenAI Semantic Conventions for Security Teams.
- Caching: exact or semantic response caching with tenant and user isolation so one caller never receives another's cached answer.
- Egress and SSRF safety: restrictions on which upstream URLs administrators can configure, blocking private and cloud metadata ranges.
Evaluation criteria
| Criterion | What good looks like | Evidence to request | Weight |
|---|---|---|---|
| Authentication failure behaviour | Invalid, expired or malformed credentials always rejected; no unauthenticated mode in production builds | Live negative tests; security advisory history | High |
| Credential model | Per-caller keys or federated identity, scoped to models and budgets, with expiry and instant revocation | Demonstration of creating, scoping and revoking a key | High |
| Admin plane | SSO, RBAC, no default keys, admin API not exposed with the data plane | Architecture diagram; hardening guide | High |
| Data handling | Configurable prompt logging and redaction; clear retention; no training on your data | Data processing terms; log samples | High |
| Consumption control | Token budgets per caller with hard stops; output token caps | Budget breach test during the proof of concept | Medium |
| Telemetry | Structured events with caller, model, tokens, cost and decision; SIEM export | Event schema; connector documentation | Medium |
| Policy integration | Pre- and post-call hooks with measured added latency | Benchmark under your traffic profile | Medium |
| Resilience | High availability, provider failover, documented behaviour when the gateway is down | Failover test results | Medium |
| Vendor security practice | Published disclosure process, timely fixes, independent testing | Penetration test summary; advisory timelines | Medium |
Questions to ask vendors
- What happens when a caller presents an invalid, expired or malformed token, on every endpoint including any MCP or admin routes? Show us.
- Does the product ship with any default, example or shared administrator key? What stops a deployment going live with it?
- How are upstream provider keys stored, who can read them, and can they live in our secrets manager?
- Can each application or agent have its own credential with model allowlists, token budgets, rate limits and expiry? How fast does revocation take effect across all nodes?
- Are budgets enforced on tokens or on requests, and are they hard stops?
- Where are prompts, responses and logs processed and stored, for how long, and are they ever used to train models?
- Can administrators configure arbitrary upstream URLs, and how do you prevent routes to internal addresses or cloud metadata services?
- How is cached content isolated between tenants and users?
- What is the added latency at our expected load with policy hooks enabled?
- Which vulnerabilities have been reported in the product in the past two years, and how quickly were fixes released and disclosed?
Red flags
- Authentication that fails open: in the 2026 AI gateway MCP authentication bypass, CVE-2026-59822 in an open-source gateway let failed token validation open a session, and the flaw was added to the CISA Known Exploited Vulnerabilities catalogue on 2 September 2026. Ask any vendor how its product behaves on validation failure, then test it.
- Documented example keys that work in production: the same incident page reports internet-facing gateways still accepting a documented example master key.
- One shared key for everyone: no per-caller credentials means no attribution, no per-caller budget and no targeted revocation.
- Request-only rate limiting: limits on requests per minute without token budgets leave Denial of Wallet open.
- Admin console on the same listener as the data plane: anything that reaches the inference endpoint can attempt administration.
- Vague data terms: no clear answer on prompt retention, location or model training.
Proof of concept
- Set written success criteria: authentication behaviour, credential scoping, budget enforcement, logging completeness, added latency and operational effort.
- Connect real callers: one internal application, one agent and one developer tool, each with its own credential and a different model allowlist.
- Run negative authentication tests: no token, expired token, malformed token and a revoked key against every endpoint, including admin and any MCP routes. Every one must fail.
- Breach a budget: drive one caller past its token budget and confirm a hard stop and an alert.
- Attempt an internal route: as an administrator, try to add an upstream pointing at a private address or a cloud metadata endpoint and confirm it is refused.
- Check the logs: confirm each request in your SIEM shows caller, model, tokens, cost and policy decision, with redaction applied.
- Test bypass: from a pilot workload, try to call a provider API directly with a personal key; egress controls should block it.
- Fail the gateway and a provider: confirm failover and fail-closed behaviour match your design.
After selection, track a few measures: share of model traffic through the gateway, callers with individual credentials, budget breaches caught before invoice, mean time to revoke a credential, and days to patch gateway advisories.
How NHI Mgmt Group can help
We provide independent market analysis, RFP strategy and evaluation support for NHI and agentic AI identity programmes, including AI gateway selection. Browse vendors in our NHI and AI products directory or contact us about independent evaluation support.
Related NHI Mgmt Group resources: AI Gateway and LLM Gateway Guide · MCP Gateway Buyers Guide · AI Guardrails and AI Firewalls Guide · AI Agent Identity Security Buyer's Guide · AI Security Platform Buyer's Guide