LiteLLM vs Pomerium: access control gaps in AI gateway stacks

At a glance

What this is: This is a comparison of LiteLLM and Pomerium showing that model routing and identity-aware access control solve different problems in AI gateway stacks.

Why it matters: IAM, IGA, PAM, and platform security teams need to separate API abstraction from enforcement so AI access does not outgrow existing identity controls.

Context

AI gateway stacks often combine model routing, user authentication, and policy enforcement, but those are not the same control layer. When teams treat them as interchangeable, they end up with a gateway that can translate requests while still exposing weak access governance around LLMs and MCP-connected services.

The governance gap is simple: an LLM gateway can simplify developer integration, but it does not by itself decide who may use the service, from which device, or under what conditions. That distinction matters because AI systems increasingly sit in front of sensitive data and internal tools, where identity-aware access control is part of production security, not an add-on.

Key questions

Q: What breaks when LLM access is integrated directly into each application instead of using a shared gateway?

A: Direct integrations usually fragment security boundaries. Teams duplicate credentials, enforce policies inconsistently, and lose a reliable audit trail for who used which model and with what data. Over time, this increases shadow usage, makes incident review harder, and raises the chance that sensitive prompts or regulated workloads bypass the intended controls.

Q: Why do AI gateways need identity, policy, and observability attached to every request?

A: Without request level identity and policy context, AI usage becomes hard to attribute, govern, and investigate. Observability alone only shows traffic, not intent or accountability. When every request carries user, team, project, and session metadata, teams can link spend to outcomes, enforce budget and safety rules, and spot misuse before it turns into runaway cost or risk.

Q: What are the signs that AI gateway logging is not enough for governance?

A: If logs show traffic volume but cannot tie each request to a person, device, or access decision, they are operational telemetry rather than audit evidence. Governance breaks when a security team can see usage but cannot explain or review the permissioning behind it.

Q: How should teams secure MCP-connected services behind an AI gateway?

A: They should apply the same identity, device, and policy checks to MCP-connected services that they use for the gateway itself. If the upstream tool layer is looser than the front door, the gateway only hides where the real exposure begins.

Technical breakdown

LLM gateway routing vs identity enforcement

LiteLLM is described as a translation layer between applications and multiple model APIs, using an OpenAI-compatible endpoint to abstract provider differences. That helps developers route traffic, fail over between models, and track usage without changing application code. Pomerium, by contrast, sits in front of HTTP services and authenticates users before policy is applied. The two layers are orthogonal: one normalises model access, the other governs who can reach the gateway at all. In practice, confusion starts when teams assume that a gateway with API keys and logging is also an access-control boundary.

Practical implication: separate model routing from identity enforcement and place explicit policy in front of the gateway.

Why context-aware policies matter for AI infrastructure

The article emphasises policies based on identity, device status, time, and other context. That matters because AI access is rarely just a yes or no decision. A policy engine can allow the same user in one condition and deny them in another, which is closer to enterprise access governance than a static allowlist. For AI infrastructure, context becomes part of the control itself, especially when the request may trigger external tool use, data retrieval, or downstream actions through MCP servers. Static API key checks cannot express those conditions well enough for production governance.

Practical implication: use contextual policy to govern AI requests, not just credential presence.

Audit logging and compliance boundaries in gateway design

Pomerium’s value proposition in the article is tied to access logging with identity metadata, tamper-resistant audit trails, and support for compliance workflows. That is a materially different control objective from LiteLLM’s usage logging and retry logic. Logging only becomes governance when the record is attributable to a person, device, or session and can be used for review, investigation, or control validation. For regulated environments, the question is whether every LLM call can be tied back to an access decision that a security or compliance team can examine later.

Practical implication: require auditable identity context on each AI request if the gateway participates in regulated workflows.

Threat narrative

Attacker objective: The objective is to use permissive AI gateway access to reach models, connected tools, or sensitive data without effective identity governance.

1. Entry occurs when an AI gateway is exposed with weak or minimal authentication, allowing users or services to reach model endpoints without meaningful identity checks.
2. Privilege escalation follows when API keys or broad gateway permissions are reused across internal services, making the gateway a high-value path into LLM-connected resources.
3. Impact is reached when unauthorised requests can access sensitive prompts, internal data, or MCP-connected services without an attributable audit trail.

Breaches seen in the wild

• LiteLLM MCP auth bypass 2026: An exploited LiteLLM MCP auth bypass and default sk-1234 master keys let attackers steal AI gateway master and provider API keys.
• LiteLLM PyPI package breach: LiteLLM PyPI supply chain attack, credentials stolen from users.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

AI gateway stacks expose an identity boundary problem, not just an integration problem: model routing and access enforcement are different controls, and treating them as one creates governance blind spots. LiteLLM can abstract provider access, but that does not satisfy the need to decide who may invoke the stack, from where, and under what conditions. Practitioners should treat gateway placement as a control design decision, not a developer convenience choice.

Context-aware policy is now the real access layer for production AI: identity, device posture, time, and session context are the variables that determine whether a request should proceed. That is especially true once MCP-connected services or internal data sources are in play, because the gateway is no longer just returning model output. Security teams should assume static API key controls are insufficient for that operating model.

Tamper-resistant audit logging is a governance requirement, not a compliance accessory: AI request logs only become useful when they can be tied to a real identity and a specific session. Without that linkage, teams can see traffic but cannot explain access decisions or reconstruct misuse. The practical conclusion is that AI gateway controls must produce evidence a security team can actually investigate.

Secure agentic access depends on reducing the trust placed in the gateway itself: once tools, models, and internal services are reachable through the same path, the gateway becomes a policy enforcement point that must be continuously constrained. Identity-aware proxying changes the problem from "can the model be called" to "should this actor be allowed to use this capability right now". Teams should evaluate AI gateway design through that lens.

What this signals

AI gateway governance is converging on policy at the request boundary: the most useful control point is no longer the model endpoint alone, but the combination of identity, device context, and session state that decides whether a request is allowed. Teams that already run zero trust patterns for web apps can extend the same logic to AI infrastructure without changing the underlying governance model.

Model abstraction does not remove accountability: when one endpoint can reach multiple providers and internal tools, the question becomes who is responsible for the access decision, the audit trail, and the downstream action. That is why security and platform teams need a shared control plane for AI access, not a developer-only gateway.

Zero Trust for AI agents is the right mental model for this stack even when the article is not about autonomous behaviour yet. The access path to models, data, and MCP servers should be treated as continuously verified, because the risk is the same: a broad trust boundary around a high-value execution layer.

For practitioners

• Implement identity-aware access in front of AI gateways Place an access proxy or equivalent control ahead of LLM gateway traffic so user identity, device status, and context are checked before requests reach model APIs.
• Separate routing from authorisation Treat model abstraction, failover, and provider selection as application concerns, and keep authorisation in a distinct control layer with explicit policy.
• Require full-context audit logs for AI requests Capture identity, session, device, and policy decision data for each LLM call so investigators can reconstruct who accessed what and why.
• Constrain access to MCP-connected services Apply the same identity and context rules to upstream MCP servers and other internal tools that AI workflows can reach through the gateway.
• Replace static API key trust with short-lived credentials Use session-scoped access and rotate credentials so a single leaked secret does not become a standing path into AI infrastructure.

Key takeaways

• LiteLLM-style routing solves integration complexity, but it does not by itself enforce who may access AI services or under what conditions.
• Identity-aware access control becomes the decisive layer once LLM gateways sit in front of sensitive data and MCP-connected services.
• Teams should separate abstraction, authorisation, and audit so AI infrastructure remains governable as usage scales.

Key terms

• AI Gateway: A control point that sits between AI applications and the models, tools, or data they call. In practice, it can authenticate requests, enforce policy, inspect runtime behaviour, and stop unsafe actions before they spread into connected systems.
• Identity-aware Proxy: An identity-aware proxy combines routing with authentication and authorization logic. It checks tokens or certificates, applies policy at the edge, and forwards verified identity context to the backend so applications do not have to re-implement security decisions inconsistently.
• Context-Aware Policy: Context-aware policy is a control model that decides access based on current conditions, not just preassigned entitlement. For AI agents and other non-human identities, this means privileges, tool use, and monitoring expectations can change as the task, environment, or risk signal changes.
• Tamper-Evident Audit Trail: A tamper-evident audit trail is a log record that captures who invoked an AI agent, which tool it called, what policy decision was made, and when it happened. It is designed for forensic reconstruction, compliance review, and incident response, especially when actions happen faster than humans can supervise them.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity lifecycle are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building or maturing an IAM programme, it is worth exploring.