LLM gateways: what access-control gaps should IAM teams close?

TL;DR: Open-source LLM gateways can abstract model routing, but LiteLLM’s gaps around built-in authentication, audit logging, and policy controls push many teams toward alternatives that better fit security and compliance needs, according to Pomerium. The real issue is not model access alone, but whether identity, logging, and policy enforcement exist at the gateway boundary.

NHIMG editorial — based on content published by Pomerium: LiteLLM Alternatives: Best Open-Source and Secure LLM Gateways in 2025

By the numbers:

• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems (39%), inappropriately sharing sensitive data (31%), and revealing access credentials (23%).
• When AWS credentials are exposed publicly, attackers attempt access within an average of 17 minutes.

Questions worth separating out

Q: How should security teams govern access to LLM gateways?

A: They should treat the gateway as the enforcement point for authentication, authorization, and audit logging.

Q: Why do LLM gateways create an NHI security problem?

A: Because many gateway paths rely on service accounts, API keys, and backend tokens that act as non-human identities.

Q: What do teams get wrong about open-source LLM gateways?

A: They often assume that API compatibility implies security completeness.

Practitioner guidance

• Map every LLM request to an identity chain Document the human user, service account, token, and backend credentials involved in each model path so you know which identity actually holds access at enforcement time.
• Centralise policy enforcement at the gateway Apply role, device, location, and request-context rules in one control point instead of duplicating checks across applications and model wrappers.
• Instrument complete request logging Log prompt metadata, model selection, policy decision, and identity context for every request so audit and investigation do not depend on application-specific traces.

What's in the full article

Pomerium's full blog post covers the operational detail this post intentionally leaves for the source:

• Side-by-side feature comparison across LiteLLM, OpenRouter, Portkey, Helicone, LangServe, TrueFoundry, and custom proxies.
• Specific access-control capabilities such as OIDC, SAML, policy engines, and request logging configuration.
• Implementation detail on rotating credentials automatically and preventing token leakage at the proxy layer.
• Practical positioning for teams deciding whether to secure LLM traffic with an identity-aware proxy or a gateway-only stack.

👉 Read Pomerium’s comparison of LiteLLM alternatives for secure LLM access →

LLM gateways: what access-control gaps should IAM teams close?

Explore further

View Full Forum →  |  NHI Foundation Course →