Notifications
Clear all
Topic starter
12/06/2026 12:12 am
TL;DR: Open-source LLM gateways can abstract model routing, but LiteLLM’s gaps around built-in authentication, audit logging, and policy controls push many teams toward alternatives that better fit security and compliance needs, according to Pomerium. The real issue is not model access alone, but whether identity, logging, and policy enforcement exist at the gateway boundary.
NHIMG editorial — based on content published by Pomerium: LiteLLM Alternatives: Best Open-Source and Secure LLM Gateways in 2025
By the numbers:
- 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems (39%), inappropriately sharing sensitive data (31%), and revealing access credentials (23%).
- When AWS credentials are exposed publicly, attackers attempt access within an average of 17 minutes.
Questions worth separating out
Q: How should security teams govern access to LLM gateways?
A: They should treat the gateway as the enforcement point for authentication, authorization, and audit logging.
Q: Why do LLM gateways create an NHI security problem?
A: Because many gateway paths rely on service accounts, API keys, and backend tokens that act as non-human identities.
Q: What do teams get wrong about open-source LLM gateways?
A: They often assume that API compatibility implies security completeness.
Practitioner guidance
- Map every LLM request to an identity chain Document the human user, service account, token, and backend credentials involved in each model path so you know which identity actually holds access at enforcement time.
- Centralise policy enforcement at the gateway Apply role, device, location, and request-context rules in one control point instead of duplicating checks across applications and model wrappers.
- Instrument complete request logging Log prompt metadata, model selection, policy decision, and identity context for every request so audit and investigation do not depend on application-specific traces.
What's in the full article
Pomerium's full blog post covers the operational detail this post intentionally leaves for the source:
- Side-by-side feature comparison across LiteLLM, OpenRouter, Portkey, Helicone, LangServe, TrueFoundry, and custom proxies.
- Specific access-control capabilities such as OIDC, SAML, policy engines, and request logging configuration.
- Implementation detail on rotating credentials automatically and preventing token leakage at the proxy layer.
- Practical positioning for teams deciding whether to secure LLM traffic with an identity-aware proxy or a gateway-only stack.
👉 Read Pomerium’s comparison of LiteLLM alternatives for secure LLM access →
LLM gateways: what access-control gaps should IAM teams close?
Explore further
12/06/2026 9:17 am
LLM gateways are becoming identity control points, not just routing layers. The article’s core signal is that model abstraction alone does not solve governance. Once a gateway handles traffic for multiple providers, it becomes the enforcement point for authentication, policy, and audit evidence across human users and non-human identities. Practitioners should treat the gateway as part of the identity plane, not an app convenience layer.
A few things that frame the scale:
- 92% agree governing AI agents is critical to enterprise security, yet only 44% have implemented any policies to do so, according to AI Agents: The New Attack Surface report.
- Only 52% of companies can track and audit the data their AI agents access, leaving 48% with a complete blind spot for compliance and breach investigation, according to SailPoint.
A question worth separating out:
Q: When should organisations add identity-aware controls to their LLM stack?
A: They should add them before model usage becomes multi-provider, multi-team, or production-critical. Once requests cross several applications and credentials are shared across workflows, the audit and access problem becomes harder to retrofit. Early control placement is cheaper than reconstructing governance later.
👉 Read our full editorial: LiteLLM alternatives expose the access-control gap in LLM gateways
