How should security teams choose between a self-hosted LLM gateway and a managed SaaS gateway?

Why This Matters for Security Teams

Gateway choice is really a governance decision about how much control must sit inside the security boundary versus a provider’s service boundary. That matters because LLM gateways are no longer just routing layers; they mediate prompts, responses, tool calls, secrets, and audit trails. When autonomous systems are involved, the gateway can become the enforcement point that determines whether a model or agent is allowed to retrieve data, call an API, or persist context. Current guidance suggests that the wrong deployment model creates blind spots in logging, policy consistency, and data residency.

Security teams should weigh this against the actual exposure pattern. NHIMG’s AI Agents: The New Attack Surface report found that 80% of organisations say their AI agents have already acted beyond intended scope, which makes runtime control more important than convenience alone. A self-hosted gateway can improve control over routing and inspection, while a managed SaaS gateway can reduce operational burden if the vendor’s logging, retention, and tenancy model match policy requirements. The risk is choosing based on procurement speed and discovering later that governance cannot be proven during an investigation. In practice, teams usually learn this only after a prompt, tool call, or data export has already crossed an ownership boundary.

How It Works in Practice

Start by mapping what the gateway must govern, not just where it will run. If the gateway brokers sensitive prompts, internal retrieval, tool execution, or agent traffic, the decision should consider data classification, identity integration, policy evaluation, and audit depth. A self-hosted design usually fits environments that need direct control over secrets, custom routing, and internal SIEM integration. A managed SaaS gateway can fit lower-risk use cases where the security team can accept provider-managed storage and standard controls.

For AI-heavy environments, the gateway should support more than request forwarding. It should enforce identity-aware policy, log model and tool usage, and apply rules consistently at request time. That aligns with the direction described in NIST AI Risk Management Framework and the OWASP Agentic AI Top 10, both of which emphasize runtime governance and misuse resistance. In practice, teams should confirm:

• Whether prompts, responses, and tool calls are retained, encrypted, and inspectable under their own policy.
• Whether identity can be tied to users, services, and agents rather than only to an API key.
• Whether policy can block sensitive destinations, redact data, and enforce allowlists before the request leaves the boundary.
• Whether logs are complete enough for incident response, compliance review, and lateral-movement analysis.

NHIMG’s The State of Non-Human Identity Security shows the visibility gap is still large, which is why gateway telemetry must be treated as security evidence rather than a convenience feature. These controls tend to break down when teams connect the gateway to sprawling multi-agent workflows, because tool chaining and delegated credentials quickly outrun static routing rules.

Common Variations and Edge Cases

Tighter gateway control often increases integration and operating overhead, requiring organisations to balance enforcement strength against deployment speed and vendor reliance. That tradeoff is usually acceptable for regulated workloads, but it is not free. Best practice is evolving, and there is no universal standard for this yet, especially for teams running mixed human and agent traffic through the same gateway.

Some environments need a hybrid model. A managed SaaS gateway may handle low-risk developer traffic, while a self-hosted gateway handles production systems, regulated data, or agentic workloads with tool access. That separation can reduce operational load without giving up the highest-value controls where they matter most. This is also where the distinction between NHI governance and agent governance matters: an agent may need just-in-time access, short-lived credentials, and context-aware approval that a basic SaaS proxy cannot express. The OWASP NHI Top 10 and Ultimate Guide to NHIs — Lifecycle Processes for Managing NHIs are useful when the real question is whether the gateway can support lifecycle control, not just traffic mediation.

Edge cases include startups that need speed first, enterprises with strict residency requirements, and agentic systems that call internal tools from ephemeral compute. Managed SaaS can be sufficient if the vendor’s controls are contractually and technically aligned with policy. Self-hosted is usually stronger when the organisation needs custom data handling, deep telemetry, or direct revocation authority. The right answer is the one that can prove governance under audit, not the one that is easiest to buy.

Related resources from NHI Mgmt Group

• How should security teams choose between managed and self-hosted CIAM?
• How should security teams choose between self-managed cloud PKI, SaaS PKI, and PKIaaS for enterprise use cases?
• How should teams choose between managed and self-hosted identity platforms?
• How should security teams decide between a lightweight gateway and a full identity provider for self-hosted apps?