Self-hosted open-weight models behind one AI gateway

At a glance

What this is: This analysis says open-weight LLMs are easiest to govern when they are exposed through one AI gateway that treats self-hosted and commercial models as the same control surface.

Why it matters: For IAM and platform teams, that means routing, budgets, tracing, and access control can stay unified while sovereignty-driven workloads move behind private inference infrastructure.

Context

Self-hosting open-weight models changes the control plane more than the model choice itself. The problem is not just where inference runs, but whether the organisation can avoid creating a second, separate stack for authentication, routing, observability, and policy enforcement when a workload cannot use a third-party API.

In practice, the governance issue is consistency. A self-hosted model that sits behind the same gateway as commercial models lets teams keep one decision surface for access and routing while still meeting sovereignty or cost requirements. That is a genuine identity and access management intersection because the gateway becomes the place where model access, model names, and policy boundaries are governed.

Key questions

Q: What breaks when self-hosted models are added outside the AI gateway?

A: The control plane fragments. Applications start seeing different API shapes, routing rules, budgets, and audit paths depending on which backend serves the request, which makes governance inconsistent and operational troubleshooting harder.

Q: Why do self-hosted models need the same access policy as commercial APIs?

A: Because the security problem is not where the model runs, but whether model access is authorised consistently. If policy lives in multiple places, users can bypass governance by choosing a different backend, and regulators will still view the workload as one system.

Q: How do teams know if model routing is creating governance drift?

A: Look for duplicate dashboards, backend-specific credentials, inconsistent tracing, or workloads that behave differently depending on model name. Those are signs that routing decisions are no longer governed by one policy layer and that the platform has become harder to audit.

Q: Should organisations separate self-hosted inference from commercial model governance?

A: No. A split governance model usually creates the very stack sprawl self-hosting was meant to avoid. The better pattern is one policy framework, one routing layer, and distinct backend choices only underneath that shared control surface.

Technical breakdown

Why open-weight models create stack duplication

Open-weight models do not remove operational complexity, they relocate it. Once a team self-hosts, it owns the serving engine, GPU capacity, scaling behaviour, reliability, and the API shape exposed to applications. Without a gateway abstraction, the organisation usually ends up with a second interface, a second observability path, and a second policy layer that diverge from the commercial-model path. The result is not just more infrastructure, but more governance drift: different routing rules, different budgets, and different audit visibility for workloads that should be managed consistently.

Practical implication: Treat self-hosting as a control-plane decision and force it through the same gateway, policy, and telemetry path as commercial model traffic.

How a shared gateway preserves access control and routing

The key mechanism is API unification. If a self-hosted Llama or Mistral instance speaks the same OpenAI-compatible request shape as a commercial model, the application does not need to know which backend it hit. That allows routing, fallback, RBAC, budgeting, and tracing to attach at the gateway rather than inside each application. In identity terms, the gateway becomes the policy enforcement point for model access, while backend choice remains an implementation detail. This reduces the risk that one class of model runs outside the organisation's normal governance model.

Practical implication: Map model access policy to the gateway layer, not to application code paths or backend-specific credentials.

Why self-hosted inference changes the supply-chain and runtime risk model

Self-hosting shifts trust from a third-party provider to the organisation's own deployment chain. The weights, serving image, and supporting dependencies become artifacts that must be pinned, verified, scanned, and rolled back like any other production workload. It also introduces operational behaviours that commercial APIs hide, such as cold starts, image streaming, shared volumes, and prefix-aware routing for KV-cache reuse. Those are performance mechanisms, but they also affect how predictable and governable the service is under load. In a self-hosted environment, the risk boundary expands from API consumption to runtime operations.

Practical implication: Govern model images, weights, and registry publishing with the same rigor used for production application artifacts.

Threat narrative

Attacker objective: The practical objective in this pattern is not intrusion but control-plane fragmentation, which weakens consistent governance over who can use which model and under what policy.

1. Entry occurs when an organisation adds self-hosted inference without collapsing it into the existing gateway and policy layer, creating a parallel stack with its own API surface.
2. Credential and access controls then fragment, because the new serving path can be governed differently from the commercial model path and may bypass established routing, tracing, or budget enforcement.
3. Impact is governance drift, where model choice, access decisions, and audit trails no longer live in one control plane, making policy inconsistent across workloads.

Breaches seen in the wild

• LiteLLM MCP auth bypass 2026: An exploited LiteLLM MCP auth bypass and default sk-1234 master keys let attackers steal AI gateway master and provider API keys.
• LiteLLM PyPI package breach: LiteLLM PyPI supply chain attack, credentials stolen from users.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Gateway unification is now an IAM control problem, not just an LLMOps convenience. Once open-weight models are routed through the same interface as commercial APIs, the main governance question becomes whether access policy, routing, and telemetry remain centralised. If they do not, the organisation has not simplified AI operations, it has duplicated its control plane. The practical conclusion is to make the gateway the authoritative enforcement point for model access.

Self-hosting only changes the risk equation when the surrounding stack is different. The model itself may be the same class of workload, but the hosting environment changes the trust boundary to include GPUs, serving engines, images, and model artifacts. That creates a broader governance surface than a third-party API ever exposed. Practitioners should treat the hosted inference path as a production service with identity, supply-chain, and runtime controls.

Open-weight adoption creates 'model sprawl' unless policy follows the routing layer. When teams can switch between commercial and self-hosted models by name, the dangerous failure mode is not lack of capability, but lack of consistent control inheritance. The named concept here is model routing sprawl: the proliferation of model endpoints with different governance rules. The practitioner response is to keep routing, budgets, tracing, and authorisation attached to the gateway rather than to individual backends.

The identity lesson is that model choice should not determine control strength. A regulated workload should not become less governed because it moves to a self-hosted model, and a commercial workload should not receive stronger policy simply because it is easier to route. That principle aligns naturally with zero standing privilege thinking for AI access: the system should grant only the minimum model route required for the task. The conclusion is to standardise policy before scaling model diversity.

From our research library:

• AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
• Read next: Agentic AI Identity Maturity Model

What this signals

Model routing sprawl: the operational risk is not that enterprises will use too many models, but that they will govern them through too many control paths. When self-hosted and commercial inference sit behind different policy surfaces, the organisation loses a single view of access, spend, and auditability, which weakens both AI governance and IAM discipline.

Open-weight adoption should be read as a control-plane consolidation problem. The technical win is not self-hosting by itself, but making model placement invisible to the application while keeping governance visible to the platform. That is the pattern that lets regulated workloads move to private inference without creating a shadow AI estate.

For practitioners

• Centralise model access at the gateway Keep self-hosted and commercial model traffic behind one policy and routing layer so applications do not learn which backend is serving the request.
• Apply consistent RBAC to model names and routes Authorize model selection at the gateway so a regulated workload cannot silently bypass approval by switching to a different backend or endpoint.
• Govern serving images and weights as production artifacts Pin model versions, verify artifacts where possible, and scan serving images and dependencies before they are promoted into the self-hosted path.
• Benchmark the serving engine against real workloads Test vLLM, SGLang, TGI, or TRT-LLM against your actual sequence lengths and concurrency before standardising on one serving stack.
• Plan for cold starts and cache reuse Use model caching, shared volumes, and prefix-aware sticky routing where applicable so autoscaling does not collapse under large weight downloads.

Key takeaways

• Self-hosted LLMs create governance value only when they do not create a second stack for access, routing, and observability.
• The central problem is control-plane fragmentation, not model quality, because fragmented routes and policies are harder to audit and standardise.
• A shared gateway lets organisations keep one policy surface while choosing commercial or self-hosted backends according to sovereignty, cost, or availability needs.

Key terms

• AI Gateway: A control point that sits between AI applications and the models, tools, or data they call. In practice, it can authenticate requests, enforce policy, inspect runtime behaviour, and stop unsafe actions before they spread into connected systems.
• Open-Weight Model: A model whose parameters can be downloaded and run locally by the operator. In practice, that means safety behaviour can be altered outside the provider’s environment, so governance has to focus on the runtime, the operator, and the abuse path, not only the model brand.
• Prefix-Aware Routing: A routing method that sends requests with shared prompt prefixes to the same model instance so cached attention state can be reused. It improves latency and efficiency on repeated or conversational workloads, but it also requires the gateway to understand request affinity.
• KV Cache: A KV cache stores key and value tensors produced during transformer attention so later tokens can reuse them instead of recomputing the entire prompt. In production serving, cache locality strongly affects latency, throughput, and GPU cost because the cache lives on the node that processed the earlier tokens.

Deepen your knowledge

NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course. It is built for security and platform teams that need to govern access, lifecycle, and trust across modern AI estates.