Notifications
Clear all
Topic starter
21/08/2026 4:56 pm
TL;DR: Self-hosting open-weight models is less about benchmark performance than about sovereignty, cost at scale, and operational control, according to TruFoundry’s analysis of running Llama and Mistral behind an AI gateway. The real governance issue is avoiding a second, separate stack so routing, budgets, tracing, and access controls stay unified across commercial and self-hosted models.
NHIMG editorial — based on content published by TruFoundry: Self-hosting open-weight models behind the AI gateway
Questions worth separating out
Q: How should security teams govern self-hosted AI models alongside commercial APIs?
A: Use a single gateway to apply access control, routing policy, tracing, and budget enforcement across both paths.
Q: Why do self-hosted models create more governance risk than commercial APIs?
A: Because the organisation now owns the serving layer, GPU capacity, scaling logic, and artefact lifecycle.
Q: What breaks when model routing is handled differently for self-hosted and external LLMs?
A: Auditability breaks first, then access policy consistency, then fallback governance.
Practitioner guidance
- Standardise on one model gateway Put commercial and self-hosted model access behind the same gateway so routing, RBAC, budgets, and tracing are applied in one control surface rather than duplicated per backend.
- Separate policy from backend choice Define which workloads may use self-hosted inference, which may fail over to external providers, and which are prohibited from leaving the environment for regulatory reasons.
- Harden model artefact supply chains Pin model and serving-image versions, verify checksums or signatures where available, control registry publishing rights, and keep rollback paths for both weights and engines.
What's in the full article
TruFoundry's full post covers the operational detail this post intentionally leaves for the source:
- Benchmark comparisons across vLLM, SGLang, TGI, and TRT-LLM for different workload profiles.
- Implementation details for model caching, shared volumes, and image streaming to reduce cold starts.
- Prefix-aware sticky routing patterns for KV-cache reuse across repeated prompts and sessions.
- Configuration examples showing how self-hosted and commercial models are governed through one gateway.
👉 Read TruFoundry's analysis of self-hosted open-weight models behind the AI gateway →
Open-weight models behind one gateway: what changes for IAM teams?
Explore further
21/08/2026 5:13 pm
Self-hosted model governance is really control-plane governance. Once a model runs on your own GPUs, the security question moves from vendor dependency to internal policy consistency. The gateway is the important control boundary because it lets routing, access control, tracing, and budgets apply uniformly across commercial and self-hosted models. Practitioners should treat this as an extension of identity governance into AI operations, not as a separate experimentation lane.
A question worth separating out:
Q: When should organisations keep a model fully self-hosted instead of allowing failover?
A: Keep it fully isolated when data sovereignty, contractual restrictions, or regulatory constraints prohibit any external processing path. In those cases, failover to a commercial provider can violate policy even if it improves uptime, so the routing rule must be explicit and non-negotiable.
👉 Read our full editorial: Self-hosted open-weight models behind one AI gateway
