TL;DR: A self-hosted AI gateway pushes the data plane into Kubernetes while keeping prompts, provider keys and logs inside infrastructure you control, according to NeuralTrust, but it still depends on external sync, telemetry and policy paths. That means private deployment changes where trust is enforced, not whether trust assumptions exist.
At a glance
What this is: This is a deployment guide for a self-hosted AI gateway on Kubernetes, with the key finding that keeping prompts and keys inside your cluster does not remove external dependency or policy risk.
Why it matters: IAM, NHI and platform teams need to treat gateway deployment as an identity and control-plane design problem because provider keys, application keys and delegated tool access all sit in the same governed path.
By the numbers:
- A standalone Private data plane needs 2 entry points, with port 8081 for LLM traffic and port 8082 for MCP.
- A standalone Private data plane needs at least 2 nodes, with 8 vCPU and 16 GiB each.
Context
A self-hosted AI gateway is a control layer that sits between applications and model providers, then routes both LLM and MCP traffic through infrastructure you operate. In practice, that means the trust boundary moves into your Kubernetes cluster, but the gateway still depends on external sync, telemetry and provider connectivity.
The governance question is not whether the gateway is self-hosted, but which identities, secrets and logs stay under your control and which still transit managed services. That distinction matters for teams responsible for AI agent authorisation and for any programme that treats application keys as governed credentials rather than simple configuration.
Key questions
Q: How should security teams govern secrets in a self-hosted AI gateway?
A: Treat gateway secrets as production credentials, not deployment metadata. Separate ownership for registry keys, sync tokens, enrollment material and provider credentials, then enforce rotation, revocation and recovery procedures. The practical goal is to prevent one gateway from becoming a long-lived concentration point for machine credentials that support both prompt routing and model access.
Q: Why do AI gateway architectures matter when organisations use multiple LLMs and MCP servers?
A: AI gateway architectures matter because they centralise control over model traffic, tool access, and auditability. Without a common enforcement layer, each integration becomes a separate risk surface with inconsistent authentication, logging, and policy. That fragmentation makes it harder to govern agentic workflows, compare providers, and respond quickly when behaviour changes or a model is swapped.
Q: What happens when a self-hosted AI gateway loses control-plane connectivity?
A: A running data plane may continue serving its last-known-good configuration for a period, but it can stop receiving updates and a restarted pod may fail to become ready. That means resilience depends on how the gateway handles stale state, cache rebuilds and reconnect behaviour, not just on Kubernetes uptime.
A: Security teams should bind provider keys to a gateway-managed access key, then let the gateway use the attached credential upstream. That keeps raw provider secrets out of application code, developer environments, and policy logic. It also gives one place to control authorization, rotate credentials, and reduce the chance that a leaked app secret exposes direct provider access.
Technical breakdown
How the data plane and control plane split changes trust boundaries
The deployment pattern separates request handling from configuration and orchestration. The data plane receives prompts, forwards them to providers and returns responses, while the control plane stores policy, console state and gateway configuration. In the Private model, the data plane runs in your cluster and pulls configuration from the vendor control plane, so the cluster initiates the connection and raw payload handling stays local. That design narrows data exposure, but it also creates a dependency on reachability, token validity and trusted certificate chains for config sync and telemetry.
Practical implication: Treat control-plane reachability and trust anchors as production dependencies, not install details.
Why separate LLM and MCP entry points matter for governance
The guide uses two distinct public interfaces: one for LLM traffic on port 8081 and one for MCP traffic on port 8082. That separation matters because tool calls and model calls carry different risk profiles. A single URL can silently break MCP resolution, and tool governance becomes ambiguous if both traffic types are collapsed into one entry point. Separate hostnames also make it easier to apply different routing, logging and policy controls to prompt traffic versus tool execution traffic. Read AI Agents vs Agentic AI.
Practical implication: Keep LLM and MCP endpoints distinct so tool governance does not collapse into generic API routing.
Why secrets management becomes the real control surface
The install requires Kubernetes Secrets for the registry key, config sync token and enrollment material, and it optionally generates or preserves additional values depending on GitOps mode. The provider credential stays inside the gateway, while application callers get their own key, which turns the gateway into a credential broker. If secret creation, rotation or preservation is handled loosely, the gateway becomes a concentration point for long-lived credentials rather than a control point for short-lived access.
Practical implication: Govern gateway secrets as privileged credentials with explicit ownership, rotation and recovery handling.
Threat narrative
Attacker objective: The attacker wants to intercept prompts, steal provider or application keys, or force the gateway into a state where policy and tool routing no longer behave as intended.
- Entry occurs through the gateway's public LLM or MCP endpoints, or through the control-plane sync path that keeps the data plane updated.
- Credential access is concentrated in registry keys, config sync tokens, provider credentials and application keys, all of which can expose the brokered trust chain if mishandled.
- Impact is loss of prompt confidentiality, provider key exposure, broken tool calls or stale policy enforcement across the routed AI workload.
Breaches seen in the wild
- Carbonato botnet 2026: A botnet puts a Hermes AI agent on exposed Docker hosts to steal AI API keys, SSH keys and tokens, powering the operators' own LLM gateway.
- Secrets in Docker Hub images (RWTH Aachen study): A 2023 RWTH Aachen study found secrets in 8.5% of container images, and 275,269 internet hosts still using the leaked private keys.
Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.
NHI Mgmt Group analysis
Self-hosted AI gateways create a new identity control plane, not just a new deployment model. The gateway becomes the broker for provider keys, application keys and delegated tool access, so the security question shifts from where the pods run to how trust is issued, scoped and revoked. That makes gateway governance adjacent to AI agent authorisation, because the same platform is deciding what an application or agent may do next. Practitioners should treat the gateway as an identity-enforcing choke point, not a transport component.
The strongest concept here is prompt-and-key locality with external-control dependence. The guide shows that prompts can remain inside the cluster while config sync, telemetry and provider connectivity still depend on outside systems. That means residency is only one layer of the control story, because updates, trust anchors and health checks can still fail or drift. Teams need to evaluate whether their residency claim covers raw data only or the full operating path.
Separate LLM and MCP traffic or you weaken governance at the point of use. The article makes clear that MCP resolution can fail silently when both flows are bootstrapped from one URL, which turns a routing shortcut into a governance problem. The operational lesson is that tool execution deserves its own control boundary, because model prompts and tool calls do not carry the same risk or oversight requirements. Practitioners should design policy and telemetry around the traffic type, not just the endpoint.
Secret concentration is the hidden fragility in self-hosted AI gateways. Registry access, sync tokens, enrollment credentials and provider keys all converge in one deployment path, so a single governance mistake can create broad exposure. That is an NHI problem as much as an AI problem, because these credentials are machine identities with lifecycle, scope and revocation needs. Teams should govern them as a credential estate, not as chart values.
High availability is a control decision before it is an uptime decision. The default replica counts, disabled autoscaling and last-known-good cache behaviour show that resilience depends on explicit operator choices, not just Kubernetes defaults. If the control plane disappears, a running pod can keep serving stale configuration for a time, but a restarted pod may not recover cleanly. The practical conclusion is that gateway resilience must be designed alongside policy and secret governance, not after it.
What this signals
Prompt-and-key locality is becoming a governance pattern, not just a deployment preference. Teams are trying to keep prompts local while still relying on sync, telemetry and provider services outside the cluster, which means the true policy boundary is wider than the pod network. The next maturity step is to govern trust anchors, outbound dependencies and log flows as one operating model.
AI gateway programmes will be judged by credential lifecycle discipline. Once the gateway brokers provider keys and application keys, the question becomes whether those machine credentials are scoped, rotated and recoverable like any other privileged identity. Organisations that cannot explain that lifecycle will struggle to prove the gateway is enforcing governance rather than merely proxying requests.
For practitioners
- Define separate governance for LLM and MCP routes Publish distinct hostnames, policies and telemetry paths for prompt traffic and tool traffic so a single URL does not collapse two different control problems.
- Treat gateway secrets as a privileged credential estate Assign ownership for registry keys, config sync tokens, enrollment material and provider credentials, then rotate and recover them as production secrets.
- Validate control-plane reachability from inside the cluster Test outbound 443 access to sync, telemetry and DataBridge endpoints from the data plane, and confirm certificates are trusted without relying on insecure settings.
- Keep application keys separate from provider keys Issue each application its own key and retain provider credentials in the gateway so callers never inherit broader model-provider access than they need.
- Build for recovery before turning on production traffic Set pod disruption budgets, verify replica placement and decide how the gateway behaves when the control plane is unavailable or the cache must be rebuilt.
Key takeaways
- Self-hosting an AI gateway changes where control sits, but it does not eliminate the need to govern trust, secrets and external dependencies.
- The deployment path concentrates registry keys, sync tokens, provider credentials and application keys, which makes credential lifecycle the main operational risk.
- Separate LLM and MCP flows, validate control-plane reachability and treat the gateway as an identity-enforcing control point rather than a simple proxy.
Related NHI Mgmt Group resources: AI Agent Authorisation Guide · AI Agent Observability, Audit and Incident Response Guide · AI Agents vs Agentic AI · AI Agent Identity Security Buyer's Guide
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Non-Human Identity Top 10 and MITRE ATT&CK address the attack and risk surface, while NIST SP 800-53 Rev 5 and NIST CSF 2.0 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Non-Human Identity Top 10 | NHI-02 — Secret Leakage | The gateway concentrates registry keys, sync tokens and provider credentials that can leak or be mishandled. |
| NHI-04 — Insecure Authentication | The gateway relies on token-based sync, enrollment and application authentication paths. | |
| NHI-05 — Overprivileged NHI | Provider credentials and application keys must be scoped so the gateway does not become over-authorised. | |
| Recommendation — Scan gateway secret paths and revoke exposed machine credentials before they can be reused. Harden gateway authentication flows and remove fallback paths that weaken credential assurance. Constrain gateway-issued and provider-bound credentials to the minimum access each workflow needs. | ||
| NIST SP 800-53 Rev 5 | IA-5 — Authenticator Management | Credential lifecycle management is central to gateway secrets, tokens and application keys. |
| Recommendation — Apply IA-5 to manage issuance, rotation and revocation for gateway credentials and tokens. | ||
| NIST CSF 2.0 | PR.AA-05 — Access Permissions, Entitlements and Authorizations | The gateway brokers access for prompts, providers and tools, making authorisation a core control. |
| Recommendation — Align gateway permissions and entitlements so tool and model access are explicitly authorised. | ||
| MITRE ATT&CK | TA0006;TA0010 — Credential Access; Exfiltration | The article's threat model centres on stealing keys or prompts through the gateway path. |
| Recommendation — Map gateway exposures to credential access and exfiltration techniques in your detections. | ||
Key terms
- Self-Hosted AI: Self-hosted AI is artificial intelligence infrastructure that runs on an organisation’s own systems rather than a third-party managed service. It keeps model execution, data handling, and control boundaries inside the operator’s environment, which can support tighter governance, custom security controls, and clearer accountability for prompts, outputs, logs, and connected identities.
- Data Plane: The data plane is where operational access to content occurs, including prompts, outputs, logs, training data, and secrets. In AI governance, this is the layer where over-privileged identities often expose sensitive information even when the control plane appears tightly managed.
- Control Plane: The control plane is the set of actions that create, configure, or manage a service. For AI workloads, it covers deployment and administration of the model platform, while data-plane permissions govern what the service and its identities can read or process.
- Machine Issued Credential Lifecycle: The set of controls that govern creation, review, rotation, and revocation of credentials issued to non-human actors. It matters because initial issuance is only one point in the life of access, and unmanaged persistence is where many identity failures begin.
Deepen your knowledge
NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security and secrets management. It helps practitioners connect identity controls to the operational realities of gateways, workloads and delegated access.
Published by the NHIMG editorial team on October 22, 2026.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org