What breaks when a raw self-hosted LLM endpoint is used without a gateway?

Why This Matters for Security Teams

A raw self-hosted LLM endpoint may look simpler to operate, but it removes the control point that security teams need most: a consistent place to mediate identity, policy, logging, and data handling. Without that layer, every application starts speaking to the model directly, which fragments governance and makes prompt, output, and usage controls inconsistent across teams. This is exactly the kind of sprawl highlighted in NHIMG coverage of the McKinsey AI platform breach, where central oversight mattered as much as the model itself.

The issue is not just operational. Once access is embedded directly into apps, it becomes harder to enforce least privilege, redact sensitive data, or prove who used the model and for what purpose. That gap matters even more as AI usage becomes a security perimeter problem, not just an application feature. Guidance from the NIST AI Risk Management Framework and the OWASP Agentic AI Top 10 both point toward governance at the interaction layer, not only at the model host. In practice, many security teams encounter the failure only after separate product teams have already hard-coded direct endpoint access across multiple environments.

How It Works in Practice

A gateway restores a security and governance control plane in front of the model. Instead of every application calling the raw endpoint, the gateway becomes the enforcement point for authentication, routing, policy checks, logging, and data controls. That lets teams centralise how prompts are inspected, how outputs are filtered, and how usage is attributed across business units. It also creates a cleaner path for rate limits, tenant separation, and model version switching without changing each application.

In practice, the gateway should treat the model like a protected service, not a public utility. A useful design usually includes:

• workload identity for each application or agent, so access is tied to cryptographic identity rather than shared static keys
• policy-as-code for request-time decisions, so sensitive prompts can be blocked or transformed based on context
• central logging for prompts, completions, tool calls, and metadata, with privacy filtering where required
• ephemeral secrets or JIT access for downstream systems, instead of long-lived credentials embedded in code
• segregated routes for production, testing, and internal experimentation, so developers do not bypass controls

This pattern aligns with current guidance in CSA MAESTRO agentic AI threat modeling framework and the AI Agents: The New Attack Surface report, which show how often agentic systems exceed intended scope when controls are fragmented. The practical lesson is simple: the raw endpoint is only acceptable for tightly controlled lab use, while production needs a mediation layer that can prove what was sent, who sent it, and what policy allowed it. These controls tend to break down when legacy apps and data science notebooks are allowed to call the endpoint directly because there is no single place to enforce governance.

Common Variations and Edge Cases

Tighter gateway control often increases latency, integration work, and operational ownership, so organisations have to balance model agility against governance. That tradeoff is real, especially in development environments where teams want fast iteration and may resist central approval steps.

Best practice is evolving for hybrid cases. Some teams keep a raw endpoint for offline testing but require the gateway for anything that touches production data. Others use separate gateway policies for internal employees, customer-facing workloads, and autonomous agents. For agentic systems, the bar is higher because an agent may chain prompts, call tools, and surface secrets in ways a human user would not. NHIMG research on the OWASP NHI Top 10 and the LLMjacking research shows why direct credentials and unmediated access are a recurring abuse path. The same design concern appears in external guidance from NIST AI 600-1 Generative AI Profile.

The main edge case is a small internal deployment with one trusted application and no sensitive data. Even there, current guidance suggests planning the gateway early, because direct endpoint access usually becomes technical debt the moment another team wants to reuse the model.

Related resources from NHI Mgmt Group

• What breaks when passkeys are used without endpoint governance?
• What breaks when DAST is used without authenticated coverage and endpoint discovery?
• What breaks when MDM is used without endpoint DLP in a BYOD programme?
• What breaks when LLM output is used directly in application logic without validation?