What breaks when organisations rely on direct model access instead of a gateway?

Why This Matters for Security Teams

Direct model access removes the control point that many security teams quietly depend on for governance, auditability, and incident response. A gateway is not just a routing layer. It is where policy checks, identity binding, content inspection, quota enforcement, and logging can be applied consistently before requests reach a model or provider. Without that layer, teams often discover that access is technically working while governance has already failed.

This matters because model traffic can contain secrets, regulated data, or prompts that trigger unsafe behavior, and without central mediation there is no reliable way to see what was sent, what came back, or who approved the interaction. The issue also extends to machine identities. If application services, agents, or automation jobs call models directly, their credentials become harder to govern and rotate, which is exactly the kind of concentration risk highlighted by the OWASP Non-Human Identity Top 10. In practice, many security teams encounter the failure only after a data exposure, cost spike, or provider outage has already made direct access visible as an operational weakness rather than a design choice.

How It Works in Practice

A gateway reintroduces control boundaries between applications and model endpoints. It can authenticate the caller, attach a workload or agent identity, enforce allowlists, inspect prompts and outputs, and decide whether a request should proceed. It also creates a single telemetry path for security monitoring, chargeback, and abuse detection. That makes it easier to apply the kinds of access and logging expectations described in NIST SP 800-53 Rev 5 Security and Privacy Controls, especially where organisations need evidence of controlled access and traceable system activity.

In mature environments, the gateway usually handles several functions at once:

• central API key or token mediation so keys are not embedded in every application
• request filtering to block disallowed topics, unsafe tool calls, or sensitive data leakage
• response inspection to detect policy violations before downstream systems act on them
• rate limits and quotas to prevent runaway token consumption
• routing logic for failover across model providers, versions, or regions
• logging and correlation so security teams can reconstruct who asked for what and when

This is especially important where agentic workflows or service accounts call models on behalf of users, because the model interaction becomes part of a broader machine-to-machine trust chain. Identity governance becomes relevant here too: if the gateway cannot bind requests to a known workload identity, it becomes much harder to distinguish approved automation from shadow usage. Current guidance suggests that organisations should treat model access like any other privileged dependency, with explicit approval, monitoring, and revocation paths. These controls tend to break down in high-volume, low-latency environments where engineering teams bypass the gateway to remove perceived latency or deployment friction, because decentralised direct calls quickly fragment identity, logging, and policy enforcement.

Common Variations and Edge Cases

Tighter gateway control often increases operational overhead, requiring organisations to balance governance against latency, developer speed, and provider flexibility. That tradeoff is real, and best practice is evolving rather than universally settled for every workload. Some teams use a hybrid pattern, where low-risk internal prototyping goes direct while production traffic is forced through a gateway. Others apply the gateway only at the edge of shared services, with local controls inside trusted clusters.

Edge cases matter. Streaming responses may need different inspection logic than single-shot completions. Multi-provider failover can complicate policy parity if each model endpoint has different moderation behavior or logging formats. Agentic systems add another layer of complexity because tool use, retrieval, and model calls may all need separate enforcement decisions. In regulated environments, the gateway may also need to preserve evidence for investigations, retention, or customer notification obligations, which makes retention policy and log integrity part of the design, not an afterthought.

Where direct access can be acceptable is usually narrow: isolated experimentation, short-lived proofs of concept, or workloads with no sensitive data and no external exposure. Even then, teams should treat that as temporary. If a system starts with direct calls and later becomes business-critical, retrofitting a gateway often reveals missing identity binding, incomplete logs, and inconsistent controls that are painful to rebuild after the fact. For that reason, OWASP Non-Human Identity Top 10 remains highly relevant wherever application tokens, agent credentials, or service identities are allowed to reach model APIs directly.

Related resources from NHI Mgmt Group

• What breaks when organisations rely on one AI gateway for content, routing, and access control?
• What breaks when organisations rely on observability instead of access control?
• What breaks when organisations rely on persistent access instead of just-in-time controls?
• What breaks when organisations rely on passwords and OTPs for high-risk access?