TL;DR: AI gateways improve routing, rate limiting, budget control, key handling and traffic observability, but Starseer argues they do not secure the model itself because surface-level classifiers miss backdoors, hidden capabilities and agent activity that never looks suspicious. The governance gap is between traffic control and model assurance.
At a glance
What this is: NHI Mgmt Group's analysis, drawing on Starseer's blog post: AI gateways are useful infrastructure for routing and observability but remain insufficient as a security layer for the model itself.
Why it matters: It matters because IAM, NHI and AI security teams need to distinguish traffic governance from model assurance when evaluating controls around AI systems, agents and provider access.
Context
AI gateways sit in the request path and help govern traffic, but they do not prove that the model behind the path is safe. In practice, teams often confuse visibility into prompts and responses with assurance about model behaviour, which leaves hidden capability and backdoor risk untreated.
That gap matters because AI systems now operate across model APIs, tools and connectors, which creates a governance problem that is broader than prompt filtering. Where identity, access and delegation are involved, the control question shifts from what crosses the gateway to what the model is allowed to do inside the workflow.
Key questions
Q: How should security teams evaluate whether an AI gateway is enough for model risk?
A: Treat the gateway as a routing and observability control, not as a model assurance layer. If a tool only inspects prompts and responses, it cannot prove the model is free from backdoors, hidden capabilities or tampering. Teams should require model validation, behavioural baselining and delegated-action controls before claiming security coverage.
Q: When does prompt and response filtering fail as an AI security control?
A: It fails when the threat lives inside the model rather than in the visible text. Backdoors, latent capabilities and manipulated behaviours can produce ordinary-looking inputs and outputs, so surface filtering misses them. Organisations need controls that inspect model integrity and runtime behaviour, not just the words around the model call.
Q: Why do agentic AI systems complicate identity governance more than traditional service accounts?
A: Traditional service accounts usually follow fixed workflows, while agentic systems can choose actions and sequence them at runtime. That makes access governance harder because the risk is not just possession of credentials, but the system's ability to combine privileges across tools and services in ways that static reviews do not capture.
Q: What is the difference between AI traffic observability and AI assurance?
A: Observability tells you what crossed the control point, while assurance tells you whether the model behind it is trustworthy. A gateway can show routing, volume and policy decisions, but it cannot by itself validate model provenance, internal behaviour or hidden capability. Assurance requires deeper inspection and validation.
Technical breakdown
What an AI gateway can actually inspect
An AI gateway observes request and response traffic at the perimeter of model usage. It can route prompts by policy, enforce rate limits and budgets, manage API keys, and run classifiers against text for known patterns such as prompt injection or sensitive data leakage. That makes it a useful control point for traffic governance, but it is still external to the model. It sees the surface of the interaction, not the internal state that drives behaviour. When organisations treat that surface as evidence of safety, they overestimate what the gateway can detect. See the AI Security Platform Buyer's Guide.
Practical implication: Use the gateway for traffic control and observability, not as proof that the model is trustworthy.
Why hidden model behaviour defeats surface monitoring
Backdoors, hidden capabilities and other model-level weaknesses can pass cleanly through a gateway because neither the input nor the output looks obviously malicious. A classifier can only inspect the text it receives, so it is blind to behaviours that are embedded in the model and only triggered under specific conditions. That is why output monitoring and prompt screening remain partial controls: they can reduce exposure to known patterns, but they do not establish model integrity or behavioural assurance. The security issue is not only what the model says, but what it is capable of doing when the surface looks normal. See the MITRE ATLAS Guide.
Practical implication: Add pre-deployment model validation and runtime behavioural checks if you need assurance beyond prompt screening.
How agent workflows expand the control boundary
Once an AI system can call tools and connectors, the control boundary extends beyond the model API. The gateway may still route the model call, but it may not see the full chain of tool use, delegation and downstream actions that the agent performs. That creates a gap between request governance and action governance. In identity terms, the important question becomes which non-human identity is permitted to invoke which tools, under what policy, and with what runtime constraints. Without that view, the organisation governs traffic while missing the execution path that actually creates risk. See the Agentic AI Security Policy Template.
Practical implication: Govern tool access and delegated actions as part of the AI control plane, not as a separate downstream concern.
Threat narrative
Attacker objective: The objective is to make a malicious or compromised model operate inside production while remaining invisible to surface-level security controls.
- Entry occurs through normal prompt traffic that passes an AI gateway and appears benign to request and response classifiers.
- Credential or control abuse emerges when the model contains a hidden capability or backdoor that is not visible at the surface and therefore is not flagged by gateway monitoring.
- Impact follows when the organisation mistakes traffic observability for assurance and allows a compromised or misaligned model to operate inside production workflows.
Breaches seen in the wild
- LiteLLM MCP auth bypass 2026: An exploited LiteLLM MCP auth bypass and default sk-1234 master keys let attackers steal AI gateway master and provider API keys.
- Microsoft SAS token exposure 2023: An over-permissive Azure SAS token in a Microsoft AI GitHub repo exposed 38TB, including workstation backups and Teams messages, for 3 years.
Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.
NHI Mgmt Group analysis
AI gateway governance is not AI security: a gateway is a control point for traffic, policy and observability, but it does not establish whether the model itself is safe. When security teams treat perimeter controls as model assurance, they confuse routing visibility with behavioural trust. The governance failure is architectural, not operational. Practitioners should separate traffic control from model assurance in their AI risk model.
Model assurance has to start inside the model: if security only inspects prompts and outputs, it will miss threats that are encoded in the model rather than expressed in the text. That is why internal-state analysis, behavioural baselining and validation matter for AI governance. The article's core contribution is the reminder that security controls must inspect the thing doing the work, not just the path around it. Practitioners should require evidence about model behaviour, not just request telemetry.
Agentic workflows create a non-human identity problem as well as an AI security problem: once models can use tools and connectors, the real risk sits in delegated action, not only in inference. That means governance must account for which non-human identity can act, which tools it can reach, and what policy constrains those actions. The identity boundary is now part of the AI security boundary, and agentic AI security policy becomes relevant for lifecycle and oversight. Practitioners should govern agent permissions with the same seriousness as human privileged access.
Validation before deployment needs more weight than runtime filtering: if a model is backdoored or tampered with, no amount of prompt screening at the gateway will turn it into a trustworthy system. Pre-deployment checks for integrity and provenance are therefore a first-class control, not a niche hardening step. This shifts the category conversation away from detection-only tooling toward assurance of the model supply chain. Practitioners should demand validation evidence before any model reaches production.
From our research library:
- Only 23% of IT leaders were very confident in their organisation's ability to manage security and governance for GenAI deployments, according to a 2025 Gartner survey of 360 IT leaders.
- Read next: AI Security Platform Buyer's Guide
What this signals
Model assurance debt: organisations that stop at prompt filtering accumulate a form of governance debt where the visible control looks complete but the underlying model remains unverified. That debt grows as more tools, connectors and agent workflows are layered on top of the same incomplete control point. Practitioners should treat each new AI workflow as an additional assurance obligation, not just another traffic route.
The next wave of AI governance will favour controls that can explain model behaviour, not just classify text at the edge. That creates pressure to align security architecture with runtime evidence, provenance checks and delegated-action policy rather than with surface monitoring alone. Where AI agents are involved, the control conversation has already moved from model usage to model accountability.
For practitioners
- Separate traffic governance from model assurance Use the gateway for routing, budgets, rate limits and key handling, but do not treat those functions as evidence that the model is secure or aligned.
- Require model-level validation before deployment Assess integrity, tampering and supply-chain provenance before a model serves production traffic, especially where external models or updates are introduced. See the MITRE ATLAS Guide.
- Govern agent tool access as delegated identity Map which non-human identities can call tools and connectors, then constrain those permissions with explicit policy and oversight.
- Evaluate AI security tools by what they inspect Ask whether a control reads the model or only the request and response surface, then reject any product claim that stops at traffic visibility.
Key takeaways
- AI gateways are useful for routing and observability, but they do not establish that a model is trustworthy or free from hidden behaviour.
- The main gap is governance at the model and agent layers, where surface classifiers cannot see backdoors, delegated actions or internal state.
- Practitioners need a split control model: traffic controls at the gateway, assurance controls at validation and runtime.
Related NHI Mgmt Group resources: AI Security Platform Buyer's Guide · MITRE ATLAS Guide · Agentic AI Security Policy Template
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
OWASP Agentic AI Top 10 and MITRE ATLAS address the attack and risk surface, while NIST AI RMF and NIST CSF 2.0 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| OWASP Agentic AI Top 10 | ASI03 — Identity & Privilege Abuse | Agent tool use and delegated action create the identity and privilege boundary discussed in the article. |
| Recommendation — Constrain agent permissions and tool access so delegated actions stay within explicit policy. | ||
| NIST AI RMF | MAP — Map Context and Risks | The article is fundamentally about mapping where traffic controls stop and model risk begins. |
| Recommendation — Map AI workflows, dependencies and control boundaries before deciding which layer each control covers. | ||
| MITRE ATLAS | TA0006 — Credential Access | The threat discussion includes hidden model behaviour and AI-adjacent abuse patterns that map to adversarial tactics. |
| Recommendation — Hunt for model abuse patterns and credential exposure paths that enable adversarial AI activity. | ||
| NIST CSF 2.0 | PR.AA-05 — Access Permissions, Entitlements and Authorizations | The article stresses permission boundaries for gateways, models and agents. |
| Recommendation — Review AI workflow entitlements so model and agent access are only granted where policy requires them. | ||
Key terms
- AI Gateway: A control point that sits between AI applications and the models, tools, or data they call. In practice, it can authenticate requests, enforce policy, inspect runtime behaviour, and stop unsafe actions before they spread into connected systems.
- Model Assurance: Model assurance is evidence that a model behaves as intended and has not been tampered with, backdoored, or otherwise compromised. It goes beyond monitoring prompts and outputs by checking integrity, provenance, and runtime behaviour that surface controls cannot fully see.
- Agentic workflow: An agentic workflow is a sequence of tasks executed by an AI agent with some level of tool access and decision authority. In security terms, the workflow matters because it can span multiple systems, identities, and permissions, which makes attribution and revocation harder than with ordinary automation.
- Surface Monitoring: Surface monitoring is detection that inspects prompts, outputs, or other externally visible text for suspicious patterns. It can catch known abuse signals, but it cannot see hidden capabilities, internal model state, or behaviour that appears normal at the surface.
Deepen your knowledge
The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity and secrets management. It helps security and IAM teams build the control model that sits beneath AI workflows and delegated access.
Published by the NHIMG editorial team on October 22, 2026.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org