TrueFoundry and TrojAI define a secure AI gateway stack

At a glance

What this is: This is an analysis of how a unified AI gateway and runtime defense model separates control from enforcement for production AI traffic.

Why it matters: It matters because IAM, NHI, and AI platform teams now have to govern AI agents and model traffic as a live access pathway, not just a hosting layer.

👉 Read TROJ.AI's analysis of the TrueFoundry and TrojAI AI gateway integration

Context

Enterprise AI traffic has become an identity and policy problem as much as an application problem. As organisations route requests across multiple model providers, they need a way to centralise authentication, quotas, logging, and routing without losing control over what the model can see or return.

For AI agents and applications, the governance gap is no longer just model quality. It is the split between who can send requests, what those requests are allowed to contain, and how responses are screened before they reach users or downstream systems.

Key questions

Q: How should security teams govern AI model access in production?

A: Security teams should govern AI model access through a centralized control layer that enforces authentication, quotas, logging, and routing policy consistently across applications. The goal is to make model access visible and reviewable as an identity pathway, not as isolated application logic. That lets IAM, platform, and security teams apply one set of rules to many AI workloads.

Q: Why do AI gateways need runtime enforcement as well as observability?

A: Observability tells you what the AI system did, but it does not prevent prompt injection, leakage, or unsafe output in the moment. Runtime enforcement adds the ability to block, redact, or pass traffic before harmful content reaches the model or exits the system. Without that inline layer, policy remains retrospective rather than protective.

Q: What do teams get wrong about protecting AI traffic at scale?

A: The common mistake is assuming a routing layer alone can provide security. Routing decides where traffic goes, but it does not decide whether content is safe, compliant, or free of sensitive data. Teams need separate security decisions, shared policy review, and clear ownership for both the control plane and the enforcement plane.

Q: Should organisations treat AI agents as part of NHI governance?

A: Yes. AI agents consume credentials, access models through APIs, and move sensitive context across services, which makes them part of the broader non-human identity control surface. Governance should therefore cover their service accounts, tokens, quota limits, offboarding, and logging just as it would for other high-value machine identities.

How it works in practice

AI gateway control plane for model access

An AI gateway sits in front of model providers and acts as the policy and routing layer for AI traffic. In this pattern, the gateway centralises authentication, rate limits, quotas, request logging, and provider selection, while also capturing operational signals such as latency, token usage, cost, and failures. The architectural value is separation of control from application code, so teams can govern model access consistently across many AI workloads. For IAM teams, the gateway becomes part of the entitlement path for AI agents and applications, not just an API management layer.

Practical implication: treat gateway policies as identity controls and review who can invoke which models, under what quotas, and with what logging depth.

Runtime AI firewall and inline enforcement

A runtime AI firewall inspects prompts and responses while traffic is flowing, rather than after the fact. That lets security teams block, redact, or pass requests based on policy, including detection of prompt injection, jailbreak attempts, system prompt leakage, data exfiltration, and unsafe content. The important technical point is enforcement happens inline, so the control can stop malicious or policy-breaking content before it reaches the model or returns to the caller. This is materially different from observability alone, which can tell you what happened but cannot prevent it.

Practical implication: place enforcement where traffic actually moves, and verify that the proxy can stop both inbound abuse and outbound leakage.

Policy separation between routing and security decisions

The strongest design pattern here is splitting traffic governance from threat enforcement. Routing logic decides where traffic should go based on cost, latency, policy, or failover, while the security layer decides whether the request or response is acceptable. That separation matters because AI systems often combine user input, retrieved context, and model output in ways that create new leakage and manipulation paths. Without distinct control points, organisations either over-trust routing layers or overload security tools with infrastructure decisions they were not built to make.

Practical implication: define which layer owns routing, which layer owns blocking, and how policy drift is detected when those responsibilities blur.

NHI Mgmt Group analysis

AI gateway governance is becoming a control plane problem, not an application feature. Once enterprises route multiple models and AI agents through a shared path, access control, logging, and policy decisions must move out of scattered application code and into a governable layer. That shifts AI security into the same discipline family as IAM and NHI oversight, where policy consistency matters more than one-off guardrails. The practitioner takeaway is to manage AI traffic as a privileged access channel.

Runtime enforcement closes the gap that observability alone cannot. Visibility into tokens, cost, latency, and failures is useful, but it does not stop prompt injection, data leakage, or unsafe output at the moment it occurs. The real value of an inline policy layer is that it converts AI risk from a post-incident review problem into a real-time containment problem. The practitioner takeaway is to require blocking or redaction capability wherever AI traffic touches sensitive data.

Identity and policy boundaries for AI agents must extend beyond the model call. AI systems increasingly combine user input, retrieved context, and downstream actions, so the security boundary cannot stop at authentication to the gateway. That creates a named concept we would call the AI traffic enforcement gap: the space between permitted model access and acceptable model behaviour. The practitioner takeaway is to define policy across request, response, and logging stages as one lifecycle.

This pattern validates a layered AI governance model, but it also complicates ownership. Platform teams may own the gateway while security teams own the firewall, yet neither can claim complete control if policies diverge or if logs do not feed a shared review process. In NIST CSF terms, the issue is not only Protect and Detect, but also governance alignment across control points. The practitioner takeaway is to assign a single accountable owner for AI request policy.

AI agents are now part of the broader NHI control surface. Even when the article focuses on model routing, the underlying problem is that non-human actors are making requests, consuming quotas, and carrying sensitive context across services. That means AI governance cannot be isolated from workload identity, secrets, and access policy. The practitioner takeaway is to review AI agent access the same way you review other high-value NHI pathways.

From our research:

• Two-thirds of enterprises have endured a successful cyberattack resulting from compromised non-human identities, with a quarter encountering multiple attacks, according to The 2024 ESG Report: Managing Non-Human Identities.
• Enterprises that have experienced a compromised NHI averaged 2.7 separate incidents in the past 12 months, which shows how quickly identity weakness compounds across environments.
• For a broader view of where governance fails across machine identities, see NHI Lifecycle Management Guide and use it to align provisioning, rotation, and offboarding around AI traffic paths.

What this signals

AI traffic governance is moving into the same risk category as workload identity management. Once model access, routing, and response filtering are part of the production path, the control question becomes who can use the AI system and under what conditions. In practice, that means teams should align gateway policy, secrets handling, and offboarding around the same operational owner. The AI traffic enforcement gap is the place where policy exists on paper but not in the request path.

The governance pressure will increase as AI agents are treated less like demos and more like production workloads with quotas, credentials, and downstream effects. Our research shows that two-thirds of enterprises have endured a successful cyberattack resulting from compromised non-human identities, which is a reminder that identity exposure scales quickly once machine actors are in the path. Teams should prepare for AI control planes to become audit evidence sources, not just infrastructure tooling.

That shift also means platform and security teams will need a shared view of AI request policy, because routing decisions and security decisions now influence the same transaction. Where the gateway chooses model paths and the firewall decides content acceptability, operational ownership must be explicit. Organisations that cannot separate those responsibilities cleanly will struggle to prove control effectiveness during reviews or incidents.

For practitioners

• Map AI gateway policies to identity controls Document which teams can access which models, what authentication is required, and how quotas and logging are enforced across environments. Treat the gateway as part of entitlement governance, not just traffic management.
• Require inline enforcement for sensitive AI traffic Use a control that can block, redact, or pass requests in real time when prompt injection, system prompt leakage, or data exfiltration is detected. Log both the decision and the policy version that made it.
• Separate routing policy from security policy Define which layer selects the model or failover path and which layer decides whether the content is acceptable. Reconcile both layers through a shared review process so drift does not create blind spots.
• Review AI agent access as NHI access Inventory the service accounts, tokens, and API credentials used by AI applications and agents, then apply least privilege, rotation, and offboarding discipline to those identities.
• Test outbound leakage as part of validation Include response filtering checks for secrets, PII, and unsafe content in every AI deployment test, not just inbound prompt screening. Validate that downstream systems cannot consume unfiltered model output.

Key takeaways

• AI gateway governance is now an identity and policy problem, because model access is a privileged production pathway.
• Inline runtime enforcement matters because visibility alone cannot stop prompt injection, leakage, or unsafe output.
• Teams should manage AI agents, credentials, and request policy as one control surface, not as separate infrastructure tasks.

Key terms

• AI Gateway: A control layer that sits between AI applications and model providers to manage access, routing, quotas, logging, and policy. In practice, it turns model usage into a governable transaction path so teams can apply consistent identity and operational controls across many workloads.
• Runtime Enforcement: A security control that inspects and acts on AI requests or responses while they are flowing through production systems. It can block, redact, or allow traffic based on policy, which makes it different from monitoring tools that only record what happened after the fact.
• AI Traffic Enforcement Gap: The space between permitted access to a model and acceptable behaviour of the resulting request or response. It appears when organisations can authenticate AI traffic but cannot reliably stop prompt injection, leakage, or unsafe output before those events affect production workflows.
• Non-Human Identity: A machine or software identity used by systems rather than people, including service accounts, API keys, tokens, certificates, and AI agents. These identities need lifecycle, privilege, and visibility controls because they can carry access across services without a human sitting behind every action.

What's in the full announcement

TROJ.AI's full blog post covers the operational detail this post intentionally leaves for the source:

• How the integration is configured in practice, including policy creation and connector setup in the platform
• The request and response handling flow for prompts, redaction, blocking, and pass-through decisions
• The specific logging, metrics, and trace data exposed by the gateway for production review
• Deployment details for using the firewall as a proxy layer or API monitoring layer

👉 TROJ.AI's full post covers the runtime enforcement flow, policy setup, and observability details.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity security are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are responsible for identity security strategy or programme maturity, it is worth exploring.