Enterprise AI gateway governance extends Ng’s three loops

At a glance

What this is: This is an analysis of how Andrew Ng's three-loop AI build model changes when it moves into enterprise production, with the key finding that the outer loop becomes a governance layer.

Why it matters: It matters because AI gateways, agent identities, and audit trails now sit inside identity programmes alongside human IAM and NHI governance, so control design has to cover policy, spend, and accountability as well as access.

By the numbers:

• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems, inappropriately sharing sensitive data, and revealing access credentials.

👉 Read TruFoundry's full analysis of enterprise AI gateway governance

Context

Enterprise AI does not fail because teams cannot build quickly enough, it fails when the build process outruns the governance model that must surround the resulting system. In this article, the primary identity question is how the AI gateway, model traffic, and agent execution layer change once they move from prototype conditions into an enterprise environment where identity, policy, audit, and accountability are mandatory.

TrueFoundry frames Andrew Ng's three-loop idea as still valid in production, but the outer loop now includes governance obligations that weekend apps do not face. That makes the article relevant to AI gateway governance, because the same control plane that routes model and MCP traffic also becomes the place where identity, budgets, and evidence are attached to AI work.

The useful lens for practitioners is not whether the loops still work, but which controls become non-optional once AI actions can touch real users, real data, and real budgets. That is where IAM, PAM, NHI lifecycle management, and request-level evidence start to converge.

Key questions

Q: How should teams govern AI gateways that route model and tool traffic?

A: Teams should treat the gateway as the control boundary for identity, spend, logging, and policy enforcement. That means registering each AI workload or agent, attaching a clear owner, and ensuring every significant model or tool call is traceable. The goal is not to slow AI down, but to make it accountable in production.

Q: Why do AI systems need request-level logs instead of only dashboards?

A: Dashboards show volume and latency, but request-level logs show accountability. For enterprise AI, you need to know which identity made the request, what tools it used, what data it touched, and whether policy allowed the action. Without that record, investigation and access review become guesswork.

Q: When should organisations use approval gates in AI workflows?

A: Use approval gates when the action is consequential, such as external side effects, sensitive data access, or budget-bearing operations. Do not use them as a blanket substitute for authorisation design. The right pattern is selective human review for high-risk steps, paired with automated policy for ordinary traffic.

Q: What is the difference between AI policy and AI governance?

A: AI policy states what the organization wants to allow, while AI governance enforces how those rules work in practice through ownership, access control, logging, and review. Without technical enforcement, policy becomes advisory text that cannot control machine identities or agent behaviour at scale.

Technical breakdown

How AI gateway identity changes the outer loop

The article distinguishes between the application logic that builds the agent and the control plane that governs its traffic. Once a model or agent uses registered MCP traffic, the gateway can attach identity, policy, spend limits, and logging to those requests. That matters because the outer loop is no longer just user feedback or A/B testing. It becomes the place where who acted, on whose behalf, against which resource, and under what budget can be answered consistently.

Practical implication: treat the gateway as the policy boundary for AI execution, not just as a routing layer.

Why request-level evidence matters for AI governance

A production AI system needs durable evidence, not just aggregated metrics. The article separates dashboard visibility from request traces and request logs, which is the difference between operational monitoring and accountability. In identity terms, aggregate telemetry tells you that usage happened, but request-level evidence tells you which identity, which tool call, which response, and which control decision produced the outcome. That is the minimum audit substrate for enterprise AI.

Practical implication: retain request traces and logs at a level that supports investigation, approval review, and policy challenge.

How approval checkpoints fit into AI execution

The article describes approval policies as execution checkpoints for sensitive tool calls, not as a substitute for upstream authorisation design. That distinction is important. If a model or agent is allowed to initiate tool use, then the organisation needs a governable step before consequential actions complete. The control is not about slowing all AI activity. It is about introducing human review where the action changes risk materially, such as data access, external side effects, or budgeted operations.

Practical implication: reserve explicit approval gates for consequential actions and keep them separate from general request routing.

NHI Mgmt Group analysis

The outer loop is now an identity and accountability problem, not just a product feedback problem. Ng's three-loop model still explains the build process, but enterprise deployment adds policy, budget ownership, audit, and access control to the outer loop. That changes the governance burden around AI gateways because the system must now answer who acted, what they touched, and whether the action was authorised. For practitioners, the implication is that AI product graduation is an IAM and NHI design problem as much as an engineering milestone.

Gateway-mediated AI traffic creates a new governance boundary for non-human identities. Once model calls and tool invocations are centralised, the gateway becomes the most practical place to attach NHI controls such as registration, request logging, policy enforcement, and spend attribution. That does not replace application logic, but it does create a single control point for identity-bound AI traffic. The field should treat AI gateways as governance infrastructure, not merely performance infrastructure.

Request traces are the accountability primitive that enterprise AI still lacks in too many programmes. Aggregate metrics are useful for operations, but they do not answer the questions auditors, incident responders, and access reviewers care about. This is especially true when agents and models interact with real data or take multi-step actions across tools. The implication is simple: if the organisation cannot reconstruct a request path, it does not yet have enterprise-grade AI governance.

Approval gates only work when they are reserved for consequential steps. The article correctly separates execution checkpoints from authorisation design, which prevents teams from confusing a human pause with real governance. Sensitive actions deserve explicit review, but routine traffic should remain policy-driven and observable rather than manually bottlenecked. Practitioners should use this distinction to avoid over-gating the AI workflow while still controlling high-impact operations.

From our research:

• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems, inappropriately sharing sensitive data, and revealing access credentials, according to AI Agents: The New Attack Surface report.
• Only 52% of companies can track and audit the data their AI agents access, leaving 48% with a complete blind spot for compliance and breach investigation.
• For a broader control lens, OWASP NHI Top 10 is the next step for mapping AI agent risk into governance and access design.

What this signals

With 52% of companies unable to track and audit the data their AI agents access, the operating model is already behind the deployment curve. That gap will widen unless identity teams treat AI gateways as governed execution points rather than transport infrastructure.

Request-path accountability: this is the control concept that now matters most for AI gateway programmes. If you cannot reconstruct the request path, you cannot reliably answer who acted, what they accessed, or whether a policy should have blocked it.

For programme owners, the signal is to align AI gateway governance with existing NHI and access-review processes, then anchor the policy model to NIST AI Risk Management Framework principles where AI decisioning changes risk posture.

For practitioners

• Define the AI gateway as a governance boundary Map identity, policy, spend, and logging responsibilities to the gateway layer before the first production rollout. Make sure every model call and registered MCP tool invocation has an owner and an audit path.
• Attach request-level evidence to production AI Retain traces and logs that show who invoked the model, which tools were called, and what response or side effect followed. Use that record for incident review, access review, and accountability checks.
• Separate approval checkpoints from routine routing Use human approval only for sensitive tool calls and consequential actions, and keep routine inference traffic policy-driven. That reduces bottlenecks while preserving control where risk is highest.
• Treat agent identities as governed NHIs Register agents, bind them to explicit policies, and assign budgets to accountable owners. Where AI systems act through registered identities, lifecycle, scope, and offboarding controls should follow the same discipline used for other NHIs.

Key takeaways

• Enterprise AI changes the outer loop by adding identity, policy, budgets, and audit to what used to be simple product feedback.
• Gateway logs and request traces are now the minimum evidence layer for accountable AI operations.
• Selective approval gates and governed agent identities are the practical controls that make production AI defensible.

Key terms

• AI Gateway: A control point that sits between AI applications and the models, tools, or data they call. In practice, it can authenticate requests, enforce policy, inspect runtime behaviour, and stop unsafe actions before they spread into connected systems.
• Session-level Evidence: Session-level evidence is the record of who accessed a resource, when the access occurred, and what activity happened during that session. It is the operational proof that supports audits, investigations, and access reviews when privileged access is shared across teams.
• Agent Identity: An agent identity is the set of attributes, credentials and permissions assigned to an autonomous software entity. It is treated as a non-human identity because it can authenticate, act on systems and accumulate access over time, which creates governance, audit and lifecycle obligations similar to other production identities.
• Approval checkpoint: An approval checkpoint is a human review step inserted before a sensitive AI action completes. It is useful for consequential operations, but it is not a substitute for authorisation design, logging, or lifecycle governance.

What's in the full article

TruFoundry's full article covers the operational detail this post intentionally leaves for the source:

• The documented AI Gateway architecture that shows where model traffic, MCP traffic, and policy enforcement intersect.
• The specific gateway capabilities for Agent Identity, budgets, traces, and request logs that support enterprise accountability.
• The staged rollout mechanics for evaluation, routing, and approval checkpoints in production AI workflows.
• The product-level limits and configuration notes that matter when you are implementing controls rather than analysing them.

👉 TruFoundry's full article covers the gateway architecture, governance controls, and production sequencing in detail.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity security are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are responsible for identity security strategy or NHI governance in your organisation, it is worth exploring.