OpenTelemetry tracing for AI gateways exposes hidden control gaps

At a glance

What this is: This is an analysis of how AI gateway tracing plus observability turns LLM and MCP activity into auditable telemetry, with the key finding that infrastructure-layer visibility can be added without touching application code.

Why it matters: It matters because IAM, NHI, and agent governance teams need evidence of who or what accessed models, tools, and data before they can enforce policy, investigate misuse, or prove control coverage.

By the numbers:

• TrueFoundry says its AI Gateway handles 350+ RPS on just 1 vCPU.
• 10ms latency even under load.

👉 Read TruFoundry's integration guide for AI gateway tracing and Logfire

Context

AI gateway observability is the practice of recording model requests, routing decisions, tool calls, and outputs in a form that can be queried and governed. In this article’s context, the key gap is not whether the gateway can move traffic fast enough, but whether the organisation can see what happened inside LLM and MCP workflows after the request completes.

For identity teams, the question is how a gateway becomes a governance point for non-human identity activity without requiring every application team to instrument its own traces. That matters for AI agents, service workloads, and human-operated applications alike, because the trace becomes the evidence layer for access, action, and accountability.

A useful framing is that OpenTelemetry can standardise the record of execution, while policy and access controls still have to be enforced elsewhere. That combination is typical for modern AI infrastructure, where visibility often arrives before governance maturity does.

Key questions

Q: How should security teams govern AI gateway traffic that carries prompts and tool calls?

A: Security teams should govern AI gateway traffic as a runtime policy problem, not just a routing problem. Inspect prompts, responses, and tool calls before they reach downstream systems, and make sure the gateway logs enough context to show what was sent, what was returned, which policy applied, and what action followed.

Q: Why do AI gateways create new identity governance concerns?

A: AI gateways sit between users, service accounts, agents, and models, so they become the place where identity, authorisation, and data controls either stay coherent or fragment. If governance is split across code, plugins, and side integrations, compliance drift and policy gaps appear quickly.

Q: What breaks when organisations rely on observability instead of access control?

A: What breaks is the assumption that telemetry can replace authorization. A trace can show that a model or tool was used, but it cannot stop misuse, limit privilege, or prevent excessive data access in real time. Organisations that confuse the two often discover too late that they have evidence of overreach, not prevention.

Q: What should teams check before using AI gateway traces for investigations?

A: Teams should confirm trace completeness, retention, access restrictions, and correlation across models, providers, and tool calls. If any of those are weak, the record may be too fragmentary to support incident response or compliance review. The goal is a defensible evidence chain, not a dashboard that is easy to read.

Technical breakdown

How OpenTelemetry turns AI gateway events into usable trace data

The gateway emits spans for each request and attaches gen_ai.* attributes such as model name, token counts, and finish reason. Those spans can also include gateway-specific fields like tfy.input, tfy.output, and tfy.span_type, which let teams distinguish ChatCompletion, AgentResponse, and MCPGateway activity. Because the export is asynchronous, the tracing path does not block the live request. This is what makes the pipeline practical in production: execution stays fast, while telemetry is delivered after completion into a central observability system.

Practical implication: treat the gateway trace as the system of record for model and tool activity, then map those spans to identity and policy controls.

Why MCP tool calls change the governance problem

MCP tool call management matters because the gateway is not just forwarding prompts, it is brokering access to tools and data sources. Once tool calls sit inside the same trace as the model interaction, teams can see whether an LLM response stayed inside the intended execution path or invoked additional actions. That creates a governance boundary around runtime behaviour rather than just around application code. For identity teams, the critical issue is whether tool access is bound to the right identity, scope, and session context.

Practical implication: classify MCP tools as governed access paths, not just application integrations, and review who can invoke them through the gateway.

What asynchronous export means for latency and evidence

Asynchronous publication to a queue, followed by OTLP export, separates telemetry delivery from user-facing execution. That design avoids stalling the request, but it also means observability is eventually consistent rather than inline. The operational trade-off is clear: you gain speed and scalable export, but you still need downstream retention, correlation, and access control on the telemetry store. In practice, the gateway and the observability platform become two halves of the same evidence chain.

Practical implication: validate trace retention, access controls, and correlation rules in the observability backend, not only in the gateway configuration.

NHI Mgmt Group analysis

AI gateway tracing is becoming the evidence layer for non-human identity governance. Once model requests, tool calls, and routing decisions are recorded in standard telemetry, teams can reconstruct what happened without changing every upstream application. That shifts AI governance from code-local logging to infrastructure-level proof, which is more sustainable for large environments. The practitioner conclusion is that observability is now part of identity control design, not just operations.

OpenTelemetry gives the sector a common contract, but not a common control model. Standard spans make AI gateways and observability tools interoperable, yet policy decisions still depend on how organisations classify model traffic, tool invocation, and agent behavior. That means the same telemetry can support very different governance outcomes, from basic troubleshooting to compliance-grade access evidence. Practitioners should treat the standard as an enabler, not a control boundary.

Runtime tool visibility is the named concept that matters here: when MCP calls are embedded in traces, tool use becomes inspectable after the fact, but not automatically constrained at the moment of execution. That distinction matters because many teams confuse traceability with authorization. A trace can prove that a tool was used, but it does not itself prove that the tool should have been reachable. The practitioner conclusion is to separate evidence capture from privilege enforcement.

Identity governance for AI systems must account for the gap between request-time controls and post-execution auditability. The gateway validates callers and routes traffic, but the observability platform records behavior after the fact. That creates a useful governance pattern, but also a blind spot if organisations assume logging equals policy enforcement. The practitioner conclusion is to align gateway authentication, routing policy, and telemetry review as a single control chain.

AI gateway telemetry will increasingly shape incident response for LLM and agent workflows. When traces show prompts, tool invocations, and response timing together, investigations become faster and more defensible. That is especially relevant where multiple teams, models, and providers share the same execution layer. The practitioner conclusion is that trace data should be treated as security evidence, not just developer convenience.

From our research:

• 92% agree governing AI agents is critical to enterprise security, yet only 44% have implemented any policies to do so, according to AI Agents: The New Attack Surface report.
• Only 52% of companies can track and audit the data their AI agents access, leaving 48% with a complete blind spot for compliance and breach investigation.
• That visibility gap makes the next step not more telemetry alone, but stronger governance patterns like OWASP Agentic AI Top 10 applied to runtime access paths.

What this signals

Runtime tool visibility: AI gateway traces give teams a way to see model, agent, and MCP activity in one place, but that does not close the authorization gap. The programme risk is that observability becomes the comfort blanket while policy remains fragmented across identity, platform, and application teams. That is why trace review needs to sit alongside access review, not in place of it.

With 52% of companies able to track and audit the data their AI agents access, the governance gap is already visible in operating practice, not just in theory. The practical response is to align telemetry with policy ownership, retention rules, and review cadence, then tie that to the AI Gateway and the OWASP Agentic AI Top 10 where runtime tool misuse is a concern.

For practitioners

• Instrument the gateway as the primary evidence source Use the AI gateway trace stream as the canonical record for model requests, routing decisions, and MCP tool calls. Make sure the trace schema preserves enough context to answer who accessed what, when, and through which execution path.
• Separate telemetry from enforcement Do not assume that seeing a tool call in observability means the call was authorised. Keep authorization policy, routing policy, and trace review as distinct controls so that evidence capture does not substitute for privilege control.
• Review MCP tools as governed access paths Inventory every tool reachable through the gateway, then assign ownership, access scope, and review cadence to each one. Treat these tools as non-human identity touchpoints because they can expose data, trigger actions, or both.
• Validate trace retention and access controls Confirm who can query the observability backend, how long traces are retained, and whether response content or tool arguments are masked where necessary. The audit trail is only useful if it is protected from casual access and preserved for investigations.

Key takeaways

• AI gateway tracing turns model and tool activity into evidence, but evidence alone is not enforcement.
• The main governance challenge is separating observability from authorization so runtime access cannot exceed policy.
• Practitioners should treat MCP tool calls, trace retention, and access review as one control chain rather than three separate problems.

Key terms

• AI Gateway: A control point that sits between AI applications and the models, tools, or data they call. In practice, it can authenticate requests, enforce policy, inspect runtime behaviour, and stop unsafe actions before they spread into connected systems.
• OpenTelemetry Span: An OpenTelemetry span is a single record of work performed during a request or task. In AI systems it can carry model, token, prompt, response, and tool-use metadata, giving security and platform teams a standardised evidence trail across systems.
• MCP Gateway: The control layer that relays assistant intent to tools and data sources through the Model Context Protocol. In practice, it becomes a policy boundary, not just a transport layer. If it trusts model output too early, it can turn unverified reasoning into real-world execution or disclosure.
• Identity Traceability: Identity traceability is the ability to link each action back to a specific identity, authorisation path, and time window. It is essential when humans, service accounts, and AI agents all operate in the same environment and auditors need a defensible record.

What's in the full article

TruFoundry's full post covers the operational detail this analysis intentionally leaves for the source:

• Step-by-step OTEL Config setup for traces and metrics export to Logfire regional endpoints
• The exact span fields and tfy.span_type values used to separate ChatCompletion, AgentResponse, and MCPGateway traffic
• Logfire Live view and SQL query examples for tracing prompts, tool arguments, and token usage across teams
• The write-token workflow and configuration details needed to stand up the integration securely

👉 TruFoundry's full post shows the OTEL configuration, trace fields, and live view setup in detail

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity security are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building or maturing an IAM programme, it is worth exploring.