OpenTelemetry tracing for AI gateways exposes hidden control gaps

At a glance

What this is: This is a vendor analysis of AI gateway tracing that shows OpenTelemetry spans can capture model routing, request context, and MCP tool calls in one audit trail.

Why it matters: It matters because IAM and platform teams need evidence of who or what initiated an AI action, which tools were used, and whether that behaviour is governable at the gateway boundary.

By the numbers:

• TrueFoundry says its AI gateway handles 350+ RPS on just 1 vCPU.

Context

AI gateway tracing is the practice of recording request flow, routing decisions, model responses, and downstream tool calls as telemetry rather than leaving them buried inside application logs. In this article, TruFoundry argues that the governance problem is not raw observability but whether execution paths, model selection, and MCP tool use can be audited consistently at the infrastructure layer.

For identity teams, the important question is where control evidence comes from when an AI gateway brokers access to models and tools on behalf of applications, users, or agents. If the gateway is the enforcement point, then tracing becomes part of the control plane, not just an operational convenience.

The article frames this as a code-light model: spans move asynchronously, the gateway records standard OpenTelemetry fields plus its own attributes, and an external viewer reconstructs the activity path. That is typical of modern AI infrastructure, where the real governance gap is not lack of data but lack of usable audit structure.

Key questions

Q: How should teams govern AI gateways that route model and tool traffic?

A: Teams should treat the gateway as the control boundary for identity, spend, logging, and policy enforcement. That means registering each AI workload or agent, attaching a clear owner, and ensuring every significant model or tool call is traceable. The goal is not to slow AI down, but to make it accountable in production.

Q: Why do AI gateways need traceable tool-call evidence?

A: Because model output alone rarely explains how a task was actually executed. Tool-call evidence shows which external actions were triggered, by which request, and in what sequence. Without that linkage, teams cannot distinguish a harmless answer from a model-driven action that reached into other systems.

Q: What breaks when AI gateway spans do not preserve MCP context?

A: The audit trail becomes fragmented. Security teams may still see traffic and latency, but they lose the ability to connect a model response to the downstream tool invocation that followed it. That gap prevents credible review of behaviour, access scope, and post-incident reconstruction.

Q: How do AI gateway traces differ from ordinary application logs?

A: Traces preserve causal sequence across the request path, while logs usually record isolated events. For AI gateways, that distinction matters because routing decisions, model responses, and tool use need to be correlated in time and context. A trace can show the path; disconnected logs usually cannot.

Technical breakdown

How OpenTelemetry spans capture AI gateway execution

The gateway emits an OpenTelemetry span for each request and enriches it with standard gen_ai attributes such as model name, token counts, and finish reason. TruFoundry also adds gateway-specific fields like tfy.input, tfy.output, and tfy.span_type, which lets teams separate chat, agent, and MCP gateway activity in later analysis. The key design choice is that telemetry is attached to the execution path itself, so the trace preserves context across routing, response generation, and tool calls without requiring application-side instrumentation.

Practical implication: treat the gateway as the trace source of record and validate that its span schema captures the fields you need for governance and audit.

Why asynchronous export matters for latency and control

The article describes a non-blocking pipeline in which the gateway publishes spans to NATS after the request completes, then an exporter service forwards them to an OTLP endpoint. That separation preserves user-facing latency while still allowing centralised telemetry collection. Architecturally, this avoids making audit export part of the critical path, which is important when an AI gateway handles many short-lived requests and tool interactions at scale. The trade-off is that governance depends on the exporter, queue, and destination being reliably configured and monitored.

Practical implication: verify exporter durability and queue health so trace loss does not become a blind spot in incident review.

MCP tool calls become visible only if the trace model is complete

The article shows that Logfire can render MCP tool calls when it detects gen_ai attributes and related span metadata. That means tool usage is not inherently visible unless the instrumentation carries enough context for the downstream observability platform to recognise it. For practitioners, this is the boundary where AI gateway governance becomes identity governance: model access, tool invocation, and request context must all remain correlated in the same trace if you want to explain behaviour after the fact.

Practical implication: ensure tool-call spans and parent request spans stay linked so investigators can reconstruct who did what through the gateway.

Breaches seen in the wild

• LiteLLM MCP auth bypass 2026: An exploited LiteLLM MCP auth bypass and default sk-1234 master keys let attackers steal AI gateway master and provider API keys.
• Moltbook AI agent keys breach: Moltbook breach exposed 1.5M AI agent keys.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

AI gateway tracing is becoming a governance control, not just an operations feature: once the gateway brokers model access and MCP tool calls, the trace is the evidence trail for execution decisions. That shifts observability into the identity-security domain because the organisation needs to prove which request led to which tool action and under what context. Practitioners should treat the trace schema as part of the control design, not as an afterthought.

Hidden control gaps appear when the telemetry contract is incomplete: a gateway can authenticate callers and still leave major blind spots if spans do not preserve model routing, tool invocation, and response lineage. The missing control is often not logging volume but semantic structure. When the audit record cannot correlate request, model, and tool use, governance breaks even though data collection appears to be in place.

AI gateway governance increasingly depends on the boundary between execution and export: asynchronous telemetry protects latency, but it also creates a dependency on queue durability, exporter configuration, and destination integrity. That means the operational control is no longer just “turn on tracing”; it is whether the organisation can trust the telemetry pipeline under load and during incidents. Practitioners should assume the observability path is part of the security boundary.

Traceability of MCP tool use is the new minimum bar for AI execution oversight: the useful unit is no longer the prompt alone but the full sequence of routing, model response, and tool call behaviour. This is where the same governance discipline used for machine identity reappears in AI infrastructure. If the gateway cannot show the full chain, the organisation does not have meaningful oversight of how AI work actually happened.

OpenTelemetry is acting as a cross-system control contract: TruFoundry’s article shows that a standard trace format can let one layer govern execution while another layer interprets behaviour without custom integration. That pattern is important because it makes AI oversight portable across tools and platforms. Practitioners should favour standards that preserve evidence across the stack rather than point solutions that only surface a local view.

From our research library:

• AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
• Read next: AI Agent Observability, Audit and Incident Response Guide

What this signals

AI gateway telemetry is now part of the governance boundary: as gateways mediate model traffic and tool calls, the trace becomes the mechanism that proves which execution path was taken. Teams that still treat tracing as optional observability will struggle to evidence accountability when AI actions cross system boundaries.

Traceability of MCP tool use is the practical threshold for AI oversight: if the gateway can correlate request identity, routing choice, and downstream tool activity, security teams can review behaviour after the fact. If not, the organisation has only partial visibility into how its AI layer actually operates.

According to the State of Secrets Sprawl 2026, AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers. That is a reminder that visibility without governed identity and secrets control only tells part of the story.

For practitioners

• Define the gateway trace schema Map the span fields you need for routing, model selection, token usage, and MCP tool calls before rollout so the trace can support later review.
• Validate asynchronous export reliability Test the queue, exporter service, and OTLP endpoint under load so trace delivery failure is visible before you depend on it for audit evidence.
• Correlate tool calls with parent requests Require parent-child span linkage for every MCP interaction so investigators can reconstruct which request led to which tool execution.
• Review gateway authentication and routing evidence Confirm that caller identity, routing rule choice, and downstream provider selection remain visible in the same trace record.

Key takeaways

• AI gateway tracing can expose model routing and MCP tool use, but only if the span model preserves enough context to reconstruct execution.
• Asynchronous export protects latency, yet it also introduces a dependency on queue and exporter reliability that governance teams must validate.
• The control question is not whether telemetry exists, but whether it can prove who or what caused an AI action across the gateway boundary.

Key terms

• AI Gateway Telemetry: The request, routing, and usage data collected at the gateway layer that sits between applications and model providers. It becomes useful for governance when it is linked to evaluation signals, attribution, and budget boundaries rather than treated as logging alone.
• MCP Tool Call: An MCP Tool Call is a request an AI agent sends through the Model Context Protocol to use an external tool or data source. It carries the action, parameters, and context needed for execution, and it should be governed like a privileged machine-to-machine transaction because it can expose data, trigger actions, or change state.
• OpenTelemetry Span: An OpenTelemetry span is a time-bounded record of a single operation within a distributed trace. For AI systems, spans can describe model calls, authentication steps, or tool invocations, and they become highly sensitive when they include prompts, responses, or identity-linked metadata.
• Asynchronous Telemetry Export: A delivery pattern where spans are published after the request path completes, usually through a queue or broker. It preserves user-facing performance, but it can also hide delivery failures and create gaps between application success and audit completeness.

Deepen your knowledge

NHI governance, agentic AI identity, and machine identity lifecycle are core topics in our NHI Foundation Level course, the industry's only accredited NHI security programme. If you are building or maturing an identity security programme, it is worth exploring.