TrueFoundry AI Gateway traces show where AI observability meets governance

At a glance

What this is: This is a technical explainer of how TrueFoundry’s AI Gateway traces requests and exports them to Honeycomb, with a clear emphasis on request-level visibility across LLM and MCP activity.

Why it matters: It matters to IAM and NHI teams because gateway traces can reveal how credentials, prompts, and tool calls move through agentic workflows, which affects access governance, data exposure, and investigation readiness.

By the numbers:

• The gateway handles 250+ requests per second with approximately 3 ms of added latency.
• TrueFoundry says its gateway can handle 350+ RPS on just 1 vCPU.

👉 Read TruFoundry's walkthrough of AI Gateway trace export to Honeycomb

Context

AI gateway observability is not only a performance problem. Once prompts, model calls, and tool invocations are traced end to end, the security question becomes how much sensitive identity and request data leaves the runtime, who can query it, and whether those traces create a durable record of NHI and agent behaviour. In AI gateway architectures, telemetry is also governance data.

TrueFoundry describes a stateless gateway that traces inbound requests, authentication, model routing, outbound provider calls, and MCP tool activity before exporting spans asynchronously. That architecture is relevant to identity and access teams because the same trace path that helps debug latency can also expose token use, payload content, and tool access patterns. This is a practitioner-level concern, not a theoretical one.

The starting position here is typical for modern AI platform teams: trace first, govern later. The problem is that AI gateway telemetry quickly becomes part of the control plane, so observability design and identity design cannot be separated.

Key questions

Q: How should security teams govern AI gateway traffic that carries prompts and tool calls?

A: Security teams should govern AI gateway traffic as a runtime policy problem, not just a routing problem. Inspect prompts, responses, and tool calls before they reach downstream systems, and make sure the gateway logs enough context to show what was sent, what was returned, which policy applied, and what action followed.

Q: Why do AI gateways create new authorization risks for NHI governance?

A: Because they concentrate many decisions into one control point while still relying on coarse scopes in many deployments. That leaves a gap between authentication and true authorization. NHI governance has to move beyond “can the caller reach the proxy” and ask whether the caller should be allowed to invoke a particular model, tool, or argument pattern.

Q: What breaks when trace export is left outside security review?

A: Teams can end up with a second data path that bypasses the controls applied to the application runtime. If export headers, datasets, or endpoints are misconfigured, traces may be redirected, over-retained, or exposed to broader audiences than intended. The result is visibility without governance, which is a common failure mode in AI platform operations.

Q: Should organisations separate observability access from platform access?

A: Yes. The people who need to operate an AI gateway do not always need access to raw prompts, full tool payloads, or every production trace. Separate those permissions so that troubleshooting, security review, and data inspection remain distinct roles. That reduces unnecessary exposure while still preserving investigatory value.

Technical breakdown

How AI gateway traces capture LLM and MCP request flow

An AI gateway trace is a structured record of what happened during a request, usually represented as spans with parent-child relationships. In this case the gateway records inbound handling, JWT authentication, model resolution, provider invocation, and streaming response handling. When MCP is involved, the trace also captures tool calls and their payloads, which means the trace becomes a map of both model behaviour and delegated action. That makes the trace useful for debugging, but also sensitive because it can expose request content and access patterns across the full agentic workflow.

Practical implication: treat trace design as part of identity governance when agent workflows can invoke tools or carry secrets.

Why asynchronous export changes the security and reliability model

The gateway publishes spans asynchronously over NATS, then an exporter forwards them to an OTLP backend. That means observability is decoupled from the request path, so trace delivery failure does not block inference availability. This is operationally sensible, but it also creates a second-plane system that must be controlled independently. If the export destination, header values, or routing configuration are wrong, traces may be misdirected or exposed even while the gateway continues serving traffic normally.

Practical implication: secure the export path as a separate trust boundary, not just as a logging convenience.

What Honeycomb’s schema-less ingestion means for investigation and control

Honeycomb stores each span as a row and indexes every attribute it receives without a predefined schema. That is powerful for ad hoc analysis because any attribute, including model identifiers, token counts, and request metadata, becomes queryable immediately. It also means whatever the gateway emits is what investigators inherit later. In practice, schema-less observability speeds search but widens the need for disciplined field selection, redaction, and dataset separation when traces may contain prompts, credentials, or agent actions.

Practical implication: define exactly which trace fields are allowed to leave the gateway before teams rely on them for analysis.

Threat narrative

Attacker objective: The attacker seeks operational intelligence and sensitive runtime data that can be used to abuse AI access paths or pivot into adjacent systems.

1. Entry occurs through a compromised or overexposed AI gateway export path, where trace data can include request content and tool invocation details.
2. Escalation follows if exported spans reveal prompts, headers, model metadata, or MCP payloads that help an attacker understand access patterns or target downstream systems.
3. Impact comes when those traces support credential abuse, data exposure, or unauthorized tool use across the AI workflow.

NHI Mgmt Group analysis

Observability has become an identity control surface in AI gateways. Once traces include prompts, headers, tool calls, and model routing, telemetry is no longer just an operations artifact. It becomes evidence of who or what acted, on which resource, and through which delegated path. That makes OpenTelemetry design relevant to IAM, PAM, and NHI governance, especially where MCP servers and AI agents are involved. Practitioners should treat exported trace fields as governed identity data, not generic logs.

MCP visibility without payload discipline creates governance debt. The gateway can trace full tool requests and responses, which is useful for debugging but risky for long-lived retention and broad analyst access. This is the same control tension that shows up in NHI programs when secrets, tokens, and service-account activity are over-collected without lifecycle rules. AI gateway trace minimization: is the practical concept here, meaning teams should separate routing metadata from content that can leak credentials or sensitive context. The conclusion is that trace usefulness must be bounded by data-classification policy.

Asynchronous export shifts the trust boundary, it does not remove it. Decoupling telemetry from inference protects availability, but it also means the export channel, destination credentials, and dataset routing become part of the security architecture. That aligns with NIST CSF and OWASP NHI thinking: control the movement of sensitive machine identity artifacts, not just the application front door. For practitioners, the lesson is to govern telemetry transport with the same discipline used for other NHI-backed integrations.

Schema-less observability can accelerate investigations and still undermine least privilege. Honeycomb’s ability to query every span attribute is valuable, but broad access to richly instrumented traces can expose too much about AI workflows, user actions, and internal routing logic. In identity terms, this is a visibility-versus-discretion problem. Teams should align dataset access to role, use separate environments for production and staging traces, and ensure only the minimum required span fields leave the runtime. The right control objective is investigability without overexposure.

This is an early example of identity-aware AI observability becoming part of the control plane. The gateway’s traces reveal not only system health but also which models, MCP servers, and tool paths were used. That is why NHI governance and AI governance are converging around the same runtime artefacts. Practitioners should expect observability pipelines to be reviewed with the same seriousness as access policies, because they now carry both operational and security meaning.

What this signals

The operational signal here is that AI gateways are becoming evidence systems as much as routing systems. If traces can expose prompts, MCP tool calls, and identity-bearing headers, then observability design needs the same governance discipline as secret handling and access review. That is especially true when the control boundary includes both human operators and non-human identities that act through the gateway.

AI gateway trace minimization: the useful unit of observability is no longer the full payload, but the minimum metadata needed for incident response and performance analysis. Teams should expect this to shape retention, dataset segmentation, and role-based access to traces across production AI estates.

For practitioners

• Define a trace data minimization policy Classify which span attributes may leave the AI gateway, then strip prompts, responses, and any secret-bearing fields before export. Keep token counts, latency, model identifiers, and error flags if they are needed for operations. This reduces the chance that observability data becomes a secondary data leak.
• Separate production and staging datasets Use distinct dataset routing for production, staging, and MCP traffic so analysts do not inherit a mixed trace corpus. Limit access to each dataset by role and retention need, and make sure the routing header cannot silently override your intended segmentation.
• Protect export credentials as machine identities Treat the Honeycomb ingest key and any similar headers as NHI secrets with rotation, scope restriction, and monitoring. The x-honeycomb-team header is sent on every export request, so compromise of that credential creates broad telemetry exposure even if inference remains available.
• Review MCP spans for tool and payload leakage Inspect whether MCP tool calls are recording request and response bodies, then decide whether that level of detail is justified. If the payloads are not required for incident response, redact or summarise them so traces do not expose delegated actions unnecessarily.

Key takeaways

• AI gateway traces can reveal enough about prompts, tool calls, and routing decisions to become part of identity governance, not just observability.
• Asynchronous export protects request latency, but it also creates a separate trust boundary for telemetry credentials, routing, and retention.
• Practitioners should minimise exported fields, segment datasets, and govern trace access with the same discipline used for other NHI-backed integrations.

Key terms

• AI Gateway: A control point that sits between AI applications and the models, tools, or data they call. In practice, it can authenticate requests, enforce policy, inspect runtime behaviour, and stop unsafe actions before they spread into connected systems.
• OpenTelemetry Span: An OpenTelemetry span is a time-bounded record of a single operation within a distributed trace. For AI systems, spans can describe model calls, authentication steps, or tool invocations, and they become highly sensitive when they include prompts, responses, or identity-linked metadata.
• MCP Gateway: The control layer that relays assistant intent to tools and data sources through the Model Context Protocol. In practice, it becomes a policy boundary, not just a transport layer. If it trusts model output too early, it can turn unverified reasoning into real-world execution or disclosure.
• Trace Minimization: Trace minimization is the practice of exporting only the telemetry fields required for operations, security, and compliance. In AI environments, it reduces the chance that prompts, secrets, or sensitive context will be copied into long-lived observability systems.

What's in the full article

TruFoundry's full article covers the implementation details this post intentionally leaves at the architecture level:

• The exact OpenTelemetry span fields emitted by the gateway, including the gen_ai.* and tfy.* attributes
• The full Honeycomb configuration surface, including endpoint, dataset routing, and authentication header values
• The request-to-trace lifecycle showing how NATS decouples export from inference availability
• The differences between TrueFoundry export behavior and the OpenTelemetry Collector's OTLP HTTP path handling

👉 The full TruFoundry article covers span fields, OTLP configuration, and dataset routing details.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, IAM, and secrets management. It is designed for practitioners who need to connect identity controls to broader security operations.