TrueFoundry AI Gateway trace export changes observability boundaries

At a glance

What this is: This is an architecture explanation of how TrueFoundry AI Gateway emits OTEL traces to OpenLIT without touching the request path, while controlling whether prompt and response content is forwarded.

Why it matters: It matters because AI gateway telemetry can become a second data plane for sensitive request content, so IAM, data governance, and observability teams need clear rules on what is exported, stored, and queryable.

By the numbers:

• A single pod on 1 vCPU and 1 GB RAM handles over 250 requests per second with approximately 3 ms of added latency.
• The gateway handles 350+ RPS on just 1 vCPU in the product positioning shown in the article.
• The traces pipeline uses a memory limiter set to 1500 MiB with a 512 MiB spike limit before forwarding to ClickHouse.

Context

AI gateway telemetry is not just operational metadata. When spans include prompts, responses, or request summaries, the observability layer becomes another place where sensitive AI content can persist, be queried, and be shared across tools.

This article is about the governance boundary created by asynchronous trace export. The architecture keeps latency off the request path, but it also shifts responsibility onto teams to decide which fields can leave the gateway and which systems may store them.

Key questions

Q: How should teams govern AI gateway traces that contain prompt and response data?

A: Treat AI gateway traces as sensitive data flows, not generic logs. Classify each span attribute, restrict content-bearing fields, and apply access control, retention, and deletion policies to the trace store itself. If a trace can reveal a user prompt or model output, the observability platform needs the same governance discipline as the application that produced it.

Q: Why does asynchronous trace export change the risk profile of an AI gateway?

A: Because it moves telemetry handling out of the request path and into a separate data store. That protects latency, but it also creates another place where sensitive AI content can persist, be indexed, and be reused for reporting or alerting. The risk increases whenever observability data contains business or personal information.

Q: What breaks when prompt data is sent to observability without filtering?

A: The observability layer starts functioning like a secondary content repository. Raw prompts and responses can be queried by operational teams, retained longer than intended, and exposed through downstream analytics workflows. Once that happens, the gateway has effectively widened the data-handling boundary beyond the application team’s original intent.

Q: How do AI gateway traces differ from ordinary application logs?

A: Traces preserve causal sequence across the request path, while logs usually record isolated events. For AI gateways, that distinction matters because routing decisions, model responses, and tool use need to be correlated in time and context. A trace can show the path; disconnected logs usually cannot.

Technical breakdown

Asynchronous OTEL export in an AI gateway

The gateway emits OpenTelemetry spans after the request completes, then publishes them through an asynchronous path rather than during live request handling. That design separates operational telemetry from the request path, which is why observability backends can be slow or temporarily unavailable without affecting user-facing latency. In practice, the exporter becomes a post-response transport layer, not part of authorization or model routing. For AI gateways, that distinction matters because the trace pipeline can be externally visible even when the application remains internal.

Practical implication: treat the export pipeline as a governed data path, not just an instrumentation feature.

Prompt, response, and metadata in trace attributes

The spans described here can carry tfy.input, tfy.output, and tfy.input_short_hand, plus standard gen_ai.* fields for tokens, models, and finish reasons. That means observability data may contain the full conversational payload, not just timing and error signals. The article also shows a toggle that strips input and output before forwarding, which is the right kind of control surface for reducing telemetry exposure. If trace attributes are queryable in ClickHouse, they are effectively part of the system of record and need the same handling assumptions as other sensitive AI artefacts.

Practical implication: classify trace fields by sensitivity before enabling export to any external observability store.

ClickHouse-backed observability and queryable AI telemetry

OpenLIT stores trace rows in ClickHouse, with span attributes preserved as a map that can be queried directly. That makes the observability layer useful for analytics, but it also creates a durable dataset that can be searched, joined, and retained beyond the gateway's runtime. The design is operationally convenient because teams can use SQL for custom reporting, but the same convenience expands the blast radius if prompts or responses are included. For AI governance, queryability is the control boundary: if a field is searchable, it is no longer transient telemetry.

Practical implication: align retention, access control, and query permissions with the sensitivity of exported AI request data.

NHI Mgmt Group analysis

AI gateway tracing is becoming a governance boundary, not just an observability choice. Once prompt and response fields are exported into a queryable store, the problem is no longer instrumentation latency but data placement. That changes who must approve the pipeline, what fields are permitted to leave the gateway, and how long those records remain retrievable. For practitioners, the control question is whether telemetry can expose more of the conversation than the application itself was meant to reveal.

Telemetry field design now carries the same governance weight as API design. The article shows that trace payloads can include full request and response content, condensed summaries, and model metadata in the same span. That is a named trust boundary because the observability layer can turn runtime interactions into durable records. Teams should treat trace schemas as policy artefacts, not engineering conveniences, because a searchable field is an accountability decision.

Queryability creates observability debt. ClickHouse-backed trace storage gives teams powerful analytics, but it also means AI request data can become broadly inspectable across reporting and alerting pipelines. The issue is not whether the gateway is internal or external, but whether the exported content is governed as sensitive data from the point of emission. For identity and platform teams, the right stance is to define export and access rules before traces become operationally entrenched.

Prompt-content export is now a data security problem as much as an AI operations problem. When AI gateways forward prompts into observability systems, they extend the sensitive-data surface beyond the model provider and the application layer. That intersects directly with IAM, access permissions, and data governance, because the question becomes who can read traces, who can query them, and which roles should never see raw content. Practitioners should govern exported telemetry like production data, not developer debug output.

From our research library:

• AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
• Read next: Agentic AI Compliance Guide

What this signals

Queryable AI telemetry is a new policy surface. When gateways forward prompt content into observability backends, teams are no longer managing only traces and metrics. They are managing a searchable record of AI interactions, which should be classified, access-controlled, and retained under explicit policy before it becomes embedded in operations.

Telemetry export is where AI governance meets identity governance. Once trace stores are shared across platform, SOC, and engineering workflows, role design matters as much as transport design. Who can read raw prompts, who can query summaries, and who can administer the store are all privilege questions, not just observability questions.

For practitioners

• Define trace field export policy Separate operational metadata from prompt and response content, and decide explicitly which attributes may leave the gateway.
• Use content stripping for external destinations Enable the exclude request data option when traces must flow to an external observability store and raw content is not required.
• Restrict query access to stored spans Limit who can search ClickHouse trace tables, especially when tfy.input and tfy.output are retained in span attributes.
• Review retention for AI telemetry Apply retention and deletion rules to trace data according to whether it contains prompts, responses, or only timing metadata.

Key takeaways

• AI gateway trace export can preserve low-latency request handling while still creating a durable copy of prompts, responses, and summaries in observability systems.
• The governance issue is the telemetry boundary, because once AI request content enters a queryable store it becomes subject to retention, access, and search controls.
• Teams should classify trace attributes before export and limit raw-content forwarding to cases where the operational need clearly justifies the data exposure.

Key terms

• AI Gateway Telemetry: Telemetry emitted by an AI gateway about request handling, routing, latency, and outcomes. In this article’s context, telemetry may also include prompt and response content, which means it must be governed as sensitive operational data rather than harmless system noise.
• OpenTelemetry Span: An OpenTelemetry span is a time-bounded record of a single operation within a distributed trace. For AI systems, spans can describe model calls, authentication steps, or tool invocations, and they become highly sensitive when they include prompts, responses, or identity-linked metadata.
• Trace Attribute: A key-value field attached to a trace or span. In AI systems, attributes can expose request bodies, outputs, model identifiers, or routing decisions, making attribute design a direct input into privacy, access control, and retention policy.
• Telemetry Boundary: A telemetry boundary is the point where operational logs stop being internal data and become governed data subject to new access, retention, and privacy rules. In migration projects, that boundary often shifts from the SIEM itself to the collection and routing layer before ingestion.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, workload identity, and secrets management. It gives practitioners a structured way to connect identity controls to the operational systems that now carry AI telemetry and access decisions.