TrueFoundry AI Gateway trace export changes observability boundaries

At a glance

What this is: This is an architectural explainer of how TrueFoundry’s AI Gateway exports traces to OpenLIT without adding code changes or request-path latency.

Why it matters: It matters because AI gateway telemetry can quietly expand the scope of sensitive data exposure, access control, and retention decisions across AI and identity governance programmes.

By the numbers:

• A single pod on 1 vCPU and 1 GB RAM handles over 250 requests per second with approximately 3 ms of added latency.
• The gateway’s performance profile includes ~10ms latency even under load and 350+ RPS on just 1 vCPU.
• When AWS credentials are exposed publicly, attackers attempt access within an average of 17 minutes.

👉 Read TruFoundry's walkthrough of AI Gateway trace export to OpenLIT

Context

AI gateway telemetry is not just operational data. When request and response content are exported into observability systems, teams create a second control plane for sensitive prompts, outputs, model identifiers, and routing metadata. That changes the security boundary for AI operations, especially when logs and traces become queryable across shared infrastructure.

The core governance issue is whether telemetry carries business content, secrets, or regulated data beyond the systems that originally processed it. In AI programmes, that intersects with access control, data minimisation, retention, and incident investigation. For identity teams, the same problem shows up whenever service-to-service telemetry is more permissive than the application that generated it.

This pattern is common in modern AI platforms because observability is often added after the gateway is already in production, not designed into the architecture from the start.

Key questions

Q: How should teams govern AI gateway traces that contain prompt and response data?

A: Treat AI gateway traces as sensitive data flows, not generic logs. Classify each span attribute, restrict content-bearing fields, and apply access control, retention, and deletion policies to the trace store itself. If a trace can reveal a user prompt or model output, the observability platform needs the same governance discipline as the application that produced it.

Q: Why do AI observability platforms create new data-governance risk?

A: Because they often centralise information that was previously transient inside the application path. Once prompts, outputs, routing metadata, and identifiers are written to a searchable backend, they can be retained, joined, and accessed by more people than intended. That turns telemetry into a secondary sensitive-data repository.

Q: How do security teams know whether telemetry export is too permissive?

A: Look for content in trace attributes, broad dashboard or SQL access, long retention periods, and uncontrolled downstream copies. If observability users can read AI interactions that operators could not access in the source system, the export boundary is too broad. The right test is whether the telemetry store reveals more than the application should.

Q: What should teams do before enabling external trace export from an AI gateway?

A: Review which fields are exported, who can read the destination, and whether the destination is isolated from other sensitive workloads. Then test failure behaviour, retention settings, and deletion paths. If those controls are not defined first, external export can create a compliance and exposure problem faster than it improves visibility.

Technical breakdown

How asynchronous OTEL export avoids request-path latency

The TrueFoundry gateway separates request handling from telemetry emission. Requests are processed in memory, then spans are published asynchronously to NATS after the response completes. The OTEL exporter forwards those spans to an external collector without touching the live request path, so collector slowness or outage does not block user traffic. This is a classic decoupled telemetry pattern: observability is treated as an eventually consistent side channel rather than a synchronous dependency. That design improves resilience, but it also means dropped spans can be invisible at the application layer.

Practical implication: treat exporter reliability and drop handling as part of the telemetry control plane, not as a monitoring afterthought.

What data rides inside AI gateway spans

The spans carry more than timing data. Attributes can include the full prompt, the full response, a shortened request summary, token counts, model identifiers, finish reasons, and request-type labels such as chat completion or agent response. In practical terms, the trace becomes a content-bearing record, not just an operational metric. That matters because observability backends often have broader read access than production systems. If teams do not constrain which attributes are exported, they may unintentionally create a parallel store of sensitive AI interaction data.

Practical implication: classify trace attributes by sensitivity and decide which fields are allowed to leave the gateway before enabling export.

Why OpenLIT and ClickHouse change the governance surface

OpenLIT ingests OTLP over HTTP or gRPC, stores traces and metrics in ClickHouse, and exposes them through a dashboard and SQL queries. That architecture is flexible, but it also turns AI telemetry into structured data that can be searched, joined, and retained like any other analytical dataset. Once prompts and outputs sit in ClickHouse tables, the governing questions shift to database access, retention, and query authorization. In identity terms, the issue is not only who can reach the collector, but who can read the stored trace estate later.

Practical implication: apply database access controls and retention rules to trace stores with the same rigor used for sensitive application data.

Threat narrative

Attacker objective: The attacker objective is to recover AI prompts, responses, routing metadata, or embedded secrets from telemetry systems that were not designed as primary data stores.

1. Entry occurs when AI request telemetry is exported from the gateway into a shared observability pipeline that can include prompt and response content.
2. Escalation happens when those traces are stored in ClickHouse and become queryable by users or systems with broader access than the originating application.
3. Impact is sensitive AI interaction data becoming searchable, retainable, and more widely exposed than intended across the observability stack.

NHI Mgmt Group analysis

AI telemetry has become a governance boundary, not a passive by-product. Once prompt and response content are exported into observability pipelines, the security model changes from application protection to data governance. That brings retention, access control, and queryability into scope alongside model operations. Practitioners should treat trace export as a controlled data movement decision, not a logging convenience.

Content-bearing traces create a new form of AI data sprawl. The problem is not just where spans go, but what they now contain: prompts, responses, model identifiers, and routing metadata. When those records land in ClickHouse or similar stores, they can be reused for troubleshooting, cost analysis, or search, which broadens exposure. The named concept here is telemetry content sprawl: operational traces that quietly become sensitive data repositories. Teams need to govern trace fields with the same discipline they apply to source datasets.

Identity and access controls must extend into the observability stack. In many environments, the collector and dashboard are treated as infrastructure, while the application remains the only protected system. That assumption fails when telemetry stores AI content, because read access to the observability tier can become equivalent to access to the application conversation itself. The governance takeaway is straightforward: who can query traces is now a security question, not an operations detail.

Decoupled export improves reliability, but it can hide data-loss and data-exposure trade-offs. Asynchronous forwarding protects latency, yet it also normalises the idea that telemetry can be dropped, buffered, or redirected without the caller noticing. That is useful operationally, but it complicates assurance because security teams may not see all the data that left the gateway or all the places it landed. Practitioners should align observability design with data handling policy before scaling export across environments.

What this signals

Telemetry content sprawl: AI programmes now need a policy for what trace data may leave the gateway, not just who may view it. If observability includes prompts or outputs, the platform becomes part of the data perimeter and should be managed accordingly, with reference to NIST AI 600-1 Generative AI Profile.

The practical signal is that trace destinations should be evaluated like any other sensitive data store. That means role separation, retention limits, and deletion controls, plus explicit review of whether the collector, database, and dashboard sit inside the same trust boundary as the gateway.

Identity teams should also watch the service-to-service path. When gateways publish telemetry over internal identities, those non-human access paths need the same governance as other machine-to-machine integrations, especially where observability systems can read content that production users never see.

For practitioners

• Restrict exported span attributes Remove prompt, response, and other content-bearing fields from telemetry unless a documented use case requires them. Use the gateway’s exclusion controls to keep tfy.input, tfy.output, and tfy.input_short_hand out of external observability paths.
• Apply trace-store access governance Treat ClickHouse tables containing AI traces as sensitive datasets. Limit dashboard and SQL access, separate analyst and operator roles, and review who can query otel_traces and otel_metrics.
• Define telemetry retention and deletion rules Set retention periods for spans and metrics based on security and compliance needs, not just storage cost. If prompts or outputs are exported, define deletion workflows that cover the collector, ClickHouse, and any downstream copies.
• Validate drop and failure behaviour Test what happens when the collector is slow, unreachable, or misconfigured. Confirm whether spans are dropped, buffered, or delayed, and document the operational and security impact of each outcome.

Key takeaways

• AI gateway traces can become sensitive data stores when prompt and response content are exported into observability systems.
• The operational benefit of asynchronous export does not remove the governance burden created by searchable, retained telemetry.
• Teams should control trace fields, destination access, and retention before scaling external telemetry across AI environments.

Key terms

• Telemetry Content Sprawl: The expansion of operational traces into repositories that contain sensitive business or security content. In AI systems, this often means prompts, responses, routing metadata, and identifiers are stored in observability tools, creating a secondary data estate that needs explicit governance.
• OTEL Exporter: A component that forwards OpenTelemetry data from an application or gateway to a collector or backend. It is part of the telemetry pipeline, not the request path, and its configuration determines where traces and metrics go, what format they use, and how failures are handled.
• Trace Store: A backend that persists distributed tracing data for inspection, search, and analysis. When trace records include content-bearing fields, the store becomes a governed data platform rather than a pure diagnostics tool, with access control and retention obligations.

What's in the full article

TruFoundry's full blog post covers the operational detail this post intentionally leaves for the source:

• Exact OTEL exporter configuration for traces and metrics, including endpoint structure and encoding choices
• ClickHouse table layout and query syntax for trace-level analysis using SpanAttributes
• Helm chart and Kubernetes deployment details for the OpenLIT collector, dashboard, and database
• The internal service DNS names and port mappings used to wire the gateway to the collector

👉 TruFoundry's full post covers the OTEL pipeline, ClickHouse storage path, and deployment wiring in detail.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, workload identity, secrets management, and the controls that matter when machine-to-machine paths expand. It helps practitioners align AI telemetry, access, and lifecycle decisions with broader identity governance.