AI gateway telemetry exposes request content unless export is governed

At a glance

What this is: This is an architectural analysis of how TrueFoundry AI Gateway generates, exports, and stores OpenTelemetry traces for LLM traffic, with the key finding that trace payloads can include full request and response content unless excluded.

Why it matters: It matters because AI gateway telemetry can become a hidden data-exposure channel for prompts, outputs, and credentials, so IAM, NHI, and AI governance teams need to treat trace export as a governed data path.

By the numbers:

• The TrueFoundry AI Gateway handles over 250 requests per second on 1 vCPU with approximately 3 ms of added latency.
• The gateway handles 350+ RPS on just 1 vCPU and advertises ~10 ms latency even under load.

👉 Read TruFoundry's explanation of AI Gateway trace export to SigNoz

Context

AI gateway telemetry is not just operational metadata when span payloads can include full prompts, responses, and model-routing details. The governance gap is that teams often treat observability exports as infrastructure plumbing, while the content being exported can contain secrets, sensitive business data, or regulated personal data. In AI systems, telemetry paths deserve the same access scrutiny as the application path itself.

This article is also relevant to identity governance because the gateway depends on cached authentication keys, in-memory authorization maps, and a workspace-scoped ingestion key to move trace data to an external backend. That makes the telemetry path part of the trust boundary around non-human identities, service credentials, and AI workload data handling. The starting position is common in modern AI platforms, where speed and portability are prioritised before telemetry governance is fully tightened.

Key questions

Q: How should security teams govern AI gateway traffic that carries prompts and tool calls?

A: Security teams should govern AI gateway traffic as a runtime policy problem, not just a routing problem. Inspect prompts, responses, and tool calls before they reach downstream systems, and make sure the gateway logs enough context to show what was sent, what was returned, which policy applied, and what action followed.

Q: Why do AI gateways create new identity governance concerns?

A: AI gateways sit between users, service accounts, agents, and models, so they become the place where identity, authorisation, and data controls either stay coherent or fragment. If governance is split across code, plugins, and side integrations, compliance drift and policy gaps appear quickly.

Q: What breaks when trace export is not governed as part of the application design?

A: Teams lose control over where prompts, outputs, and routing details are stored, who can query them, and whether they leave the intended trust boundary. That creates both confidentiality risk and audit blind spots, especially when export failures are silent and trace delivery is asynchronous.

Q: Should organisations use external observability platforms for AI workloads by default?

A: Only if they can enforce field-level minimisation, tenant separation, and credential governance for the export path. External observability is workable, but it should be a deliberate choice with explicit data handling rules, not a default assumption that all spans are safe to ship.

Technical breakdown

How AI gateway traces become a data exposure channel

The gateway generates spans around request handling, authentication, model resolution, and provider calls. That trace is useful for latency and error analysis, but the same span can carry tfy.input and tfy.output, which means full prompt and response content may leave the gateway unless explicitly filtered. In practice, observability systems become downstream consumers of application data, not just measurements. When export is asynchronous, the request stays fast, but the confidentiality decision shifts to the telemetry pipeline. The key architectural issue is not whether traces exist, but whether their payload is governed like sensitive production data.

Practical implication: Classify trace content before export and apply data minimisation controls to any span that can carry prompts, outputs, or secrets.

Why asynchronous export changes the failure model

TrueFoundry separates request processing from telemetry delivery by publishing spans to NATS after the response is already on the wire. The exporter then forwards them over OTLP/HTTP to the regional ingestion endpoint. This architecture protects latency, but it also means telemetry delivery failures do not surface to callers and dropped spans may go unnoticed unless the export pipeline is monitored separately. From a security perspective, that split is important because the operational path and the audit path can diverge. If the audit path is lossy, governance teams lose visibility exactly when they need it most.

Practical implication: Monitor exporter health, drop rates, and queue depth as first-class control signals rather than assuming trace delivery is reliable.

What workspace-scoped ingestion keys really protect

The SigNoz side uses a per-workspace ingestion key and regional endpoint routing to place telemetry into the correct tenant pipeline. That key authenticates ingestion, but it is not a substitute for content control, because anyone or anything that can export to the endpoint can still send sensitive payloads if the exporter allows it. In AI environments, this is the same pattern seen with NHI credentials: possession of the secret determines where data flows, while payload policy determines what is allowed to move. Both layers matter because identity alone does not constrain content.

Practical implication: Treat ingestion keys as one control in a broader telemetry governance model that also enforces payload filtering and tenant separation.

Threat narrative

Attacker objective: The attacker objective is to harvest sensitive AI request and response content from telemetry pathways without needing to compromise the primary application workload.

1. Entry occurs when an AI gateway exports spans that include prompts, responses, and routing metadata to an external telemetry backend.
2. Escalation happens if the exporter or ingestion key is misused, allowing sensitive workload data to be forwarded outside the intended trust boundary.
3. Impact is exposure of business context, secrets, or regulated data through observability systems that were meant for monitoring, not data transport.

NHI Mgmt Group analysis

Telemetry export is now a governance surface, not an operations detail. AI gateways increasingly emit spans that can include full prompts and completions, which means observability platforms can become accidental data processors. That changes the control model from simple monitoring to content-aware governance, especially where secrets, customer data, or regulated information may appear in traces. Practitioners should treat trace export like any other sensitive data flow, with policy, minimisation, and auditability built in.

Non-human identity controls do not end at the API key. The ingestion key authenticates the target backend, but it does not decide what content is permitted to move. That distinction matters for NHI governance because machine-to-machine trust often stops at possession of a credential, while the real risk sits in the payload that credential can deliver. The correct control plane extends from the gateway’s service identity through the export policy to the receiving telemetry tenant.

Asynchronous architecture reduces latency but can widen blind spots. By decoupling request handling from telemetry delivery, the gateway avoids visible user impact when the backend is slow or unreachable. That is architecturally sound, but it also means dropped spans, delayed delivery, and failed exports can disappear from the operator’s line of sight unless separately measured. In governance terms, the control gap is assuming that invisible failure is acceptable because the user path still works.

AI observability needs least-privilege content handling, not just least-privilege access. The most useful named concept here is telemetry payload governance: the discipline of deciding which fields may leave the runtime, which may be masked, and which must never be exported. This is where AI security, data security, and identity governance meet. Teams that fail to define it will keep shipping rich traces into systems that were never meant to receive raw production content.

What this signals

Telemetry payload governance: AI programmes now need a policy for what may leave the runtime, not just who may access the runtime. The practical signal is simple: if trace export can carry prompts or outputs, then observability sits inside the AI data boundary and must be governed accordingly.

The hidden risk for practitioners is that a fast gateway can still leak sensitive content at machine speed. Use the control patterns in NIST AI 600-1 Generative AI Profile and pair them with OWASP Non-Human Identity Top 10 thinking where ingestion keys and service identities move data between systems.

For practitioners

• Inventory span fields before enabling external export Review every attribute the gateway can emit, including tfy.input, tfy.output, and routing metadata, then classify which fields are allowed to leave the runtime and which must be removed or masked.
• Use trace exclusion for sensitive workloads Enable the Exclude Request Data toggle for workloads that need latency visibility but should not transmit prompt or response content to SigNoz or any other external backend.
• Monitor exporter loss as a control failure Track dropped spans, queue backlog, and ingestion failures so the team can tell when audit visibility has degraded even though request latency remains normal.
• Separate ingestion authentication from payload policy Treat the signoz-ingestion-key as tenant routing control only, then add policy checks that decide whether the trace content is permitted to cross the telemetry boundary at all.
• Apply NHI governance to telemetry credentials Rotate and scope workspace ingestion keys as non-human identities, restrict who can read them, and tie their use to the minimum telemetry path required for operations.

Key takeaways

• AI gateway traces can expose prompts, responses, and model-routing data, so telemetry export is a data-governance decision as much as an observability choice.
• Asynchronous export protects latency, but it also creates silent failure modes that can erase audit visibility unless the pipeline is monitored independently.
• The right control posture combines payload minimisation, scoped ingestion credentials, and explicit NHI governance for every telemetry path.

Key terms

• Telemetry Payload Governance: The set of rules that determine which telemetry fields may leave a system, which must be masked, and which are prohibited from export. In AI platforms, this governs prompts, outputs, routing metadata, and other content that may be embedded inside spans or logs.
• Ingestion Key: A workspace-scoped credential used to authenticate telemetry delivery to an external observability backend. It routes data to the correct tenant or pipeline, but it does not by itself control what content is allowed to move or how long it is retained.
• Asynchronous Telemetry Export: A delivery pattern where spans are published after the request path completes, usually through a queue or broker. It preserves user-facing performance, but it can also hide delivery failures and create gaps between application success and audit completeness.
• Request Data Exclusion: A configuration choice that strips sensitive request and response fields from exported telemetry. It is a practical minimisation control for AI systems where trace usefulness must be balanced against the risk of leaking prompts, completions, or secrets into downstream tools.

What's in the full article

TruFoundry's full article covers the operational detail this post intentionally leaves for the source:

• Exact OTLP exporter settings for traces and metrics, including endpoint format, protocol, and headers.
• SigNoz ingestion and storage pipeline details, including how spans flow through the shared gateway into per-tenant ClickHouse.
• Field-by-field explanation of the span attributes such as tfy.span_type, tfy.input_short_hand, and gen_ai semantic conventions.
• Practical configuration examples for filtering by service.name and destination in SigNoz explorers.

👉 The full TruFoundry article covers the OTEL configuration surface, span handling, and SigNoz ingestion flow in detail.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management for practitioners responsible for access paths like telemetry export. It is a fit for teams that need to connect identity controls to real operational workflows.