Notifications
Clear all
Topic starter
20/08/2026 6:08 pm
TL;DR: AI gateways can emit OpenTelemetry traces for every LLM request and export them asynchronously to Arize without adding latency, according to TruFoundry. The architectural takeaway is that observability, routing, and data leakage controls now sit in the same control plane for AI systems, which makes gateway governance part of identity and access governance for models, tools, and agent workflows, while preserving privacy controls that can strip prompt and completion content before anything leaves the gateway.
NHIMG editorial — based on content published by TruFoundry: Ask TFY: Debug, Analyze, and Act on Everything Happening Inside Your AI Gateway
By the numbers:
- While 71% of IT teams have been advised on AI agent data access, only 47% of compliance teams, 39% of legal teams, and 34% of executives have the same visibility.
Questions worth separating out
Q: What breaks when AI gateway traces include prompt content by default?
A: Prompt content in exported traces can expose secrets, personal data, and proprietary instructions to downstream observability systems.
Q: Why do AI gateways need both authorization and telemetry controls?
A: Authorization decides who can call a model or tool, while telemetry controls decide what information leaves the boundary after the call.
Q: How can security teams know whether AI trace export is working safely?
A: Check that traces contain the expected structural fields, such as model identifier, latency, token counts, and status, while sensitive request content is absent when exclusion is enabled.
Practitioner guidance
- Classify gateway traces as governed security data Define which span attributes may contain sensitive content, then map them to data handling rules before enabling external export.
- Redact prompt and completion fields at the gateway boundary Enable request-data exclusion wherever observability leaves the trust boundary, and validate that only structural metadata such as latency, token counts, model metadata, and error status is exported.
- Standardise AI trace schemas across gateway and backend tools Use a consistent span taxonomy and resource attributes so model_id, model_version, and provider context are searchable across observability systems.
What's in the full article
TruFoundry's full article covers the operational detail this post intentionally leaves for the source:
- Step-by-step OTEL exporter configuration for the AI Gateway and Arize integration
- Attribute mapping examples for model_id and model_version across trace views
- Gateway-level exclusion behaviour for tfy.input, tfy.output, and tfy.input_short_hand
- Verification workflow for checking span trees, filtering, and trace grouping in Arize
👉 Read TruFoundry's guide to AI Gateway trace export and prompt redaction →
AI gateway trace export and privacy controls for LLM teams?
Explore further
20/08/2026 6:42 pm
AI gateway telemetry is becoming a governance control, not just an observability feature. Once a gateway validates identity, resolves model access, and exports traces, it is part of the decision boundary for AI systems. That makes the gateway relevant to IAM, PAM, and NHI governance because it can see and shape who or what is allowed to call models and with which context. Practitioners should treat gateway instrumentation as an access-control-adjacent function, not a separate monitoring layer.
A question worth separating out:
Q: Who is accountable when AI gateway telemetry leaks sensitive prompts?
A: Accountability sits with the team that owns the gateway policy, the telemetry configuration, and the data classification rules for exported traces. If the gateway forwards prompt content to a third-party backend without redaction, that is a governance failure, not just an observability bug. The owning team must define the boundary before deployment and verify it continuously.
👉 Read our full editorial: AI gateway traces and content controls for LLM observability
