Topic starter
20/08/2026 6:08 pm
TL;DR: AI gateways can emit OpenTelemetry traces for every LLM request and export them asynchronously to Arize without adding latency, according to TruFoundry. The architectural takeaway is that observability, routing, and data leakage controls now sit in the same control plane for AI systems, which makes gateway governance part of identity and access governance for models, tools, and agent workflows, while preserving privacy controls that can strip prompt and completion content before anything leaves the gateway.
Editorial analysis by NHI Mgmt Group, based on content published by TruFoundry: “Arize integration with TrueFoundry”.
By the numbers:
- TrueFoundry says the gateway handles 350+ RPS on just 1 vCPU in its production-ready platform description.
Key questions
Q: How should teams stop prompt content from leaving an AI gateway through telemetry export?
A: Treat the gateway as the privacy boundary.
Q: Why do AI gateways change the authorisation model for LLM traffic?
A: Because the gateway is now where identity, model selection, and tool access are resolved together.
Q: What are the signs that LLM observability is not working well enough?
A: Common warning signs include unexplained latency, inconsistent outputs, missing traces, weak session context, and no clear link between prompts and downstream actions.
Practitioner guidance
- Define a telemetry allowlist for LLM traces Only export the span attributes needed for performance, reliability, and cost analysis.
- Separate access control from observability export Review whether JWT validation, model authorisation, and trace export are governed by different policies or by a single shared boundary.
- Use asynchronous export with internal buffering Ensure trace delivery cannot block inference traffic.
Bottom line: AI gateway tracing can provide detailed LLM observability without forcing prompt content to leave the gateway, which makes export policy part of the security design.
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
20/08/2026 6:42 pm
AI observability is becoming a governance boundary, not just a monitoring feature. When traces can include prompts, completions, routing decisions, and token counts, the telemetry pipeline itself becomes part of the security model. The practical implication is that teams should treat observability export as a controlled data flow, not an engineering afterthought.
A few things that frame the scale:
- AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
A question worth separating out:
Q: When is it better to keep LLM content inside the gateway instead of exporting it?
A: Keep content inside the gateway whenever the prompt, completion, or tool payload may contain personal data, confidential business data, or regulated material. Export the minimum telemetry needed for troubleshooting and evaluation, and retain the richer content only in systems that are explicitly approved to store it.
👉 Read our full editorial: AI gateway traces and content controls for LLM observability