OpenTelemetry dual export gives AI traces both observability and governance

At a glance

What this is: This walkthrough shows how one OpenTelemetry stream can send the same AI traces to both Arize AX and Databricks Unity Catalog, creating separate operational and governed analytical views of the same production telemetry.

Why it matters: It matters because AI, data, and identity teams increasingly need shared telemetry without losing control over who can access, query, or repurpose production traces.

👉 Read Arize's walkthrough on sending OpenTelemetry traces to Arize AX and Unity Catalog

Context

The governance gap here is not trace collection itself, but what happens when observability data becomes a reusable enterprise asset. Agent traces can contain prompts, tool calls, model outputs, and evaluation signals, which makes them useful for debugging and analytics but also sensitive from an access and reuse standpoint. In practice, teams often split those needs across separate systems and end up duplicating pipelines or losing consistency.

OpenTelemetry changes that pattern by allowing one telemetry stream to fan out to multiple destinations at source. In identity and AI governance terms, that creates a control problem as much as an integration benefit: the same trace may be operational evidence, analytical input, and potentially retraining material. For practitioners, the core question is not whether to centralise traces, but how to govern reuse across debugging, analytics, and downstream model workflows.

Key questions

Q: How should teams govern AI traces that are exported to both observability and analytics platforms?

A: Teams should treat dual-export traces as governed data, not just logging output. Set policy at the instrumentation or collector layer, classify span fields, and apply destination-specific access rules before traces reach shared analytics. The key is to control who can read, join, retain, and reuse telemetry across debugging, reporting, and AI lifecycle workflows.

Q: Why do production AI traces create security and governance risk?

A: Production traces can include prompts, tool calls, model outputs, and evaluation signals, which means they may expose sensitive context beyond standard application logs. If those traces move into a shared data platform without controls, they can be copied, queried, and reused far more widely than the original operational need justifies.

Q: How do organisations know whether trace data is being reused safely?

A: Look for clear lineage from capture to destination, named owners for each reuse class, and a documented reason for every move from debugging evidence into analytics or training. If teams cannot explain who approved reuse and which policy enabled it, the governance model is too weak for production AI telemetry.

Q: What should security teams do before production traces become training data?

A: They should require explicit approval, provenance tracking, and field-level review before any production trace enters a training or evaluation dataset. That prevents operational telemetry from being repurposed without consent, and it keeps AI lifecycle decisions tied to data governance rather than convenience.

Technical breakdown

Why OpenTelemetry fan-out changes trace governance

OpenTelemetry can export a single span to multiple destinations by attaching more than one span processor to the same TracerProvider. Each processor receives the same telemetry at emission time, so the system does not need a second replication job or a post-processing ETL step to duplicate the data. That matters because trace pipelines are often built to serve one purpose at a time, either low-latency debugging or long-term storage and analytics. Fan-out makes the instrumentation layer the control point, not the downstream copy job.

Practical implication: govern export destinations at the collector or SDK layer, not only in the lakehouse.

What changes when traces move into governed storage

When traces land in Unity Catalog, they become queryable alongside other enterprise data and inherit the access and governance model of the platform. That shifts traces from being purely operational artefacts to becoming governed analytical records. In AI systems, those records may include prompts, model responses, tool invocation metadata, latency, and evaluation output. The security question is no longer just who can see a trace during incident response, but who can query, join, retain, or reuse it later.

Practical implication: classify trace fields and apply data access controls before traces reach shared analytics layers.

How dual export supports AI evaluation and lifecycle reuse

A production trace can be reused across multiple stages of the AI lifecycle. The same span that explains a failed response in an observability tool can become an evaluation example, a regression test case, a human review item, or a training input later on. That reuse is efficient, but it also collapses the boundary between operational telemetry and model development data. Once that boundary disappears, governance has to track provenance, purpose limitation, and retention with more precision than a single-system workflow requires.

Practical implication: define when production traces may move from debugging evidence into evaluation or training datasets.

NHI Mgmt Group analysis

Dual-export telemetry creates governance debt if the organisation cannot explain downstream reuse. The same trace can support debugging, analytics, and model improvement, but each use case implies a different access model and retention rule. Without clear lineage, teams will treat production traces as harmless metadata when they can also expose prompts, tool calls, and decision context. Practitioners should treat trace fan-out as a governance design decision, not just an observability convenience.

Trace data is becoming an identity-adjacent asset because access controls determine whether it stays operational or becomes enterprise-wide input. In a governed lakehouse, traces may be queried by analysts, ML engineers, and platform teams who are far removed from the original incident. That makes least privilege, purpose limitation, and dataset-level accountability central to AI telemetry governance. Practitioners should decide who can read, enrich, and reuse traces before they are merged into shared data planes.

Agent traces now sit at the intersection of AI governance and data security posture management. This is not only about observability retention, it is about whether sensitive production telemetry is discoverable, joinable, and exportable at scale. The right lens is closer to NIST AI RMF governance and data control discipline than to traditional logging alone. Practitioners should align telemetry policy with data classification and AI lifecycle boundaries.

OpenTelemetry makes multi-destination export easy, which means the control problem shifts to policy enforcement. If every instrumented LLM call and tool invocation can flow into multiple systems, the hard part becomes ensuring each destination receives only the data it should. That requires explicit decisions about field filtering, masking, access segmentation, and retention. Practitioners should build policy around telemetry flow, not assume downstream platforms will compensate for upstream overexposure.

What this signals

Trace fan-out will push more organisations to think about AI telemetry as governed content rather than disposable logs. The practical shift is toward access models that distinguish incident response, analytics, and model improvement. NIST AI RMF governance and data handling discipline matter here, especially when traces cross into shared platforms that also hold business data.

Telemetry reuse boundary: the point at which operational traces stop being debug artefacts and start becoming governed enterprise data. Once that boundary is crossed, data teams, ML teams, and security teams all need a common policy for retention, lineage, and permissible reuse.

For readers building AI platforms, the next control gap is not whether traces can be exported to multiple systems. It is whether the organisation can prove which traces were sent where, who could read them, and whether downstream reuse matched the original collection purpose.

For practitioners

• Define trace reuse classes Separate production traces into operational, analytical, and training-eligible classes before they leave the instrumentation layer. Each class should have its own access policy, retention period, and approved downstream destinations.
• Enforce export controls at the SDK or collector Treat the TracerProvider and collector pipeline as enforcement points, not just plumbing. Apply filtering, masking, and destination allowlists there so sensitive prompts or tool payloads do not flow into every sink by default.
• Classify trace fields before lakehouse ingestion Identify which span attributes contain prompts, model output, customer identifiers, tool parameters, or evaluation data. Apply access controls and retention rules in Unity Catalog based on those field classes, not on the table as a whole.
• Track provenance for evaluation reuse Record when a production trace is promoted into an evaluation set, regression suite, or training corpus. Provenance needs to show where the trace came from, who approved reuse, and which policy allowed the transfer.

Key takeaways

• OpenTelemetry dual export solves a workflow problem, but it also expands the governance surface of AI production traces.
• The same trace can become debugging evidence, analytical input, and training material, so provenance and access control need to travel with it.
• Practitioners should set policy at the instrumentation layer, because downstream platforms cannot reliably compensate for uncontrolled upstream fan-out.

Key terms

• OpenTelemetry: An open standard for collecting and moving traces, metrics, and logs in a consistent format. It helps teams avoid re-instrumenting every system when tools change, and it supports cross-platform analysis by keeping telemetry structured and portable.
• Trace fan-out: Trace fan-out is the practice of sending the same trace to more than one destination at the point of emission. It reduces duplication, but it also creates governance questions about access, retention, masking, and whether downstream reuse stays within approved boundaries.
• Governed storage: Governed storage is a data repository where access, retention, lineage, and usage rules are centrally enforced. For production AI traces, that means telemetry can be queried and analysed without losing the controls needed to limit sensitive data exposure.

What's in the full article

Arize's full article covers the operational implementation details this post intentionally leaves for the source:

• The exact Databricks notebook setup for dual export, including the working code path and runtime assumptions
• The OpenTelemetry and OpenInference dependency details needed to reproduce the integration in a live workspace
• The verification queries used to confirm that spans landed in both Arize AX and Unity Catalog
• The secrets handling and environment configuration steps required to run the example end to end

👉 The full Arize article includes the notebook, code snippet, and verification steps for dual trace export.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, IAM, and secrets management. It helps practitioners align identity controls with the broader security and data workflows that production AI systems now depend on.