OpenTelemetry for LLM tracing exposes the gap in agent observability

At a glance

What this is: Braintrust argues that OpenTelemetry can standardise LLM and agent tracing, but conventional monitoring still misses whether model outputs are grounded, safe, or correct.

Why it matters: This matters because IAM, NHI, and AI governance teams need visibility into what agents did and whether those actions and outputs were acceptable, not just whether the service was up.

Context

LLM observability is not the same thing as application health monitoring. Traditional APM can confirm that a request completed quickly and without error, yet still leave teams blind to whether an agent used the right retrieval context, chose the right tool, or produced an answer that was safe to act on.

The governance gap becomes sharper once an application can call models, retrieve data, and invoke tools inside a single execution path. That makes agent tracing relevant not only to platform engineering, but to identity and access governance because the trace now reflects who or what was allowed to act, and on which data or tools.

For security and identity teams, the issue is not whether a span exists, but whether the span tells a trustworthy story about delegated action. That is why tracing and evaluation need to sit together when agentic systems begin operating across production workflows.

Key questions

Q: What breaks when teams monitor agents only with standard APM?

A: Standard APM can show that an agent request completed quickly and without transport errors, but it cannot prove that the answer was grounded, policy-compliant, or safe to use. That creates a governance blind spot where healthy infrastructure masks bad model behaviour, unsupported retrieval, or incorrect tool choice.

Q: Why does clean core matter for identity and access governance?

A: Clean core matters because it changes where controls can live. When the SAP digital core is kept minimal, identity governance must operate through supported integrations and policy layers instead of bespoke code. That improves upgrade resilience, but only if IAM and GRC teams redesign controls for portability rather than assuming legacy extensions will carry forward.

Q: How do security teams know if agent observability is actually working?

A: Observability is working only when teams can tie together token activity, tool calls, and latency for a specific agent session. If those signals are disconnected, the logs may show traffic but not governance. The test is whether investigators can reconstruct the path of a meaningful agent action from start to finish.

Q: Should organisations keep tracing and evaluation in the same workflow?

A: Yes, because tracing without evaluation only explains what happened, while evaluation without trace context cannot explain why it happened. Keeping them connected lets teams inspect the run, judge the result, and preserve the failure as evidence for the next release.

Technical breakdown

Why standard APM misses agent behaviour

Traditional monitoring was designed around latency, status codes, and error rates, which are useful but incomplete for LLM applications. A request can succeed at the transport layer while still producing an unsupported, misgrounded, or policy-breaking answer. In agentic systems, the meaningful security and governance questions sit inside the workflow: what prompt was sent, what context was retrieved, what tool was chosen, and what output was returned. OpenTelemetry makes those steps visible as spans and traces, while GenAI semantic conventions give backends a common schema for reading them. That creates portability, but it does not by itself judge correctness or safety.

Practical implication: Treat APM as system-health telemetry and tracing as the evidence layer for agent behaviour, then add separate evaluation controls for output quality.

How GenAI semantic conventions structure LLM traces

OpenTelemetry's GenAI semantic conventions define a shared vocabulary for model interactions so traces can be understood across tools and backends. Fields such as model name, input tokens, output tokens, and operation name let different systems interpret the same run consistently. That matters because LLM applications often span multiple services and frameworks, and ad hoc instrumentation quickly becomes fragmented. With a consistent schema, teams can capture retrieval, tool calls, parsing, and agent steps as structured telemetry instead of opaque HTTP calls. The value is standardisation of evidence, not standardisation of judgment. The trace can describe what happened, but not whether the result was fit for use.

Practical implication: Instrument GenAI spans once with a shared schema so your observability and review workflows can inspect the same execution path.

Why tracing and evaluation must stay separate

Tracing tells you how an agent executed, while evaluation tells you whether the outcome met the criteria you care about. That separation is important because an agent can follow the expected path and still produce an unsafe or incorrect result, and a fast response can still be operationally wrong. Evaluation uses the traced inputs, outputs, metadata, and token context to score quality, then converts failures into reusable tests. For governance, this creates a control loop: observe the run, score the result, and reuse the failure as a regression case. Without that loop, teams have visibility but not assurance.

Practical implication: Build an evaluation layer on top of traces so production failures become test cases instead of recurring surprises.

NHI Mgmt Group analysis

Agent observability is becoming a governance control, not just an engineering aid. Once model calls, retrieval, and tool execution sit inside one request path, the trace becomes part of the control evidence for delegated action. That means the question is not only whether the service responded, but whether the agent used authorised data and tools in a way the business can trust. For practitioners, observability now has to support governance review as well as debugging.

OpenTelemetry solves portability, but portability is not assurance. A shared GenAI schema reduces backend lock-in and lets traces move across tooling, yet a clean trace still does not tell you whether the answer was right. That distinction matters for AI risk management because instrumentation standardises collection, not policy judgment. Teams should treat trace portability as a prerequisite, not a control outcome.

AI agent traceability creates a new kind of identity evidence. When an agent retrieves context or invokes a tool, the security question is effectively who or what exercised that authority and under what scope. This intersects directly with NHI governance because the agent's runtime behaviour depends on service identities, delegated permissions, and tool access boundaries. The control gap is not visibility alone, but provable accountability over delegated execution.

Quality failures are now a production control problem, not a model-review problem. If a bad answer can be traced, scored, and turned into a regression case, then output governance moves from periodic review to continuous assurance. That is the practical direction the market is heading: traces will increasingly serve as evidence for both operations and assurance. Practitioners should align observability, evaluation, and access governance around the same execution record.

Agent observability debt: organisations that instrument only for latency will keep missing the evidence needed to govern model behaviour. The result is a blind spot where operational uptime looks healthy while answer quality, tool choice, and retrieval grounding remain ungoverned. Practitioners should close that gap by treating trace quality, not just trace volume, as the governance target.

From our research library:

• AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
• Read next: AI Agent Identity Security Buyer's Guide

What this signals

Traceability is becoming a control requirement for agentic systems. Once model calls, retrieval, and tool execution are bundled into one request path, teams need evidence of delegated behaviour, not just system uptime. That shifts observability from a performance function into an assurance function, especially where agents can act across sensitive workflows.

Evaluation is the missing second half of observability. If a production trace can be scored and then reused as a regression case, security and AI teams gain a durable feedback loop for controlling quality drift. The practical signal for practitioners is clear: trace coverage is only useful when it feeds decision-making, review, and release gating.

For practitioners

• Instrument prompts, retrieval, tool calls, and outputs as one trace Capture the full execution path so model calls, context retrieval, tool usage, and final responses can be reviewed together. Without that end-to-end trace, teams only see a healthy request, not the steps that produced it.
• Separate operational monitoring from output evaluation Keep latency, error rates, and service health in your APM stack, but score the same spans for groundedness, policy alignment, and task success in a quality layer. That separation prevents false confidence from green infrastructure metrics.
• Use the Collector to route one trace to multiple backends Forward the same GenAI spans to your observability platform and your evaluation backend through a collector pipeline so you can inspect system health and model quality from one source of truth.
• Turn failed traces into regression cases Save the exact failing input, output, and context as a dataset case, then rerun it before release when prompts, models, or tools change. That makes production failure evidence reusable instead of one-off.

Key takeaways

• Standard monitoring can confirm that an LLM service is healthy without showing whether the answer was grounded, safe, or correct.
• OpenTelemetry improves portability by standardising how prompts, retrieval, tool calls, and agent steps are captured across backends.
• The governance value comes from pairing traces with evaluation so production failures become reusable assurance evidence.

Key terms

• OpenTelemetry GenAI semantic conventions: A shared set of attributes for representing model calls, token usage, tool invocations, and related metadata in a portable trace format. They let different observability backends read the same AI execution data without custom schemas or one-off instrumentation.
• Agent observability: Agent observability is the collection and correlation of traces, logs, metrics, and evaluations across an AI agent’s execution path. It explains what happened during model calls, tool use, handoffs, and downstream effects, but it does not itself authorize, deny, or revoke access.
• Evaluation Layer: An evaluation layer is the control surface that scores agent traces against defined quality or policy criteria. It turns observability data into enforcement by detecting regressions, generating reusable test cases, and supporting release gates that prevent broken behaviour from reaching production.
• Distributed Tracing: Distributed tracing is a method for following a request across multiple services so operators can see where latency or failure occurs. It is especially valuable in Kubernetes, but it also captures fine-grained runtime behaviour that should be restricted to those with a clear operational need.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, machine identity security, and secrets management. It helps practitioners connect delegated access, runtime accountability, and identity control across modern security programmes.