Explainable AI for LLM apps needs traces, not token scores

At a glance

What this is: Braintrust argues that explainable AI for LLM apps depends on execution traces plus evaluation scores, because final-response inspection alone misses retrieval, tool-use, and intermediate-step failures.

Why it matters: AI and governance teams need trace-level evidence to diagnose agent behaviour, validate controls over tool use and memory, and avoid fixing symptoms instead of the failing step in the workflow.

Context

Explainability for LLM applications is not the same problem as feature attribution for a single predictive model. Once an AI system can retrieve context, call tools, and carry state across multiple steps, a final answer can look plausible even when an earlier stage failed. That creates a governance gap for AI security teams, because the visible output no longer tells you where the failure started.

The practical issue for AI governance is diagnosis, not just interpretation. A trace gives the execution path, and evaluations give the quality signal on each span, so teams can see whether the breakdown came from retrieval, tool selection, memory, or generation. That distinction matters most for LLM applications and agents, where the control surface spans both the model and the surrounding workflow.

Key questions

Q: How should security teams explain failures in LLM applications and agents?

A: They should use trace-based evaluation, because the failure may originate in retrieval, tool use, memory, or generation rather than in the final response itself. A useful explainability process records each step, scores each span against task criteria, and turns failed runs into regression tests. That approach gives teams repeatable evidence and a clearer governance trail.

Q: Why do final LLM responses hide the real source of errors?

A: Because the output is only the last step in a longer chain. A wrong retrieval, an incorrect tool call, or a bad intermediate state can produce a plausible answer that looks correct at the surface, while the root cause sits earlier in the run. Traces expose that upstream failure so teams can fix the right control point.

Q: When should teams use trace-based explainability instead of token attribution?

A: Use trace-based explainability whenever the system does more than one model call, especially when retrieval, memory, or tools affect the result. Token attribution can help inside a single call, but it cannot show how the surrounding application logic changed the outcome. Multi-step workflows need span-level evidence.

Q: What is the difference between LLM observability and explainability?

A: Observability tracks how the system behaves across traffic, including latency, cost, reliability, and quality trends. Explainability starts from one output and works backward through the evidence that produced it. A trace can support both, but only explainability asks which specific step caused the failure in that run.

Technical breakdown

Why token attribution misses multi-step LLM failures

Token attribution explains how input text influenced one model call, but it cannot reconstruct the application steps that surrounded that call. In an LLM app, the wrong answer may begin with a bad retrieval result, continue through an incorrect tool argument, and only become visible in the final response. That is why a token map of the answer often points to the symptom, not the cause. Execution traces solve the missing context problem by recording the path through retrieval, memory, tool use, and generation, giving evaluators a span-by-span view of failure.

Practical implication: Instrument the full application path so reviewers can identify the failing step instead of chasing the last visible output.

How traces and evals work together in LLM observability

A trace records the sequence of spans inside an AI application, while evals score the outputs or intermediate steps against a defined quality standard. Together they let teams ask two different questions at once: what happened, and how well did it perform. That pairing turns observability into explainability for LLM systems, because the evidence is tied to both execution and quality. Without the scoring layer, a trace only shows movement through the system. Without the trace, a score cannot tell you which stage introduced the defect.

Practical implication: Link every trace to a reusable scoring rubric so failures can be compared, triaged, and rerun consistently.

Why agentic workflows need span-level governance

Agents create longer and less predictable execution chains than single-shot model calls, which makes intermediate-step governance essential. The important object is not only the output, but each span that handled retrieval, planning, tool invocation, and memory. If a wrong step is allowed to propagate, later evaluation may correctly flag the final answer while still leaving the true failure hidden upstream. That is why application-level explainability overlaps with AI observability: both require evidence at the span level, not just the response level.

Practical implication: Treat each agent span as a governance checkpoint and review tool, retrieval, and memory decisions independently.

Threat narrative

Attacker objective: The objective is not an external attacker in this article’s model but the operational outcome of hiding the true failure source inside a multi-step AI workflow.

1. Entry occurs when an LLM application receives a request and begins a multi-step run that may include retrieval, memory, and tool calls.
2. Credential or tool abuse can emerge when the application selects the wrong context or invokes the wrong external function, causing the error to propagate through the chain.
3. Impact appears in the final response, where the output looks plausible even though the underlying execution path was broken.

Breaches seen in the wild

• Moltbook AI agent keys breach: Moltbook breach exposed 1.5M AI agent keys.
• DeepSeek database exposure 2025: An unauthenticated DeepSeek ClickHouse database exposed over a million log lines with plaintext chat history and API keys in 2025.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Trace coverage is now a governance control, not a debugging convenience. LLM systems can fail in retrieval, memory, planning, or tool use before they ever produce a bad answer, which means post-hoc response review is structurally insufficient. The article’s core point is that explainability for AI applications depends on evidence across the execution path, not just the end state. For AI governance, the practitioner conclusion is clear: if you cannot trace a decision chain, you cannot govern it.

LLM explainability has become an identity and access problem as much as a model problem. Once an agent can choose tools, consume context, and act on behalf of a user or workflow, the question shifts from output quality to action authority. That creates a direct intersection with agentic AI identity, because the security issue is often whether the right step had the right permission at the right time. The practitioner conclusion is to govern agent actions as well as model outputs.

Execution traces create the named control gap this article exposes: the intermediate-step visibility gap. Final-response inspection assumes the answer contains enough evidence to reconstruct causality, but multi-step AI systems break that assumption. Traces plus evals close the gap by preserving span-level context and quality signals together. The practitioner conclusion is to make trace completeness part of the AI operating model.

Braintrust’s strongest analytical contribution is not a product comparison but a workflow thesis. The meaningful boundary is between application-level explanation and model-level attribution, and teams need both only when they are solving different problems. For LLM apps, the operational question is whether the system can show which step failed and why, not whether a feature contributed to a prediction. The practitioner conclusion is to match the explanation method to the layer being governed.

What this article signals for the field is a move from model-centred AI assurance to system-centred AI assurance. As agents absorb more context and tool authority, evaluation has to follow the run, not the token. That shift aligns more closely with AI observability, agent governance, and the emerging need to treat execution traces as first-class evidence. The practitioner conclusion is to build assurance around workflow evidence, not isolated model outputs.

From our research library:

• AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
• Read next: AI Agent Observability, Audit and Incident Response Guide

What this signals

Intermediate-step visibility is becoming the real assurance boundary for AI teams. Once an LLM application can retrieve, call tools, and retain state, the final answer no longer tells the full governance story. Teams should expect trace completeness to matter as much as output quality, especially where AI agents act with delegated authority.

Agentic AI identity is part of the explainability problem. When a system can choose actions and tools at runtime, security teams need to know which span exercised which permission. That pushes AI governance closer to identity governance, because the control question becomes who or what was authorised to do each step.

Trace-plus-eval workflows align well with emerging AI risk practices. They support the kind of evidence-based control testing that AI governance programmes need, and they fit naturally alongside AI observability and agent review. For practitioners, the next move is to make execution traces a standard artefact in release and incident workflows.

For practitioners

• Instrument full execution traces Capture retrieval, memory, model calls, and tool invocations for every LLM run so the full decision path is available for review.
• Pair traces with scoring rubrics Attach evaluators that measure factuality, format compliance, and task success to each span or run, then reuse the same scoring logic in staging and production.
• Treat low-scoring runs as regression cases Promote failed traces into datasets and rerun them as fixed test cases so the same defect cannot reappear silently in later releases.
• Review agent tool authority separately Check whether an agent had the right permission to select a tool, fetch context, or write back state before you judge the final answer quality.

Key takeaways

• LLM explainability fails when teams inspect only the last answer, because the real defect often happened earlier in retrieval, planning, or tool execution.
• Trace-based evaluation creates the evidence needed to separate symptom from cause in agentic workflows, which is essential for AI governance.
• The practical control shift is from output inspection to span-level review, with traces, evaluators, and regression cases working as one loop.

Key terms

• Session Trace: A complete record of one agent run from the initial prompt to the final output. It usually contains nested spans for model calls, tool invocations, and handoffs, giving teams a timeline that shows sequence, duration, and context across the full task.
• Span: A span is one discrete operation inside a trace, such as a retrieval query, prompt assembly step, or model call. Spans make it possible to pinpoint which part of an AI workflow consumed time, introduced error, or passed bad context to the next stage.
• Evaluation Score: An evaluation score is a measured output that represents how well an AI system performed against defined criteria such as accuracy, safety, or task completion. It is useful only when paired with context about the dataset, audience, and failure mode it was intended to represent.
• Trace-based Explainability: An approach to AI explainability that reconstructs how an application reached an output by examining the full execution path. It captures retrieval, tool calls, memory, and generation so teams can diagnose the step that introduced an error, not just the final response.

Deepen your knowledge

NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course. Explore nhimg.org for resources that connect identity governance to the broader security disciplines your programme depends on.