Notifications
Clear all
Topic starter
19/08/2026 3:28 pm
TL;DR: Wrong LLM answers are often caused upstream in retrieval, prompt assembly, tool execution, or model routing, but standard logs usually capture only the final output, according to Braintrust. The practical shift is to treat trace trees and regression datasets as the default for incident analysis, because observability without span-level evidence leaves teams unable to prove root cause.
NHIMG editorial — based on content published by Braintrust: How to trace an LLM failure back to its root cause
Questions worth separating out
Q: How should teams debug wrong answers from an LLM in production?
A: Start with the trace, not the final response.
Q: Why do standard application logs fail for LLM root cause analysis?
A: Because they usually collapse a multi-step workflow into a single output record.
Q: How do you know whether an LLM failure came from retrieval or generation?
A: Check the trace spans in sequence.
Practitioner guidance
- Capture span-level traces for every production AI request Record retrieval results, rendered prompts, tool calls, model versions, token counts, and response metadata so engineers can reconstruct the execution path after a failure.
- Filter incidents by request metadata before opening the trace tree Attach stable identifiers such as user, session, environment, and model route at request time.
- Promote confirmed failures into regression datasets Turn each root-caused production miss into a dataset row with the original input and expected output, then run it in CI before release.
What's in the full article
Braintrust's full guide covers the operational detail this post intentionally leaves for the source:
- Step-by-step trace filtering examples for finding a failed request by user, timestamp, error, or text fragment.
- Query patterns that isolate spans anywhere in the trace tree, including ANY_SPAN logic for nested failures.
- How to promote a failed production trace into a dataset row and connect it to evaluation coverage.
- Examples of how trace metadata helps separate retrieval misses, prompt regressions, and model changes.
👉 Read Braintrust's guide to tracing an LLM failure back to its root cause →
LLM trace analysis: what root-cause workflows should teams use?
Explore further
19/08/2026 4:11 pm
Trace visibility is now part of AI governance, not just observability. When AI systems fail, the critical evidence is usually spread across retrieval, prompt assembly, tool calls, and model output. That makes span-level telemetry a governance control because it determines whether teams can explain behaviour after the fact. For AI programmes, the practical standard is not whether a model responded, but whether the organisation can reconstruct how it got there.
A question worth separating out:
Q: What should teams do after they fix an LLM production incident?
A: Add the failing request to a regression dataset and run it in evaluation before the next release. That turns a one-off incident into a repeatable control. If the same input can fail again after a prompt, retrieval, or model change, the organisation has not actually closed the issue.
👉 Read our full editorial: LLM trace analysis shows why logs miss the real failure
