Notifications
Clear all
Topic starter
19/08/2026 4:04 pm
TL;DR: Production AI agent failures often hide in retrieval, tool calls, state updates, or model inputs, and Braintrust’s guide shows why trace-level visibility is needed to isolate the failing step rather than rerun the agent blindly. The core implication is that agent observability must span execution paths, not just final outputs, if teams want reliable regression coverage.
NHIMG editorial — based on content published by Braintrust: Agent tracing: how to trace and debug AI agents in production
Questions worth separating out
Q: How should security teams trace AI agent failures in production?
A: Trace the whole agent run, not just the final model call.
Q: Why do AI agents need more than LLM tracing and logs?
A: Because logs and model-call traces miss the workflow around the model.
Q: How do security teams know if agent observability is actually working?
A: Observability is working only when teams can tie together token activity, tool calls, and latency for a specific agent session.
Practitioner guidance
- Instrument the full execution path Capture model calls, tool invocations, retrievals, memory reads, state updates, and handoffs as connected spans so failure analysis does not stop at the final response.
- Separate model issues from workflow issues Use trace review to determine whether the model, the retrieval layer, the tool call, or the state transition caused the failure before you change prompts or policies.
- Convert failed traces into regression cases Label each confirmed failure with a root cause, attach the source spans, and add it to a pre-release eval set so the same pattern is caught earlier next time.
What's in the full article
Braintrust's full guide covers the operational detail this post intentionally leaves for the source:
- Framework-specific setup steps for OpenAI Agents SDK, LangGraph, and Mastra instrumentation
- Code examples for adding manual spans around custom logic and data transformation steps
- OpenTelemetry export configuration for teams already using distributed tracing
- Workflow details for converting a failed trace into a regression eval with scorers and release gates
👉 Read Braintrust's guide to tracing and debugging AI agents in production →
AI agent tracing: what it means for debugging production failures?
Explore further
19/08/2026 4:38 pm
Agent tracing is becoming a governance control, not just an engineering convenience. Once agents can retrieve data, call tools, and update state, the security question is no longer whether the final answer looks plausible. The question is whether the organisation can reconstruct the decision path well enough to explain error, misuse, or credential exposure. That makes traceability part of AI governance, NHI oversight, and incident review.
A question worth separating out:
Q: How should organisations use failing traces to improve agent governance?
A: Turn each confirmed failure into a labelled regression case with source spans, expected output, and a scorer that tests for the same pattern before release. That creates a feedback loop from production evidence to evaluation coverage and makes agent governance measurable rather than anecdotal.
👉 Read our full editorial: Agent tracing exposes where AI agent failures actually begin
