Topic starter
19/08/2026 4:00 pm
TL;DR: Stateful agents cannot be evaluated with single-prompt replay because memory, live systems, and multi-step trajectories create failure modes that only appear over time, from wrong-but-consistent narratives to stale assumptions, according to Braintrust. The practical problem is not just scoring quality, but governing state, side effects, and regression risk as agent workflows move into production.
Editorial analysis by NHI Mgmt Group, based on content published by Braintrust: “How to eval stateful agents”.
Key questions
Q: What breaks when stateful agent evals rely on prompt replay?
A: Prompt replay misses the state transitions that make stateful agents risky.
Q: Why do live tools and memory create more eval risk for agents?
A: Live tools and memory make the agent depend on mutable conditions outside the prompt.
Q: What do security teams get wrong about testing stateful agents?
A: They often optimise for scoring when the real issue is state control.
Practitioner guidance
- Implement trace capture for every tool step Log prompts, tool calls, arguments, responses, metadata, and intermediate state so each agent run can be reconstructed as a timeline and tree.
- Promote failed traces into regression datasets Turn real production failures into versioned test cases so the exact stateful scenario is exercised again in future eval runs.
- Define which environment state must be real Document the systems, records, and permissions that need to be authentic for the eval to be meaningful, and mock the rest.
Bottom line: Stateful agents are governed by trajectory, not by isolated prompt quality, because memory and tool use create downstream effects that single-turn replay cannot expose.
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
19/08/2026 4:37 pm
Trace fidelity is becoming a governance control, not just an engineering convenience. Once an agent can act on live systems, the trace is the only durable record of how delegated authority was used. That makes observability part of accountability, because teams need evidence of both the decision path and the state that shaped it. For IAM and NHI programmes, this is the point where access governance meets runtime assurance.
A question worth separating out:
Q: How should teams govern agent workflows that can write to live systems?
A: Treat the workflow as a change-controlled system, not a chatbot. Require trace logging, explicit permission boundaries, regression tests built from real failures, and rollback-aware validation before shipping updates. If the agent can create tickets, modify records, or trigger deployment, those actions need the same scrutiny as any other production change.
👉 Read our full editorial: Stateful agent evaluation needs trace-based controls, not prompt replay