Notifications
Clear all
Topic starter
19/08/2026 4:00 pm
TL;DR: Stateful agents cannot be evaluated with single-prompt replay because memory, live systems, and multi-step trajectories create failure modes that only appear over time, from wrong-but-consistent narratives to stale assumptions, according to Braintrust. The practical problem is not just scoring quality, but governing state, side effects, and regression risk as agent workflows move into production.
NHIMG editorial — based on content published by Braintrust: How to eval stateful agents
By the numbers:
- When AWS credentials are exposed publicly, attackers attempt access within an average of 17 minutes and as quickly as 9 minutes in some cases.
Questions worth separating out
Q: How should security teams evaluate agents that change state across multiple steps?
A: They should test the full trajectory, not just the final response.
Q: Why do stateful agents create more governance risk than stateless ones?
A: Because they can carry forward mistaken assumptions, stale context, and side effects into later steps.
Q: What do teams get wrong when they try to test agent memory with simple replay?
A: They assume the same prompt will produce the same meaningful test case.
Practitioner guidance
- Instrument every agent step as a trace span Capture tool calls, intermediate outputs, metadata, and the before-and-after state of any record the agent touches.
- Promote real failures into regression datasets Convert bad traces from production into versioned eval cases so the next release is tested against the same failure mode.
- Separate step scoring from end-to-end scoring Use span-level checks for tool choice and argument quality, then add trace-level checks for task completion and final environment state.
What's in the full article
Braintrust's full blog post covers the operational detail this post intentionally leaves for the source:
- Trace instrumentation examples for LLM calls, tool invocations, and application-layer spans
- Dataset pipeline patterns for turning production failures into versioned regression cases
- Span-level and trace-level scorer examples for step validation and end-to-end outcome checks
- CI integration details for blocking regressions before agent changes ship
👉 Read Braintrust's guide to evaluating stateful agents →
Stateful agents and evals: what changes when agents do real work?
Explore further
19/08/2026 4:37 pm
Stateful agents are becoming privileged software identities, not just models with memory. Once an agent can hold context, call tools, and modify records, the governance question moves from prompt quality to control of persistent operational authority. That intersects directly with IAM and NHI because the agent is acting like a software identity with memory, permissions, and a lifecycle. Practitioners should treat these systems as governed entities, not transient interfaces.
A question worth separating out:
Q: Who is accountable when a stateful agent creates an unsafe side effect?
A: The organisation operating the agent remains accountable, because the agent is acting as a governed software entity inside a controlled workflow. That makes ownership, logging, approval boundaries, and rollback capability part of the control model, not optional operational extras. If the agent can act, the programme must be able to explain and contain that action.
👉 Read our full editorial: Stateful agent evaluation needs trace-based controls, not prompt replay
