The six generations of AI agents and why evals now matter

At a glance

What this is: Braintrust maps six generations of AI agents and argues that evals, not architecture alone, are what keep changing agent behaviour governable as capability expands.

Why it matters: IAM, NHI, and AI governance teams should treat evals as a control layer for agentic systems because changing behaviour, tool use, and state make static review insufficient.

Context

AI agents are no longer just prompt wrappers. As tool use, retrieval, memory, sandboxes, approvals, and durable state get added, the governance problem shifts from simple output quality to whether the system still behaves safely when its architecture changes.

That creates a direct identity and access question for agentic AI programmes. Once an agent can choose tools, retain state, and act through connected systems, teams have to govern what it can do, when it can do it, and how they know behaviour has drifted.

The article frames evals as the missing control plane because older tests only covered narrow prompt behaviour. In practice, that is a familiar failure pattern in fast-moving platforms: capability grows faster than the organisation's ability to certify safe operation.

Key questions

Q: How should security teams evaluate AI agents that make tool calls and update systems across multiple steps?

A: Security teams should evaluate agents at the workflow level, not just the response level. Test the final outcome, the tool choices, the parameter quality, and each intermediate decision that leads to system changes. Use repeatable tasks, multiple trials, and controlled data so failures can be traced to the exact step that caused them.

Q: Why do older tests stop working as AI agents become more capable?

A: Because each new capability widens the failure surface. A prompt-only system can be judged on the response, but a chain, workflow, or harness can fail in retrieval, routing, state handling, or tool use before the final answer appears. Tests that only measure output miss those intermediate breakdowns.

Q: What are the signs that AI agent testing is too shallow?

A: If evaluation only checks refusal behaviour, overall benchmark scores, or end-to-end task success, it is probably missing the real attack surface. Shallow testing ignores whether the model can be pushed into tool misuse, data leakage, or indirect prompt injection at the exact decision point where failures occur.

Q: When should organisations use shadow runs instead of direct release for agent changes?

A: Use shadow runs when the agent can affect live systems, when tool paths are expensive, or when behaviour has changed enough that offline evals no longer predict real outcomes. Shadow runs let teams compare candidate behaviour against production traffic without giving the new version execution authority.

Technical breakdown

Why prompt-only agents break once tool use enters the stack

A prompt-only agent is a single model call with no tools, no retrieval, and no runtime state. That makes evaluation relatively simple because the unit under test is the answer itself. Once tool use appears, the system stops being a static response generator and becomes a composed system of prompts, context, and external actions. At that point, a good answer can still hide a bad process, and a bad process can be masked by a fluent output. The architecture has changed, so the test surface must change with it. For agentic systems, the critical question is no longer only what the model said, but what evidence it used and what actions it took.

Practical implication: Evaluate the agent as a system, not just as a response generator, and include tool selection and evidence use in test coverage.

How chains and retrieval create hidden failure paths

Chains add fixed sequencing, usually fetch evidence first and then ask the model to interpret it. Retrieval-augmented generation, multi-step prompts, and router patterns all fit this class. The benefit is grounding in current context, but the risk is that each step inherits the errors of the previous step. If retrieval misses the right service, picks the wrong time window, or pulls noisy logs, the model reasons over the wrong evidence with high confidence. This is why chain evals have to measure retrieval recall, intermediate output quality, and end-to-end success together. The control problem is not only answer correctness. It is whether the pipeline retrieved the right context in the first place.

Practical implication: Test retrieval quality and intermediate state, not just the final response, before trusting chained agent behaviour.

Why workflow graphs and harnesses need contract-level evals

Workflow graphs move routing, retries, and guardrails into deterministic code while keeping the LLM inside bounded nodes. That improves predictability, but it also creates graph-level failure modes such as branch drift, schema mismatch, and unreachable paths. A harness then adds live tools, memory, simulations, replay, and shadow runs, which means the system can now fail in runtime interactions rather than only in static outputs. In that environment, evals have to operate like software tests and contract checks. Each node, transition, and policy boundary needs a measurable expected behaviour. For agentic AI governance, this is the difference between hoping the model behaves and proving the runtime still obeys control intent.

Practical implication: Use node-level, contract-level, and replay-based evals so workflow and harness changes remain governable as the agent evolves.

NHI Mgmt Group analysis

Evals are becoming the governance boundary for agentic AI. As agent architecture shifts from prompts to chains, workflows, and harnesses, the meaningful control point moves from model output review to behavioural certification. That matters because agent capability expands faster than human confidence in the system, especially when tools and durable state are involved. The practical conclusion is that eval coverage, not architecture preference, is what determines whether an agent can be shipped safely.

The named concept here is eval drift: the widening gap between what a previous generation of tests can see and what the current agent can actually do. Each new capability introduces failure modes that older evals were never designed to catch, which means a green test suite can become stale without any obvious warning. Teams should treat eval drift as a governance defect, not a testing inconvenience.

Agentic AI creates an identity and authorization problem, not only a model-quality problem. Once an agent can choose tools, use memory, and execute write actions, the security question becomes what authority it holds at runtime and how that authority is constrained. That is directly relevant to AI agent identity and NHI governance, because the agent's access pattern is effectively an access-control policy with decision-making attached. Practitioners should align evaluation with issued privilege, not just with generated text.

Workflow determinism does not eliminate risk, it relocates it. Graphs and bounded nodes make behaviour more inspectable, but they also create new control dependencies around routing, state contracts, and branch coverage. The result is a programme that looks safer on paper while still failing at the seams if transitions are not evaluated. The lesson is to govern the runtime, not just the model.

Production traces should be treated as control evidence. The strongest signal in the article is the move from offline testing to replay, shadow, and online scoring. That is the point where evaluation becomes operational assurance rather than pre-release validation. Teams that do not turn real traces into new evals will keep learning from incidents instead of from the system's normal behaviour.

From our research library:

• 62% of organisations are at least experimenting with AI agents, but only 23% have scaled one into a single business function, according to McKinsey's 2025 State of AI survey.
• Read next: AI Agent Identity Security Buyer's Guide

What this signals

The governance lesson for practitioners is that evals must become a standing control, not a pre-launch checklist. As agentic systems gain tools, memory, and write paths, the team that only measures final answers will miss the failures that actually create operational risk.

Eval drift: the control gap between a changing agent and the tests written for an older version. Once that gap opens, a passing benchmark can coexist with a materially different runtime risk profile, which is why replay and shadow testing matter for release decisions.

Agent identity also becomes a practical governance issue here. If the system can call tools, persist state, and act across sessions, practitioners need evidence that its authority is bounded to the task and that evals verify that boundary before production changes ship.

For practitioners

• Instrument agent behaviour end to end Log prompts, retrieved context, tool calls, model choices, latency, cost, and human corrections so behaviour can be replayed and scored later.
• Build evals around the agent architecture you actually run Use answer-quality tests for prompt-only systems, retrieval and coverage tests for chains, node and contract tests for workflows, and replay plus shadow-run scoring for harnesses.
• Gate every model or prompt change with production-derived cases Convert failures and near misses into golden cases, then require candidate versions to match expected behaviour before they ship.
• Tie eval coverage to tool authority Include tests for destructive actions, escalation paths, and uncertain states so the agent is only allowed to act when the evaluated behaviour matches its permitted scope.

Key takeaways

• AI agent programmes should be judged by behaviour across the full runtime path, not only by the quality of the final response.
• Each architectural generation expands the failure surface, which is why older evals quickly become insufficient as systems add tools, memory, and state.
• Replay, shadow runs, and trace-driven golden cases turn production behaviour into an operational control for safer release decisions.

Key terms

• Evals: Evals are structured measurement systems for judging whether an AI tool performs well across many runs. They do not try to prove one exact output is correct. Instead, they use scoring, thresholds, and repeated scenarios to show whether the tool is reliably useful in practice.
• Workflow Graph: A deterministic control structure that routes an agent through named steps, branches, retries, and guardrails. It reduces model freedom by moving sequencing into code, but it also turns state schemas and branch logic into governed interfaces that must be tested and maintained.
• Shadow Run: A deployment test where a candidate agent processes live traffic in parallel with the production version without taking action. It is used to compare quality, safety, latency, and cost under real conditions before granting the candidate operational authority.
• Replay Testing: A method that re-executes historical production traces against a new agent configuration to see whether behaviour improves or regresses. It turns real incidents and near misses into regression evidence, which is especially valuable for systems that learn through runtime interaction.

Deepen your knowledge

NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course. Explore the NHI Foundation Level course, the industry's only accredited NHI security programme, to build stronger governance for AI agents and machine identities.