Topic starter
20/08/2026 6:06 pm
TL;DR: AI agent architectures have evolved through six generations, and each jump in capability widened the failure surface that previous evals could no longer see, according to Braintrust. The key lesson is that evals become the durable control plane for shipping changing agent systems safely, because implementation changes faster than confidence in behaviour.
Editorial analysis by NHI Mgmt Group, based on content published by Braintrust: “The six generations of AI agents and how to eval them”.
Key questions
A: Security teams should evaluate agents at the workflow level, not just the response level.
Q: Why do older tests stop working as AI agents become more capable?
A: Because each new capability widens the failure surface.
Q: What are the signs that AI agent testing is too shallow?
A: If evaluation only checks refusal behaviour, overall benchmark scores, or end-to-end task success, it is probably missing the real attack surface.
Practitioner guidance
- Instrument agent behaviour end to end Log prompts, retrieved context, tool calls, model choices, latency, cost, and human corrections so behaviour can be replayed and scored later.
- Build evals around the agent architecture you actually run Use answer-quality tests for prompt-only systems, retrieval and coverage tests for chains, node and contract tests for workflows, and replay plus shadow-run scoring for harnesses.
- Gate every model or prompt change with production-derived cases Convert failures and near misses into golden cases, then require candidate versions to match expected behaviour before they ship.
Bottom line: AI agent programmes should be judged by behaviour across the full runtime path, not only by the quality of the final response.
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
20/08/2026 6:41 pm
Evals are becoming the governance boundary for agentic AI. As agent architecture shifts from prompts to chains, workflows, and harnesses, the meaningful control point moves from model output review to behavioural certification. That matters because agent capability expands faster than human confidence in the system, especially when tools and durable state are involved. The practical conclusion is that eval coverage, not architecture preference, is what determines whether an agent can be shipped safely.
A few things that frame the scale:
- 62% of organisations are at least experimenting with AI agents, but only 23% have scaled one into a single business function, according to McKinsey's 2025 State of AI survey.
A question worth separating out:
Q: When should organisations use shadow runs instead of direct release for agent changes?
A: Use shadow runs when the agent can affect live systems, when tool paths are expensive, or when behaviour has changed enough that offline evals no longer predict real outcomes. Shadow runs let teams compare candidate behaviour against production traffic without giving the new version execution authority.
👉 Read our full editorial: The six generations of AI agents and why evals now matter