Notifications
Clear all
Topic starter
20/08/2026 6:06 pm
TL;DR: AI agent architectures have evolved through six generations, and each jump in capability widened the failure surface that previous evals could no longer see, according to Braintrust. The key lesson is that evals become the durable control plane for shipping changing agent systems safely, because implementation changes faster than confidence in behaviour.
NHIMG editorial — based on content published by Braintrust: The six generations of AI agents and how to eval them
By the numbers:
- When AWS credentials are exposed publicly, attackers attempt access within an average of 17 minutes and as quickly as 9 minutes in some cases.
Questions worth separating out
Q: What breaks when AI agents are treated like standard human users?
A: You lose visibility into effective permissions, expected behaviour, and real blast radius.
Q: Why do AI agents complicate access governance more than ordinary automation?
A: AI agents complicate access governance because they can branch at runtime, wait on external services, and continue later with the same operational context.
Q: How do you know if AI agent monitoring is actually working?
A: It is working when you can explain why a sequence of actions was allowed, blocked, or escalated, using evidence from the full chain rather than a single request.
Practitioner guidance
- Build evals for the full agent runtime Cover prompt quality, tool selection, state transitions, escalation behaviour, and destructive-action safety in one controlled test set.
- Separate node tests from workflow tests Treat each workflow node as a unit test target, then add contract checks for the data passed between nodes.
- Use replay and shadow runs before production action Re-run historical traces against candidate versions and compare behaviour on quality, safety, latency, and cost.
What's in the full article
Braintrust's full blog covers the implementation detail this post intentionally leaves for the source:
- Concrete examples of the six agent generations and the exact eval patterns used for each one
- Code snippets showing how to structure golden datasets, scorers, and workflow contracts
- Practical guidance on replay, shadow runs, and production trace scoring for release gates
- Examples of failure modes such as hallucinations, retrieval misses, branch drift, and unsafe actions
👉 Read Braintrust's analysis of the six generations of AI agents and their eval strategies →
AI agent generations and the governance gap teams are missing?
Explore further
20/08/2026 6:41 pm
AI agent evals are becoming the missing control plane for delegated action. As agents add tools, memory, approvals, and persistent state, the security problem is no longer limited to prompt quality. The system can now choose actions, sequence work, and carry context across steps, which means assurance has to cover runtime behaviour as well as code. In practice, this makes evals a governance artifact, not just an engineering benchmark.
A question worth separating out:
Q: Who is accountable when an AI agent makes the wrong change?
A: Accountability sits with the governance chain that approved the access model, not with the agent alone. Teams need a trace from requester to policy decision to identity issuance to action results. If that chain is missing, incident review becomes guesswork and access governance cannot be defended to auditors.
👉 Read our full editorial: The six generations of AI agents and why evals now matter
