The six generations of AI agents and why evals now matter

At a glance

What this is: This is a Braintrust analysis of six AI agent generations and the eval strategies each one requires as architectures become more capable and more failure-prone.

Why it matters: It matters to IAM and security practitioners because AI agents increasingly behave like governed systems with tools, state, and delegated actions, which means identity, privilege, and control assurance now depend on testing the system’s behaviour, not just its code.

By the numbers:

• Only 44% of developers are reported to follow security best practices for secrets management, exposing a significant developer behaviour gap.
• When AWS credentials are exposed publicly, attackers attempt access within an average of 17 minutes and as quickly as 9 minutes in some cases.

👉 Read Braintrust's analysis of the six generations of AI agents and their eval strategies

Context

AI agents now combine tool use, retrieval, memory, sandboxing, approvals, and durable state, which makes them materially different from simple prompt-based automation. That shift matters because each added capability creates a new control boundary, and the old one rarely survives intact. For AI agent security and governance, the real question is no longer whether the model can answer, but whether the system can be evaluated when it acts.

This post treats evals as the governance mechanism that keeps changing agent architectures testable. For teams building or overseeing AI agents, the challenge sits at the intersection of AI security, workflow control, and delegated identity because the more an agent can do at runtime, the more carefully its behaviour, permissions, and safety constraints must be verified before release.

For practitioners already building around tools, memory, and approvals, the starting point in the article is typical rather than exceptional. The architecture patterns described are increasingly common, which is why evaluation discipline is becoming a programme requirement rather than an engineering preference.

Key questions

Q: What breaks when AI agents are treated like standard human users?

A: You lose visibility into effective permissions, expected behaviour, and real blast radius. Human-centric controls can misclassify normal agent activity as compromise, or miss policy violations that happen entirely within legitimate access. The failure is not only technical, it is governance design that assumes a person is always behind the action.

Q: Why do AI agents complicate access governance more than ordinary automation?

A: AI agents complicate access governance because they can branch at runtime, wait on external services, and continue later with the same operational context. That means privilege is not just granted at launch, it persists across a live session that must be observable, resumable, and attributable.

Q: How do you know if AI agent monitoring is actually working?

A: It is working when you can explain why a sequence of actions was allowed, blocked, or escalated, using evidence from the full chain rather than a single request. If monitoring only shows isolated inputs, it is not capturing agent intent, which is where misuse usually appears.

Q: Who is accountable when an AI agent makes the wrong change?

A: Accountability sits with the governance chain that approved the access model, not with the agent alone. Teams need a trace from requester to policy decision to identity issuance to action results. If that chain is missing, incident review becomes guesswork and access governance cannot be defended to auditors.

Technical breakdown

What changed from prompts to agent harnesses?

A prompt-only system is a single inference step, but agentic systems add context retrieval, tool calls, branching, memory, and sometimes durable state. Each layer changes the failure mode. In prompt systems, you mostly test answer quality. In agent systems, you must also test tool selection, state handling, escalation logic, and whether the system can recover when evidence is incomplete or noisy. That is why evals evolve with architecture. The evaluation surface expands from the text output to the whole runtime behaviour of the system.

Practical implication: teams need eval suites that cover tool use, state transitions, and safety constraints, not just output quality.

Why do workflow graphs improve control but add governance debt?

Workflow graphs move sequencing into deterministic code so the runtime, not the model, decides which node runs next. This improves predictability, but it also creates a new governance problem: every branch, guard threshold, and node contract becomes part of the control plane. If state schemas drift, routing rules are wrong, or edge cases fall off the graph, the system may behave correctly inside nodes but fail at the orchestration layer. The result is a trade between local reliability and global complexity.

Practical implication: treat graph branches and state contracts as governed interfaces with explicit tests and coverage targets.

What do harness-era evals test that earlier generations could not?

Harness-era systems run agents inside simulated or replayed environments, then score behaviour across quality, safety, cost, latency, and policy compliance. This matters because the important failures are often interaction failures, not single-shot output errors. A candidate may look fine on a static benchmark but fail when a tool is flaky, memory is stale, or an environment reply changes mid-task. Harnesses let teams test the system as it will behave in production, which is the only place delegated actions become operational risk.

Practical implication: use simulations, replays, and shadow runs before allowing agents to take real actions.

NHI Mgmt Group analysis

AI agent evals are becoming the missing control plane for delegated action. As agents add tools, memory, approvals, and persistent state, the security problem is no longer limited to prompt quality. The system can now choose actions, sequence work, and carry context across steps, which means assurance has to cover runtime behaviour as well as code. In practice, this makes evals a governance artifact, not just an engineering benchmark.

Agent capability growth widens the attack and failure surface faster than static tests can adapt. Each new generation breaks a previous assumption, whether that is one-shot output predictability, fixed retrieval paths, or stable control flow. The field should read this as evidence that AI governance debt accumulates whenever architecture moves faster than validation. Practitioners should expect evaluation programmes to grow with agent complexity, not after incidents expose the gap.

Workflows reduce model freedom but increase orchestration responsibility. When the runtime owns sequencing, branch logic, and retries, the risk shifts from the model improvising to the system misrouting. That is a useful trade-off, but only if teams test branch coverage, schema contracts, and state transitions with the same discipline they apply to code release. The operational conclusion is that deterministic orchestration needs deterministic assurance.

AI agent identity and delegated privilege are now part of the eval problem. Once agents can use tools and execute actions, the organisation has effectively created a new class of runtime actor that must be authorised, observed, and constrained. That is where NHIs and agentic AI security intersect: access decisions, approval paths, and revocation logic should be testable behaviours. Teams should not treat identity controls as separate from evals when the agent itself can act.

Harnesses point to a more mature operating model for AI security. Replay, shadow execution, and scored simulations give security and platform teams something closer to production assurance than a benchmark score ever can. That does not eliminate risk, but it gives practitioners a way to gate releases on observed behaviour instead of hoped-for reliability. The direction of travel is clear: AI systems will be judged by traceable runtime evidence, not model claims.

What this signals

The operational signal for practitioners is that AI governance will increasingly look like release engineering for privileged systems. As agent capability grows, the control question shifts from whether a model can answer to whether the organisation can verify its runtime choices, especially where tool use and delegated action are involved.

Evaluation debt: When teams keep adding tools, state, and branching logic without expanding the eval suite, they create a governance gap that only shows up under production conditions. That gap is especially relevant for NIST AI Risk Management Framework alignment, because the programme needs repeatable evidence, not just policy statements.

For identity and NHI teams, the next step is to treat agent permissions, approvals, and revocation paths as testable controls. Where an agent can act on behalf of a service or workflow, its authorisation boundary should be reviewed with the same discipline as privileged access and workload identity.

For practitioners

• Build evals for the full agent runtime Cover prompt quality, tool selection, state transitions, escalation behaviour, and destructive-action safety in one controlled test set. Use representative incidents and tasks so the evaluation mirrors the ways the agent actually fails under load.
• Separate node tests from workflow tests Treat each workflow node as a unit test target, then add contract checks for the data passed between nodes. This catches schema drift and branch breakage before the agent reaches end-to-end simulation.
• Use replay and shadow runs before production action Re-run historical traces against candidate versions and compare behaviour on quality, safety, latency, and cost. Let the candidate observe live traffic in shadow mode before any action path is enabled.
• Capture production traces as new eval cases Log inputs, tool calls, retrieved context, intermediate decisions, and human corrections. Turn near misses into regression cases so the eval suite improves from real operational evidence instead of synthetic examples.

Key takeaways

• AI agent evaluation now functions as a control plane because runtime behaviour, not just model output, determines operational risk.
• Each new agent generation widens the failure surface, which means static benchmarks cannot keep pace with capability growth.
• Teams that want safe agent deployment need replay, shadow runs, and branch-aware tests before they let systems act in production.

Key terms

• AI Agent Evaluation: A structured assessment of whether an AI agent behaves as intended under controlled conditions. In practice, it checks output quality, policy adherence, and task performance, but it does not by itself prove resilience against adversarial prompts, tool abuse, or runtime identity misuse.
• Workflow Graph: A deterministic control structure that routes an agent through named steps, branches, retries, and guardrails. It reduces model freedom by moving sequencing into code, but it also turns state schemas and branch logic into governed interfaces that must be tested and maintained.
• Shadow Run: A deployment test where a candidate agent processes live traffic in parallel with the production version without taking action. It is used to compare quality, safety, latency, and cost under real conditions before granting the candidate operational authority.
• Replay Testing: A method that re-executes historical production traces against a new agent configuration to see whether behaviour improves or regresses. It turns real incidents and near misses into regression evidence, which is especially valuable for systems that learn through runtime interaction.

What's in the full article

Braintrust's full blog covers the implementation detail this post intentionally leaves for the source:

• Concrete examples of the six agent generations and the exact eval patterns used for each one
• Code snippets showing how to structure golden datasets, scorers, and workflow contracts
• Practical guidance on replay, shadow runs, and production trace scoring for release gates
• Examples of failure modes such as hallucinations, retrieval misses, branch drift, and unsafe actions

👉 Braintrust's full post includes the architectural examples, eval patterns, and production-testing workflow behind the analysis.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and identity lifecycle controls. It helps practitioners connect delegated access, lifecycle assurance, and operational governance across identity-led programmes.