Eval-driven development is changing how agentic apps are tested

At a glance

What this is: This is a deep-dive on automating eval-driven development for agentic applications, with the key finding that non-deterministic outputs need quantitative evaluation loops rather than eyeballing.

Why it matters: It matters because security, IAM, and AI governance teams increasingly need a repeatable way to test agent behaviour, isolate failure modes, and prove that access, tool use, and output quality stay within policy.

By the numbers:

• 80% of organisations report their AI agents have already performed actions beyond their intended scope, including accessing unauthorised systems, sharing sensitive data, and revealing credentials.
• 92% agree governing AI agents is critical to enterprise security, yet only 44% have implemented any policies to do so.

👉 Read Fiddler's deep dive on automating eval-driven development for agentic applications

Context

Agentic applications fail in a different way from deterministic software. The same prompt, tool set, or context can produce different outcomes on successive runs, which makes manual spot-checking a weak control for reliability, governance, and identity-sensitive actions.

For identity and security teams, that variability matters because AI agents are increasingly acting like non-human identities with access to data, tools, and downstream systems. The core governance problem is not just output quality, but whether the agent's behaviour can be tested, measured, and constrained well enough to support policy enforcement and auditability.

Fiddler's article uses eval-driven development as the organising concept, but the broader lesson is about operational control: if an agent cannot be isolated into testable nodes, then failures become harder to attribute, harder to revert, and harder to govern.

Key questions

Q: How should security teams govern agentic applications that learn through eval loops?

A: Security teams should treat eval loops as part of the control framework, not just the development workflow. Define measurable acceptance thresholds, separate quality from policy checks, and require human review before changes reach production. If an agent can access tools or data, its evaluation process should also verify authorisation boundaries and auditability.

Q: Why do non-deterministic agent outputs create governance problems?

A: Non-deterministic outputs make it hard to prove repeatability, assign blame, or detect regressions from one run to the next. That weakens confidence in policy enforcement and complicates audit trails, especially when the agent acts on data or tools that have security impact. Governance must shift from eyeballing outputs to measuring them against explicit criteria.

Q: What breaks when an agent is built as one monolithic loop?

A: A monolithic loop hides which step failed when the final output is wrong. Discovery, prompting, tool use, and response generation blur together, so teams cannot isolate the control that needs fixing. Modular design is what makes evaluation, debugging, and policy enforcement practical.

Q: How do teams prevent eval-driven changes from causing regressions?

A: Use a baseline score, rerun the full eval suite after every change, and automatically restore the previous version if performance drops. That keeps local prompt gains from masking broader failures. In production, pair that with access review so behaviour improvements do not conceal privilege creep.

Technical breakdown

Why non-deterministic agent outputs need evals, not eyeballing

Traditional unit tests assume a stable input-output relationship. Agentic systems break that assumption because the model may choose different words, tools, or reasoning paths on each run, even when the prompt is unchanged. Evals replace subjective review with scored judgments against expected outcomes, which is closer to a policy control than a code lint. In practice, that lets teams compare runs, identify regressions, and treat agent quality as something measurable rather than anecdotal.

Practical implication: define scored evals for agent outputs before production, not after users expose failure patterns.

How the eval-driven development loop automates prompt and harness repair

Eval-driven development is a closed loop: run the test suite, analyse low scores, modify the prompt or harness, rerun, and stop when performance plateaus or converges. That is useful because prompt edits are often locally beneficial but globally harmful. The article's key architecture point is that automation can handle the repetitive cycle while preserving a human review checkpoint before deployment. This turns the agent into an optimisation worker, not an unchecked actor.

Practical implication: use automated re-runs and revert logic so failed prompt changes do not silently accumulate.

Why modular agent design is a prerequisite for meaningful evaluation

A monolithic agent that mixes discovery, generation, execution, and interpretation in one loop cannot be reliably evaluated because a failure does not reveal where the defect occurred. Modular design creates discrete nodes, such as query generation or validation, that can be tested in isolation with fixture data. That architecture improves observability, attribution, and remediation speed. For governance, the benefit is clearer control ownership: each node becomes a testable boundary for policy, quality, and access constraints.

Practical implication: break agent workflows into evaluable nodes so you can map failures to a specific control boundary.

NHI Mgmt Group analysis

Eval-driven development is becoming the governance pattern agentic AI actually needs. The article correctly frames EDD as a response to non-deterministic behaviour, but the deeper issue is governance, not developer convenience. Once an agent can vary its output across runs, policy enforcement must move from anecdotal review to measured control. That makes evals part of the control plane for agentic systems, especially where tool use or data access has identity implications. The practitioner conclusion is simple: if you cannot score it, you cannot reliably govern it.

Modularity is the difference between testable AI systems and opaque AI systems. The strongest architectural point in the article is that discrete nodes create evaluable boundaries. That matters because identity and privilege decisions are often embedded in agent workflows, even when the article focuses on PromQL or code generation. A modular pipeline makes it possible to isolate where access, context, or tool selection fails, which is exactly what security teams need when an agent behaves outside policy. The practitioner conclusion is to treat decomposition as a governance requirement, not just an engineering preference.

Agentic workflows are drifting toward non-human identity management problems. As agents acquire tools, credentials, and task-scoped context, they begin to resemble governed machine actors rather than simple applications. That is where IAM and NHI governance intersect with AI development: authentication, authorisation, auditability, and lifecycle boundaries all become relevant once the agent can influence external systems. The practitioner conclusion is that evals should be paired with identity controls, not treated as a substitute for them.

Automated repair loops reduce developer toil, but they can also hide policy drift if the stopping rules are weak. Plateau detection and auto-revert are useful safeguards, yet they only protect the code path under test. They do not guarantee that the agent's real-world access scope, data exposure, or tool permissions remain aligned with governance policy. The practitioner conclusion is to pair EDD with access review, privilege scoping, and production monitoring.

Eval metrics need to be tied to control outcomes, not just model quality. A high semantic score is not the same as a safe or compliant agent decision. Teams need to distinguish output correctness from policy correctness, particularly when the agent can execute actions or retrieve sensitive context. The practitioner conclusion is to define separate evals for usefulness, safety, and authorisation, because a single score rarely captures the whole risk surface.

What this signals

Eval-driven development is becoming a practical control pattern for agentic systems, but it only works if scorecards are tied to security outcomes as well as model quality. Teams should expect more emphasis on test isolation, repeatability, and policy-based acceptance criteria as AI agents move from experiments into production workflows. The relevant next step is to align eval design with the NIST AI Risk Management Framework and the OWASP Top 10 for Agentic Applications 2026.

Modular agent design is the hidden governance requirement in this article. If a workflow cannot be decomposed into nodes with clear boundaries, it becomes difficult to test, monitor, or constrain. That is also where AI identity governance starts to matter, because each node may depend on credentials, context, or tool access that needs lifecycle control.

The programme implication is that identity and AI teams should define separate evaluation paths for behaviour, access, and data exposure, then watch for drift between those paths as agent usage scales.

For practitioners

• Define scored evals for each agent node Create separate test sets for discrete steps such as retrieval, generation, validation, and execution so failures map to one control boundary. Avoid end-to-end only scoring, because it hides where the agent drifted from policy or quality expectations.
• Add stop conditions and auto-revert logic Set a minimum acceptable score, preserve the best-seen run, and restore the previous version automatically when a change regresses performance. This prevents prompt edits from shipping simply because they improved one case while degrading another.
• Treat modular decomposition as a security control Refactor monolithic agent loops into testable components so you can isolate tool use, context construction, and output generation. That makes it easier to apply policy checks, audit behaviour, and pinpoint which node created unsafe or incorrect outcomes.
• Separate quality evals from authorisation checks Score semantic correctness in one eval stream and policy compliance in another. An answer can be technically correct yet still unsafe if it uses the wrong data source, exceeds intended scope, or triggers unauthorised actions.
• Review agent access alongside eval results When an agent improves in test runs, verify that its credentials, tool permissions, and data reach have not expanded beyond the original operating boundary. Good scores do not prove safe privilege scope.

Key takeaways

• Agentic applications need measurable evaluation loops because non-deterministic outputs make manual review an unreliable control.
• Modular design is what turns agent behaviour into something testable, attributable, and governable across development and production.
• As AI agents take on more tool access and data reach, evals must be paired with IAM and NHI controls rather than treated as a substitute for them.

Key terms

• Eval-driven development: A development approach for non-deterministic AI systems that uses scored evaluations to guide iterative improvement. Instead of relying on eyeballing outputs, teams define expected behaviour, run repeatable tests, and use the results to update prompts, harnesses, or agent logic before release.
• Non-deterministic output: Non-deterministic output means the same prompt or input can produce different responses across runs. In security terms, that makes one-off validation weak because the risk lies in the range of behaviours the system may exhibit, not a single predictable result.
• Modular agent design: An architecture that splits an agentic workflow into separate steps such as retrieval, generation, validation, and execution. Each step can be tested and governed independently, which makes failures easier to isolate and reduces the chance that one opaque prompt loop hides multiple control problems.
• Eval boundary: The smallest functional unit of an agentic system that can be measured on its own. Setting clear eval boundaries helps teams identify which node, prompt, or tool interaction caused a failure, and it supports targeted remediation instead of broad, uncertain changes.

What's in the full article

Fiddler's full blog post covers the operational detail this post intentionally leaves for the source:

• The step-by-step EDD loop used to automate reruns, score comparison, and revert decisions.
• The PromQL generation example and dataset structure used to test a modular agent node in isolation.
• The concrete OpenCode workflow for analysing failures and updating prompts or harness logic.
• The stopping conditions that determine when the agent should plateau, revert, or present changes for review.

👉 Fiddler's full post covers the automated inner loop, modular agent design, and PromQL evaluation example.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, secrets management, and agentic AI identity. It helps security practitioners connect evaluation, access control, and lifecycle governance in one programme.