Eval-driven development is changing how agentic apps are tested

At a glance

What this is: Fiddler argues that eval-driven development turns non-deterministic agentic app testing into a measurable workflow by scoring outputs, iterating automatically, and stopping when results plateau or regress.

Why it matters: This matters because teams building AI agents need testable boundaries and repeatable quality signals before they can govern behaviour, isolate failures, and trust changes in production.

Context

Agentic applications are difficult to test with traditional pass or fail methods because the same prompt and tool chain can produce different results on each run. In practice, that makes eyeballing outputs a weak control for release decisions, especially when the agent is doing multi-step work such as query generation, orchestration, or insight synthesis.

Eval-driven development addresses that gap by treating scored evaluations as the spec for non-deterministic behaviour. The identity and governance angle appears when modular design turns each node into a separately testable control point, which is the same logic IAM and NHI programmes rely on when they split lifecycle, privilege, and execution responsibilities instead of managing one opaque runtime.

The article’s starting position is typical of fast-moving agent programmes: the logic is already being shipped before the governance model catches up.

Key questions

Q: What breaks when agentic applications are tested with pass or fail methods only?

A: Traditional pass or fail tests assume stable outputs, but agentic applications can produce different valid or invalid responses on each run. That means a single check can miss variance, regressions, and failure patterns across the wider input set. Scored evals are needed because they measure behaviour consistently enough to support release decisions.

Q: Why do non-deterministic agent outputs need quantitative evals?

A: Quantitative evals create a repeatable way to compare outputs when exact matching is unrealistic. They let teams measure semantic correctness, detect regressions, and see whether a prompt or orchestration change improved the system overall. Without scores, improvement is mostly subjective and hard to govern.

Q: What do security teams get wrong about monolithic agent workflows?

A: They often treat one large prompt and tool loop as easier to manage, but that design hides which step caused the failure. If metric discovery, query generation, and response synthesis are inseparable, the team cannot isolate risk or improve one control without disturbing the rest. Modularity is what makes evaluation and rollback workable.

Q: When should teams stop automating eval-driven development and bring in human review?

A: Human review should take over when scores plateau, regress, or fail to improve after repeated iterations. At that point, prompt edits alone are usually exhausted and the issue is more likely architectural, such as poor context design, weak tool boundaries, or missing decomposition. The automation should stop before it normalises a broken pattern.

Technical breakdown

Why traditional testing fails for agentic applications

Agentic applications combine LLM reasoning, tool use, orchestration, and stateful iteration, so the same input can legitimately produce different outputs. That breaks the assumption behind classic unit testing, which expects stable outputs from stable inputs. In this setting, manual spot checks create a false sense of confidence because they sample too few paths and do not capture variation across hundreds of cases. The result is a testing problem, but also a governance problem: if the control cannot measure behaviour consistently, it cannot be used to decide whether a change is safe.

Practical implication: teams need scored evaluation criteria before release, not informal output review after the agent is already in use.

How evals turn non-deterministic behaviour into measurable quality

Evals work by defining inputs, expected outcomes, and a scoring method, often using an LLM-as-judge to assess semantic correctness rather than exact string match. That makes them suited to AI systems where acceptable outputs may vary but still need to stay within a bounded quality envelope. The key architectural point is that the eval becomes the specification, while the agent is the thing being measured against it. This is especially relevant for multi-step systems because you can compare scores across runs, identify regressions, and separate real improvement from random variation.

Practical implication: define quantitative eval criteria for the behaviours you intend to govern, then use those scores as the release gate.

Why modular agent design is a prerequisite for safe automation

An agent that performs metric discovery, query generation, execution, and insight creation inside one opaque loop is hard to evaluate because failure in one step is hidden by the rest of the chain. Modular design breaks the workflow into discrete nodes that can be tested in isolation, so each node has a clearer quality boundary. That matters operationally because it lets teams pinpoint which prompt, tool call, or post-processing step is responsible for a regression. The broader governance lesson is that observability starts at design time: without separable components, there is no trustworthy place to attach evaluation, review, or rollback.

Practical implication: refactor monolithic agent flows into testable nodes before relying on automation to tune or approve changes.

Threat narrative

Attacker objective: The objective is not an external attacker but a software failure mode: shipping unreliable agent behaviour that escapes review because the system lacks measurable test boundaries.

1. Entry begins when an untested agentic workflow is promoted on the strength of a few local runs and informal eyeballing rather than a scored eval suite.
2. Escalation occurs when the same non-deterministic flow is allowed to run across more cases, hiding regressions inside prompt edits, tool changes, or orchestration noise.
3. Impact is failed query generation or other incorrect agent output reaching users without a reliable gate to catch quality loss before deployment.

Breaches seen in the wild

• Moltbook AI agent keys breach: Moltbook breach exposed 1.5M AI agent keys.
• DeepSeek database exposure 2025: An unauthenticated DeepSeek ClickHouse database exposed over a million log lines with plaintext chat history and API keys in 2025.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Eval-driven development is becoming a governance control, not just a developer workflow. The article shows that scoring agent output is no longer a nicety for prompt tuning. It is the mechanism that makes non-deterministic behaviour reviewable, comparable, and safe to change. For identity and security teams, the same principle applies wherever runtime behaviour cannot be trusted without a measurable boundary.

Modularisation is the real precondition for assurance. A monolithic agent cannot be governed well because failures are not attributable to a single control point. Once the workflow is split into discrete nodes, each step can be assessed on its own merits, which is exactly how mature security programmes separate access, execution, and validation responsibilities. Practitioners should treat modular design as an assurance prerequisite, not an architecture preference.

The named concept here is evaluation trust boundary. This is the point at which an AI workflow becomes sufficiently separable that scores can meaningfully describe whether the system is behaving within its intended limits. If that boundary is absent, automation can accelerate bad decisions just as efficiently as good ones. Teams should define the trust boundary before they automate improvement loops.

Automated iteration shifts the risk from detection to control design. Once an agent can run experiments, inspect scores, and revert regressions, the human role changes from manual tester to spec owner and final approver. That is useful, but it also means policy has to be written into the eval logic, the rollback condition, and the stop criteria. Practitioners should govern the loop, not admire the loop.

This pattern will push agent governance closer to software change control. When scores plateau, regress, or converge becomes as important as the final output itself. That creates a new operating expectation for AI teams: quality is proven through repeatable measurement, not intuition. Security and platform leaders should align agent release processes with that reality.

From our research library:

• AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
• Read next: AI Agent Identity Security Buyer's Guide

What this signals

Evaluation trust boundary: agent programmes now need a clear line between behaviour that can be scored automatically and behaviour that still requires human judgment. Once that line is undefined, prompt iteration can speed up failure as easily as it improves quality.

For NHI and AI governance teams, the practical shift is to treat agent modularity like control segmentation. If a workflow cannot be broken into measurable steps, release confidence remains weak no matter how polished the final answer looks.

AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.

For practitioners

• Define scored evals before deployment Write quantitative eval cases for the specific agent behaviours you expect, including acceptable output variation and the failure modes that should block release.
• Split monolithic agents into testable nodes Refactor workflows so metric discovery, tool execution, query generation, and response synthesis can each be evaluated separately.
• Add regression rollback to the eval loop Use stop conditions that compare each run against the best seen score and revert prompt or harness changes when quality drops.
• Treat evals as the release spec Make the eval suite the formal definition of acceptable agent behaviour so human review focuses on changes that pass the threshold.
• Set plateau thresholds for human intervention Define when repeated non-improvement means the agent should stop iterating and a person should review architecture, context, or tool design.

Key takeaways

• Agentic applications cannot be governed reliably with traditional output checking because non-deterministic behaviour makes pass or fail logic too coarse.
• Eval-driven development shifts assurance from subjective review to repeatable scoring, which is the only practical way to compare agent changes at scale.
• Modular design is the enabling control because it creates testable boundaries, attributable failures, and a sane rollback point for regressions.

Key terms

• Eval-driven development: A development approach for non-deterministic AI systems that uses scored evaluations to guide iterative improvement. Instead of relying on eyeballing outputs, teams define expected behaviour, run repeatable tests, and use the results to update prompts, harnesses, or agent logic before release.
• Non-deterministic output: Non-deterministic output means the same prompt or input can produce different responses across runs. In security terms, that makes one-off validation weak because the risk lies in the range of behaviours the system may exhibit, not a single predictable result.
• Metadata Trust Boundary: A metadata trust boundary is the line between tool content that can be safely consumed and tool content that must be validated before use. For agentic systems, descriptions, examples, and schemas are security-relevant inputs because they can influence decisions and trigger actions with real-world impact.
• Modular agent design: An architecture that splits an agentic workflow into separate steps such as retrieval, generation, validation, and execution. Each step can be tested and governed independently, which makes failures easier to isolate and reduces the chance that one opaque prompt loop hides multiple control problems.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, and secrets management. It helps practitioners translate identity control principles into operational decisions for AI and automation programmes.