Eval-driven development

Expanded Definition

Eval-driven development is the discipline of using repeatable evaluations as the primary feedback loop for improving AI behaviour. For non-deterministic systems, especially LLM-based features and agents, the goal is not to “test once and trust forever”, but to define measurable expectations, run the same suite across versions, and compare score shifts over time. This makes the process closer to quality engineering than casual prompt tuning.

In practice, the term covers prompt changes, tool-use constraints, retrieval settings, guardrails, and agent routing logic, because all of these can alter outcomes. It also sits alongside broader governance practices in the NIST Cybersecurity Framework 2.0, where risk management depends on evidence, repeatability, and defined outcomes. Definitions vary across vendors on whether evals should measure only task success or also safety, refusal quality, latency, and policy adherence, so organisations should document what each score actually means.

The most common misapplication is treating ad hoc prompt checking as eval-driven development, which occurs when teams change prompts after a few manual examples without a fixed dataset, scoring rubric, or regression baseline.

Examples and Use Cases

Implementing eval-driven development rigorously often introduces process overhead, requiring teams to balance faster iteration against the cost of maintaining test sets, rubrics, and review discipline.

• A product team creates a regression suite for a support chatbot so that answer quality, hallucination rate, and refusal behaviour can be compared after each prompt change.
• An agentic workflow team scores whether an AI agent chooses the correct tool, follows permitted actions, and stops when it should, rather than relying on one-off demonstrations.
• A security team tests whether an LLM-powered assistant leaks secrets, accepts prompt injection, or ignores policy boundaries, using the results to block unsafe releases.
• A knowledge retrieval system uses evaluations to compare retrieval settings, chunking strategies, and reranking models before rollout, with each release measured against the same benchmark.
• A governance team maps model behaviour checks to broader AI risk management practices described in the NIST Cybersecurity Framework 2.0, so that quality and risk evidence are reviewed together.

Why It Matters for Security Teams

Eval-driven development matters because AI systems can appear safe in a demo and fail under real inputs, especially when prompts, tools, or retrieval sources change. For security teams, that means the release decision should be based on measured behaviour, not intuition. It is particularly important where AI agents have execution authority, because a small regression can turn into unwanted tool use, policy bypass, or exposure of sensitive data.

This approach also supports control validation. If a team cannot show that unsafe outputs, prompt injection, or policy violations were scored before deployment, it is much harder to prove that safeguards are working in a disciplined way. The operational lesson aligns with NIST Cybersecurity Framework 2.0 because evidence-based governance depends on repeatable checks, not informal confidence. Organisations typically encounter the real cost of weak evals only after a model update, when an AI feature regresses in production and rollback becomes the only viable response.

Related resources from NHI Mgmt Group

• How should security teams use AI-driven testing in the development lifecycle?
• Why do AI-driven development cycles create identity governance risk?
• Why do AI-driven development pipelines make remediation slower even when visibility improves?
• Spec-driven Development