Dfbench raises the bar for evaluating defensive AI security agents

At a glance

What this is: Dfbench is a benchmark for frontier models and agentic systems that tests whether they can sustain open-ended defensive security work across real repositories and changing codebases.

Why it matters: It matters because security teams evaluating AI-assisted code review, AppSec, and NHI-adjacent automation need evidence of coverage, precision, and lifecycle handling, not just isolated bug-finding performance.

By the numbers:

• Developers pushed 986 million commits to GitHub in 2025, 25% more than the year before.
• NIST reports that CVE submissions increased 263% between 2020 and 2025.
• Dfbench includes 253 total examples and 910 total vulnerabilities, averaging 3.6 vulnerabilities per example.

👉 Read depthfirst's benchmark analysis for defensive AI security evaluation

Context

Frontier model evaluation has a coverage problem. Many security benchmarks still measure whether a system can reproduce known findings, but real defensive work depends on finding what is not yet known, validating which findings matter, and keeping pace as code changes. That makes this article about cybersecurity benchmark design, but it also intersects with agentic AI governance because the benchmark is really testing whether a security agent can be trusted to operate as part of a defensive workflow.

The first-order issue for practitioners is not whether an agent can produce an answer, but whether it can maintain useful security coverage at acceptable cost. That is especially relevant for organisations exploring AI-assisted AppSec, code review, and broader NHI governance, where the system’s identity, scope, and persistence matter as much as its raw capability. In that sense, the article is a good proxy for how AI security tooling will be judged in production, not just in demos.

Key questions

Q: What breaks when an AI security agent only optimises for vulnerability recall?

A: It can look effective while generating findings that teams cannot validate or act on. High recall without precision creates noise, and without lifecycle awareness the agent may miss what changed between scans. That combination wastes analyst effort and leaves security coverage stale when code moves quickly.

Q: Why do changing codebases make AI security evaluation harder?

A: Because the task is no longer just finding a bug once. The system has to preserve context, distinguish patched from unresolved issues, and recognise new weaknesses introduced by later commits. That means security evaluation becomes a lifecycle problem, not a one-off detection problem.

Q: What do security teams get wrong about AI agent benchmarks?

A: They often measure end-to-end complexity or general model quality instead of the exact failure moment. The result is a score that looks useful but does not isolate whether the backbone, tool access, or orchestration caused the problem. Benchmarks need a specific state, attack vector, and scoring function to be operationally meaningful.

Q: How should organisations evaluate AI agents used for security review?

A: Use realistic audit scopes, multi-repository examples, and rescans across changing commits. Score the agent on useful coverage, finding quality, and the cost of keeping results current. If the tool cannot support that workflow, it is not ready for defensive operations.

Technical breakdown

Why open-ended security evaluation is harder than bug reproduction

Bug reproduction benchmarks test bounded recall: can a model identify a vulnerability already described in the prompt? Open-ended defensive evaluation is different. The agent must discover issues across a live security scope, decide which findings are real, and maintain that coverage as software evolves. That introduces three separate problems: search breadth, precision, and lifecycle continuity. If any one of those fails, the system may look effective while still missing exploitable issues or generating unusable noise. In practice, defensive AI needs to behave more like a security analyst with memory than a classifier over static examples.

Practical implication: Practitioners should evaluate AI security tools on multi-step audit work, not just single-vulnerability prompts.

How detect, validate, and differential analysis fit together

Dfbench separates defensive security into three tasks because each one measures a different operational property. Detect measures vulnerability recall across a scope, validate measures precision of reported findings, and differential analysis checks whether the system can carry forward context when code changes from a base commit to a target commit. That separation matters because a model can be strong at broad scanning but weak at verification, or accurate on one code state but ineffective at tracking what changed. The benchmark design mirrors real security operations, where findings, triage, and rescan logic are all part of the same control loop.

Practical implication: Teams should assess whether an AI agent can preserve context across rescans before allowing it into recurring review workflows.

Why security coverage becomes a lifecycle problem in fast-moving codebases

Security coverage is not a one-time scan property. As repositories evolve, patches land, new flaws appear, and known weaknesses become stale or resolved. Dfbench treats that as a state-machine problem, with base and target commits representing transitions in the vulnerability profile. That model is especially relevant to agentic systems because the cost of re-establishing trust after each change can erase any efficiency gains. The architectural point is simple: coverage only has value if it remains current enough to support action, and precision only matters if the results stay aligned with the code state being assessed.

Practical implication: Practitioners should measure whether AI-assisted review can refresh security state incrementally without losing accuracy.

Threat narrative

Attacker objective: The objective is to exploit weaknesses faster than defenders can find, validate, and maintain coverage for them.

1. Entry begins when vulnerable code or a security scope contains exposed issues that a scanning agent must discover without being told where they are.
2. Escalation occurs when the agent must reason across multiple repositories, shared libraries, and authorization assumptions to confirm a real exploit path.
3. Impact is a missed or noisy assessment that leaves exploitable issues unaddressed or wastes analyst time, reducing defensive coverage when code changes quickly.

NHI Mgmt Group analysis

Open-ended security evaluation is now an AI governance problem, not just a benchmark-design problem. Once frontier systems are asked to perform defensive work over live code, their identity as tools, their scope of action, and their persistence of context become governance questions. That is where agentic AI and NHI thinking meet: a security agent is not merely a model, but a controlled runtime actor whose permissions and auditability shape its utility. Practitioners should treat evaluation design as a proxy for operational control design.

Coverage, precision, and maintenance cost form a single control triangle. The article makes clear that high recall is not enough if findings are noisy, and high precision is not enough if coverage cannot keep up with code change. That is a named governance gap we can call detection-response latency: the delay between a vulnerability appearing, being found, and being acted on. Shortening that gap is the real objective for security teams, whether they are using AI agents or traditional review pipelines.

Distributed attack paths are the right mental model for modern software security. dfbench’s multi-repository design mirrors how real weaknesses emerge across services, shared components, and implicit trust relationships. This matters for identity and access governance because authorization assumptions often live outside the code that ultimately fails. Practitioners should stop evaluating systems as if each repository were an island.

Benchmarks should reward lifecycle awareness, not static cleverness. The differential analysis task is the most interesting part of the article because it tests whether a system can remember, update, and retire findings as the environment changes. That is the same challenge faced by NHI and agentic AI governance: controls need to follow runtime state, not just initial approval. Practitioners should ask whether their tooling can track change without resetting trust from scratch.

The next evaluation frontier will be defensive realism. The article points toward a market where AI security tooling is judged on whether it can operate like a durable control, not a flashy assistant. That implies better testing of context retention, multi-system reasoning, and false-positive discipline. Practitioners should favour evaluation methods that look like production workflows, because that is where failure becomes expensive.

What this signals

The practical signal for security programmes is that AI-assisted review will be judged less like a model demo and more like a control surface. That shifts procurement and internal evaluation toward context retention, false-positive discipline, and change-aware rescanning. For identity-heavy environments, the more relevant question is whether the system can operate safely as a governed non-human workload rather than a one-off assistant.

Detection-response latency: the shorter the gap between issue discovery and decision quality, the more value the agent creates. That is why lifecycle-aware review matters, especially where code, permissions, and deployment paths evolve faster than human review cycles. Security teams should benchmark tools against operational change, not static repositories.

The article also reinforces a broader market trend: security automation is becoming accountable for maintenance cost as much as raw output. That has implications for SOC, AppSec, and identity governance teams that are considering AI agents as part of their operating model. If the tool cannot keep its findings aligned with current state, it adds work instead of reducing it.

For practitioners

• Define security-agent acceptance criteria around lifecycle performance Require evidence for detect, validate, and rescan performance before placing an agent into review or triage workflows. Treat recall, precision, and update fidelity as separate gates rather than a single score.
• Test multi-repository reasoning explicitly Use scopes that cross service boundaries, shared libraries, and permission layers so the agent must reconstruct the full attack path. Single-repository tests are useful, but they understate real audit complexity.
• Measure cost per useful finding Track analyst time, token cost, and false-positive burden together. A system that surfaces more findings but consumes disproportionate review effort does not improve security coverage.
• Preserve context across rescans Carry forward prior findings, component relationships, and threat-model context when code changes. This is essential for differential analysis and prevents every scan from becoming a full restart.

Key takeaways

• Dfbench argues that defensive AI must be measured on sustained security coverage, not just its ability to rediscover known bugs.
• The benchmark’s focus on detect, validate, and differential analysis shows why precision and change-awareness matter as much as recall.
• For practitioners, the key question is whether an AI agent can preserve useful context as code evolves without multiplying analyst workload.

Key terms

• Open-ended Security Evaluation: A testing approach that measures whether a system can find and manage unknown issues across a real security scope, not just solve a fixed prompt. It focuses on coverage, verification, and continuity as the environment changes over time.
• Differential Analysis: A method for comparing security findings across two states of the same system, usually a base and a target commit. It checks whether an agent can recognise what was fixed, what remains open, and what new risk has appeared after change.
• Detection-Response Latency: The elapsed time between identifying a security issue and executing a bounded, auditable fix. In data security programmes, long latency means exposure persists after discovery, which undermines the value of detection and weakens compliance evidence.
• Precision of Findings: The degree to which reported issues are real, actionable, and correctly scoped. In defensive AI, precision matters because a high volume of false positives can overwhelm analysts and erode trust in the system’s output.

What's in the full report

depthfirst's full research covers the operational detail this post intentionally leaves for the source:

• The benchmark construction method, including how the authors reconstruct vulnerable repository states and filter unusable examples.
• The scoring approach for detect, validate, and differential analysis, including how frontier models are compared across cost and recall.
• The repository-scale and language diversity data that underpins the benchmark and explains why some evaluations are harder than others.
• The appendix methodology for estimating latent vulnerability risk and measuring precision at scale.

👉 depthfirst's full post covers the benchmark methodology, scoring logic, and validation process in more detail.

Deepen your knowledge

NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course, the industry's only accredited NHI security programme. It is designed for teams that need a practical governance lens across human and machine identity.