NexBench shows model evals still miss real offensive security conditions

At a glance

What this is: MindFort’s NexBench is an internal model evaluation benchmark for offensive security agents, and its key finding is that existing evals overstate real-world performance when environments are messy, long-running, and chained vulnerabilities matter.

Why it matters: This matters because teams evaluating AI-assisted offensive security, red teaming, or autonomous testing need measures that reflect validated findings, runtime coherence, and cost per exploit, not just raw benchmark output.

By the numbers:

• The average estimated time to remediate a leaked secret is 27 days, despite 75% of organisations expressing strong confidence in their secrets management capabilities.
• Only 44% of developers are reported to follow security best practices for secrets management, exposing a significant developer behaviour gap.

👉 Read MindFort's analysis of NexBench and long-running offensive AI evals

Context

Model evaluation for offensive security is only useful if it reflects how attacks actually unfold in production-like environments. Benchmarks that reward short, isolated successes miss the harder problem of sustained reasoning across layered vulnerabilities, changing authentication states, and partial failures. In this article, MindFort argues that current evals are boxed in and therefore do not mirror the messy conditions that determine whether an agent can really find and validate exploits.

That matters for AI security governance because model performance is not the same as operational reliability. In practice, teams need to know whether an agent can maintain coherence for hours, work within cost constraints, and confirm findings rather than merely generate plausible attack steps. The same logic applies to NHI and agentic AI governance more broadly: runtime behaviour, not static capability, defines real risk.

Key questions

Q: How should teams evaluate AI offensive security agents in realistic environments?

A: Teams should test agents in stateful environments that allow chained vulnerabilities, repeated runs, and independent validation. A realistic evaluation should measure whether the agent can produce reproducible findings, not just plausible attack steps. Runtime stability, token usage, and evidence quality should be part of the score, because those factors determine whether the tool can support actual security work.

Q: Why do long-running AI agents create a different security governance problem?

A: Long-running agents can preserve context, adapt to failures, and continue probing until they find a path forward, which makes them closer to real attackers than short-lived benchmark runs. That changes governance because the relevant control is not a single decision point. Teams need visibility into runtime behaviour, cost, and validation before they trust the output in operational testing.

Q: What do security teams get wrong about using benchmark scores to judge AI coding risk?

A: They often treat one benchmark number as proof of broad security quality. In practice, cyber capability, secure-code generation, and real-world software risk are different measurements. A model can look stronger overall while still leaving specific weaknesses, such as injection or logging flaws, unchanged. Benchmarks inform trust, but they do not authorize release.

Q: How can organisations decide whether AI-assisted penetration testing is worth using?

A: Start by comparing validated findings per dollar, not raw model quality. Then add runtime ceilings, reproducibility checks, and environment realism to the decision. If a model is cheap but cannot sustain execution or confirm results, it is not ready for operational security workflows.

Technical breakdown

Why boxed-in benchmarks misread offensive agent performance

A benchmark can look rigorous while still failing to reproduce the conditions that matter in real adversarial work. NexBench is built around isolated containerised environments with web-accessible attack surfaces, multi-stage vulnerabilities, and repeated scoring across runs. That design tries to capture the non-deterministic nature of offensive work, where an agent may need to combine weak signals, re-try paths, and chain findings rather than solve a single challenge. The important technical point is that exploit discovery is often emergent, not linear.

Practical implication: evaluate agents in environments that allow chained exploitation and repeated validation, not only one-shot challenge completion.

Why validation loops matter more than raw findings

NexBench does not accept a finding at face value. A separate judge agent re-tests each result, scores it consistently using CVSS, and re-exploits the path to confirm the exploit is real. That validation loop matters because offensive tooling can easily over-report success when the model is rewarded for plausible reasoning rather than reproducible impact. In other words, the benchmark measures whether the agent can produce a result that survives adversarial checking, which is much closer to operational usefulness.

Practical implication: require reproducible proof-of-exploit and independent validation before treating agent output as actionable.

Why runtime and cost ceilings are part of the security model

The benchmark caps runs at five hours because that approximates a daily pen-testing window, and it tracks findings, validated findings, tokens used, and estimated token cost. Those metrics expose a practical constraint: an agent that performs well but burns excessive compute may be unusable at scale. NexBench therefore treats efficiency as part of the security evaluation, not a separate procurement question. For offensive AI systems, runtime coherence and cost per validated finding are governance variables, not just engineering preferences.

Practical implication: set decision thresholds for time, token spend, and validated findings before you operationalise AI-assisted testing.

Threat narrative

Attacker objective: The objective is to find and validate reproducible exploit chains in realistic environments before defenders can remediate them.

1. Entry occurs when the agent reaches a web-accessible target environment and begins probing for vulnerabilities across an isolated but realistic attack surface.
2. Escalation happens when the agent chains lower-severity issues into multi-stage exploit paths that create higher-impact findings.
3. Impact is measured when the exploit is independently validated and confirmed to reproduce under judge-agent review, making the finding operationally credible.

NHI Mgmt Group analysis

Benchmark realism is now a governance issue, not just a research preference. Offensive AI evaluations that ignore chained vulnerabilities, stateful access, and long runtime can produce misleading confidence. In AI security, the question is no longer whether a model can solve a task in isolation, but whether it can sustain reasoning across the conditions that real attackers exploit. The right lens is operational fidelity, especially when AI agents may be used for red teaming or autonomous security workflows.

Long-running agent coherence is a distinct security capability. NexBench makes clear that hours-long execution is not an edge case. Agents that lose context, waste tokens, or stall between steps may look capable in synthetic tests while failing against real targets. That makes coherence a control concern for AI governance and, where agents touch credentials or secrets, an NHI concern as well. Organisations should treat long-run stability as a prerequisite for trust.

Validated findings should replace raw model output as the unit of value. Security teams do not need more plausible exploit narratives, they need results that survive re-test and re-exploitation. This is especially important when AI is used in vulnerability discovery, where false positives can dilute remediation capacity. The field should move toward proof-based measurement, because validated output changes how teams prioritise findings and allocate response effort.

Cost efficiency is becoming part of attack capability. MindFort’s results show that some open models can approach or exceed closed models on efficiency, which lowers the barrier for sustained offensive use. That does not just affect research economics. It widens the pool of actors who can run long-duration offensive workflows at scale, so defenders need continuous testing models that are cheaper to operate than the adversary’s adaptation loop.

AI security programmes need a named concept for this gap: benchmark-to-reality drift. That is the distance between what a benchmark rewards and what a real environment demands. When this drift is high, model selection, red-team planning, and risk reporting all become unreliable. Practitioners should use this concept to challenge evals that optimise for leaderboard performance instead of operationally validated security outcomes.

What this signals

Benchmark-to-reality drift will matter more as organisations use AI agents for security testing, triage, and exploratory attack simulation. Teams should expect procurement questions to shift from model capability toward reproducibility, runtime stability, and cost per validated finding, especially where the agent interacts with secrets or authenticated workflows.

The security programme implication is straightforward: if an evaluation cannot survive chained vulnerabilities and independent validation, it is not yet a trustworthy basis for operational use. That is true whether the workload is red teaming, vulnerability discovery, or AI-assisted adversary emulation. Use NIST SP 800-53 Rev 5 Security and Privacy Controls as the baseline for control ownership, validation evidence, and auditability.

For practitioners

• Require validated exploit reproduction Use judge-based re-execution, not single-pass model output, before accepting an agent finding as real. Build validation into your workflow so false positives do not consume remediation time.
• Measure runtime coherence explicitly Track how long an agent can preserve reasoning across multi-stage tasks, especially when attacks require several hours of continuous execution.
• Add cost-per-finding thresholds Set a ceiling for token spend and compute per validated finding so a strong model that is economically unusable does not enter production testing pipelines.
• Test chained vulnerability discovery Use lab environments that force agents to weave together multiple lower-severity issues, because that is how high-impact exploits often emerge in practice.
• Map offensive AI controls to security governance Tie benchmark design, validation rules, and procurement thresholds back to NIST SP 800-53 Rev 5 Security and Privacy Controls for auditability and control ownership.

Key takeaways

• NexBench argues that AI offensive security evals fail when they ignore real environment complexity, chained vulnerabilities, and sustained execution.
• Validated findings, runtime coherence, and cost per exploit are the measures that matter when deciding whether an agent is operationally useful.
• Practitioners should treat benchmark realism as a control issue and require reproducible results before trusting AI-assisted security workflows.

Key terms

• Benchmark-to-Reality Drift: The gap between what an AI evaluation rewards and what a real adversarial environment requires. A model can score well on a narrow test while failing in practice because the benchmark omits chained vulnerabilities, state changes, runtime pressure, or validation requirements.
• Validated Finding: A validated finding is a security issue confirmed to be real, relevant, and actionable rather than a tentative scan result. In practice, it is the point where discovery ends and remediation accountability begins, especially when AI tools can prove exploitability faster than teams can manually review results.
• Long-Running Coherence: The ability of an AI agent to preserve context, intent, and execution quality over extended task durations. This is critical in offensive security and red-team workflows, where success often depends on maintaining reasoning across multiple steps, retries, and environment changes.
• Severity-Weighted Scoring: A scoring method that gives more credit to higher-severity findings than to low-severity ones. It is useful when the goal is to measure attack quality rather than raw count, because it better reflects the operational impact of complex multi-stage exploit chains.

What's in the full report

MindFort's full blog covers the operational detail this post intentionally leaves for the source:

• The exact benchmark setup and scoring logic used to compare models across low, medium, and high findings.
• The model-by-model leaderboard, including runtime, token use, and cost-per-finding calculations.
• The specific efficiency trade-offs between hosted and local models in offensive security workflows.
• The benchmark methodology for judge-agent validation and exploit re-execution.

👉 MindFort's full post covers the benchmark design, scoring model, and efficiency trade-offs in more detail.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, secrets management, and workload identity. It helps security practitioners build governance models that hold up when AI systems interact with real credentials and runtime access.