Cybench defensive results show accuracy and cost gaps in model security tasks

At a glance

What this is: Cotool’s Cybench defensive subset benchmark compares 10 models across 18 security tasks and shows that accuracy, cost, and latency diverge sharply.

Why it matters: For IAM, PAM, and broader security teams, the result underscores that agentic workflows need governance around model selection, task scope, and failure tolerance, not just raw capability.

By the numbers:

• Three models tied for the highest accuracy at 61.1% in the defensive subset.
• GPT-5.2 cost about $2.62 per task, while Claude Opus 4.5 cost $9.71 for the same 61.1% accuracy.
• Seven models achieved 100% task completion in the benchmark.

👉 Read Cotool's Cybench defensive benchmark analysis for model accuracy, cost, and latency details

Context

Cybench is a benchmark for evaluating cybersecurity capabilities in language models, and this defensive subset focuses on forensics, reverse engineering, miscellaneous challenges, and hardware tasks. The primary question is not whether a model can answer a prompt, but whether it can sustain analysis inside a constrained environment and produce a correct result under hard-mode conditions.

For security programmes, the deeper issue is governance of AI-assisted work. These results matter because they show a gap between task completion and useful performance, which affects how teams should think about analyst augmentation, triage support, and agentic workflows in environments where identity, access, and execution privileges must remain tightly controlled.

Key questions

Q: How should security teams evaluate AI models for defensive cyber work?

A: Use representative tasks, not generic prompts. Score the model on accuracy, runtime cost, latency, and completion reliability against the actual workflows you expect it to support, then decide whether the result is good enough for triage, investigation, or analyst assistance. A benchmark only matters if it reflects your operational boundary.

Q: Why does model accuracy alone not tell you whether an AI tool is usable in security operations?

A: Because security operations depend on more than correct answers. Cost, speed, auditability, and permission scope all affect whether the tool can be trusted in production. A model that is accurate but slow or expensive may still be the wrong fit for high-volume work, while a faster model may be acceptable for bounded tasks.

Q: What do organisations get wrong when they adopt AI for security?

A: Organisations often assume that AI capability automatically means security value. In practice, the mistake is failing to define the boundary between decision support and delegated action. If the organisation cannot explain what the AI is allowed to do, it cannot govern the risk it introduces into identity and response workflows.

Q: How should teams govern AI agents that can reach APIs, events, and memory?

A: Teams should govern those agents as runtime identities, not as isolated integrations. That means enforcing policy at execution time, logging every tool and data access, and binding actions back to a clear initiating workflow or identity. If the control plane cannot show who acted, what they reached, and why, the programme does not have usable governance.

Technical breakdown

Why defensive cyber benchmarks stress reasoning, not recall

Defensive CTF-style tasks force a model to inspect files, infer structure, and choose tools without step-by-step guidance. That is different from ordinary question answering, because the model must chain actions across analysis, parsing, and validation while staying inside a sandbox. In practice, the benchmark is testing an agentic work loop, not just model knowledge. That distinction matters for security because a model that is accurate in chat can still fail when it must sustain stateful investigation under constraints.

Practical implication: evaluate AI tools on end-to-end task execution, not prompt response quality alone.

Why accuracy, cost, and latency diverge in defensive workflows

The benchmark shows a classic trade-off. Some models reach similar accuracy but at very different per-task costs, while the faster models are not always the most accurate. That happens because deeper reasoning often increases token use and wall-clock time, especially on tasks that require iteration or confirmation. For security operations, the important point is that automation economics are now part of control design. A tool that is cheap but unreliable can create hidden analyst overhead, while a slower tool may be acceptable for deeper investigations.

Practical implication: treat model selection as a control and cost decision, not a pure performance comparison.

What defensive benchmarks reveal about agentic security governance

A sandboxed model still needs scoped tools, bounded permissions, and clear execution rules. Otherwise, the same qualities that help it solve a challenge can create operational risk in production, especially if it is allowed to read, write, or execute beyond its intended boundary. For identity teams, this is an NHI governance problem as much as an AI problem, because the model behaves like a runtime identity with delegated access. The control question becomes who grants that access, how it is constrained, and how its actions are audited.

Practical implication: apply least privilege, logging, and task-scoped delegation before putting AI agents into security workflows.

NHI Mgmt Group analysis

Benchmark performance is now an identity and governance issue, not just an AI evaluation issue. When a model can inspect files, execute tools, and complete multi-step tasks, it is acting like a delegated runtime identity. That means access scope, tool permissions, and auditability matter as much as model accuracy. Security teams should treat defensive agents as governed non-human identities rather than as interchangeable automation.

The named concept here is agentic evaluation gap: the distance between benchmark success and operational trust. Cybench’s defensive subset shows that a model can complete tasks reliably while still varying widely in cost and latency. That gap is what creates procurement and governance friction, because the cheapest or fastest option is not always the one that fits the programme’s risk tolerance. Practitioners should evaluate workloads by control impact, not leaderboard position.

Task completion rates can mask important control weaknesses. Seven models completed every task, but completion alone does not prove safe behaviour, bounded execution, or predictable resource use. For identity-led programmes, this is the same mistake seen in over-trusting “works in test” automation. The governance burden is to define where delegated execution begins, where it ends, and what evidence is retained.

Defensive AI is converging with NHI security requirements. Once a system can browse, write files, and invoke tools, it inherits the same questions that already govern service accounts, API keys, and workload identities. The control model should therefore combine AI evaluation with NHI lifecycle management, least privilege, and short-lived access. Teams that separate these disciplines will miss the real risk surface.

Operational fit should replace abstract model ranking as the buying criterion. A model with 61.1% accuracy and low per-task cost may be the right fit for routine triage, while a slower model may be better for high-stakes forensic work. The practical conclusion is simple: match model capability to the control objective, then document the boundary conditions under which it is allowed to act.

What this signals

AI evaluation is moving toward operational realism, which means security teams will need to measure more than benchmark accuracy. The programme signal is clear: if an AI system can act, it must be governed like a privileged runtime identity, with scoped permissions, logging, and revocation controls.

Agentic evaluation gap: the difference between a model that completes a benchmark and a model that is safe to deploy. That gap will matter more as teams use AI for triage, reverse engineering, and defensive analysis, because governance must account for both task success and delegated authority.

For practitioners

• Benchmark AI tools on real defensive workflows Test models on the same file analysis, reverse engineering, and evidence extraction patterns your team actually uses, then compare accuracy, cost, and latency together, not in isolation.
• Treat model access as delegated identity Assign each AI workflow a scoped runtime identity, separate credentials, and explicit tool permissions so the system cannot move beyond the intended task boundary.
• Log every tool invocation and output path Capture command use, file access, and decision traces so investigators can reconstruct how an agent reached a result and whether it touched data it should not have accessed.
• Set cost and latency thresholds per use case Define acceptable spend and response-time limits for each security workflow, because the cheapest model is not always the most operationally efficient option.

Key takeaways

• Cybench’s defensive subset shows that model capability is uneven once tools, files, and multi-step reasoning are required.
• The same benchmark result can hide very different cost, latency, and governance profiles, which changes deployment decisions.
• Security teams should govern AI agents as scoped identities with auditable authority, not as generic automation.

Key terms

• Agentic Evaluation Gap: The gap between a model’s benchmark success and its readiness for real operational use. It captures differences in cost, latency, auditability, and permission scope that benchmarks may not expose, even when the model completes tasks correctly.
• Delegated Runtime Identity: A non-human identity assigned to an AI system or workflow so it can act within a defined boundary. The identity carries credentials, tool permissions, and logging obligations that determine what the system may access and how its actions are traced.
• Defensive Cyber Benchmark: A controlled evaluation that measures how well a model handles security tasks such as forensics, reversing, or incident analysis. It is useful only when it reflects realistic workflows, because isolated task completion does not guarantee safe or efficient production use.

What's in the full report

Cotool's full benchmark write-up covers the operational detail this post intentionally leaves for the source:

• Per-model result tables with cost, latency, and completion metrics for each defensive task
• Methodology notes on sandbox setup, scoring, and task selection across the benchmark subset
• Challenge category breakdowns that show where models performed well and where they struggled
• Raw benchmark context that implementation teams can use to compare model behaviour across security workflows

👉 Cotool's full benchmark write-up includes the per-model results and methodology behind the defensive subset.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps security practitioners build the control thinking needed for delegated systems, agentic workflows, and identity-aware automation.