Notifications
Clear all
Topic starter
18/08/2026 2:38 pm
TL;DR: Cybench’s defensive subset shows three models tied at 61.1% accuracy, but GPT-5.2 delivered that result at roughly $2.62 per task versus Claude Opus 4.5 at $9.71, according to Cotool’s benchmark analysis. The wider signal is that defensive cyber workloads still reward careful reasoning, but cost, latency, and reliability vary enough to change operational choices.
NHIMG editorial — based on content published by Cotool: Cybench Defensive Subset Jan 2026 benchmark analysis
By the numbers:
- GPT-5.2 cost about $2.62 per task, while Claude Opus 4.5 cost $9.71 for the same 61.1% accuracy.
- Seven models achieved 100% task completion in the benchmark.
Questions worth separating out
Q: How should security teams evaluate AI models for defensive cyber work?
A: Use representative tasks, not generic prompts.
Q: Why does model accuracy alone not tell you whether an AI tool is usable in security operations?
A: Because security operations depend on more than correct answers.
Q: What do organisations get wrong when they adopt AI for security?
A: Organisations often assume that AI capability automatically means security value.
Practitioner guidance
- Benchmark AI tools on real defensive workflows Test models on the same file analysis, reverse engineering, and evidence extraction patterns your team actually uses, then compare accuracy, cost, and latency together, not in isolation.
- Treat model access as delegated identity Assign each AI workflow a scoped runtime identity, separate credentials, and explicit tool permissions so the system cannot move beyond the intended task boundary.
- Log every tool invocation and output path Capture command use, file access, and decision traces so investigators can reconstruct how an agent reached a result and whether it touched data it should not have accessed.
What's in the full report
Cotool's full benchmark write-up covers the operational detail this post intentionally leaves for the source:
- Per-model result tables with cost, latency, and completion metrics for each defensive task
- Methodology notes on sandbox setup, scoring, and task selection across the benchmark subset
- Challenge category breakdowns that show where models performed well and where they struggled
- Raw benchmark context that implementation teams can use to compare model behaviour across security workflows
👉 Read Cotool's Cybench defensive benchmark analysis for model accuracy, cost, and latency details →
Cybench defensive benchmark: what the accuracy and cost spread means?
Explore further
18/08/2026 3:22 pm
Benchmark performance is now an identity and governance issue, not just an AI evaluation issue. When a model can inspect files, execute tools, and complete multi-step tasks, it is acting like a delegated runtime identity. That means access scope, tool permissions, and auditability matter as much as model accuracy. Security teams should treat defensive agents as governed non-human identities rather than as interchangeable automation.
A question worth separating out:
Q: How should teams govern AI agents that can reach APIs, events, and memory?
A: Teams should govern those agents as runtime identities, not as isolated integrations. That means enforcing policy at execution time, logging every tool and data access, and binding actions back to a clear initiating workflow or identity. If the control plane cannot show who acted, what they reached, and why, the programme does not have usable governance.
👉 Read our full editorial: Cybench defensive results show accuracy and cost gaps in model security tasks
