Topic starter
18/08/2026 2:38 pm
TL;DR: Three models matched 61.1% accuracy on Cybench's defensive subset, according to Cotool, but their per-task costs diverged sharply from about $2.62 to $9.71. Security workflows now have to weigh cost, latency, and completion reliability alongside benchmark score when selecting a model.
Editorial analysis by NHI Mgmt Group, based on content published by Cotool: “Cybench (Defensive Subset)”.
By the numbers:
- Claude Opus 4.5 cost about $9.71 per task for the same 61.1% accuracy.
Key questions
Q: Why does model cost matter when accuracy is similar?
A: Because similar accuracy can hide very different operating costs.
Q: What breaks when a model is accurate but unreliable in security workflows?
A: Analyst trust and workflow predictability break down.
Q: How should security teams choose a model for defensive cybersecurity tasks?
A: Choose it by workload fit, not by benchmark accuracy alone.
Practitioner guidance
- Define model acceptance thresholds for defensive tasks Set minimum accuracy, maximum cost per task, and maximum latency for each defensive workload type before any model is approved for analyst use.
- Separate exploratory from operational use cases Allow lower-confidence or lower-cost models only for exploration, while reserving production analysis for models that meet reliability and turnaround targets.
- Benchmark by task family, not one blended score Test forensics, reverse engineering, miscellaneous analysis, and hardware-style challenges separately so a strong result in one category does not hide weakness in another.
Bottom line: The benchmark shows that several models can reach the same defensive accuracy while still creating very different operational trade-offs.
Glossary terms: Benchmark Reliability, Task Fit, Cost Per Task, Latency Budget
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
This topic was modified 4 hours ago by NHI Mgmt Group
18/08/2026 3:22 pm
Defensive model evaluation is becoming an operating-control problem, not a leaderboard problem. The Cybench defensive subset shows that several models can land at the same accuracy while still diverging sharply in cost and runtime. That means the question for practitioners is no longer which model wins a single benchmark, but which model fits a defensible operating envelope. The practitioner conclusion is to govern model choice as a service decision, not a score chase.
A question worth separating out:
Q: Should teams prioritise latency or accuracy for defensive AI use cases?
A: It depends on the workflow. For interactive triage and live analyst support, latency can matter as much as accuracy because slow responses reduce usefulness. For offline investigations, higher accuracy may justify longer runtime. Teams should match the latency budget to the operational purpose instead of assuming one measure should always dominate.
👉 Read our full editorial: Cybench defensive results show accuracy and cost gaps in model security tasks