Cybench defensive results show accuracy and cost gaps in model security tasks

At a glance

What this is: Cotool’s Cybench defensive benchmark shows a tight accuracy race at the top, but a wide spread in cost and latency across models solving the same defensive security tasks.

Why it matters: Security teams evaluating AI for forensic, reversing, and miscellaneous defensive work need to weigh cost, speed, and reliability alongside raw accuracy, because the benchmark shows those trade-offs can materially change operating decisions.

By the numbers:

• Three models tied for highest accuracy at 61.1% in the defensive subset.
• GPT-5.2 delivered the best value at about $2.62 per task.
• Claude Opus 4.5 cost about $9.71 per task for the same 61.1% accuracy.

Context

Cybench is a benchmark for evaluating how language models handle defensive cybersecurity tasks such as forensics, reverse engineering, miscellaneous analysis, and hardware-oriented challenges. The key governance issue is not whether a model can solve a single task, but whether it can do so reliably enough, quickly enough, and cheaply enough to fit real operational workflows.

For security teams, this is an AI governance and operating-model question as much as a model-evaluation question. If a model is used in incident analysis, reverse engineering support, or defensive triage, the relevant control is not only accuracy but also latency, completion reliability, and cost predictability across repeated use cases.

Key questions

Q: How should security teams choose a model for defensive cybersecurity tasks?

A: Choose it by workload fit, not by benchmark accuracy alone. Security teams should compare accuracy, cost per task, latency, and completion reliability against the specific job they want the model to do. A model that is accurate but expensive or slow may be the wrong choice for recurring analysis, while a cheaper model may be suitable for exploratory work only.

Q: Why does model cost matter when accuracy is similar?

A: Because similar accuracy can hide very different operating costs. If two models solve the same defensive task at roughly the same quality, the cheaper option reduces budget pressure and can be used more broadly. Cost matters most when the model will be invoked repeatedly in analyst workflows, where small per-task differences compound quickly.

Q: What breaks when a model is accurate but unreliable in security workflows?

A: Analyst trust and workflow predictability break down. A model that sometimes fails to complete a task forces rework, interrupts triage, and makes it hard to plan around turnaround times. In security operations, reliability is part of the control surface because repeated failure can be more disruptive than lower average quality.

Q: Should teams prioritise latency or accuracy for defensive AI use cases?

A: It depends on the workflow. For interactive triage and live analyst support, latency can matter as much as accuracy because slow responses reduce usefulness. For offline investigations, higher accuracy may justify longer runtime. Teams should match the latency budget to the operational purpose instead of assuming one measure should always dominate.

Technical breakdown

Why accuracy alone misses the operational picture

Accuracy on a benchmark measures how often a model reaches the correct outcome, but it does not describe how much compute, time, or supervision the model needed to get there. In defensive security tasks, that matters because a model that is accurate but slow may still be unsuitable for time-sensitive workflows, while a cheaper model may be fine for exploratory analysis but not for repeatable production support. The real issue is workload fit. A benchmark that includes forensics, reversing, and miscellaneous tasks is testing end-to-end reasoning, not just classification skill. That makes cost and latency part of the security control decision, not an afterthought.

Practical implication: Treat benchmark accuracy as one input to model selection, not the decision criterion.

Why cost and latency change model governance

Cost per task is a practical control signal because security teams rarely run a model once. They run it across many investigations, escalations, and analysis loops, so small per-task differences compound quickly. Latency matters for the same reason: a model that finishes in minutes may support interactive analysis, while one that takes much longer may only fit offline work. In governance terms, the team is balancing throughput, analyst experience, and budget against outcome quality. That balance is especially important when a model is placed inside an agentic sandbox or a security workflow where human reviewers depend on predictable turnaround times.

Practical implication: Set explicit thresholds for spend and turnaround before allowing a model into recurring security workflows.

Why reliability is a separate control from accuracy

A model can score well on accuracy and still fail some tasks outright. That is why task completion rate needs to sit beside accuracy in any evaluation of security models. In defensive work, intermittent failure creates operational drag because analysts cannot know in advance which cases will succeed, which raises rework and trust issues. For AI governance, this is a reliability-and-resilience problem: a tool that completes every time at moderate quality may be more usable than one that is occasionally better but unpredictably fails. The benchmark data makes that distinction visible and operationally relevant.

Practical implication: Require completion consistency in addition to quality scores before using a model in security operations.

NHI Mgmt Group analysis

Defensive model evaluation is becoming an operating-control problem, not a leaderboard problem. The Cybench defensive subset shows that several models can land at the same accuracy while still diverging sharply in cost and runtime. That means the question for practitioners is no longer which model wins a single benchmark, but which model fits a defensible operating envelope. The practitioner conclusion is to govern model choice as a service decision, not a score chase.

Cost-per-task is now a first-class governance metric for security AI. When the same level of output can cost materially different amounts, the hidden risk is budget drift and uncontrolled scaling across repeated analyst workflows. For teams building AI support into forensics or reversing, cost controls are part of safe adoption. The practitioner conclusion is to set cost ceilings alongside quality thresholds.

Reliability should be treated as a control requirement, not a convenience metric. The benchmark’s completion rates show that performance is not only about the final answer but also whether the model finishes consistently enough to support operational use. That matters most when the model is embedded in an analyst loop or automated workflow. The practitioner conclusion is to separate promising demos from production-worthy capabilities.

Security workload fit: the decisive issue is matching model behaviour to the actual task mix, not assuming one model profile suits every defensive use case. Cybench’s mix of forensics, reversing, and hardware tasks makes that mismatch visible because different workloads stress different forms of reasoning and persistence. The practitioner conclusion is to classify workloads before selecting the model.

What this signals

Security workload fit: the model question is shifting from raw benchmark score to whether an AI system can sustain the cost, speed, and reliability profile a SOC or investigation team actually needs. That is a governance decision, not just a technical one.

For AI-assisted defensive work, the practical boundary is whether the model can be evaluated and approved per task family. A single blended score obscures the different demands of forensics, reversing, and hardware-style analysis, so programme owners should separate those pathways before comparing vendors.

For practitioners

• Define model acceptance thresholds for defensive tasks Set minimum accuracy, maximum cost per task, and maximum latency for each defensive workload type before any model is approved for analyst use.
• Separate exploratory from operational use cases Allow lower-confidence or lower-cost models only for exploration, while reserving production analysis for models that meet reliability and turnaround targets.
• Benchmark by task family, not one blended score Test forensics, reverse engineering, miscellaneous analysis, and hardware-style challenges separately so a strong result in one category does not hide weakness in another.
• Track completion rate alongside accuracy Use task completion rate as a formal criterion in vendor or model reviews, because partial failures create analyst rework and undermine trust in repeated use.

Key takeaways

• The benchmark shows that several models can reach the same defensive accuracy while still creating very different operational trade-offs.
• Cost, latency, and completion reliability are now part of model governance for security teams using AI in analysis workflows.
• Teams should evaluate models by task family and operational fit, not by a single headline score.

Key terms

• Benchmark Reliability: Benchmark reliability is the degree to which a model completes tasks consistently enough for repeated operational use. It is distinct from accuracy because a model can sometimes be correct while still failing too often to support dependable security workflows.
• Task Fit: Task fit is the match between a model’s behaviour and the specific work it is expected to do, such as forensics or reverse engineering. A model may score well overall but still be a poor fit if its cost, latency, or failure pattern does not suit the workflow.
• Cost Per Task: Cost per task is the total expense required to complete one discrete unit of work. It is calculated by combining labor, infrastructure, software, and overhead costs, then dividing by the number of tasks completed. In identity and security operations, it helps measure efficiency, automation impact, and operational scalability.
• Latency Budget: A latency budget is the maximum acceptable delay a control or service can add before the user experience or business process breaks. For AI safety, it defines whether enforcement remains part of the live interaction or becomes a deferred review step.

Deepen your knowledge

NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course. It is designed for practitioners who need a structured way to govern identities, access, and AI-linked operational risk.