Notifications
Clear all
Topic starter
18/08/2026 2:42 pm
TL;DR: GPT-5.1 and Claude Opus 4.5 tied at 65% accuracy on BOTSv3 SecOps tasks, according to Cotool’s benchmark update, while Opus 4.5 finished in 122 seconds on average and GPT-5.1 delivered top-tier accuracy at about $1.67 per task. The result is a clearer enterprise signal: model choice for security operations should be driven by task efficiency, not raw latency alone.
NHIMG editorial — based on content published by Cotool: LLM benchmarking update for real-world SecOps tasks on BOTSv3
By the numbers:
- GPT-5.1 achieved top-tier accuracy at roughly $1.67 per task in Cotool’s cost comparison.
Questions worth separating out
Q: How should security teams evaluate AI models for defensive cyber work?
A: Use representative tasks, not generic prompts.
Q: Why does reasoning efficiency matter in security operations?
A: Reasoning efficiency matters because security investigations often require multiple evidence steps, not a single answer.
Q: What breaks when AI models are chosen only on raw speed?
A: Teams can end up with fast models that produce incomplete investigations, weak evidence chaining, or poor convergence on complex log data.
Practitioner guidance
- Benchmark models against your real SecOps task mix Replicate the kinds of investigations your analysts actually perform, including cloud incidents, account abuse, and long-context log review.
- Measure completion rate alongside accuracy Track whether a model finishes the task, how many tool calls it needs, and how often it converges without analyst rescue.
- Segment models by investigation tier Use faster models for enrichment and first-pass triage, then reserve higher-reasoning models for deep investigations that need multi-step evidence chaining.
What's in the full report
Cotool's full research covers the operational detail this post intentionally leaves for the source:
- Per-model benchmark tables for GPT-5.1, Claude Opus 4.5, Gemini 3 Pro, Sonnet 4.5, and Haiku 4.5 across accuracy, cost, completion, and time.
- Task-level methodology for the BOTSv3 blue-team CTF environment, including how the agent harness was configured for investigation workflows.
- Extended notes on tool-call efficiency and token consumption that help teams decide where a model fits in the SecOps pipeline.
- The authors' planned failure analysis for the longer-context cases where some models needed more convergence than expected.
👉 Read Cotool's benchmark update on GPT-5.1, Claude Opus 4.5, and Gemini 3 Pro in SecOps →
AI models for SecOps: what changes when reasoning beats speed?
Explore further
18/08/2026 3:26 pm
Model benchmarking for security operations is becoming a governance problem, not just an engineering exercise. Once an AI system is trusted to inspect logs, correlate evidence, and recommend actions, the question shifts from model capability to operational accountability. Security teams need evidence that a model can sustain reasoning under real SecOps conditions, especially where identity signals, cloud telemetry, and access anomalies intersect. The benchmark format matters because it creates a repeatable way to judge whether an AI assistant is fit for controlled use.
A question worth separating out:
Q: How should organisations assign different AI models to different SecOps tasks?
A: Use lower-latency models for enrichment and routine triage, and reserve stronger reasoning models for forensic work or multi-step investigations. That segmentation matches model capability to task difficulty and keeps cost aligned with business need. A single-model strategy often wastes both budget and analyst time.
👉 Read our full editorial: AI model selection for SecOps now hinges on reasoning efficiency
