Topic starter
18/08/2026 2:42 pm
TL;DR: GPT-5.1 and Claude Opus 4.5 tied at 65% accuracy on BOTSv3 SecOps tasks, according to Cotool’s benchmark update, while Opus 4.5 finished in 122 seconds on average and GPT-5.1 delivered top-tier accuracy at about $1.67 per task. The result is a clearer enterprise signal: model choice for security operations should be driven by task efficiency, not raw latency alone.
Editorial analysis by NHI Mgmt Group, based on content published by Cotool: “Evaluating AI Agents in Security Operations (December 2025)”.
By the numbers:
- Claude Opus 4.5 completed tasks in 122 seconds on average.
- GPT-5.1 averaged 14.5 tool calls per task.
- GPT-5.1 achieves top-tier accuracy at roughly $1.67 per task.
Key questions
Q: How should security teams choose AI models for SecOps workflows?
A: Choose models by task, not by brand.
Q: Why can a slower model be better for security operations?
A: A slower model can still be better if it reaches the correct answer with fewer failed turns, less analyst intervention, and lower total investigation overhead.
Q: What are the signs that an AI model is inefficient in SecOps?
A: Look for high tool-call counts, repeated dead-end searches, frequent failure to complete tasks, and long tail latency on complex cases.
Practitioner guidance
- Define workload tiers for SecOps models Assign separate models to alert triage, enrichment, and deep investigation so that expensive reasoning is reserved for cases that justify it.
- Benchmark end-to-end investigation flows Test models against full SecOps tasks, including tool use, multi-step reasoning, and case completion, rather than relying on single-turn benchmarks.
- Set cost and completion thresholds Establish acceptable limits for cost per task, task completion rate, and average investigation time before moving a model into production workflows.
Bottom line: The benchmark shows that SecOps model choice should be governed by end-to-end investigation efficiency, not by latency alone.
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
18/08/2026 3:26 pm
Reasoning efficiency is becoming a governance variable, not just a model-quality metric. In SecOps, the key question is no longer whether a model can answer correctly in isolation. It is whether it can do so with enough efficiency, stability, and observability to fit an operational control environment. Practitioners should treat investigation throughput as part of the control design, not an implementation detail.
A few things that frame the scale:
- 53% of security leaders expect AI to run major portions of their infrastructure autonomously within the next three years, according to the 2026 Infrastructure Identity Survey.
- 19% of organisations give AI systems dramatically more access than human employees, nearly one in five granting unrestricted privilege, according to the 2026 Infrastructure Identity Survey.
A question worth separating out:
Q: How should teams decide when to use cheaper models versus frontier models?
A: Use cheaper models for low-risk triage, enrichment, and summarisation when the cost of a miss is limited. Reserve frontier models for complex investigations, high-confidence conclusions, or response workflows where failed reasoning would slow containment. The decision should be based on task criticality, not on model brand or benchmark rank alone.
👉 Read our full editorial: AI model selection for SecOps now hinges on reasoning efficiency