Topic starter
18/08/2026 2:45 pm
TL;DR: Frontier models still vary widely on blue team CTF work, with GPT-5.2 reaching about 69% accuracy, GPT-5.1 and Claude Opus 4.5 at 65%, and open-weight models trailing far behind, according to Cotool. The result is a reminder that SOC-grade investigation still depends on reliable tool use, context handling, and workflow discipline, not raw model capability alone.
Editorial analysis by NHI Mgmt Group, based on content published by Cotool: “BOTSv3 Blue Team CTF”.
By the numbers:
- The benchmark covered 51 scenarios.
Key questions
Q: What breaks when AI agents are used for SOC investigation without scoped tool access?
A: The investigation boundary breaks first.
Q: Why do long-horizon blue team tasks expose weaknesses that normal benchmarks miss?
A: Because they require the system to preserve context, sequence actions, and recover from mistakes across multiple steps.
A: Organisations should track operational metrics such as mean time to respond, alert reduction, and automation success rate.
Practitioner guidance
- Define SOC task boundaries for AI agents Limit AI to discrete incident-response and threat-hunting tasks where the evidence source, query surface, and expected output are explicit.
- Scope tool permissions to investigative need Grant only the Splunk actions required for the workflow, then review whether search, dataset enumeration, and source-type inspection are too broad for the use case.
- Measure completion quality, not just accuracy Track unrecoverable errors, retry loops, and abandoned tasks alongside accuracy so the evaluation reflects real SOC reliability.
Bottom line: Blue team CTF results show that agentic SOC performance still depends on disciplined tool use and state management, not just model scale.
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
18/08/2026 3:28 pm
Agentic SOC evaluation is now an identity and governance problem, not only a model-quality problem. The benchmark shows that security agents succeed or fail based on tool discipline, recovery, and bounded execution as much as on reasoning quality. That makes the agent's operating model the real control surface. Practitioners should treat blue-team AI as governed runtime identity, not just as an assistant.
A question worth separating out:
Q: Should organisations use AI for triage before they trust it for full incident investigation?
A: Yes, usually. Triage and alert enrichment tolerate more uncertainty than evidence synthesis and final attribution. Teams should first prove the agent can narrow cases, pull the right logs, and hand off cleanly to humans before expanding into higher-stakes investigative work that can affect containment decisions.
👉 Read our full editorial: Blue team CTF benchmarking shows agentic SOC tasks remain hard