Blue team CTF benchmarking shows agentic SOC tasks remain hard

At a glance

What this is: Cotool's BOTSv3 benchmark reproduction measures how well AI agents handle blue team CTF-style incident response, threat hunting, and log analysis across 51 scenarios and 2.7 million log events.

Why it matters: For IAM, SOC, and security architecture teams, the findings show that agentic workflows still need tight governance, because tool access, context quality, and answer reliability directly shape operational risk.

By the numbers:

• GPT-5.2 achieved the highest accuracy at 68.6% in Cotool's BOTSv3 benchmark reproduction.
• The benchmark spans 51 scenarios and more than 2.7 million log events across 13 months.
• GPT-5.1 delivered 64.7% accuracy at $1.67 per task, the strongest value-to-performance result among frontier models.
• Open-weight models in the benchmark ranged from 25% to 43% accuracy, well below the frontier group.

👉 Read Cotool's benchmark analysis of AI agent performance in blue team CTF scenarios

Context

Blue team CTF benchmarks matter because they approximate the investigative workflow that security operations teams use in practice: search logs, correlate events, and answer incident questions under time pressure. The core governance gap is not whether a model can produce a plausible answer, but whether it can do so reliably enough to support SOC decision-making without introducing false confidence.

In this article, agentic AI is the relevant security lens because the benchmark gives models tools and asks them to act across a long investigation chain. That makes the identity question practical as well: if an AI system can query security tooling, it needs scoped access, clear auditability, and controls that treat it as a governed non-human identity rather than a generic automation script.

Key questions

Q: How should security teams govern AI SOC agents that use SIEM and EDR tools?

A: They should treat AI SOC agents as controlled investigative systems, not generic automation. That means granting only the tool access needed for specific investigation tasks, logging every query and pivot, and requiring human review for high-consequence decisions. If the agent can change state, that privilege should be separately approved and tightly scoped.

Q: Why do benchmark scores not fully capture SOC readiness for AI agents?

A: Because incident work depends on more than answer accuracy. Teams also need completion reliability, bounded latency, cost control, and explainable tool use. A model that scores well but fails mid-task, loops excessively, or queries the wrong data sources can still create operational risk in a live security workflow.

Q: What breaks when AI agents are given broad standing access?

A: Broad standing access breaks governance because the agent can move from one task to another without a fresh authorization check. That creates a control gap between intended scope and actual runtime behaviour. The result is weak accountability, limited containment, and audit trails that show activity without explaining why the activity was allowed.

Q: How do you know if AI-assisted investigations are actually working?

A: Look for defensible closure, not just shorter handling time. A working system should consistently correlate evidence from independent sources, reduce reopen rates, and produce conclusions that analysts trust enough to act on. If cases are closed quickly but frequently retriggered or manually corrected, the AI is speeding up uncertainty rather than resolving it.

Technical breakdown

Why blue team CTF benchmarks expose tool-use reliability gaps

Blue team CTF environments test more than pattern matching. They require retrieval across long log timelines, tool selection, and iterative reasoning under uncertainty. A model can answer a simple question correctly yet still fail when the task depends on source-type discovery, query refinement, and consistent state tracking across many steps. That is why benchmark scores are useful, but completion rate, latency, and cost matter too. In SOC settings, a weak model may still look convincing while missing the investigative trail that operators need to trust the result.

Practical implication: treat benchmark scores as only one part of AI operational readiness, not proof of SOC suitability.

How agent harnesses change the risk profile of AI in security operations

When a model is given Splunk search tools, list access, and source description functions, it stops being a text generator and becomes a governed runtime actor. That creates a non-human identity problem: the system now has credentials, permissions, and observable actions inside security tooling. The control question becomes whether those permissions are bounded to the task, logged, and revocable. Without that governance, a model can overreach, query too broadly, or be misled by incomplete context while still producing a confident answer.

Practical implication: assign agent-specific identities, audit their queries, and scope access to the minimum log sources needed for each task.

What long-context investigations reveal about operational resilience

Long-horizon investigative tasks stress more than model intelligence. They expose context degradation, recoverable versus unrecoverable errors, and the operational cost of repeated tool loops. In the benchmark, some models completed every task, while others showed notable failure rates, which is a warning for real SOC workflows where partial answers can be worse than clear failure. Resilience in this setting means predictable completion, bounded tool use, and fallback paths when an agent cannot sustain the investigation chain.

Practical implication: build human escalation thresholds and fallback workflows before allowing AI into incident triage.

NHI Mgmt Group analysis

Agentic SOC benchmarking is now an identity governance problem, not just an AI accuracy problem. Once a model can query Splunk, it is operating as a non-human identity with tool access, decision latency, and audit requirements. That means SOC teams must govern permissions, logging, and revocation with the same seriousness they apply to service accounts. The practical conclusion is simple: if an AI can act inside your detection stack, it needs identity controls, not just model evaluation.

Benchmark accuracy alone underestimates the real control gap. The article shows a spread between frontier and open-weight models, but the deeper lesson is that task completion, latency, and token use all shape whether an AI workflow is operationally safe. A model that is accurate but slow, expensive, or brittle can still fail under incident pressure. Practitioners should therefore measure reliability under workflow conditions, not only exact-match scoring.

Long-context blue team work creates governance debt when AI is treated like a disposable tool. Security teams often assume the investigative process is the only thing being automated, but the AI system itself accumulates access, state, and trust across the session. That is where agentic AI security intersects with IAM: session-scoped permissions, audit trails, and revocation must be designed upfront. The takeaway is that AI-assisted SOC work should be managed as a governed runtime, not an ad hoc helper.

Agentic AI security and NHI management are converging in the SOC. The same controls that matter for service accounts, API keys, and workload identities now apply to AI agents that search logs, enrich alerts, and draft incident answers. That convergence will make identity lifecycle, access scoping, and evidence retention core requirements for AI-enabled operations. Teams that separate AI governance from identity governance will miss the real operating model shift.

Detection-response latency: the time between an alert, a log query, and a trustworthy answer is becoming a measurable security control. In this benchmark, speed and reliability varied materially across models, which means the gap between detection and decision can no longer be treated as a soft metric. Practitioners should define acceptable response latency for AI-assisted investigations and tie it to escalation rules, otherwise AI will expand the SOC surface without improving outcomes.

What this signals

As AI agents move from demos into operational security workflows, the governance burden shifts from model selection to runtime control. The practical challenge is no longer whether a system can answer a question, but whether its access, evidence trail, and escalation path are strong enough to support incident decisions without widening the attack surface.

Detection-response latency: the useful metric for AI-assisted SOC work is the time from query to trusted answer, not the time to first output. Teams should expect to pair AI tooling with strict audit logging and a documented fallback path, because speed without evidentiary confidence is not operationally useful.

The next maturity step is to treat AI investigation systems as governed identities inside the SOC stack. That means aligning access scoping and revocation with identity lifecycle controls, then anchoring the programme in guidance such as NIST SP 800-53 Rev 5 Security and Privacy Controls.

For practitioners

• Define agent-specific access boundaries Issue distinct identities for AI investigation agents, limit them to the log sources and actions they actually need, and revoke access automatically when the task ends.
• Measure investigation quality beyond accuracy Track task completion, query efficiency, latency, and escalation frequency alongside benchmark accuracy so you can see whether the agent is dependable under SOC conditions.
• Require audit-ready tool activity Log every search, dataset listing, and source-type description request from AI agents so analysts can reconstruct what the system saw and how it reached its answer.
• Build human fallback paths for fragile cases Route complex or high-impact investigations to analysts whenever the agent hits repeated recoverable errors, ambiguous evidence, or a confidence threshold you cannot justify.

Key takeaways

• Blue team benchmarks show that AI agents still struggle with the reliability needed for operational SOC work, even when headline accuracy looks strong.
• Once an AI system can query security tooling, it behaves like a governed non-human identity and needs scoped access, logging, and revocation.
• The practical standard is no longer raw accuracy alone, but trustworthy completion, bounded latency, and human fallback when investigations become fragile.

Key terms

• Agentic SOC platform: A security operations platform that can investigate alerts and choose actions at runtime rather than relying entirely on pre-authored workflows. In practice, it combines reasoning, policy, and execution so teams can automate response while still enforcing approval, rollback, and audit requirements.
• Detection-Response Latency: The elapsed time between identifying a security issue and executing a bounded, auditable fix. In data security programmes, long latency means exposure persists after discovery, which undermines the value of detection and weakens compliance evidence.
• Non-Human Identity (NHI): A digital identity assigned to a non-human entity such as a software application, service account, API key, bot, machine, or AI agent that enables it to authenticate and interact with systems without direct human involvement. NHIs now outnumber human identities in most enterprises by 25 to 50 times.

What's in the full report

Cotool's full benchmark write-up covers the operational detail this post intentionally leaves for the source:

• Per-model accuracy, cost, latency, and completion data across all 15 tested models for side-by-side evaluation
• Methodology notes on the Splunk BOTSv3 environment, including tool access and task construction
• Scenario-level examples that show how the benchmark questions map to real incident response workflows
• Benchmark caveats that explain how human context was added or removed from the evaluation set

👉 Cotool's full benchmark write-up includes the per-model ranking table, methodology, and scenario details.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, and secrets management. It helps security practitioners turn identity principles into practical controls for modern automation and AI-enabled operations.