Blue team CTF benchmarking shows agentic SOC tasks remain hard

At a glance

What this is: Cotool benchmarked frontier and open-weight models on a reproduced Splunk BOTSv3 blue team CTF and found that even the best performers did not solve the majority of agentic SOC tasks.

Why it matters: This matters because SOC teams evaluating AI for incident response and threat hunting need to separate fast-looking demos from models that can sustain accurate, tool-driven investigation over real log volumes.

By the numbers:

• GPT-5.2 achieved the highest accuracy at 68.6% in the benchmark.
• The benchmark covered 51 scenarios.
• The reproduced dataset contained over 2.7M logs spanning 13 months.

Context

Blue team CTF benchmarking matters because it tests whether an agent can investigate, correlate, and explain security events under realistic SIEM conditions. In this case, the problem is not model fluency but whether the system can use tools reliably across long, noisy log sets and still produce a correct forensic answer.

For IAM, PAM, NHI, and agentic AI practitioners, the identity angle is governance of the agent itself: tool access, task scope, and failure containment. If a security agent cannot maintain context, recover from errors, or stay within bounded workflows, then the control point moves from model selection to operational oversight and access design.

Key questions

Q: What breaks when AI agents are used for SOC investigation without scoped tool access?

A: The investigation boundary breaks first. An agent that can search broadly, enumerate datasets, and iterate without tight permissions may produce plausible answers while drifting outside the intended case. The result is weak traceability, wider blast radius, and a higher chance that analysts cannot reconstruct how the conclusion was reached.

Q: Why do long-horizon blue team tasks expose weaknesses that normal benchmarks miss?

A: Because they require the system to preserve context, sequence actions, and recover from mistakes across multiple steps. A model that scores well on short prompts can still fail when it must chain queries, interpret intermediate evidence, and avoid compounding errors. That is why workflow realism matters more than isolated answer quality.

Q: How do organisations measure whether AI-powered security workflows are actually improving SOC performance?

A: Organisations should track operational metrics such as mean time to respond, alert reduction, and automation success rate. Those signals show whether workflows are reducing analyst load and improving containment speed. Good measurement also includes audit readiness and the quality of workflow outputs, because fast automation is only useful if it remains accurate and defensible.

Q: Should organisations use AI for triage before they trust it for full incident investigation?

A: Yes, usually. Triage and alert enrichment tolerate more uncertainty than evidence synthesis and final attribution. Teams should first prove the agent can narrow cases, pull the right logs, and hand off cleanly to humans before expanding into higher-stakes investigative work that can affect containment decisions.

Technical breakdown

Why long-horizon SOC tasks break down

Blue team CTFs combine search, correlation, and explanation across many steps, which makes them more demanding than single-turn classification. Agents must keep the investigation state intact while switching between queries, findings, and final answers. In practice, that creates failure modes around context loss, incorrect assumption carryover, and tool-call drift. A model can look capable on isolated prompts and still fail when the task requires disciplined sequencing across a large log corpus. Practical implication: evaluate AI against multi-step investigation workflows, not only against one-shot benchmark questions.

Practical implication: Test agentic SOC use cases across multi-step cases, not only isolated prompt responses.

How tool access shapes agentic SOC reliability

The benchmark gave agents access to Splunk search, dataset listing, and source-type description tools, which is a realistic but constrained operating model. That matters because the agent is not reasoning in a vacuum. It is translating intent into tool calls, parsing results, and deciding whether the evidence is sufficient to answer. When the tool chain is weak, the agent's final answer quality is capped no matter how strong the underlying model appears. Practical implication: govern AI SOC systems as tool-using identities with explicit permissions and observable action trails.

Practical implication: Treat AI SOC systems as identities with scoped tool permissions and auditable action trails.

What the benchmark says about reliability and cost

The results show that accuracy, speed, and cost do not move together in a simple way. Some models completed every task but still produced only moderate accuracy, while others were faster or cheaper yet less dependable under the same workload. That is a governance signal, not just a model-selection detail. For operational security work, unrecoverable errors and inconsistent completion can matter more than raw throughput because they interrupt triage and force human rework. Practical implication: set success criteria around completion quality, not just latency or price.

Practical implication: Set acceptance thresholds for completion quality, not just speed or cost.

NHI Mgmt Group analysis

Agentic SOC evaluation is now an identity and governance problem, not only a model-quality problem. The benchmark shows that security agents succeed or fail based on tool discipline, recovery, and bounded execution as much as on reasoning quality. That makes the agent's operating model the real control surface. Practitioners should treat blue-team AI as governed runtime identity, not just as an assistant.

Long-horizon investigation exposes a context-integrity gap. The important issue is not whether a model can answer a single log question, but whether it can preserve investigative state across dozens of steps without drifting. That is the same class of failure that affects agentic workflows in SOC operations and NHI governance. The practitioner conclusion is to validate state retention and error recovery before any production deployment.

Tool access defines the security boundary for AI SOC systems. The benchmark's constrained Splunk access is a reminder that permissions, not prompts, determine what an agent can safely do. Once a model can search, enumerate datasets, and iterate on evidence, access governance becomes central. Agentic investigation boundary: the limit is not model intelligence, but the scope and traceability of the tools it can invoke. Practitioners should design AI SOC access like any other privileged workflow.

Benchmarks should measure operational trust, not just answer accuracy. Accuracy alone misses whether a system can complete the task without unrecoverable errors or expensive retry cycles. In a SOC, that gap translates directly into analyst time, triage delay, and missed escalation windows. The field needs evaluation methods that reflect investigation resilience, not just correctness on final output.

This result validates a staged maturity model for AI in security operations. Teams should distinguish between interactive enrichment, assisted triage, and autonomous investigation. The benchmark suggests the technology is uneven across those stages, which means governance should advance by task criticality. The practitioner conclusion is to bind autonomy to proven reliability, not to model class or vendor claims.

What this signals

Agentic SOC governance is becoming a control-design issue. The practical question is no longer whether an AI can query logs, but whether the workflow constrains tool use tightly enough to keep investigations reproducible. Teams should expect procurement, architecture, and operational ownership to converge around the same design choice: how much action an AI identity is allowed to take before a human review point.

Investigative reliability will matter more than benchmark spectacle. In production, a security agent that is faster but brittle creates more work than a slower one that completes the case cleanly. That means evaluation programs should track error recovery, auditability, and analyst handoff quality as first-class metrics, not afterthoughts.

For practitioners

• Define SOC task boundaries for AI agents Limit AI to discrete incident-response and threat-hunting tasks where the evidence source, query surface, and expected output are explicit.
• Scope tool permissions to investigative need Grant only the Splunk actions required for the workflow, then review whether search, dataset enumeration, and source-type inspection are too broad for the use case.
• Measure completion quality, not just accuracy Track unrecoverable errors, retry loops, and abandoned tasks alongside accuracy so the evaluation reflects real SOC reliability.
• Require auditability for every agent action Log each tool call, intermediate result, and final answer so analysts can reconstruct how the agent reached a conclusion.

Key takeaways

• Blue team CTF results show that agentic SOC performance still depends on disciplined tool use and state management, not just model scale.
• The reproduced benchmark used 51 scenarios and 2.7M+ log events, which is enough complexity to surface real investigation failures.
• Practitioners should judge AI security workflows by completion quality, traceability, and containment of tool permissions before operational adoption.

Key terms

• Agentic workflow: An agentic workflow is a sequence of tasks executed by an AI agent with some level of tool access and decision authority. In security terms, the workflow matters because it can span multiple systems, identities, and permissions, which makes attribution and revocation harder than with ordinary automation.
• Long-horizon Investigation: A long-horizon investigation is a security task that requires many dependent steps, such as querying logs, checking intermediate evidence, and refining a final conclusion. These tasks are difficult for AI systems because errors compound across steps and the system must preserve state accurately under changing context.
• Integration Reliability: Integration reliability is the ability of a connector or sync layer to keep reading and writing identity data accurately under production conditions. In identity security, it is not enough for an integration to exist. It must continue to handle rate limits, schema changes, retries, and failures without producing silent gaps.
• Tool-Use Boundary: The approved set of systems, actions, and data sources an AI model may access. This boundary is central to security because once a model can call tools or act on records, access control and permission scope become part of AI risk management, not just integration design.

Deepen your knowledge

NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course. Explore the NHI Foundation Level course, the industry's only accredited NHI security programme, if your programme needs a stronger model for governed machine identity and agent oversight.