AI investigation benchmarks need real intrusions, not CTF puzzles

At a glance

What this is: Cotool says AI security benchmarks built on a real macOS infostealer intrusion produced materially different results from CTF-style testing, with top models varying sharply by incident response, hunting, and detection tasks.

Why it matters: This matters because SOC leaders and AI security teams need evaluation methods that reflect real investigative work, where telemetry is noisy, cross-source correlation is hard, and reasoning quality matters more than puzzle solving.

By the numbers:

• GPT-5.4 achieved 87% overall accuracy in the benchmark, according to Cotool.
• Gemini 3.1 Pro reached 89% on threat hunting, according to Cotool.
• Kimi K2.5 delivered 68% accuracy at $0.36 per task, according to Cotool.

Context

AI incident response benchmarks are only useful if they resemble the work defenders actually do. A benchmark built around a real intrusion tests correlation, reasoning, and triage under noisy conditions, while CTF-style puzzles mainly measure narrow offensive problem solving. For SOCs and AI security teams, the core issue is whether the model can reason across endpoint, network, and identity evidence when the path is messy.

The article centres on AI agents and investigative workflows, which makes identity and access governance relevant even though the primary topic is detection and response. If an AI system is going to participate in incident handling, it must be evaluated as a tool with bounded authority, observable actions, and reliable task completion. That is an AI governance problem as much as a security benchmarking problem.

Cotool's comparison also reflects a broader market problem: many organisations are still using the wrong yardstick to judge AI utility in security operations. A benchmark that rewards memorisation or exploit execution can look impressive while missing the ability to reconstruct real attack chains. That is atypical of mature SOC evaluation, not a normal proxy for production readiness.

Key questions

Q: What breaks when AI security benchmarks use CTF puzzles instead of real incidents?

A: CTF puzzles overstate offensive problem solving and under-test the reasoning required for incident response, threat hunting, and detection engineering. They usually lack noisy telemetry, ambiguous signals, and multi-source correlation, so a model can score well without proving it can support a real SOC investigation.

Q: Why do real intrusion datasets give a better signal for AI incident response?

A: They force models to work through messy, cross-source evidence the way analysts do in production. That exposes whether the system can reconstruct a kill chain, triage alerts, and write useful detections instead of simply recognising a familiar puzzle pattern.

Q: Why does Python matter for threat hunting and detection engineering in modern security operations?

A: Python matters because it lets teams analyze large security datasets, automate repetitive investigations, and express detection logic in a language that humans and AI can both understand. That combination improves speed without forcing analysts into opaque tooling. It is especially useful when teams need to correlate logs, enrich events, and rapidly iterate on new detections as threat patterns change.

Q: Should security teams trust benchmark scores when buying AI offensive tools?

A: Not on their own. Benchmark scores are useful, but only when the benchmark is realistic, variant-based, and unfamiliar to the model. Teams should demand evidence from modified environments and application flows that mirror production complexity, including authentication, role boundaries, and multi-step interactions.

Technical breakdown

Why CTF benchmarks mismeasure defensive AI

Capture the Flag exercises are built to reward exploit discovery, puzzle solving, and rapid tactical execution. Defensive operations are different. Incident response and threat hunting require stitching together weak signals from multiple logs, separating signal from noise, and reasoning about what happened before, during, and after an intrusion. A model can excel at exploitation tasks and still fail at triage because the reasoning path is different. In practice, CTF scores are a poor proxy for whether an AI system can support a SOC investigation under realistic conditions.

Practical implication: Use benchmark designs that test investigation workflows, not just offensive challenge completion.

Why real telemetry changes model performance

Real investigations involve messy, cross-domain evidence from endpoint telemetry, network metadata, security alerts, and identity signals. That environment forces a model to correlate events across sources, handle ambiguity, and maintain state across a long reasoning chain. In a controlled puzzle, the answer is usually local to the task. In production-like security work, the answer emerges only after comparing sources and ruling out false leads. This is why seemingly strong model scores in synthetic settings can collapse when the data looks like an actual incident.

Practical implication: Evaluate AI tools against multi-source datasets that mirror production SOC data flows.

How benchmark contamination inflates apparent capability

When public benchmark material is widely available, frontier labs may ingest it during pre-training or post-training, which blurs the line between recall and reasoning. That is especially dangerous for security evaluations because a model that memorises known answers can appear operationally strong without actually generalising to new incidents. The article's concern is not just accuracy, but whether the benchmark can survive contamination and still measure practical investigative skill. A benchmark that leaks answers becomes a memory test, not a readiness test.

Practical implication: Prefer evaluation sets built from fresh, controlled incidents with limited public exposure.

Threat narrative

Attacker objective: Steal credentials and browser data while maintaining persistence for continued access and follow-on abuse.

1. Entry occurred through a trojanized Ledger Live application carrying Odyssey Stealer into an isolated macOS environment.
2. The malware harvested credentials, exfiltrated keychain and browser data, and established LaunchDaemon persistence.
3. The resulting intrusion generated enough telemetry to support incident response, threat hunting, and detection engineering tasks across 14 log sources.
4. The objective was credential theft and durable access for downstream exploitation of the victim environment.

Breaches seen in the wild

• ClawHub malicious skills 2026: Hundreds of malicious ClawHub skills for the OpenClaw agent pushed credential stealers in early 2026; Koi found 341 of 2,857 in the ClawHavoc campaign.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Real intrusion data is a better test of AI security value than CTF scoring. Benchmarks that use simulated puzzles mainly validate narrow problem solving, while incident response depends on investigation logic across noisy, incomplete evidence. For AI-enabled SOC workflows, the decisive question is whether the system can support correlation and triage under production-like conditions.

AI investigation tools must be judged against operational reasoning, not model fluency. The article shows that top performers split by task, which is exactly what defenders should expect when the work spans hunting, triage, and detection writing. A single aggregate score hides whether the model is dependable in the track that matters most to the programme.

Benchmark contamination creates false confidence in defensive AI. When public challenge material is available for training, scores can drift toward recall rather than competence. That means evaluation design is itself a governance control, and practitioners should treat benchmark provenance as part of AI assurance.

Investigation benchmarks expose an identity and authorisation question for AI systems. If an AI agent is being given investigative tools, it needs scoped access, auditable actions, and task-level accountability. Otherwise the benchmark may measure raw capability while ignoring the control model needed to use that capability safely.

BlueBench is really a category signal about where AI security evaluation is heading. The market is moving from contrived tests toward evidence-based assessments rooted in real attack chains. Practitioners should expect more emphasis on incident realism, source diversity, and workflow fidelity in how security AI is procured and governed.

From our research library:

• 53% of security leaders expect AI to run major portions of their infrastructure autonomously within the next three years, according to the 2026 Infrastructure Identity Survey.
• 19% of organisations give AI systems dramatically more access than human employees, nearly one in five granting unrestricted privilege, according to the 2026 Infrastructure Identity Survey.
• Read next: AI Agent Observability, Audit and Incident Response Guide

What this signals

Benchmark realism now matters more than benchmark elegance. Security teams will get misleading answers if they continue to judge AI defenders on synthetic challenges that do not resemble live investigations. The programme implication is straightforward: move procurement, red-teaming, and validation toward controlled intrusions and away from puzzle performance. The 4.5x incident delta for over-privileged AI systems in our research library is a reminder that capability without governance is not operational safety.

Investigation AI needs bounded authority, not just stronger prompts. If a system can query telemetry, propose containment, or write detections, it has entered the identity and access governance perimeter. That means teams need scoped permissions, logging, and review points that mirror how they would control any other privileged workflow, including non-human identities.

Realistic evaluation changes how security leaders buy and trust AI. The market is moving toward evidence-based assessment of whether a model can work in noisy, adversarial environments, and that will affect vendor claims, pilot design, and internal assurance. Practitioners should expect AI governance to become part of SOC tooling decisions rather than a separate policy conversation.

For practitioners

• Build evaluation sets from real incidents Use controlled intrusions, live telemetry, and genuine attack chains instead of toy puzzles when assessing security AI performance. Include endpoint, network, and identity evidence so the benchmark reflects how analysts actually work.
• Score by investigation task, not one composite number Separate incident response, threat hunting, and detection engineering into distinct tracks, then compare models by the job they will actually perform. A strong hunting result should not mask weak triage or poor detection logic.
• Test for benchmark contamination before trusting results Review whether source challenges are public, heavily discussed, or likely to have been seen in training. If the answers are already well known online, the benchmark is measuring recall more than reasoning.
• Constrain investigative AI with scoped access If an agent can query telemetry or support triage, limit its permissions, log its actions, and separate its tool access from human analyst privileges. Evaluation should include the governance controls that will exist in production, not just the model output.
• Measure cost and latency alongside accuracy Track task cost, completion time, and failure rate together so the programme can choose between high-accuracy, fast-triage, and cost-efficient models based on operational need.

Key takeaways

• CTF-style benchmarks are a weak proxy for defensive AI because they reward puzzle solving instead of real investigation work.
• The real-intrusion benchmark produced different leaders by task, which shows that incident response, threat hunting, and detection engineering require distinct capabilities.
• Security teams should favour benchmark designs built on controlled intrusions, multi-source telemetry, and scoped AI access, because that is closer to production SOC reality.

Key terms

• Benchmark contamination: A failure mode where training or fine-tuning data contains benchmark material, allowing a model to recall answers instead of reasoning them out. In security evaluation, contamination inflates scores and makes a tool look more capable than it is in production settings.
• Investigation Track: An investigation track is a distinct evaluation path that measures one security workflow, such as incident response, threat hunting, or detection engineering. Separating tracks reveals where a model is strong or weak instead of hiding those differences inside a single averaged score.
• Real Intrusion Dataset: A real intrusion dataset is evidence captured from an actual attack chain, usually in a controlled environment rather than a production system. It provides the messy, multi-source telemetry that defensive AI must interpret if it is to be useful in SOC operations.
• Standing AI Access: Persistent access granted to AI workloads, service accounts or agents beyond the shortest practical task window. It creates a wider exposure period for abuse and makes revocation harder, which is why lifecycle control matters as much as initial authorisation.

Deepen your knowledge

NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course. Explore nhimg.org for resources that connect identity governance to the broader security disciplines your programme depends on.