AI investigation benchmarks need real intrusions, not CTF puzzles

At a glance

What this is: This is Cotool’s analysis of why real-intrusion benchmarks outperform CTFs for testing AI security operations, with results showing different models excel at different defensive tasks.

Why it matters: It matters because SOC, detection engineering, and identity-aware investigation workflows need models that can reason across messy telemetry, not just solve contrived challenges.

By the numbers:

• GPT-5.4 achieved 87% overall accuracy across incident response, threat hunting, and detection engineering.
• Gemini 3.1 Pro dominated threat hunting at 89%, the highest score on any track in the benchmark.
• Kimi K2.5 delivered 68% accuracy at $0.36 per task, making it the benchmark's lowest-cost option.

👉 Read Cotool's benchmark analysis of AI security investigation performance

Context

Security teams are increasingly asked to evaluate AI systems on defensive work, but many benchmarks still measure offensive puzzle solving rather than incident response, hunting, and detection logic. That mismatch matters because real investigations depend on noisy telemetry, cross-log correlation, and identity signals that are absent from synthetic tests.

Cotool's benchmark addresses that gap by using a real macOS infostealer intrusion captured in a controlled lab environment, then testing how models handle triage, hunt hypotheses, and detection queries. The identity angle is real here: the intrusion harvested credentials, browser data, and keychain material, which is exactly the kind of evidence chain that modern SOC and IAM teams must reason through.

The starting point is typical of a broader benchmark problem, not an isolated case. If AI security evaluation is based on toy environments, the results will overstate capability and understate the operational controls needed for production use.

Key questions

Q: How should security teams evaluate AI-augmented threat hunting platforms?

A: Start by testing whether the platform can execute a complete hunt from hypothesis to evidence across your existing SIEM, EDR, and cloud sources. Then check whether it explains how it reached the conclusion, reduces analyst workload, and handles novel scenarios without relying on rigid playbooks. If it cannot do those things, it is automation support, not full hunting capability.

Q: Why do CTF-style benchmarks often overstate defensive AI capability?

A: CTFs usually reward exploitation and deterministic problem solving, while defensive work depends on noisy logs, partial evidence, and multi-step reasoning. That gap means a model can look strong in a lab and still struggle in a SOC. Real investigations demand correlation, prioritisation, and uncertainty handling across multiple data sources.

Q: What breaks when AI tools are tested only on synthetic security tasks?

A: Synthetic tasks hide the ambiguity that defines production security work. They can mask failures in evidence correlation, identity context, and query writing, which are central to incident response and detection engineering. The result is a benchmark that measures memorisation or prompt fitness instead of investigative reliability.

Q: Who is accountable when an AI investigation cannot confirm a verdict?

A: The operating team remains accountable for the workflow design, the controls around uncertainty, and the decision to allow autonomy at all. If a tool cannot verify its evidence, the safe outcome is escalation and re-verification, not closure. Governance must assign ownership before the incident happens.

Technical breakdown

Why CTFs undermeasure defensive reasoning

Capture the Flag environments are designed to reward exploitation, puzzle completion, and deterministic success paths. Defensive operations are different. Incident response asks a model to reconstruct a kill chain from incomplete telemetry, threat hunting asks it to form and test hypotheses across log sources, and detection engineering asks it to translate observed behaviour into reliable queries. That requires uncertainty handling, sequence reasoning, and evidence selection, not just task completion. A model that can solve a CTF may still fail when the evidence is fragmented, contradictory, or spread across endpoint, network, and identity telemetry.

Practical implication: evaluate AI against defensive workflows, not only exploit-style challenges.

How real intrusion datasets change benchmark design

A realistic dataset preserves the messiness of actual operations. In this case the evaluation used more than 416K events across 14 log sources, including endpoint telemetry, macOS Unified Logs, Zeek metadata, and security alerts. That gives models a chance to perform the same joins and correlations a SOC analyst would make in practice. It also reduces the risk that benchmark scores merely reflect memorised public answers from widely circulated CTF material. The result is a more credible signal of whether a model can handle investigative work under operational conditions.

Practical implication: require multi-source telemetry and provenance controls when validating AI security tools.

Why identity evidence matters in AI-assisted investigations

The intrusion described in the article is an infostealer chain, which means the decisive evidence includes credential theft, browser session data, and persistence artefacts. That makes identity data part of the investigation surface, not a side issue. In modern security operations, identity context helps distinguish user compromise, token abuse, and malicious process behaviour. For AI agents assisting analysts, the challenge is not only parsing logs but understanding which identity signals matter at each stage of the attack. That is where investigative reliability becomes an identity governance problem as much as a detection problem.

Practical implication: ensure AI-assisted investigations can correlate identity, endpoint, and network evidence before using them for triage.

Threat narrative

Attacker objective: The attacker aimed to steal credentials and session material that could support downstream account compromise and data theft.

1. Entry began with a trojanized Ledger Live application delivering the Odyssey Stealer infostealer onto macOS systems.
2. Credential access followed as the malware harvested browser data and keychain material, then exfiltrated the collected secrets over HTTP.
3. Impact came through compromised credentials and persistence, which created the basis for broader account abuse and investigative complexity.

NHI Mgmt Group analysis

Real intrusion data is the only credible benchmark for defensive AI work. CTFs measure whether a model can solve contrived tasks, but they do not test the sustained reasoning required for incident response, threat hunting, and detection engineering. When investigations depend on ambiguous telemetry and identity-linked signals, toy environments systematically overestimate readiness. Practitioners should treat real-intrusion datasets as the minimum standard for AI validation.

Identity evidence is now part of AI security evaluation, not a separate concern. The intrusion in this benchmark harvested credentials, browser data, and keychain material, which means the model had to reason across identity compromise as well as malware behaviour. That makes this a governance issue for IAM and SOC teams, not just a telemetry issue. The broader lesson is that AI helpers must understand credential lineage and access context before they can assist reliably.

Benchmark design now needs a named concept: defensive reasoning depth. That means the ability to reconstruct attack sequences, prioritise uncertain evidence, and turn observations into actions without overfitting to familiar patterns. Models can score well on isolated prompts and still fail when the task spans multiple log sources and competing hypotheses. Security teams should favour evaluations that expose reasoning depth rather than memorisation.

Cost and accuracy should be assessed together, but not as a substitute for workflow fit. The article shows a clear spread between overall accuracy, task-specific performance, and per-task cost. That is useful, but a low-cost model that cannot sustain investigative quality across all tracks will still create operational drag. The right procurement question is whether a model can support the full defensive workflow under the organisation's logging and identity conditions.

BlueBench signals a broader shift in AI governance for security operations. The next generation of evaluations will likely be tied to live attack patterns, platform-specific telemetry, and the same evidence chains analysts use in production. That will push practitioners toward stronger data governance, better access control for investigative tools, and more rigorous model oversight. The programme implication is simple: validate AI on the work, not on the marketing case.

What this signals

Security teams should expect AI evaluation to shift from benchmark theater toward evidence-rich, workflow-specific testing. That will favour organisations that can preserve high-quality telemetry, including identity context, and expose models to the same uncertainty analysts face in production.

Defensive reasoning depth: the organisations that win here will be those that validate whether a model can correlate logs, credentials, and attacker behaviour under real constraints. A fast answer is not the same as a correct investigation, and procurement should reflect that distinction.

For identity-heavy environments, the practical signal is that AI governance and IAM governance are converging. If the investigative stack cannot trust credential lineage, session state, and access boundaries, the model will inherit those blind spots and amplify them at speed.

For practitioners

• Use real-intrusion benchmarks for model selection Test AI tools against investigations that include endpoint, network, and identity telemetry, then score them on triage quality, hunt depth, and detection usefulness rather than only completion rate.
• Separate offensive and defensive evaluation criteria Stop treating exploit-solving performance as evidence of SOC readiness. Define separate pass criteria for incident response, threat hunting, and detection engineering so each workflow is measured on its own reasoning demands.
• Preserve identity context in investigative data Retain credential, session, and authentication signals in the data used for AI-assisted investigations so models can correlate account abuse, token theft, and persistence with the wider attack chain.
• Benchmark cost against operational fit Compare per-task cost only after confirming the model can sustain acceptable accuracy in the workflows you actually run, because cheap output that misses key evidence creates hidden analyst overhead.

Key takeaways

• AI security benchmarks that rely on CTF-style puzzles are a poor proxy for defensive operations because they miss the ambiguity, telemetry noise, and identity context of real incidents.
• Cotool’s benchmark shows that model performance varies sharply by task, with GPT-5.4 leading overall, Gemini 3.1 Pro excelling at hunting, and Kimi K2.5 competing on cost, which makes workflow fit more important than a single headline score.
• Security teams should validate AI on live investigative tasks, preserve identity evidence in their datasets, and treat benchmark design as part of operational governance.

Key terms

• Defensive reasoning depth: The ability of an AI system to reconstruct an attack, weigh incomplete evidence, and choose useful next steps in defensive security work. It goes beyond correct answers on isolated prompts and measures whether the model can behave reliably in messy, multi-source investigations.
• Benchmark contamination: A failure mode where training or fine-tuning data contains benchmark material, allowing a model to recall answers instead of reasoning them out. In security evaluation, contamination inflates scores and makes a tool look more capable than it is in production settings.
• Identity context: The entitlement, ownership, and purpose information that explains why an action occurred and whether it was expected. For security operations, identity context turns raw alerts into decisions by showing which human or non-human identity acted and what it was allowed to do.

What's in the full report

Cotool's full research covers the operational detail this post intentionally leaves for the source: the benchmark setup, scoring rubric, and per-model track breakdown.

• The exact 12-question structure used for incident response, threat hunting, and detection engineering.
• The scoring rubric that weights technical accuracy, completeness, and specificity across tasks.
• Per-model performance patterns, including where each model overperformed or failed under real intrusion conditions.
• The live research site's interactive results for analysts who want to inspect the dataset and outputs directly.

👉 Cotool's full research includes the track-level results, methodology, and interactive benchmark output.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, workload identity, secrets management, and identity lifecycle control. It helps security practitioners connect identity governance to the operational realities that shape modern investigation and response.