Notifications
Clear all
Topic starter
18/08/2026 2:20 pm
TL;DR: A benchmark of nine frontier models against a real macOS infostealer intrusion found that GPT-5.4 reached 87% overall accuracy, Gemini 3.1 Pro led threat hunting at 89%, and Kimi K2.5 delivered 68% accuracy at $0.36 per task, according to Cotool. The result is a clear warning that CTF-style tests can overstate defensive capability and miss the reasoning SOC work actually requires.
NHIMG editorial — based on content published by Cotool: LLMjacking: How Attackers Hijack AI Using Compromised NHIs
By the numbers:
- GPT-5.4 achieved 87% overall accuracy across incident response, threat hunting, and detection engineering.
Questions worth separating out
Q: How should security teams evaluate AI-augmented threat hunting platforms?
A: Start by testing whether the platform can execute a complete hunt from hypothesis to evidence across your existing SIEM, EDR, and cloud sources.
Q: Why do CTF-style benchmarks often overstate defensive AI capability?
A: CTFs usually reward exploitation and deterministic problem solving, while defensive work depends on noisy logs, partial evidence, and multi-step reasoning.
Q: What breaks when AI tools are tested only on synthetic security tasks?
A: Synthetic tasks hide the ambiguity that defines production security work.
Practitioner guidance
- Use real-intrusion benchmarks for model selection Test AI tools against investigations that include endpoint, network, and identity telemetry, then score them on triage quality, hunt depth, and detection usefulness rather than only completion rate.
- Separate offensive and defensive evaluation criteria Stop treating exploit-solving performance as evidence of SOC readiness.
- Preserve identity context in investigative data Retain credential, session, and authentication signals in the data used for AI-assisted investigations so models can correlate account abuse, token theft, and persistence with the wider attack chain.
What's in the full report
Cotool's full research covers the operational detail this post intentionally leaves for the source: the benchmark setup, scoring rubric, and per-model track breakdown.
- The exact 12-question structure used for incident response, threat hunting, and detection engineering.
- The scoring rubric that weights technical accuracy, completeness, and specificity across tasks.
- Per-model performance patterns, including where each model overperformed or failed under real intrusion conditions.
- The live research site's interactive results for analysts who want to inspect the dataset and outputs directly.
👉 Read Cotool's benchmark analysis of AI security investigation performance →
AI incident response benchmarks: are your tests measuring real work?
Explore further
18/08/2026 3:12 pm
Real intrusion data is the only credible benchmark for defensive AI work. CTFs measure whether a model can solve contrived tasks, but they do not test the sustained reasoning required for incident response, threat hunting, and detection engineering. When investigations depend on ambiguous telemetry and identity-linked signals, toy environments systematically overestimate readiness. Practitioners should treat real-intrusion datasets as the minimum standard for AI validation.
A question worth separating out:
Q: Who is accountable when an AI investigation cannot confirm a verdict?
A: The operating team remains accountable for the workflow design, the controls around uncertainty, and the decision to allow autonomy at all. If a tool cannot verify its evidence, the safe outcome is escalation and re-verification, not closure. Governance must assign ownership before the incident happens.
👉 Read our full editorial: AI investigation benchmarks need real intrusions, not CTF puzzles
