Topic starter
18/08/2026 2:20 pm
TL;DR: A benchmark of nine frontier models against a real macOS infostealer intrusion found that GPT-5.4 reached 87% overall accuracy, Gemini 3.1 Pro led threat hunting at 89%, and Kimi K2.5 delivered 68% accuracy at $0.36 per task, according to Cotool. The result is a clear warning that CTF-style tests can overstate defensive capability and miss the reasoning SOC work actually requires.
Editorial analysis by NHI Mgmt Group, based on content published by Cotool: “Beyond CTFs: Evaluating AI Agents on Real Intrusion Data”.
By the numbers:
- GPT-5.4 achieved 87% overall accuracy in the benchmark, according to Cotool.
- Gemini 3.1 Pro reached 89% on threat hunting, according to Cotool.
- Kimi K2.5 delivered 68% accuracy at $0.36 per task, according to Cotool.
Key questions
Q: What breaks when AI security benchmarks use CTF puzzles instead of real incidents?
A: CTF puzzles overstate offensive problem solving and under-test the reasoning required for incident response, threat hunting, and detection engineering.
Q: Why do real intrusion datasets give a better signal for AI incident response?
A: They force models to work through messy, cross-source evidence the way analysts do in production.
Q: Why does Python matter for threat hunting and detection engineering in modern security operations?
A: Python matters because it lets teams analyze large security datasets, automate repetitive investigations, and express detection logic in a language that humans and AI can both understand.
Practitioner guidance
- Build evaluation sets from real incidents Use controlled intrusions, live telemetry, and genuine attack chains instead of toy puzzles when assessing security AI performance.
- Score by investigation task, not one composite number Separate incident response, threat hunting, and detection engineering into distinct tracks, then compare models by the job they will actually perform.
- Test for benchmark contamination before trusting results Review whether source challenges are public, heavily discussed, or likely to have been seen in training.
Bottom line: CTF-style benchmarks are a weak proxy for defensive AI because they reward puzzle solving instead of real investigation work.
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
18/08/2026 3:12 pm
Real intrusion data is a better test of AI security value than CTF scoring. Benchmarks that use simulated puzzles mainly validate narrow problem solving, while incident response depends on investigation logic across noisy, incomplete evidence. For AI-enabled SOC workflows, the decisive question is whether the system can support correlation and triage under production-like conditions.
A few things that frame the scale:
- 53% of security leaders expect AI to run major portions of their infrastructure autonomously within the next three years, according to the 2026 Infrastructure Identity Survey.
- 19% of organisations give AI systems dramatically more access than human employees, nearly one in five granting unrestricted privilege, according to the 2026 Infrastructure Identity Survey.
A question worth separating out:
Q: Should security teams trust benchmark scores when buying AI offensive tools?
A: Not on their own. Benchmark scores are useful, but only when the benchmark is realistic, variant-based, and unfamiliar to the model. Teams should demand evidence from modified environments and application flows that mirror production complexity, including authentication, role boundaries, and multi-step interactions.
👉 Read our full editorial: AI investigation benchmarks need real intrusions, not CTF puzzles