Notifications
Clear all
Topic starter
18/08/2026 2:35 pm
TL;DR: CyberMetric’s 10,180-question benchmark found all 13 tested models clustered between 86.9% and 90.8% accuracy, a 3.9-point spread that suggests static cybersecurity knowledge is close to saturation, according to Cotool. The practical issue is no longer recall alone, but whether models can reason, use tools, and operate reliably in live workflows.
NHIMG editorial — based on content published by Cotool: CyberMetric Feb 2026 multiple-choice cybersecurity knowledge evaluation across 10,000 questions
By the numbers:
- All 13 models scored between 86.9% and 90.8%, creating a gap of just 3.9 percentage points from top to bottom.
- GPT-5 Mini scored 25.45% without answer choices, which matches random guessing at 25%.
- Frontier models achieved 100% completion on the benchmark, while GLM-4.7 finished at 97.4%.
Questions worth separating out
Q: How should security teams evaluate LLMs for cybersecurity use beyond benchmark scores?
A: Use benchmark scores as a baseline, then test the model in realistic workflows.
Q: Why do high cybersecurity benchmark scores not guarantee safe enterprise use?
A: Because multiple-choice benchmarks mainly test stored knowledge, not judgment.
Q: What do security teams get wrong about model selection for SOC and IAM workflows?
A: They often optimise for leaderboard rank instead of workflow fit.
Practitioner guidance
- Separate recall from operational fitness Use static knowledge benchmarks only as one input.
- Tier models by sensitivity and privilege Assign cheaper or smaller models to low-risk classification tasks, and reserve higher-trust workflows for systems with stronger audit logging, retrieval controls, and approval gates around IAM or secrets actions.
- Test identity workflows explicitly Run scenarios that force the model to reason about privileged access, service accounts, token handling, and access exceptions so you can see whether it respects least privilege in practice.
What's in the full report
Cotool's full benchmark analysis covers the methodology and model-by-model detail this post intentionally leaves for the source:
- Per-model accuracy, cost, and completion comparisons across all 13 tested systems
- Method notes on the 10,180-question dataset and the human verification process
- Evidence behind the memorization check using answerless prompts and random-guess baselines
- Benchmarks that Cotool recommends for applied reasoning and agentic evaluation after static knowledge saturates
👉 Read Cotool's CyberMetric benchmark analysis of cybersecurity model accuracy and cost →
CyberMetric benchmark results: what do they mean for security teams?
Explore further
18/08/2026 3:21 pm
Static benchmark saturation is changing what “good” means for security AI. When 13 models cluster within 3.9 percentage points on a 10,180-question test, leaderboard separation stops being a strong proxy for operational fitness. That does not make evaluation irrelevant. It means practitioners need to value tool use, traceability, and policy adherence over trivia recall. For identity teams, the same logic applies to assistants that advise on access, secrets, and privileged workflows.
A question worth separating out:
Q: How should organisations decide which model can touch sensitive identity data?
A: Base the decision on sensitivity, privilege, and logging requirements. If a workflow involves identity records, privileged access, or secret handling, require stronger retrieval controls, narrower permissions, and explicit human review for actions that could alter access or exposure.
👉 Read our full editorial: CyberMetric shows cybersecurity knowledge is near saturation in LLMs
