Topic starter
02/08/2026 9:46 am
TL;DR: Benchmarking 13 AI models against 26 known CVEs shows GPT-5.6 leading with 23/26, while pooled three-run results often beat a single expensive pass and cheaper mid-tier models can narrow the gap substantially, according to Aikido. The practical lesson is that evaluation design, not raw model price, determines how much vulnerability coverage security teams actually get.
Editorial analysis by NHI Mgmt Group, based on content published by Aikido: “Benchmarking 13 AI models on rediscovering known CVEs”.
Key questions
Q: When does pass@3 outperform a single AI security scan?
A: Pass@3 outperforms a single scan when model outputs vary materially from run to run and the task involves probabilistic search over code paths.
Q: Why do some CVEs separate strong models from average ones?
A: The gap appears when the vulnerability depends on missing checks, hidden data flows, or reasoning across several files.
Q: What are the signs that an AI code review benchmark is too easy?
A: A benchmark is too easy if most models find the same issues, especially when the bugs are obvious sinks, short flows, or familiar injection patterns.
Practitioner guidance
- Benchmark with repeated runs, not single passes Run security models multiple times and compare pooled results to the best single-run result, because variance materially changes discovery rates.
- Test on indirect and multi-file vulnerabilities Include missing authorization checks, indirect injections, and longer exploit chains in your benchmark set, not only obvious sink-based CVEs.
- Evaluate the harness as part of the control Assess whether the analysis workflow can inspect entry points, follow suspicious flows, and triage evidence in a repository-like environment.
Bottom line: Repeated runs can materially improve AI vulnerability discovery because model output varies from pass to pass.
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
02/08/2026 12:39 pm
Pass@3 is the right lens for probabilistic security review. A single benchmark run overstates confidence because model outputs vary materially from pass to pass. For security teams, that means the governance question is not only which model finds the most issues, but whether the evaluation process captures run-to-run variance before procurement or operational adoption. Repeated testing is the more defensible measure of detection value.
A few things that frame the scale:
- 13% of organisations suffered breaches of their AI models, according to IBM's Cost of a Data Breach 2025 report.
A question worth separating out:
Q: How should security teams evaluate AI tools for code vulnerability discovery?
A: Use a fixed vulnerable corpus, repeat each test several times, and score the union of findings as well as the average single-pass result. That approach reveals variance, missed paths, and cost efficiency. A model that looks weak once may be useful when pooled, while a headline score can hide unreliable performance.
👉 Read our full editorial: AI model benchmarking shows pass@3 beats one flagship scan