Notifications
Clear all
Topic starter
02/08/2026 9:46 am
TL;DR: Benchmarking 13 AI models against 26 known CVEs shows GPT-5.6 leading with 23/26, while pooled three-run results often beat a single expensive pass and cheaper mid-tier models can narrow the gap substantially, according to Aikido. The practical lesson is that evaluation design, not raw model price, determines how much vulnerability coverage security teams actually get.
NHIMG editorial — based on content published by Aikido: Benchmarking 13 AI models on rediscovering known CVEs
Questions worth separating out
Q: How should security teams evaluate AI tools for code vulnerability discovery?
A: Use a fixed vulnerable corpus, repeat each test several times, and score the union of findings as well as the average single-pass result.
Q: Why do repeated model runs often outperform a single flagship pass?
A: Because vulnerability discovery is stochastic.
Q: What do security teams get wrong about AI agent benchmarks?
A: They often measure end-to-end complexity or general model quality instead of the exact failure moment.
Practitioner guidance
- Benchmark AI review tools on pooled runs, not single-pass scores Test candidate models using at least three runs against the same vulnerable corpus, then compare the union of findings to the average single-pass result.
- Measure findings per dollar and findings per analyst hour Build a procurement scorecard that compares cost against unique vulnerabilities found, not just raw benchmark recall.
- Use a purpose-built analysis harness for code security Place model evaluation inside a workflow that navigates repositories, isolates likely attack paths, and validates exploitability before surfacing a result.
What's in the full report
Aikido's full blog covers the operational benchmarking detail this post intentionally leaves for the source:
- Per-model run breakdowns across all 13 tested models, including the pass@3 union scores that show where variance matters most.
- The exact 26-vulnerability dataset composition, including severity mix and which vulnerability chains were hardest to rediscover.
- Cost-per-run comparisons for flagship, mid-tier, and open-weight models, useful if you need to justify an evaluation budget.
- Kimi K3 follow-up results and the late-added comparison that changed the final ranking picture.
👉 Read Aikido's benchmark of 13 AI models on rediscovering known CVEs →
AI model vulnerability review: is pass@3 better than one flagship run?
Explore further
02/08/2026 12:39 pm
AI-assisted AppSec is now an orchestration problem, not just a model problem. The benchmark shows that the harness determines whether model reasoning is aimed at the vulnerable path or wasted on irrelevant code. That means teams should treat tool routing, snippet selection, and triage as part of the security control, not just the model choice. In practice, the control boundary is the workflow around the model.
A question worth separating out:
Q: How should teams decide whether a cheaper model is good enough?
A: Compare unique vulnerabilities found per dollar, not just per-run recall. If a mid-tier model repeated three times matches or beats a flagship once, it may be the better operational choice. The real decision is about stable yield, analyst effort, and how much variance your programme can absorb.
👉 Read our full editorial: AI model benchmarking shows pass@3 beats one flagship scan
