AI model benchmarking shows pass@3 beats one flagship scan

At a glance

What this is: Aikido benchmarked 13 AI models on 26 known CVEs and found that repeated runs can outperform a single more expensive pass.

Why it matters: IAM, AppSec, and AI security teams should care because AI-assisted code review is only as trustworthy as the evaluation design, run consistency, and harness that govern it.

By the numbers:

• GPT-5.6 gets the highest recall score, topping out at 23/26.

Context

This is a benchmarking question about how well AI models rediscover known vulnerabilities, not just whether they can generate plausible security commentary. The security governance gap is that teams often judge model value by single-pass output, even though detection quality varies materially from run to run and from one model tier to another.

For IAM and AppSec programmes, the important issue is not model novelty but control reliability: how evaluation harnesses, repetition, and scoring design shape whether AI can support real code review decisions. In practice, the question becomes how to measure consistent security signal from probabilistic systems rather than assuming one expensive run is enough.

Key questions

Q: When does pass@3 outperform a single AI security scan?

A: Pass@3 outperforms a single scan when model outputs vary materially from run to run and the task involves probabilistic search over code paths. Repeating the model captures findings missed on one pass, especially in indirect flows and multi-step bugs. The method is most useful when you care about discovery coverage rather than a single polished answer.

Q: Why do some CVEs separate strong models from average ones?

A: The gap appears when the vulnerability depends on missing checks, hidden data flows, or reasoning across several files. Simple sinks and obvious injection points are often found by most models, so they do not test real analytical depth. Stronger models show value when the bug cannot be identified from a single dangerous line.

Q: What are the signs that an AI code review benchmark is too easy?

A: A benchmark is too easy if most models find the same issues, especially when the bugs are obvious sinks, short flows, or familiar injection patterns. That setup measures pattern matching more than security reasoning. A better benchmark includes indirect routes, absent authorisation checks, and cases where the exploit path must be inferred rather than observed directly.

Q: How should security teams evaluate AI tools for code vulnerability discovery?

A: Use a fixed vulnerable corpus, repeat each test several times, and score the union of findings as well as the average single-pass result. That approach reveals variance, missed paths, and cost efficiency. A model that looks weak once may be useful when pooled, while a headline score can hide unreliable performance.

Technical breakdown

Why pass@3 changes vulnerability detection results

Pass@3 is a pooling method that treats a finding as discovered if any of three separate runs surfaces it. That matters because probabilistic models do not produce the same search path every time, especially when the target is a codebase with multiple possible entry points and indirect flows. A single run can miss a bug simply because it explored a different branch of reasoning. Pooling does not make the model smarter, but it captures variance that a one-shot benchmark hides. In this article, that variance is large enough to change the ranking between cheaper models and premium ones.

Practical implication: Measure AI security tools with repeated runs, not one-off outputs, when deciding whether they can support code review.

Why obvious CVEs are a weak test of AI security capability

The benchmark shows that straightforward issues such as dangerous sinks, simple injection paths, and clear access-control failures are relatively easy for modern models to rediscover. Those findings are mostly pattern recognition, because the exploit path is short and the vulnerable operation is visible. The harder cases are the ones with missing checks or long, indirect data flows across several files, where no single line announces the bug. That distinction is important for governance: a model that finds obvious CVEs may still struggle with the control failures that matter most in real review work.

Practical implication: Use test sets that include indirect flows and absent checks, not only obvious sink-based bugs.

Why harness design determines whether the model is evaluating code or guessing

The article uses a dedicated AI Code Analysis harness rather than a chat interface, and that changes the task entirely. The harness lets the model navigate a repository, inspect likely entry points, and triage candidate findings with tooling. In other words, the evaluation measures code investigation behaviour, not generic conversation quality. The authors also point out that prompts were kept short and model-agnostic, which reduces wording bias. This is a good example of benchmark hygiene: the surrounding system can determine whether a model is actually reasoning over code or just producing fluent guesses.

Practical implication: Test AI review systems inside a controlled harness that mirrors real investigation workflows.

NHI Mgmt Group analysis

Pass@3 is the right lens for probabilistic security review. A single benchmark run overstates confidence because model outputs vary materially from pass to pass. For security teams, that means the governance question is not only which model finds the most issues, but whether the evaluation process captures run-to-run variance before procurement or operational adoption. Repeated testing is the more defensible measure of detection value.

The real benchmark is not raw recall on easy CVEs. Obvious sinks, short exploit chains, and common injection patterns do not separate frontier models from cheaper ones in a meaningful way. The article shows that the differentiator appears in missing checks, indirect flows, and multi-file reasoning, which is closer to real AppSec work. That makes benchmark composition a control issue, not just a research choice.

AI-assisted code review should be governed like a probabilistic control, not a deterministic scanner. If teams treat one expensive pass as proof of coverage, they will misread both capability and risk. The more durable governance model is to compare pooled outcomes, cost per finding, and the failure modes that remain invisible after a single run. That is the practical threshold for deciding where AI can augment review and where human verification remains essential.

Harness quality is part of the security control, not an implementation detail. When the model is evaluated inside a purpose-built analysis environment, you are testing a workflow, not a prompt. That shifts attention to orchestration, tool access, and case selection, which is exactly how emerging AI security programmes should be designed. For practitioners, the lesson is that benchmarking without workflow realism can produce false confidence in detection coverage.

From our research library:

• 13% of organisations suffered breaches of their AI models, according to IBM's Cost of a Data Breach 2025 report.

What this signals

Benchmark variance is now a governance variable. If a model can miss a vulnerability on one run and find it on another, the control question moves from raw capability to repeatability under the same workload. For practitioners, that means procurement and validation should measure pooled performance and not assume one-pass results represent stable detection quality.

AI code review should be treated as an investigative workflow. The article's strongest signal is that the harness, the task framing, and the repository context shape the result as much as the model itself. That makes AI security adoption a workflow-design problem, not just a model-selection problem, which is why programme owners need to test realistic operating conditions before depending on the output.

For practitioners

• Benchmark with repeated runs, not single passes Run security models multiple times and compare pooled results to the best single-run result, because variance materially changes discovery rates.
• Test on indirect and multi-file vulnerabilities Include missing authorization checks, indirect injections, and longer exploit chains in your benchmark set, not only obvious sink-based CVEs.
• Evaluate the harness as part of the control Assess whether the analysis workflow can inspect entry points, follow suspicious flows, and triage evidence in a repository-like environment.
• Track cost per found vulnerability Compare discovery yield against spend so you can choose between a single flagship run and multiple mid-tier runs based on actual output.

Key takeaways

• Repeated runs can materially improve AI vulnerability discovery because model output varies from pass to pass.
• Obvious CVEs are not a sufficient benchmark for code security tools because they mostly measure pattern recognition.
• Security teams should evaluate both model capability and harness design before trusting AI-assisted code review in production.

Key terms

• Pass@3: A repeat-evaluation method where the same model is run multiple times and a finding counts if any run detects it. It is useful when outputs vary between runs, because the pooled result can show the model’s practical coverage better than a single score.
• Recall Score: Recall score measures how many of the known target issues a model successfully identifies out of the full set. In security benchmarking, it shows coverage rather than precision, so it tells you what the model found, not whether every finding was equally useful or actionable.
• Benchmark Harness: A benchmark harness is the evaluation setup used to measure model performance under defined conditions. It may include tools, prompts, iteration, grading logic, and execution constraints. Results from a harness describe a specific testing environment, not the full range of production behavior.
• Model Variance: Model variance is the difference in output quality or findings across repeated runs of the same system on the same task. For security teams, it is a critical property because inconsistent discovery means a single pass cannot be treated as a dependable measure of detection ability.

Deepen your knowledge

NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course. It is built for practitioners who need to govern non-human access and AI-driven decision-making with the same discipline they apply to human identity.