AI model benchmarking shows pass@3 beats one flagship scan

At a glance

What this is: This benchmark compares 13 AI models on rediscovering 26 known CVEs and finds that repeated runs with result pooling often outperform a single flagship pass.

Why it matters: It matters to AppSec, AI security, and code-review teams because model selection, run strategy, and harness design change how much real vulnerability coverage you can expect from AI-assisted analysis.

By the numbers:

• GPT-5.6 gets the highest recall score, topping out at 23/26, ahead of grok-4.5 (20), the Claude Opus models (15 to 18), and everything else we tested.

👉 Read Aikido's benchmark of 13 AI models on rediscovering known CVEs

Context

AI-assisted vulnerability discovery only works well when the evaluation setup matches the job you want the model to do. A chat-style assistant can miss issues because it stops at the first plausible answer, while a purpose-built harness can force deeper code-path analysis and better triage. For AppSec teams, the real question is not whether an LLM can spot a CVE, but whether the surrounding workflow turns model output into reliable coverage of code security risk.

This article is primarily about AI model benchmarking for application security, but it also raises an identity-adjacent governance issue: the model, harness, and tooling together behave like a non-human system with delegated authority over code analysis. That makes access to repositories, prompts, and findings part of the control surface, even when the subject is vulnerability discovery rather than identity itself.

Key questions

Q: How should security teams evaluate AI tools for code vulnerability discovery?

A: Use a fixed vulnerable corpus, repeat each test several times, and score the union of findings as well as the average single-pass result. That approach reveals variance, missed paths, and cost efficiency. A model that looks weak once may be useful when pooled, while a headline score can hide unreliable performance.

Q: Why do repeated model runs often outperform a single flagship pass?

A: Because vulnerability discovery is stochastic. Different runs explore different reasoning paths, so one pass may miss an obscure chain that another finds. Repetition increases coverage and reduces the risk of treating a model’s first answer as complete when the task actually requires broader search and validation.

Q: What do security teams get wrong about AI agent benchmarks?

A: They often measure end-to-end complexity or general model quality instead of the exact failure moment. The result is a score that looks useful but does not isolate whether the backbone, tool access, or orchestration caused the problem. Benchmarks need a specific state, attack vector, and scoring function to be operationally meaningful.

Q: How should teams decide whether a cheaper model is good enough?

A: Compare unique vulnerabilities found per dollar, not just per-run recall. If a mid-tier model repeated three times matches or beats a flagship once, it may be the better operational choice. The real decision is about stable yield, analyst effort, and how much variance your programme can absorb.

Technical breakdown

Why pass@3 changes vulnerability discovery outcomes

Pass@3 means running the same model multiple times and treating a CVE as found if any run surfaces it. That matters because model outputs are stochastic: one run may follow a different reasoning path, inspect a different code region, or miss an obscure chain that another run catches. In this benchmark, pooling reduced the impact of variance and recovered additional findings that a single pass missed. The architectural point is simple: when the model is inconsistent but directionally useful, repetition becomes a detection strategy rather than mere redundancy.

Practical implication: evaluate AI code-review tools using repeated runs and pooled results, not a single headline score.

Why harness design matters more than prompt polish

The benchmark used an AI Code Analysis harness that sends investigator agents directly to known vulnerable snippets, then forces them to reason through exploitability and evidence. That is different from a general coding assistant, which is optimized to produce code, not audit it. By constraining the search space, the harness measures reasoning over the vulnerable flow instead of wasted exploration. In security terms, the orchestration layer is part of the control plane, because it determines whether the model sees the right context to make a defensible call.

Practical implication: place model-assisted code review inside a purpose-built analysis workflow, not a generic chat interface.

Why frontier model price does not map cleanly to security value

The article shows diminishing returns at the top end of the model market. Higher-cost tiers can add one or two findings, but in several cases a cheaper model run multiple times and pooled its results to match or exceed a single flagship pass. This is not just a pricing story. It reflects the gap between raw reasoning capacity and operational yield, which depends on how many independent tries you can afford and how stable the model is across runs. For AppSec programmes, budget decisions should track usable findings per dollar, not model prestige.

Practical implication: compare models on findings per dollar and variance, not on maximum capability alone.

Threat narrative

Attacker objective: The objective is to keep exploitable code paths undiscovered so they remain available for later exploitation.

1. Entry occurs when attacker-controlled code, inputs, or repository flows reach a dangerous sink such as deserialization, command execution, or an HTML output path.
2. Credential or authorization failures matter when the model has to reason across missing checks, indirect data flows, or obscure chains that do not expose a single obvious flaw.
3. Impact is the failure to rediscover a real vulnerability in code that still contains exploitable paths, leaving the issue unreviewed and untriaged.

NHI Mgmt Group analysis

AI-assisted AppSec is now an orchestration problem, not just a model problem. The benchmark shows that the harness determines whether model reasoning is aimed at the vulnerable path or wasted on irrelevant code. That means teams should treat tool routing, snippet selection, and triage as part of the security control, not just the model choice. In practice, the control boundary is the workflow around the model.

pass@3 reveals the value of variance management. A single run underestimates what a model can find because different passes expose different reasoning paths. This is the same governance lesson IAM teams already know from access review and entitlement discovery: one snapshot rarely captures the full state. For security teams, repeated analysis is not inefficiency, it is how you recover hidden findings.

Cost efficiency is becoming a security metric. The article shows that cheaper models, when run multiple times, can rival a more expensive one-shot pass. That shifts procurement and programme design toward measurable yield, not brand strength or maximum benchmark score. Security leaders should ask which setup produces the most valid findings per dollar and per analyst hour.

Model-assisted code review creates a delegated decision surface that needs governance. Once a model and harness can inspect repositories, surface findings, and influence remediation priorities, they are part of the control environment. The relevant question is not whether the model is autonomous, but whether its access, outputs, and escalation paths are bounded. Practitioners should govern the system as a controlled non-human workflow.

What this signals

AI-assisted vulnerability discovery will increasingly be judged on operational yield, not model prestige. Teams that still evaluate tools on a single benchmark score will miss the effect of run-to-run variance and orchestration quality. The procurement question should be which setup produces the most credible findings per dollar, with the least analyst churn, using repeatable evidence.

Delegated code analysis should be governed like any other privileged non-human workflow. The model, its harness, and the repository access it receives collectively create a non-human decision path. That means prompt logging, access scoping, and review gates belong in the control design, not as afterthoughts. For teams already thinking about service accounts and machine identities, the same governance logic applies here.

From our research, 27 days is the average time to remediate a leaked secret and 75% of organisations still report strong confidence in their secrets management capabilities. That gap matters because AI-assisted review can increase finding volume faster than remediation capacity if workflow ownership is unclear. The practical signal is that detection improvements must be matched by lifecycle controls and analyst routing, or risk simply accumulates faster.

For practitioners

• Benchmark AI review tools on pooled runs, not single-pass scores Test candidate models using at least three runs against the same vulnerable corpus, then compare the union of findings to the average single-pass result. This exposes variance, which is often more important than the top-line score.
• Measure findings per dollar and findings per analyst hour Build a procurement scorecard that compares cost against unique vulnerabilities found, not just raw benchmark recall. Include the overhead of reruns, triage, and false-positive review so the economics reflect real operations.
• Use a purpose-built analysis harness for code security Place model evaluation inside a workflow that navigates repositories, isolates likely attack paths, and validates exploitability before surfacing a result. A generic chatbot interface is too shallow for code review at scale.
• Govern model access to repositories and findings as a control surface Restrict which repositories a model can inspect, log every prompt and retrieval action, and require human review for any result that will drive remediation or risk reporting. Treat the model pipeline like any other privileged non-human workflow.

Key takeaways

• AI code-security benchmarking is as much about orchestration and variance as it is about model capability.
• Repeated runs with pooled results can outperform a single expensive pass, making cost efficiency a security metric rather than a finance metric.
• Model-assisted code review should be governed as a privileged non-human workflow with scoped access, logging, and human validation.

Key terms

• Pass@3: A repeat-evaluation method where the same model is run multiple times and a finding counts if any run detects it. It is useful when outputs vary between runs, because the pooled result can show the model’s practical coverage better than a single score.
• AI Code Analysis harness: A purpose-built workflow that gives a model tools for repository navigation, targeted inspection, and result triage. It is designed for audit-style analysis, not code generation, so it can evaluate vulnerable flows more effectively than a generic chat interface.
• Run-to-run variance: The difference in outputs produced by the same model across repeated runs on the same task. In security analysis, high variance can hide findings in one pass and reveal them in another, which is why repeated evaluation often gives a truer operational picture.
• Operational yield: The amount of useful security output a tool produces in real conditions, relative to cost, analyst effort, and workflow overhead. For AI-assisted vulnerability discovery, yield matters more than headline benchmark scores because it reflects actual programme value.

What's in the full report

Aikido's full blog covers the operational benchmarking detail this post intentionally leaves for the source:

• Per-model run breakdowns across all 13 tested models, including the pass@3 union scores that show where variance matters most.
• The exact 26-vulnerability dataset composition, including severity mix and which vulnerability chains were hardest to rediscover.
• Cost-per-run comparisons for flagship, mid-tier, and open-weight models, useful if you need to justify an evaluation budget.
• Kimi K3 follow-up results and the late-added comparison that changed the final ranking picture.

👉 Aikido's full post includes the per-model score table, cost comparisons, and the Kimi K3 update.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, workload identity, secrets management, and agentic AI identity. It helps security and identity practitioners build the control discipline needed for non-human systems that operate across modern security programmes.