How should teams improve vulnerability scan coverage with AI assistants?

Why repeated sampling improves AI-assisted scan coverage

AI assistants can improve vulnerability scanning, but they are not deterministic enough to treat one pass as complete. The same repository can yield different findings depending on prompt wording, traversal order, context window limits, and which files or call paths the assistant explores first. Repeated sampling helps expose blind spots that a single run can miss, especially when findings depend on how the model reasons through the codebase.

Coverage should be treated as a sampling problem, not just a detection problem. If the assistant is used as part of a scan workflow, the goal is to increase the chance of reaching distinct vulnerable paths, not to inflate the total number of alerts. That is why deduplicated unique findings are the meaningful unit, while raw alert count mainly reflects repetition or model verbosity.

One practical way to think about this is to compare runs for overlap and novelty. A stable workflow will surface the same core issues repeatedly, while additional passes should contribute a shrinking but still measurable set of new unique findings. That pattern tells you whether the assistant is expanding coverage or merely restating the same results in different language.

How to make scan results comparable across passes

Comparable sampling requires control of the run conditions. Keep model settings, seed, and filters consistent so differences between passes reflect exploration breadth rather than configuration drift. If those variables change, you cannot tell whether a new finding is genuinely new coverage or simply the effect of a different scan recipe.

It also helps to define a deduplication rule before you start. Teams should decide whether two alerts refer to the same underlying vulnerability, the same file, the same sink, or the same remediation item. Without that rule, repeated sampling can look more productive than it really is, because one issue may appear in several forms across multiple passes.

The best operational pattern is to use a fixed baseline configuration, run several passes, and track the incremental yield from each pass. When the marginal number of new unique vulnerabilities drops sharply, you have a better signal that coverage is approaching a practical plateau for that repository and prompt set.

What good coverage measurement looks like in practice

Coverage measurement should answer whether the assistant is discovering more real issues, not whether it is generating more output. A useful metric is unique vulnerabilities found per pass, paired with the cumulative total after deduplication. That gives teams a clearer view of discovery rate, diminishing returns, and whether a new prompt strategy is materially better than the previous one.

This approach also supports tuning decisions. If one prompt style consistently increases unique findings while another mostly repeats prior alerts, the team can keep the higher-yield pattern and retire the weaker one. The same logic applies to traversal strategy, file scoping, and review depth: keep the variant that adds new coverage, not the one that sounds more thorough.

For larger codebases, it is useful to separate coverage of known hotspots from broad repository exploration. A scan that only finds issues in already-suspect areas may be useful, but it is not the same as broader coverage across the full repository. Repeated sampling should therefore be judged against both depth and spread.

Risk and Threat Considerations

AI-assisted scanning can create a false sense of completeness if teams rely on a single pass or on raw alert volume. The main risk is missed vulnerabilities that remain latent because the assistant explored only one reasoning path, one traversal path, or one prompt interpretation. That matters most when teams use the scan to decide whether a repository is safe to merge or release.

Failure mechanism: the assistant produces a partial view of the codebase, then the team treats repeated wording or high alert counts as evidence of broad coverage. The result is under-detection of issues that only emerge under different prompts, paths, or sampling conditions.

Impact: vulnerabilities can survive review, remediation prioritisation becomes distorted, and assurance decisions may be based on coverage that was never actually achieved.

Practitioner Guidance

What to verify: Confirm that each pass uses the same model settings, seed, and filters, and that the deduplication logic is stable enough to compare runs fairly. If those inputs vary, stop treating the result as a coverage benchmark.

What to measure: Track unique vulnerabilities per pass, cumulative unique findings, and the marginal yield of each additional sample. The most useful signal is not total output, but whether new passes still uncover genuinely new issues.

Common mistake: Teams often chase alert volume because it looks impressive in a report. For AI-assisted scanning, a noisy run is not a better run unless it increases deduplicated discovery.

Practitioner takeaway: Use repeated sampling to improve discovery breadth, then judge the workflow by incremental unique findings, not by how much text the assistant returns.

Related resources from NHI Mgmt Group

• Guide: Vulnerability Scanning Guide
• Scan coverage
• Vulnerability Scan
• Why do traditional vulnerability and scan metrics fall short for AI agents?
• What breaks when AI assistants are used for vulnerability remediation without scanner integration?