Repeated sampling changes the cost curve for vulnerability detection

At a glance

What this is: NHI Mgmt Group's analysis, drawing on depthfirst's blog post: Repeated sampling can materially improve AI-assisted vulnerability detection because multiple runs on the same code surface different findings and improve coverage.

Why it matters: This matters to security teams because detection quality depends not only on model choice but on how findings are sampled, deduplicated and confidence-scored across repeated runs.

Context

Vulnerability detection is a coverage problem as much as a classification problem. When an attacker only needs one exploitable weakness, defenders need repeated, consistent discovery across changing repositories, and probabilistic scanners can miss different issues from one run to the next.

That creates a governance issue for AI-assisted security tooling: one pass rarely defines the full risk picture, so teams need a way to combine runs, suppress noise, and decide what confidence threshold is acceptable. The central question is not whether an LLM can spot defects, but whether the operating model can turn stochastic outputs into reliable coverage.

Key questions

Q: How should teams improve vulnerability scan coverage with AI assistants?

A: Use repeated sampling on the same repository and compare the deduplicated findings, because a single AI-assisted scan can miss issues that appear only under different prompts or traversal paths. Measure coverage by unique vulnerabilities found per pass, not by raw alert count, and keep the model settings, seed, and filters consistent so scan results are comparable.

Q: Why do repeated AI scans sometimes find more vulnerabilities than one expensive pass?

A: Repeated AI scans can find more vulnerabilities because each pass explores code, dependencies, or prompts in a slightly different way, so one pass may miss paths that another pass surfaces. The gain usually comes from variation in sampling, context window use, and ranking thresholds, while deduplication removes duplicates and leaves additional unique findings.

Q: How do confidence thresholds affect vulnerability detection results?

A: Confidence thresholds control which candidate findings are reported, suppressed, or sent for further review. Lower thresholds usually increase recall but also noise, while higher thresholds reduce false positives and can hide real issues. When deduplicated findings are kept, the threshold should be tuned so repeated scans preserve unique vulnerabilities without flooding the queue with near-duplicates.

Q: What is the best way to compare AI vulnerability scanners?

A: Compare scanners by the attack paths they cover, the prompts or model interfaces they test, and how they report exploitable findings. Use cost-adjusted recall to balance detection coverage against runtime and false positives, and verify whether they can test prompt injection, data leakage, tool abuse, and model misconfiguration in your own environment.

Technical breakdown

Why repeated sampling improves vulnerability recall

LLM-based scanners do not produce identical outputs every time because the model samples from a probability distribution rather than executing a deterministic rule set. On the same repository, different runs can surface different candidate weaknesses, which means the union of findings across runs can uncover more issues than one pass alone. This is the same basic reason that inference-time scaling can help coding and reasoning tasks: more samples widen the search over plausible outputs. The trade-off is that additional runs cost more compute and can also introduce more low-quality findings, so coverage gains only matter if the pipeline can reconcile them. See the Vulnerability Management Buyer's Guide.

Practical implication: Treat repeated runs as a coverage multiplier, not a replacement for validation, and measure recall against cost before setting scan policy.

How confidence aggregation controls false positives

Repeated sampling increases the pool of candidate findings, but that only helps if the system can rank them by credibility. In practice, aggregation means deduplicating overlapping alerts, assigning a confidence score to each candidate, and discarding items below a chosen threshold. A lower threshold raises recall but also admits more noise, while a higher threshold protects precision at the risk of missing real issues. The important design choice is that the threshold should be tuned to the team’s risk appetite and review capacity, not treated as a fixed default. Otherwise, more scans simply create more operational churn. See the Detection Engineering Guide.

Practical implication: Set and review a confidence cutoff that matches triage capacity, then revisit it as the model and codebase change.

Why cheaper models can outperform expensive single passes

A single strong model run is not always the best use of inference budget. If a lower-cost model is run several times, the combined findings may match or exceed the recall of a more expensive model run once, especially when the model’s outputs vary enough to expose different paths through the code. This changes buying and operating decisions: the relevant question becomes how much detection coverage one dollar buys, not which model looks strongest in isolation. For vulnerability management programmes, that pushes optimisation toward throughput, aggregation logic, and quality control rather than model prestige.

Practical implication: Compare scanners on cost-adjusted recall, not headline model quality, and budget for multiple passes where coverage gaps matter most.

Threat narrative

Attacker objective: The attacker wants to find at least one exploitable weakness quickly enough to turn code exposure into practical compromise or follow-on access.

1. Entry occurs when an attacker uses AI-assisted scanning or code analysis to search repositories for exploitable weaknesses faster than manual review can keep up.
2. Credentialed or code-level access is not the issue here; the critical step is discovery of one high-payoff flaw that can be reused for compromise.
3. Impact follows when defenders miss a vulnerability that the attacker finds, leaving an exploitable path in production code or adjacent infrastructure.

Breaches seen in the wild

• Meta Muse agent hijack 2026: An undocumented Muse setting let local malware hijack Meta's personal AI agent, steal its authentication material and abuse user access.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Repeated sampling creates a new control plane for detection quality: the core security decision moves from single-pass accuracy to how well a programme can combine stochastic outputs into a dependable result. That matters because the same code base can produce different findings on different runs, which makes coverage an operational property rather than a model property. Teams that do not govern this layer will overtrust one pass and undercount exposure. The practical conclusion is that scan orchestration now belongs in security governance, not just tool configuration.

AI governance for security tooling now includes sampling policy: when an AI system contributes to vulnerability detection, the issue is not only whether the model is allowed, but how its uncertainty is managed. Repeated runs, confidence thresholds, and deduplication rules are governance choices because they shape what the programme will consider evidence. That places detection pipelines inside the same accountability logic used for other AI-enabled decision systems. Practitioners should define sampling policy, confidence thresholds, and escalation rules as part of tool governance.

The named concept here is detection coverage elasticity: the ability to raise or trade off recall by increasing repeated scans and adjusting confidence filters. This is distinct from raw model quality because it depends on orchestration, aggregation, and threshold design, not just the model itself. Programmes that understand this can buy better coverage from cheaper models without losing discipline over precision. The practitioner takeaway is to manage elasticity explicitly instead of treating scan frequency as a tuning detail.

Confidence thresholds are now a security control, not a post-processing preference: the balance between recall and precision determines whether a vulnerability programme drowns in noise or quietly misses exploitable issues. That makes threshold selection part of risk governance, especially when AI output is feeding remediation queues or board-level metrics. A stable threshold chosen once and never revisited will drift out of alignment as repositories, models, and attacker tactics change. The practical conclusion is to review threshold policy as part of detection assurance.

For NHI and AI governance, this is a reminder that machine-generated security findings behave like other probabilistic systems: they need oversight, inventory, and quality controls before they can be trusted in operational workflows. Where scanners are integrated into CI/CD, the same governance logic that applies to reviewing AI-generated code also applies to AI-generated findings. The practitioner implication is to treat the scanner as a governed decision source, not an oracle.

From our research library:

• Only 5.7% of organisations have full visibility into their service accounts, according to the Ultimate Guide to NHIs.
• Read next: Vulnerability Management Buyer's Guide

What this signals

Detection coverage elasticity: security teams will increasingly need to design for variable recall rather than assume a single pass is authoritative. Repeated sampling, confidence scoring, and deduplication together create a more realistic operating model for AI-assisted vulnerability discovery. The programme implication is that scanner governance must become a measurable control surface, not an afterthought.

As more security tools use probabilistic models, the operational question shifts to assurance. Teams should expect to monitor coverage drift, tune thresholds as codebases change, and keep a clear line between candidate generation and verified findings.

For practitioners

• Define a repeated-scan policy for high-risk repositories Run the same scanner multiple times on critical code paths and specify how many passes are required before a result is considered complete.
• Tune confidence cutoffs to match triage capacity Set a numeric threshold for deduped findings, then validate that it preserves the balance between missed vulnerabilities and review noise.
• Measure cost-adjusted recall before choosing a model Compare scanners using coverage per dollar, not just a single-pass result, because repeated runs can change the economics of detection.
• Deduplicate findings across runs before routing to remediation Merge overlapping alerts, assign each candidate a confidence score, and remove low-confidence duplicates before they enter the backlog.

Key takeaways

• Probabilistic scanners do not reveal the same issues every time, so repeated runs can materially improve vulnerability discovery.
• The benefit of extra runs depends on disciplined aggregation, because more samples can also increase false positives.
• Security teams should govern scan frequency, confidence thresholds and deduplication as part of detection assurance, not as tool tuning.

Related NHI Mgmt Group resources: Vulnerability Management Buyer's Guide · Detection Engineering Guide · AI-Generated Code Review Guide

Key terms

• Repeated Sampling: Repeated Sampling is the practice of running the same AI-assisted analysis multiple times on the same target to expose different findings. It is used in vulnerability scanning, code review, and other detection workflows where stochastic model output, prompt variation, or context limits can change what is discovered.
• Recall: The share of real weaknesses that the scanner successfully identifies. High recall reduces the chance of missed vulnerabilities, but if it is achieved by over-reporting, teams can become overwhelmed and stop trusting the results.
• Precision: The share of reported findings that are actually real weaknesses. High precision means fewer false positives, but it does not guarantee that the scanner is finding all of the important problems, so it must be read alongside recall to understand practical effectiveness.
• Confidence Score: A confidence score is a numeric estimate of how likely a system’s identity-related assessment, detection, or decision is correct. In identity security, it often reflects evidence quality, signal consistency, and model certainty for actions such as authentication, risk scoring, anomaly detection, or agent authorization.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, machine identity security and secrets management. It helps practitioners connect identity control to the wider security workflows their programmes depend on.