AI vulnerability detection depends more on harness design than model choice

At a glance

What this is: Semgrep benchmarks AI vulnerability detection and concludes that harness design drives materially better recall, precision, and cost efficiency than model choice alone.

Why it matters: Security teams evaluating AI-assisted code scanning need to care about coverage, determinism, and workflow design, because a strong model still misses vulnerabilities if the harness fails to point it at the right code.

Context

AI vulnerability detection is not just a model selection problem. In security scanning, the hard part is often deciding what code should be examined, in what order, and with what context, because a model can only reason about code it is actually shown.

This article sits at the intersection of AI-assisted application security and detection engineering. The identity angle is indirect but real: the output of these systems influences who or what gets trusted in code review and remediation workflows, so governance depends on repeatable coverage rather than persuasive reasoning alone.

Key questions

Q: What breaks when AI vulnerability scanners do not enumerate the full codebase?

A: They can only reason about the code slices they are shown, so vulnerabilities in unnoticed endpoints never enter the finding set. That creates a false sense of coverage because the model may be accurate on the small subset it sees while still missing most of the attack surface. In practice, incomplete enumeration turns AI detection into selective review, not reliable assurance.

Q: Why does harness design matter more than model choice for AI code scanning?

A: Because the harness decides what the model sees, how much context it gets, and whether the scan is repeatable. A strong model can still miss vulnerabilities if it is pointed at the wrong code or asked to infer coverage on its own. The right workflow turns model reasoning into a governed detection process instead of an ad hoc chat with code.

Q: What are the signs that an AI security scanner is under-covering a repository?

A: The clearest signal is a finding set that looks plausible but remains small relative to the size and complexity of the codebase, especially when later manual review finds vulnerabilities in untouched endpoints. Inconsistent results across runs are another warning. If the tool cannot show what it inspected, teams should assume hidden blind spots until proven otherwise.

Q: How should teams compare AI-assisted detection against guided prompting?

A: Compare them on repeatable recall, precision, and cost per true positive using the same ground-truth codebase. Guided prompts can look competitive on a narrow sample, but they often fail when coverage is the bottleneck. The better question is which workflow finds more real issues across the full repository without inflating noise or cost.

Technical breakdown

Why deterministic code enumeration changes vulnerability coverage

A language model is good at judging exploitability once it sees the right function, endpoint, or sink. It is not reliable at traversing a repository and deciding which parts deserve attention. Deterministic analysis solves that first problem by enumerating code paths, endpoints, and sinks repeatably, so the model is not left guessing where to look. That matters because coverage is the hidden variable behind recall. If the scanner never surfaces a vulnerable endpoint, the model cannot flag it, no matter how capable it is. The architecture therefore separates discovery from reasoning: exact program analysis first, probabilistic judgment second.

Practical implication: benchmark AI scanners on whether they reliably enumerate reviewable targets, not just on whether they produce plausible findings.

Why precision can stay flat while recall changes dramatically

Precision measures how many reported findings are real, while recall measures how many real vulnerabilities are found. A harness can improve recall without moving precision much if it changes which code the model sees rather than how the model judges a code slice. That is what makes this article’s comparison meaningful: the model is not being praised or blamed in isolation. Instead, the workflow around it determines whether the model spends its effort on the right evidence. In production security tooling, that distinction matters because a high precision tool that misses most endpoints still leaves exploitable code untreated.

Practical implication: evaluate scanners against both true-positive rate and coverage of the full attack surface, not one metric alone.

How cost per true positive reframes AI security economics

Price per scan is a misleading measure when the real objective is risk reduction. A cheap scan that misses vulnerabilities creates hidden cost later through defects, rework, or exposure. Cost per true positive is a better operational metric because it ties spend to actionable outcomes. In this benchmark, harness design reduces wasted token use by avoiding repeated rereads and by focusing the model on slices of code that already matter. That makes orchestration part of security economics, not just engineering convenience. For teams building or buying AI-assisted detection, the key question is how much validated risk reduction each dollar actually buys.

Practical implication: compare AI detection systems on cost per actionable finding, not token cost or scan count alone.

Threat narrative

Attacker objective: The practical attacker objective is to keep exploitable code outside the scanner’s review scope so findings never surface before release.

1. Entry begins when a scanner is pointed at a repository and asked to find vulnerabilities, but the repository is only partially enumerated for review.
2. Credential access is not the issue here, but the functional equivalent is selective visibility: the model only reasons over endpoints the harness exposes, leaving hidden attack surfaces unanalyzed.
3. Impact is missed vulnerabilities that survive code review and reach production because the scanner never inspected the relevant code paths.

Breaches seen in the wild

• Meta Muse agent hijack 2026: An undocumented Muse setting let local malware hijack Meta's personal AI agent, steal its authentication material and abuse user access.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Harness design is now the primary control surface for AI-assisted vulnerability detection. The article’s central lesson is that model capability matters less than the workflow that constrains where the model looks and what evidence it receives. In security terms, the control failure is not weak reasoning alone but incomplete coverage architecture. Practitioners should treat orchestration quality as part of the security control, not as implementation detail.

Coverage is the governance gap most teams under-measure. Security leaders are accustomed to judging scanners by output quality, but this benchmark shows that endpoint enumeration and candidate selection can dominate the result. That means a tool can appear strong in demos while still leaving blind spots in real repositories. The practitioner implication is to govern AI detection as a pipeline with measurable coverage, not as a single model score.

Risk reduction, not model novelty, should define procurement and tuning decisions. The article shows that a better harness can outperform a more direct prompt using the same model, which means buyer decisions should focus on determinism, repeatability, and economic efficiency. For IAM and governance teams, the broader lesson is familiar: trustworthy outcomes come from controlled process design, not from assuming intelligence alone will fill the gaps.

AI detection systems need an assurance model comparable to other security automation. The same scrutiny applied to access reviews, policy enforcement, and privileged workflows should apply to AI-assisted code analysis. If the system cannot prove what it inspected, it cannot prove what it missed. Practitioners should demand traceable coverage evidence before treating AI findings as operationally complete.

Deterministic scaffolding is becoming the practical differentiator in agentic security workflows. As models improve, the market will still reward systems that combine probabilistic reasoning with exact program analysis, because security teams need repeatable outcomes they can defend. The field is moving toward governed AI pipelines, not standalone model calls. Practitioners should expect orchestration quality to become a procurement criterion, not an optional enhancement.

What this signals

Coverage is becoming the real quality bar for AI security operations. Teams that only ask whether a model can identify bugs are missing the more important question of whether the workflow can systematically expose every relevant code path. That is especially important where security findings drive remediation prioritisation and risk acceptance.

Deterministic analysis is the bridge between AI reasoning and operational trust. Security teams should expect AI-assisted tools to prove what they reviewed, not merely what they concluded. Without that evidence, a strong-looking finding set is still vulnerable to blind spots that only surface after independent review.

For practitioners

• Define coverage as a required control objective Measure whether the scanner enumerates the same endpoints, sinks, and slices on repeated runs, and treat unexplored code as unassessed risk rather than a clean result.
• Track cost per true positive in evaluations Compare tools on validated findings per dollar, not token spend or scan count, so procurement reflects actual risk reduction instead of superficial efficiency.
• Separate candidate discovery from reasoning Require the platform to show how deterministic analysis selects the code sent to the model, because coverage should not depend on whatever the model happens to notice.
• Validate AI findings against independent review Use pentest or expert review on a sample of the same codebase to check for endpoints the model never surfaced, especially in large repositories with many routes.
• Set governance rules for non-deterministic outputs Require repeatability thresholds and auditability for security findings so teams can explain why a vulnerability was reported on one run and not another.

Key takeaways

• AI vulnerability detection depends on the workflow around the model as much as the model itself, because coverage determines what can be found.
• A scanner that is precise but under-covers the repository can still miss exploitable code, which makes recall and enumeration central governance concerns.
• Security teams should evaluate AI-assisted detection by coverage, repeatability, and cost per true positive so that findings reflect real risk reduction.

Key terms

• Harness Design: The harness design is the orchestration layer around an AI model that decides what context it sees, how it is guided, and how outputs are filtered. In security detection, it often matters more than model choice because it governs coverage, repeatability, and the path from raw code to actionable findings.
• Recall: The share of real weaknesses that the scanner successfully identifies. High recall reduces the chance of missed vulnerabilities, but if it is achieved by over-reporting, teams can become overwhelmed and stop trusting the results.
• Precision: The share of reported findings that are actually real weaknesses. High precision means fewer false positives, but it does not guarantee that the scanner is finding all of the important problems, so it must be read alongside recall to understand practical effectiveness.
• Cost per true positive: Cost per true positive measures how much it costs to produce one real, actionable finding. It is a better operational metric than scan price because it combines economic efficiency with detection quality, which is what matters when a security team has to decide whether a workflow is worth running at scale.

Deepen your knowledge

NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course. It is designed for teams that need to govern access, trust, and lifecycle risk across modern security programmes.