Topic starter
02/08/2026 11:24 am
TL;DR: A benchmark finds that a multimodal approach detects 3.5x more true positives than a tuned guided prompt at similar precision and lower cost per finding, while a raw model can miss up to 90% of endpoints in a repository, according to Semgrep. The result is a reminder that coverage and orchestration matter more than model novelty for usable security detection.
Editorial analysis by NHI Mgmt Group, based on content published by Semgrep: “3.5x more true positives: How we benchmark AI-powered detection”.
Key questions
Q: What breaks when AI vulnerability scanners do not enumerate the full codebase?
A: They can only reason about the code slices they are shown, so vulnerabilities in unnoticed endpoints never enter the finding set.
Q: Why does harness design matter more than model choice for AI code scanning?
A: Because the harness decides what the model sees, how much context it gets, and whether the scan is repeatable.
Q: What are the signs that an AI security scanner is under-covering a repository?
A: The clearest signal is a finding set that looks plausible but remains small relative to the size and complexity of the codebase, especially when later manual review finds vulnerabilities in untouched endpoints.
Practitioner guidance
- Define coverage as a required control objective Measure whether the scanner enumerates the same endpoints, sinks, and slices on repeated runs, and treat unexplored code as unassessed risk rather than a clean result.
- Track cost per true positive in evaluations Compare tools on validated findings per dollar, not token spend or scan count, so procurement reflects actual risk reduction instead of superficial efficiency.
- Separate candidate discovery from reasoning Require the platform to show how deterministic analysis selects the code sent to the model, because coverage should not depend on whatever the model happens to notice.
Bottom line: AI vulnerability detection depends on the workflow around the model as much as the model itself, because coverage determines what can be found.
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
02/08/2026 12:41 pm
Harness design is now the primary control surface for AI-assisted vulnerability detection. The article’s central lesson is that model capability matters less than the workflow that constrains where the model looks and what evidence it receives. In security terms, the control failure is not weak reasoning alone but incomplete coverage architecture. Practitioners should treat orchestration quality as part of the security control, not as implementation detail.
A question worth separating out:
Q: How should teams compare AI-assisted detection against guided prompting?
A: Compare them on repeatable recall, precision, and cost per true positive using the same ground-truth codebase. Guided prompts can look competitive on a narrow sample, but they often fail when coverage is the bottleneck. The better question is which workflow finds more real issues across the full repository without inflating noise or cost.
👉 Read our full editorial: AI vulnerability detection depends more on harness design than model choice