Notifications
Clear all
Topic starter
02/08/2026 11:24 am
TL;DR: A benchmark finds that a multimodal approach detects 3.5x more true positives than a tuned guided prompt at similar precision and lower cost per finding, while a raw model can miss up to 90% of endpoints in a repository, according to Semgrep. The result is a reminder that coverage and orchestration matter more than model novelty for usable security detection.
NHIMG editorial — based on content published by Semgrep: benchmark analysis of AI vulnerability detection with multimodal harnessing
By the numbers:
- The benchmark shows a cost of $0.62 per true positive for Semgrep Multimodal, compared with $0.77 for the guided prompt.
- Claude Security performs with roughly the same precision and recall as the guided prompt, but at $72.55 per true positive.
- 90% of the endpoints in a repository when, 90% of the endpoints in a repository when asked to search on its own.
Questions worth separating out
Q: How should security teams evaluate AI tools for code vulnerability discovery?
A: Use a fixed vulnerable corpus, repeat each test several times, and score the union of findings as well as the average single-pass result.
Q: Why do raw LLMs miss so many vulnerabilities in code review?
A: Raw LLMs are limited by probabilistic attention and incomplete repository search.
Q: What do security teams get wrong about AI scan pricing?
A: Teams often look at price per scan, but that ignores whether the scan finds actionable issues.
Practitioner guidance
- Test coverage before trust: Require evidence that the detection workflow enumerates endpoints and sinks deterministically before any model output is accepted into triage.
- Benchmark on actionable findings: Measure tools using cost per true positive, precision, and recall on code your team actually owns, then compare those results against developer review capacity.
- Separate search from reasoning: Design the workflow so static analysis selects candidate code regions and the model reasons only within those slices.
What's in the full article
Semgrep's full article covers the operational detail this post intentionally leaves for the source:
- Benchmark methodology across open-source repositories with established ground truth
- Per-model comparison tables for Opus 4.8 and GPT 5.5 under different prompting and harness conditions
- How candidate generation, program slices, and incremental analysis are orchestrated in practice
- The earlier architecture discussion that explains why deterministic analysis changes coverage
👉 Read Semgrep's benchmark analysis of AI vulnerability detection and model harnessing →
AI vulnerability detection: is the harness doing the real work?
Explore further
02/08/2026 12:41 pm
Coverage debt is the central failure mode in AI-assisted AppSec. The article shows that a model can produce sharp reasoning and still miss most of the repository if it does not first discover the right endpoints. That is a governance problem, not a model problem. For AppSec teams, the lesson is that visibility into what was examined is as important as the findings themselves, because unexamined code is ungoverned code.
A question worth separating out:
Q: How can AppSec teams make AI findings consistent enough to use?
A: They should reduce variability by constraining what the model sees and by using repeatable candidate selection. Consistency comes from a stable harness, not from hoping the model behaves the same way every time. If the same code produces different results across runs, teams cannot build reliable triage, reporting, or remediation workflows.
👉 Read our full editorial: AI vulnerability detection depends more on harness design than model choice
