Notifications
Clear all
Topic starter
02/08/2026 11:45 am
TL;DR: None of five tested models reliably identified two vulnerabilities from the Mythos post when asked to inspect whole files, while narrowing scope to a function improved results but still left false positives and misses, according to Semgrep. The evidence suggests security teams should treat LLMs as targeted assistants, not standalone discovery engines.
NHIMG editorial — based on content published by Semgrep: LLM-assisted vulnerability discovery in Mythos benchmark testing
Questions worth separating out
Q: How should security teams use LLMs in vulnerability research without overtrusting them?
A: Use LLMs as structured reasoning aids, not as final arbiters of exploitability.
Q: Why do whole-file AI scans miss vulnerabilities more often than focused reviews?
A: Whole-file scans force the model to solve an open-ended discovery problem across too much context.
Q: What do security teams get wrong about model accuracy in code analysis?
A: They often treat a partially correct answer as proof of capability.
Practitioner guidance
- Use hotspot-first triage before model review Feed the model only code that deterministic scanners, taint analysis, or static rules have already flagged as suspicious.
- Require human validation for primary findings Treat model output as a candidate signal until a reviewer confirms the vulnerable function, preconditions, and plausible exploit path.
- Measure false positives and false negatives separately Track whether the model identifies the right function, the right mechanism, and the right exploit conditions.
What's in the full report
Semgrep's full post covers the experimental detail this analysis intentionally leaves for the source:
- Per-model result tables showing how each model performed across full-file and sliced-function tests.
- Per-iteration breakdowns that expose where models corrected themselves, missed the right function, or produced null responses.
- Appendix score definitions that explain how the team distinguished full, partial, broad, secondary, miss, and null outcomes.
- Operational comparison of naive whole-file prompting versus hotspot interrogator workflows with deterministic pre-filtering.
👉 Read Semgrep's analysis of LLM-assisted vulnerability discovery results →
Whole-file vulnerability discovery with LLMs: where do teams draw the line?
Explore further
02/08/2026 12:56 pm
LLM-assisted vulnerability discovery is a triage problem before it is a detection problem. Security teams are already using models to reduce review load, but this article shows that broad discovery remains unreliable without strong pre-filtering. That matters because code-security workflows often sit upstream of identity risk, secret leakage, and trust boundary failures in delivery pipelines. Practitioners should treat AI output as a prioritisation signal, not a finding.
A question worth separating out:
Q: How do teams decide whether model-assisted review is good enough for production use?
A: Judge it by precision, recall, consistency, and reviewer time saved on real code samples. If the model needs repeated runs, heavy prompting, or extensive correction to find the same bug, then it is useful for assistance but not yet reliable as a standalone control.
👉 Read our full editorial: LLM-assisted vulnerability discovery still fails on whole files
