Topic starter
02/08/2026 11:45 am
TL;DR: None of five tested models reliably identified two vulnerabilities from the Mythos post when asked to inspect whole files, while narrowing scope to a function improved results but still left false positives and misses, according to Semgrep. The evidence suggests security teams should treat LLMs as targeted assistants, not standalone discovery engines.
Editorial analysis by NHI Mgmt Group, based on content published by Semgrep: “Needles and haystacks: Can open-source & flagship models do what Mythos did?”.
Key questions
Q: Why do whole-file LLM prompts fail more often for vulnerability discovery?
A: Whole-file prompts force the model to infer the bug, the relevant code path, and the preconditions from too much context at once.
Q: When should teams prioritise hotspot analysis over broad LLM scanning?
A: Prioritise hotspot analysis when the goal is finding likely vulnerabilities rather than summarising a codebase.
Q: How can security teams tell whether LLM evaluation is actually working?
A: Look for continuous coverage across live traffic, regression testing on new prompts or models, and sampled human review of judge outputs.
Practitioner guidance
- Define scope before model invocation Use function-level or hotspot-level prompts for vulnerability discovery, and reserve whole-file prompting for broad triage only when you can tolerate lower precision.
- Pair LLMs with deterministic pre-filtering Use code search, static rules, or pattern filters to identify suspicious regions before asking a model to assess exploitability or root cause.
- Set validation thresholds for AI findings Require human verification, repeatability checks, and a documented false-positive tolerance before an LLM finding enters the remediation workflow.
Bottom line: Whole-file LLM vulnerability discovery still underperforms because the model is being asked to infer too much from too much context at once.
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
02/08/2026 12:56 pm
Whole-file prompting is not a governance model for secure code review: it is a search strategy with known failure modes. The article demonstrates that discovery degrades when the model must infer the bug from too much context, which means teams cannot treat a broad prompt as a control. Practitioners should see this as a workflow design problem, not a model marketing problem.
A few things that frame the scale:
- Only 5.7% of organisations have full visibility into their service accounts, according to the Ultimate Guide to NHIs.
A question worth separating out:
Q: What is the difference between LLM-assisted triage and autonomous vulnerability discovery?
A: LLM-assisted triage narrows attention, explains suspicious code, and helps humans review faster. Autonomous discovery implies the model can reliably find the bug, justify the finding, and do so with stable precision across runs. The article shows the first use case is realistic today, while the second still breaks down on whole files.
👉 Read our full editorial: LLM-assisted vulnerability discovery still fails on whole files