Notifications
Clear all
Topic starter
02/08/2026 9:46 am
TL;DR: Frontier and open-source models can be grounded on IDOR findings, according to Semgrep’s benchmark, but all models still miss the same hard majority of vulnerabilities, with per-scan recall around 15% for most systems and only modest gains from ensembling. The lesson is that reasoning quality matters, but coverage gaps still define operational risk in AI-assisted code security.
NHIMG editorial — based on content published by Semgrep: LLM groundedness, F1, and IDOR detection benchmarking
By the numbers:
- Of the 27 real IDORs in this repo, 19 were missed by every model.
- When AWS credentials are exposed publicly, attackers attempt access within an average of 17 minutes.
- 98% of companies plan to deploy even more AI agents within the next 12 months, despite documented rogue behaviour in 80% of current deployments.
Questions worth separating out
Q: Where do AI code scanners fail in practice on IDOR vulnerabilities?
A: They fail most often when authorization is distributed across multiple request steps, resource lookups, or downstream services.
Q: Why do grounded findings matter more than raw F1 scores for security tools?
A: Because F1 can look strong even when a model is arriving at the right answer for the wrong reason.
Q: How do security teams know whether an AI-assisted scanner is trustworthy?
A: Look for repeatability, sensitivity to small code changes, and consistent discrimination between real vulnerabilities and safe look-alikes.
Practitioner guidance
- Test AI scanners for groundedness, not just accuracy. Run counterfactual, metamorphic, selectivity, and stability checks on the same codebase so you can see whether findings survive patching, renaming, and repeated scans.
- Map authorization paths before trusting model output. Inventory every multi-hop access decision where object ownership, ACL checks, or downstream validation are split across services.
- Pair AI-assisted review with identity governance controls. For applications that expose user-scoped resources, tie scanning results to ownership enforcement, entitlement reviews, and change control so a missed finding does not become a silent access gap.
What's in the full report
Semgrep's full analysis covers the benchmarking details this post intentionally leaves for the source:
- The exact per-model grounding results across counterfactual, metamorphic, selectivity, and stability tests.
- Repository-specific scan behaviour that shows where open-weight and frontier models diverge on IDOR detection.
- The full method for validating whether repeated scans are stable enough to use in production triage.
- A second benchmark direction on SSRF grounding that extends the same reasoning test to another vulnerability class.
👉 Read Semgrep's analysis of groundedness, F1, and IDOR detection →
IDOR detection benchmarks: what groundedness changes for security teams?
Explore further
02/08/2026 12:38 pm
Groundedness is now a baseline requirement for AI security tooling, not a research curiosity. A scanner that cannot distinguish real authorization failures from surface similarity is not fit for operational use. This benchmark shows why reasoning quality matters separately from headline precision and recall, especially in code security where false confidence can be worse than a missed alert. Security teams should treat groundedness as a control property, not a nice-to-have metric.
A question worth separating out:
Q: What does weak IDOR coverage mean for code security programmes?
A: It means the organisation may believe it has coverage because the tool finds obvious cases, while deeper authorization flaws remain untouched. That creates a false sense of control, especially in systems with layered handlers and mixed resource types. Programmes should combine AI review with manual threat modelling and targeted access-control testing.
👉 Read our full editorial: LLM grounding beats F1, but IDOR recall still hits a wall
