Topic starter
02/08/2026 9:46 am
TL;DR: Frontier and open-source models can be grounded on IDOR findings, according to Semgrep’s benchmark, but all models still miss the same hard majority of vulnerabilities, with per-scan recall around 15% for most systems and only modest gains from ensembling. The lesson is that reasoning quality matters, but coverage gaps still define operational risk in AI-assisted code security.
Editorial analysis by NHI Mgmt Group, based on content published by Semgrep: “Grounded or Gamed? We Audited Our Own Cyber Benchmark”.
By the numbers:
- Outside of Semgrep Multimodal, all models achieve around 15% per-scan recall on IDOR detection.
- Stacking five different models buys exactly one additional vulnerability, with union recall at 29.6%.
Key questions
Q: Where do AI code scanners fail in practice on IDOR vulnerabilities?
A: They fail most often when authorization is distributed across multiple request steps, resource lookups, or downstream services.
Q: How should security teams evaluate groundedness in AI-assisted code review?
A: Use groundedness to test whether a finding survives patching, identifier renaming, and safe look-alikes, but do not treat that as a substitute for recall.
Q: What breaks when a model only recognises textbook IDOR patterns?
A: It misses structurally similar vulnerabilities that do not match the training shape, such as document, prompt, or tool endpoints with the same missing ownership check.
Practitioner guidance
- Measure recall on hard authorization cases Test AI-assisted code review against deep and unfamiliar IDOR paths, not only file-download examples, and compare results across repeated scans.
- Separate grounded findings from coverage claims Track whether a model’s detections survive patching, renaming, and safe look-alikes, then review whether that same model still misses structurally similar bugs.
- Use AI as a triage layer, not the control plane Keep human review and targeted authorization testing in the loop for endpoints where object ownership and request-path logic determine access.
Bottom line: AI-assisted IDOR detection can be grounded without being comprehensive, so reasoning quality and coverage must be measured separately.
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
02/08/2026 12:38 pm
Groundedness is not the control teams think they are buying. This benchmark shows that a model can be consistent, selective, and still operationally incomplete. For security governance, that means the important question is not whether the model sounds reasoned, but whether it materially improves discovery of authorization failures in code review.
A few things that frame the scale:
- 43% of security professionals are concerned about AI systems learning and reproducing sensitive information patterns from codebases, according to the State of Secrets in AppSec.
- AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
A question worth separating out:
Q: Should teams rely on ensembles to close AI detection gaps?
A: Not by default. Ensembles help only when different systems have genuinely different blind spots, but this benchmark shows many models miss the same hard cases. Teams should validate whether a second or third model adds unique findings before assuming coverage meaningfully improves.
👉 Read our full editorial: LLM grounding beats F1, but IDOR recall still hits a wall