LLM grounding beats F1, but IDOR recall still hits a wall

At a glance

What this is: Semgrep’s benchmark finds that LLMs can reason soundly about IDORs they detect, but they still miss most real vulnerabilities and ensembling delivers only modest gains.

Why it matters: Security teams evaluating AI-assisted code review should separate reasoning quality from detection coverage, because grounded findings are not the same as reliable vulnerability discovery.

By the numbers:

• Outside of Semgrep Multimodal, all models achieve around 15% per-scan recall on IDOR detection.
• Stacking five different models buys exactly one additional vulnerability, with union recall at 29.6%.

Context

IDOR remains a classic authorization failure, but benchmarked AI assistants expose a second problem: the model may reason correctly about a vulnerability and still fail to find most instances that matter in production. In code review, that distinction matters because grounded answers do not automatically translate into broad coverage across unfamiliar handlers, nested resource checks, or non-textbook patterns.

The article’s security relevance sits at the intersection of application security and AI-assisted analysis. For IAM practitioners, the connective tissue is authorization logic: if the review system cannot reliably surface missing ownership checks, the control gap remains regardless of how fluent the model sounds. That makes this less about LLM quality in the abstract and more about whether AI can safely augment real code governance.

Semgrep’s results also show a familiar trap in AI security programmes: optimising for benchmark quality can hide the operational difference between a believable finding and a complete one. That is a typical failure mode for teams adopting AI-assisted triage too quickly.

Key questions

Q: Where do AI code scanners fail in practice on IDOR vulnerabilities?

A: They fail most often when authorization is distributed across multiple request steps, resource lookups, or downstream services. A scanner may recognise a familiar unsafe pattern, but still miss the deeper ownership logic that should block access. That means teams need to review not just the vulnerable line, but the full control path from request to resource.

Q: How should security teams evaluate groundedness in AI-assisted code review?

A: Use groundedness to test whether a finding survives patching, identifier renaming, and safe look-alikes, but do not treat that as a substitute for recall. A grounded system can still miss most defects, so teams need separate measurements for reasoning quality and vulnerability coverage.

Q: What breaks when a model only recognises textbook IDOR patterns?

A: It misses structurally similar vulnerabilities that do not match the training shape, such as document, prompt, or tool endpoints with the same missing ownership check. The result is selective confidence in familiar code and blind spots everywhere else, which is dangerous in production review workflows.

Q: Should teams rely on ensembles to close AI detection gaps?

A: Not by default. Ensembles help only when different systems have genuinely different blind spots, but this benchmark shows many models miss the same hard cases. Teams should validate whether a second or third model adds unique findings before assuming coverage meaningfully improves.

Technical breakdown

Why groundedness and recall measure different things in IDOR detection

Groundedness asks whether a model’s finding survives perturbation: if you patch the bug, rename symbols, or introduce safe look-alikes, does the finding still behave like a real vulnerability? Recall asks a harder question: how many real vulnerabilities did the model find in the first place? In this benchmark, those are separate axes. A model can be logically consistent on the issues it catches while still missing deep authorization flaws or less familiar code shapes. That split is why F1 can look healthy while operational coverage remains poor.

Practical implication: Treat groundedness as a quality signal, but never as proof that an AI review system has sufficient vulnerability coverage.

Why IDOR is hard for models that recognise textbook patterns

IDOR detection depends on recognising missing object-level authorization, often after a request traverses several functions or handlers. The article shows two recurring blind spots: depth, where the ownership check is buried downstream, and familiarity, where models recognise file download examples but miss structurally similar document or tool endpoints. That means the model may be detecting a learned pattern rather than generalising the security rule. The problem is not just syntax, but whether the model can infer where authorization ought to exist across a request path.

Practical implication: Use code-path review and authorization mapping to test whether the model is actually generalising beyond textbook examples.

Why ensembling improves coverage only a little

Ensembling helps when systems fail for different reasons, but the article shows a shared ceiling: multiple models mostly miss the same hard IDORs. That indicates overlapping blind spots rather than complementary strengths. A small lift in union recall is useful, but it does not change the underlying fact that deep and unfamiliar authorization bugs remain invisible to most scans. In practical terms, the benchmark suggests the field is hitting an evidence problem, not a scoring problem.

Practical implication: Do not assume more models automatically equal materially better vulnerability discovery; validate whether each model adds unique coverage.

NHI Mgmt Group analysis

Groundedness is not the control teams think they are buying. This benchmark shows that a model can be consistent, selective, and still operationally incomplete. For security governance, that means the important question is not whether the model sounds reasoned, but whether it materially improves discovery of authorization failures in code review.

IDOR is a reminder that AI inherits the same governance blind spots as human review, only faster. If reviewers only recognise canonical patterns, then the system detects the vulnerabilities that resemble training examples and misses the rest. That is a governance problem, not a model branding problem, because the failure mode is incomplete authorization coverage.

The recall ceiling is the real category limit. When five models converge on the same missed findings, the issue is not evaluator bias or a weak benchmark. It is shared blindness to deep and unfamiliar authorization paths, which means practitioners should judge AI-assisted analysis by marginal coverage, not headline F1.

Benchmark quality matters because code-security AI can still game the appearance of competence. The article’s counterfactual and selectivity checks matter precisely because they reduce the chance that a system is merely pattern matching. For IAM and application security teams, this is the difference between a tool that finds familiar IDORs and one that can support governance over real access-control risk.

Grounded detection still leaves a trust gap between model output and security decision-making. A finding that survives perturbation is useful, but only if the system can also surface the hard majority of unsafe access paths. Practitioners should treat AI review as an augmenting signal, not a substitute for authorization design and targeted manual verification.

From our research library:

• 43% of security professionals are concerned about AI systems learning and reproducing sensitive information patterns from codebases, according to the State of Secrets in AppSec.
• AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
• Read next: AI Coding Agents Security Guide

What this signals

Groundedness improves trust in the findings a model does produce, but it does not solve the coverage problem that governs real risk. Security teams should expect AI-assisted review to behave like a narrow detector unless they prove otherwise with repeated scans on difficult authorization paths. The operational metric that matters is not how convincing the answer sounds, but how many materially different bugs it actually finds.

IDOR exposes a broader benchmark reality for AI security programmes: a system can be logically consistent and still leave most of the attack surface uninspected. That makes recall testing, confidence calibration, and endpoint-specific validation central to governance of AI-assisted code analysis. The same logic applies to other AI review use cases, especially where access control is distributed across multiple handlers.

Model grounding will not close the gap if the surrounding security programme does not change its review design. 43% of security professionals are concerned about AI systems learning and reproducing sensitive information patterns from codebases, according to the State of Secrets in AppSec. That concern is relevant here because AI can sound precise while still missing the access-control logic that matters most.

For practitioners

• Measure recall on hard authorization cases Test AI-assisted code review against deep and unfamiliar IDOR paths, not only file-download examples, and compare results across repeated scans.
• Separate grounded findings from coverage claims Track whether a model’s detections survive patching, renaming, and safe look-alikes, then review whether that same model still misses structurally similar bugs.
• Use AI as a triage layer, not the control plane Keep human review and targeted authorization testing in the loop for endpoints where object ownership and request-path logic determine access.
• Validate ensemble gains with unique findings Compare union recall against the single best model and check whether extra systems actually find different vulnerabilities or simply repeat the same partial coverage.

Key takeaways

• AI-assisted IDOR detection can be grounded without being comprehensive, so reasoning quality and coverage must be measured separately.
• The benchmark shows that most models still miss the same hard authorization bugs, which is why recall remains the operational constraint.
• Security teams should treat AI scanners as augmentation for code review, not as a substitute for authorization analysis and manual verification.

Key terms

• Groundedness: Groundedness is the degree to which an AI response can be supported by verifiable source material. In practice, it measures whether the model answered from evidence rather than inference, memory, or fabrication, which is critical for RAG systems and any workflow that drives decisions from model output.
• IDOR: Insecure Direct Object Reference is a vulnerability where an application exposes internal object identifiers and fails to verify whether the requesting principal is entitled to access them. The flaw often produces unauthorized reads, writes, or account-level actions.
• Recall: The share of real weaknesses that the scanner successfully identifies. High recall reduces the chance of missed vulnerabilities, but if it is achieved by over-reporting, teams can become overwhelmed and stop trusting the results.
• Selectivity: Selectivity is the ability of a detection system to avoid flagging safe code that merely resembles a vulnerability. It is a useful measure because real application logic often contains look-alike paths, and weak selectivity creates false positives that reduce trust in the tool.

Deepen your knowledge

NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course. It is a practical option for teams that need a stronger basis in identity control design, review, and governance.