Open-weight models are closing the gap in IDOR detection

At a glance

What this is: Semgrep’s benchmark shows that an open-weight model can outperform a frontier coding agent on IDOR detection when both receive the same prompt.

Why it matters: This matters because identity and access review workflows increasingly rely on AI-assisted analysis, and the quality of the surrounding harness can shape whether those tools actually find authorization failures.

By the numbers:

• GLM 5.2 scored a 39% F1 on IDOR detection, beating Claude Code at roughly $0.17 per vulnerability found.
• Semgrep Multimodal reached 61% F1 in the benchmark, with another configuration at 53% F1.

👉 Read Semgrep’s benchmark analysis of open-weight models and IDOR detection

Context

IDOR, or Insecure Direct Object Reference, is an access control failure, not a syntax bug. It appears when an application exposes an object identifier but does not verify that the requester is allowed to access that object, which makes it a useful test case for AI-assisted security analysis and for broader identity governance thinking.

The article’s real question is how much detection quality comes from the model versus the harness around it. That distinction matters for IAM and NHI programmes because analysis tooling is increasingly delegated to AI systems, yet the surrounding workflow still determines whether the system sees enough context to reason about authorisation boundaries.

Key questions

Q: How should security teams evaluate AI tools for IDOR detection?

A: Evaluate the full workflow, not just the model. A useful tool should show how it discovers endpoints, gathers context, and reasons about object ownership checks. Compare precision, recall, and cost per true positive on the same codebase so you can see whether the harness or the model is doing the real work.

Q: Why do access-control failures remain hard for LLM-based security analysis?

A: Because the flaw is usually a missing entitlement check, not an obviously dangerous API call. The evidence is often spread across routes, framework code, and request handling, so the model needs enough surrounding context to infer who should own the object. Without that context, it can miss the governance failure entirely.

Q: What breaks when endpoint discovery is missing from an AI security harness?

A: The model may read code without seeing the paths that matter most for authorisation review. That leads to blind spots around route mapping, object lookup, and ownership validation, which are central to IDOR detection. In practice, the tool becomes a code reader instead of an access-control analyst.

Q: How do teams decide when a lower-cost model is good enough for security scanning?

A: Use task-specific evidence, not price alone. If a cheaper model maintains acceptable precision and recall on the organisation’s own access-control patterns, it may be good enough for broad scanning. Keep human review for high-impact findings and use cost per finding to decide where automation should stop.

Technical breakdown

Why harness design changes IDOR detection outcomes

A harness is the orchestration layer around a model. In security workflows it can enumerate endpoints, curate repository context, guide search, and loop the model through a task until it reaches a result. That matters because IDORs are often missing-check problems, not obvious vulnerable-function patterns, so a model given raw code may miss the business logic that reveals an access control flaw. A purpose-built harness reduces that blind spot by shaping the search space before the model reasons over it.

Practical implication: evaluate AI security tools as a system, not just as a model, and measure whether the harness improves authorisation coverage.

Why IDOR is hard for both static analysis and LLMs

IDOR sits between application security and identity governance. The flaw is not that a user can reach a function, but that the function fails to verify object ownership or entitlement before returning data. Static analysis struggles because there is no universal dangerous sink to flag, while LLMs struggle because the missing authorisation check is often distributed across files, frameworks, and request flows. That is why endpoint discovery, route mapping, and context gathering matter so much in evaluation.

Practical implication: prioritise control-path discovery and ownership checks when testing tools for access-control findings.

Why cost per finding now matters in security AI

Security teams do not buy detection quality in isolation. They buy throughput, repeatability, and economics at scale. A model that performs acceptably but costs materially less per true positive can be more practical for large codebases, many services, or repeated scans. The benchmark’s cost-per-vulnerability figure shows why token economics and run structure increasingly influence whether AI-assisted security analysis is usable operationally.

Practical implication: track cost per true positive alongside precision and recall before standardising an AI-assisted review workflow.

Threat narrative

Attacker objective: The attacker wants to access another user’s data or objects without possessing the corresponding authorisation.

1. Entry occurs when a developer or application path exposes an object identifier in a request without validating ownership or entitlement.
2. Credential or access misuse follows when the caller changes the identifier and reuses a legitimate session to reach another user’s object.
3. Impact occurs when the missing check returns unauthorised records, creating data exposure and downstream privilege misuse potential.

NHI Mgmt Group analysis

Harness design is becoming an identity control point, not just a tooling detail. When AI systems are used to find access-control flaws, the orchestration layer determines what the model can actually inspect, which directly affects whether authorisation gaps are discovered or missed. That means AI-assisted security workflows now sit inside the governance boundary for identity and access analysis, not outside it. Practitioners should treat harness design as part of the control environment.

Access-control analysis is where application security and identity governance overlap most clearly. IDOR is a simple example of a deeper problem: systems can authenticate a user and still fail to prove that the user is entitled to the object being requested. That makes authorisation review a first-class identity discipline, not just an AppSec concern. Security teams should expect AI tools to help with discovery, but not to replace entitlement logic or ownership checks.

Open-weight models are changing the economics of security analysis, but not the governance model. The article shows that cost and deployment flexibility are now competitive variables, especially where sensitive repositories or code paths cannot leave the enterprise environment. That is relevant to NHI and agentic AI programmes because the same question applies to AI agents operating on internal systems: where the model runs matters, but so does what it is allowed to see and do. Practitioners should separate deployment convenience from control assurance.

Precision in security AI should be judged on scoped outcomes, not headline benchmark scores. An IDOR benchmark rewards endpoint discovery, context selection, and correct interpretation of ownership checks, which are all task-specific capabilities. A higher score on one dataset does not guarantee reliable access-control analysis across services, frameworks, or data shapes. Teams should demand evidence on their own code patterns before trusting AI output for governance decisions.

Cost discipline is now part of the security architecture conversation. If two systems produce useful findings but one does it at materially lower cost per true positive, the lower-cost path may be the only one that scales across large portfolios. That is especially relevant where identity review, code scanning, and agent-assisted triage are all competing for budget. Practitioners should evaluate unit economics as a control sustainability issue.

What this signals

Access-control review is becoming a machine-assisted governance function. As teams use AI to inspect code and find object-reference flaws, they need to decide which parts of the analysis can be automated and which still require human judgement. The practical signal is that identity governance will increasingly depend on the quality of analysis orchestration, not just the scanner itself.

For programmes that already struggle with secrets, object ownership, and delegated access, the bigger issue is not whether an AI model can spot one vulnerability class. It is whether the organisation can prove that its detection pipeline sees enough of the system to make authorisation findings reliable. That means measurement, scoping, and reviewer oversight become part of the control story.

For practitioners

• Define harness requirements before comparing models Require endpoint enumeration, repository scoping, and output parsing as explicit capabilities in any AI-assisted access-control review workflow. A bare prompt is not equivalent to a guided analysis system.
• Test AI tools against your own authorisation patterns Benchmark detection on the frameworks, route structures, and object ownership conventions actually used in your codebase. IDOR performance is highly pattern dependent, so generic scores can mislead.
• Measure cost per true positive alongside F1 Track spend divided by real findings, not just aggregate model cost. For large scanning runs, per-vulnerability economics often decide whether an AI-assisted workflow can be adopted consistently.
• Keep human review on ownership-sensitive findings Route AI-generated IDOR candidates through a reviewer who can validate whether the request path, object lookup, and entitlement check actually line up. That prevents noisy results from becoming false confidence.

Key takeaways

• IDOR remains an access-control problem, which makes it relevant to both AppSec and identity governance teams.
• The benchmark shows that harness design can matter more than model brand when AI is asked to find authorisation failures.
• Operational adoption will depend on cost per true positive, scoped evaluation, and human review for high-impact access findings.

Key terms

• Insecure Direct Object Reference: Insecure direct object reference is an access control flaw where an application exposes an identifier that allows callers to reach records or objects they should not be able to access. In practice, it means the system trusts the request too much and fails to verify object-level permission on each access.
• Harness: The harness is the layer of instructions, policies, and approval logic wrapped around an AI agent. It is where organisations try to constrain behaviour, but it only works if the rules are explicit, current, and enforced outside the model itself.
• F1 Score: A combined metric that balances precision and recall in one number. In security scanning, it shows whether a tool is both finding real issues and avoiding excessive noise, which makes it more useful than either metric alone when comparing performance on real code.

What's in the full report

Semgrep’s full analysis covers the operational detail this post intentionally leaves for the source:

• The complete benchmark table with model-by-model F1, precision, recall, and cost comparisons
• The exact harness differences between Semgrep Multimodal and the prompt-only runs
• The prompt strategy, endpoint-discovery workflow, and scoring methodology used in the experiment
• The grounding audit update explaining whether the models were reasoning or pattern-matching

👉 Semgrep’s full post covers the benchmark setup, harness differences, and model-by-model results

Deepen your knowledge

NHI Mgmt Group’s NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps security practitioners connect identity control design to the operational realities of modern security programmes.