Open-weight models are closing the gap in IDOR detection

At a glance

What this is: Semgrep’s benchmark shows an open-weight model narrowing the gap on IDOR detection, but the surrounding harness still produces the best security-task results.

Why it matters: IAM and application security teams should treat model selection and harness design as separate governance decisions, because access-control findings depend as much on context routing as on model output.

By the numbers:

• GLM 5.2 scored 39% F1 on IDOR detection, beating Claude Code at 32% in Semgrep’s benchmark.
• Semgrep’s multimodal pipeline led the benchmark with 53% to 61% F1.

Context

IDOR detection is a test of whether a system can spot missing object-level access checks, not just recognise suspicious code patterns. That makes it a useful proxy for how security tooling handles application authorization problems where the control gap is subtle and context dependent.

The article’s real question is governance as much as model quality: how much detection performance comes from the model itself, and how much comes from the harness that decides what the model sees, what context is added, and how results are interpreted. That distinction matters wherever AI is being used to support security analysis, because the workflow around the model can dominate the outcome.

Key questions

Q: What breaks when IDOR detection is run without endpoint discovery?

A: Detection quality drops because the model sees code fragments without the request paths, ownership checks, or routing context needed to spot missing authorization. In practice, that turns a policy-verification problem into a pattern-matching exercise, which is why harness design can matter more than the model name.

Q: Why do models miss IDOR issues even when they can read the code?

A: They often lack the surrounding context that proves whether an object reference is actually authorised. A bare model may identify the route or parameter, but it still has to infer entitlement, session state, and ownership from incomplete evidence. That is why model output alone is not a reliable control.

Q: How should security teams evaluate AI agent authorization tools?

A: Score every tool on whether it enforces policy before execution, covers all relevant domains, makes decisions with runtime context, and can prove the basis for each decision. If the product only detects activity or needs manual workflows to change permissions, it is not giving you full authorization control. The safest test is a live denied action, not a feature checklist.

Q: What is the difference between model capability and harness engineering in agentic security tools?

A: Model capability is the ability to reason about a task and generate outputs. Harness engineering is the surrounding control layer that turns those outputs into safe, observable, and reproducible actions. In practice, the harness decides whether the agent is a useful security tool or just an unsafe automation experiment.

Technical breakdown

Why IDOR is hard for static analysis and LLMs

Insecure Direct Object Reference is an authorization failure, not a syntax error or a taint-flow mistake. The application exposes an object identifier, such as a user ID, and fails to verify whether the caller is entitled to access it. That means detection depends on understanding request context, ownership checks, and data-flow across code paths. Models that only inspect local patterns often miss the absence of an authorization gate because there is no obvious dangerous function to flag. For AI-assisted AppSec, this is exactly the kind of problem where narrow prompting is weakest.

Practical implication: Treat IDOR detection as an authorization-analysis problem, not a generic code-scanning task.

How the harness changes model behaviour

A harness is the orchestration layer around the model. It can enumerate endpoints, select relevant files, add search strategy, parse outputs, and loop the model through a task until it converges on a finding. In this article, Semgrep’s multimodal pipeline used a purpose-built harness that pointed the model directly at application endpoints and structured context, while the open-weight runs used a simpler prompt-only setup. That difference matters because a security model rarely fails only on reasoning quality. It also fails when it is shown too little of the codebase, too late in the workflow, or without the context needed to rank evidence.

Practical implication: Evaluate AI security tooling by the quality of its task harness, not by model output alone.

Why open-weight models can still be operationally attractive

Open-weight models are not the same as open source, but they can be run inside a customer-controlled environment and tuned to sensitive workflows. For security teams, that changes the governance profile: data locality, controllability, and deployment flexibility become part of the risk discussion, not just benchmark score. The article also shows that open-weight performance is no longer trivial on narrow security tasks, even if the best results still came from a more structured pipeline. The key lesson is not that open weights replace frontier models, but that they are now credible options in controlled security workflows.

Practical implication: Assess open-weight models for sensitive security tasks where local deployment and controllable context matter.

Threat narrative

Attacker objective: The attacker aims to reach data or functions reserved for other users by changing a reference and bypassing object-level authorization.

1. Entry occurs when an application exposes object identifiers without a robust ownership check, creating a path to unauthorized object access.
2. Credential or identity context is then used to pivot from a valid session into records that belong to other users, because the authorization decision is missing or misapplied.
3. The impact is unauthorized disclosure or modification of another user’s data, which is why IDOR remains a high-value application security finding.

Breaches seen in the wild

• CI/CD pipeline exploitation case study: Credentials in an exposed .git/config let a researcher edit a Bitbucket pipeline so it planted their SSH key on the server. No victim was named.

Read and download The State of NHI & AI Agent Breach Report 2026, covering 200+ breaches impacting Non-Human Identities including AI Agents.

NHI Mgmt Group analysis

Harness design is now a security control, not just an engineering convenience. The article shows that the biggest performance gap was between prompt-only runs and a harness that enumerates endpoints and supplies structured context. That means AI-assisted security outcomes depend on what the system is allowed to see, in what order, and with what task scaffolding. For identity and access use cases, the control point is the workflow around the model, not the model alone.

IDOR detection is really a policy-verification problem dressed up as code analysis. The vulnerability exists when object-level access rules are missing or bypassed, so the model has to reason about entitlement, ownership, and session context. That places the problem close to authorization governance rather than simple pattern matching. Practitioners should expect AI tools to struggle unless the surrounding process makes authorization semantics explicit.

Open-weight models are becoming operationally relevant for security workloads. A year ago, a bare prompt from an open-weight model would have been easy to dismiss. This result shows that some open-weight systems can now compete with frontier agents on targeted security tasks, which complicates procurement assumptions about where capability comes from. Teams should therefore evaluate deployment control, reproducibility, and cost alongside benchmark scores.

Model choice and task design are converging into one governance decision. Security leaders can no longer evaluate an AI detector as if it were a static product. If the harness supplies endpoint discovery, the model is solving a different task than a prompt-only run. Practitioners should govern both layers together, because that is where performance, trust, and repeatability are actually determined.

From our research library:

• Claude Code-assisted commits leaked secrets at a rate of 3.2%, more than double the human-only baseline of 1.5%, with peaks reaching 31 secrets per 1,000 commits in August 2025, according to the State of Secrets Sprawl 2026.
• Read next: AI Agent Observability, Audit and Incident Response Guide

What this signals

IDOR detection is becoming a workflow governance problem. As security teams embed AI into code review and vulnerability discovery, the question shifts from whether a model can recognise an issue to whether the surrounding process presents the right context at the right time. The practical risk is false confidence in a detector whose harness is doing most of the work.

Context routing will separate useful security automation from noisy automation. When the task depends on ownership checks and request-path reasoning, teams need systems that can surface endpoint discovery, authorization paths, and relevant files before the model answers. That is where AI-assisted application security either becomes operationally credible or stays a demo.

For practitioners

• Define the harness before judging the model Measure endpoint discovery, context selection, and output parsing separately from raw model accuracy so you can see where the security workflow is actually succeeding or failing.
• Test IDOR workflows against authorization semantics Build evaluation sets that require the system to reason about ownership, session state, and object-level permissions instead of only spotting suspicious parameter names.
• Separate deployment control from benchmark score If a model may handle sensitive code, compare local deployment, data handling, and reproducibility options before treating the score as the main procurement criterion.
• Benchmark prompt-only and scaffolded runs side by side Use the same dataset and the same task prompt, then vary only the harness so you can isolate whether gains come from model reasoning or orchestration.

Key takeaways

• IDOR is difficult to detect because the real failure is missing object-level authorization, not an obvious code pattern.
• Semgrep’s results show that a purpose-built harness can outperform prompt-only model usage by a wide margin on the same benchmark.
• Security teams evaluating AI for application testing should judge orchestration, context selection, and reproducibility alongside raw model performance.

Key terms

• Insecure Direct Object Reference: Insecure direct object reference is an access control flaw where an application exposes an identifier that allows callers to reach records or objects they should not be able to access. In practice, it means the system trusts the request too much and fails to verify object-level permission on each access.
• Harness: The harness is the layer of instructions, policies, and approval logic wrapped around an AI agent. It is where organisations try to constrain behaviour, but it only works if the rules are explicit, current, and enforced outside the model itself.
• Object Level Authorization: Object level authorization controls access to individual records or resources, not just to an endpoint. It prevents users from reaching data that is technically available through a route but should remain hidden according to policy, role, or ownership. This is a key safeguard against broken access control.
• Prompt-Only Evaluation: A testing setup in which the model receives a prompt and limited context without extra orchestration such as endpoint discovery or guided navigation. This approach isolates raw model reasoning, but it often understates the value of surrounding workflow controls in security tasks.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, agentic AI identity, machine identity security, IAM, and secrets management. It is designed for practitioners who need to connect identity control to broader security programmes.