Notifications
Clear all
Topic starter
02/08/2026 9:46 am
TL;DR: GLM 5.2 scored 39% F1 on an IDOR benchmark, outperforming Claude Code at about $0.17 per vulnerability found, while Semgrep’s multimodal pipeline led with 53% to 61% F1 because its harness adds endpoint discovery and structured context, according to Semgrep. The result suggests model choice matters, but harness design remains the dominant control for security-task performance.
NHIMG editorial — based on content published by Semgrep: open-weight models in IDOR detection benchmarks
By the numbers:
- GLM 5.2 scored a 39% F1 on IDOR detection, beating Claude Code at roughly $0.17 per vulnerability found.
- Semgrep Multimodal reached 61% F1 in the benchmark, with another configuration at 53% F1.
Questions worth separating out
Q: How should security teams evaluate AI tools for IDOR detection?
A: Evaluate the full workflow, not just the model.
Q: Why do access-control failures remain hard for LLM-based security analysis?
A: Because the flaw is usually a missing entitlement check, not an obviously dangerous API call.
Q: What breaks when endpoint discovery is missing from an AI security harness?
A: The model may read code without seeing the paths that matter most for authorisation review.
Practitioner guidance
- Define harness requirements before comparing models Require endpoint enumeration, repository scoping, and output parsing as explicit capabilities in any AI-assisted access-control review workflow.
- Test AI tools against your own authorisation patterns Benchmark detection on the frameworks, route structures, and object ownership conventions actually used in your codebase.
- Measure cost per true positive alongside F1 Track spend divided by real findings, not just aggregate model cost.
What's in the full report
Semgrep’s full analysis covers the operational detail this post intentionally leaves for the source:
- The complete benchmark table with model-by-model F1, precision, recall, and cost comparisons
- The exact harness differences between Semgrep Multimodal and the prompt-only runs
- The prompt strategy, endpoint-discovery workflow, and scoring methodology used in the experiment
- The grounding audit update explaining whether the models were reasoning or pattern-matching
👉 Read Semgrep’s benchmark analysis of open-weight models and IDOR detection →
IDOR detection is shifting, but the harness still matters most?
Explore further
02/08/2026 12:38 pm
Harness design is becoming an identity control point, not just a tooling detail. When AI systems are used to find access-control flaws, the orchestration layer determines what the model can actually inspect, which directly affects whether authorisation gaps are discovered or missed. That means AI-assisted security workflows now sit inside the governance boundary for identity and access analysis, not outside it. Practitioners should treat harness design as part of the control environment.
A question worth separating out:
Q: How do teams decide when a lower-cost model is good enough for security scanning?
A: Use task-specific evidence, not price alone. If a cheaper model maintains acceptable precision and recall on the organisation’s own access-control patterns, it may be good enough for broad scanning. Keep human review for high-impact findings and use cost per finding to decide where automation should stop.
👉 Read our full editorial: Open-weight models are closing the gap in IDOR detection
