Topic starter
02/08/2026 9:46 am
TL;DR: GLM 5.2 scored 39% F1 on an IDOR benchmark, outperforming Claude Code at about $0.17 per vulnerability found, while Semgrep’s multimodal pipeline led with 53% to 61% F1 because its harness adds endpoint discovery and structured context, according to Semgrep. The result suggests model choice matters, but harness design remains the dominant control for security-task performance.
Editorial analysis by NHI Mgmt Group, based on content published by Semgrep: “We have Mythos at Home: GLM 5.2 beats Claude in our Cyber Benchmarks”.
By the numbers:
- GLM 5.2 scored 39% F1 on IDOR detection, beating Claude Code at 32% in Semgrep’s benchmark.
- Semgrep’s multimodal pipeline led the benchmark with 53% to 61% F1.
Key questions
Q: What breaks when IDOR detection is run without endpoint discovery?
A: Detection quality drops because the model sees code fragments without the request paths, ownership checks, or routing context needed to spot missing authorization.
Q: Why do models miss IDOR issues even when they can read the code?
A: They often lack the surrounding context that proves whether an object reference is actually authorised.
Q: How should security teams evaluate AI agent authorization tools?
A: Score every tool on whether it enforces policy before execution, covers all relevant domains, makes decisions with runtime context, and can prove the basis for each decision.
Practitioner guidance
- Define the harness before judging the model Measure endpoint discovery, context selection, and output parsing separately from raw model accuracy so you can see where the security workflow is actually succeeding or failing.
- Test IDOR workflows against authorization semantics Build evaluation sets that require the system to reason about ownership, session state, and object-level permissions instead of only spotting suspicious parameter names.
- Separate deployment control from benchmark score If a model may handle sensitive code, compare local deployment, data handling, and reproducibility options before treating the score as the main procurement criterion.
Bottom line: IDOR is difficult to detect because the real failure is missing object-level authorization, not an obvious code pattern.
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
02/08/2026 12:38 pm
Harness design is now a security control, not just an engineering convenience. The article shows that the biggest performance gap was between prompt-only runs and a harness that enumerates endpoints and supplies structured context. That means AI-assisted security outcomes depend on what the system is allowed to see, in what order, and with what task scaffolding. For identity and access use cases, the control point is the workflow around the model, not the model alone.
A few things that frame the scale:
- Claude Code-assisted commits leaked secrets at a rate of 3.2%, more than double the human-only baseline of 1.5%, with peaks reaching 31 secrets per 1,000 commits in August 2025, according to the State of Secrets Sprawl 2026.
A question worth separating out:
Q: What is the difference between model capability and harness engineering in agentic security tools?
A: Model capability is the ability to reason about a task and generate outputs. Harness engineering is the surrounding control layer that turns those outputs into safe, observable, and reproducible actions. In practice, the harness decides whether the agent is a useful security tool or just an unsafe automation experiment.
👉 Read our full editorial: Open-weight models are closing the gap in IDOR detection