Notifications
Clear all
Topic starter
21/08/2026 4:48 pm
TL;DR: A reinforcement-learning trained security agent reached state-of-the-art pass@8 on EVMBench Detect while running 10 to 30 times cheaper than frontier-model baselines, according to depthfirst. This suggests that specialised post-training can improve vulnerability recall without relying on broader model generality. The practical question is no longer whether agents can audit code, but how teams govern false positives, context limits, and tool scope as detection workloads scale.
NHIMG editorial — based on content published by depthfirst: Training State of the Art Vulnerability Discovery Agents through Reinforcement Learning
By the numbers:
- dfs-mini1 achieves pass@8 on EVMBench Detect while running 10-30x lower total cost than evaluated frontier-model baselines.
Questions worth separating out
Q: How should security teams evaluate AI tools for code vulnerability discovery?
A: Use a fixed vulnerable corpus, repeat each test several times, and score the union of findings as well as the average single-pass result.
Q: When do specialised tools make vulnerability discovery agents worse?
A: Specialised tools become a problem when they narrow the agent's search strategy before it has explored the codebase.
Q: What breaks when long-horizon security agents cannot preserve context?
A: The agent loses the thread of the investigation.
Practitioner guidance
- Define the agent's minimum viable toolset Expose only the primitives required for audit work and avoid adding specialised scanners unless they are demonstrably necessary.
- Measure audit systems on pass@k and precision together Do not evaluate security agents on recall alone.
- Treat context compaction as a governed control Test summarisation steps against long, multi-file investigations and compare compressed summaries with source state to detect information loss.
What's in the full report
depthfirst's full article covers the operational detail this post intentionally leaves for the source:
- Training setup details for the RL harness, including how thousands of Kubernetes sandboxes were orchestrated
- Benchmark methodology for EVMBench Detect, including pass@k scoring and rollout aggregation
- Comparative cost and token-usage analysis across pass@1 to pass@8 rollouts
- The authors' discussion of precision-recall trade-offs and the planned next step of end-to-end verification
👉 Read depthfirst's analysis of RL-trained vulnerability discovery agents →
Reinforcement learning for vulnerability discovery agents: what changes now?
Explore further
21/08/2026 5:07 pm
Agentic vulnerability discovery is becoming an access-governance problem, not just a model-quality problem. Once an audit agent can inspect code, run commands, and iterate across rollouts, its permissions and tool boundaries become part of the control model. That makes the security posture of the harness as important as the benchmark score, especially when the same agent is later used against production repositories. Practitioners should treat agent audit environments as governed execution zones, not convenience wrappers.
A question worth separating out:
Q: How should organisations govern shell-enabled audit agents in practice?
A: Treat them as privileged workloads with explicit scope, logging, and review. The environment should constrain filesystem reach, outbound network behaviour, and command execution, while every action remains traceable enough to support investigation and rollback.
👉 Read our full editorial: Reinforcement learning agents are reshaping vulnerability discovery
