Topic starter
18/08/2026 2:53 pm
TL;DR: Four agentic pentesting tools produced detection rates from 75% to 5% on a vulnerable FastAPI and React app, according to Escape. Orchestration, authentication handling and action-loop design can determine whether pentesting output becomes usable evidence.
Editorial analysis by NHI Mgmt Group, based on content published by Escape: “Benchmarking AI Pentesting Tools: A Practical Comparison”.
By the numbers:
- Escape found 15 of 20 vulnerabilities on Duck Store, or 75.0%, in the benchmark.
Key questions
Q: Why do agentic pentesting tools with the same model perform so differently?
A: Because the model is only one component.
Q: What are the signs that an agentic pentesting tool is producing noisy results?
A: Look for a high volume of unconfirmed findings, repeated probing of irrelevant paths, and output that lacks reproducible proof of exploitation.
Q: What does a low confirmed-detection rate mean for agentic pentesting tools?
A: It usually means the tool is not reliably turning reconnaissance into verified exploitation, so the output is not yet operationally trustworthy.
Practitioner guidance
- Define acceptance criteria around confirmed exploitation Measure agentic pentesting tools by reproducible proof of concept, not by raw issue counts or marketing claims about autonomy.
- Test authentication state handling under grey-box conditions Verify whether the tool can log in, retain session context and distinguish public from protected routes without manual intervention.
- Compare orchestration behaviours, not just model names Assess how the action loop plans requests, retries failures and sequences tool calls before you compare any underlying model.
Bottom line: Agentic pentesting tools are not interchangeable just because they share a model or market label.
Glossary terms: Agentic Pentesting, Confirmed Exploit, CI Orchestration Layer, Gray Box Testing
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
This topic was modified 11 hours ago by NHI Mgmt Group
18/08/2026 3:36 pm
Orchestration, not model branding, is the decisive variable in agentic pentesting. This benchmark shows that the same underlying model can produce sharply different outcomes once the action loop, session handling and tool selection change. That shifts evaluation away from model marketing and toward operational design. For identity and security teams, the governance lesson is that autonomy is only as reliable as the state machine that surrounds it.
A few things that frame the scale:
- 67% of organisations still rely heavily on static credentials despite the risks they pose to agentic AI deployments, according to the 2026 Infrastructure Identity Survey.
- 53% of security leaders expect AI to run major portions of their infrastructure autonomously within the next three years, according to the 2026 Infrastructure Identity Survey.
A question worth separating out:
Q: Should teams trust autonomous pentesting claims without benchmark evidence?
A: No. Autonomous claims only become meaningful when the tool is tested against realistic applications, authentication states and access-control boundaries. Without benchmark evidence, teams cannot tell whether the agent is finding exploitable issues or simply generating plausible-looking output that increases triage load.
👉 Read our full editorial: Agentic pentesting benchmarks show orchestration matters more than models