Agentic pentesting benchmarks show orchestration matters more than models

At a glance

What this is: This benchmark compares four agentic pentesting tools on a deliberately vulnerable FastAPI and React app and finds that tool scaffolding matters more than the underlying model.

Why it matters: Security teams evaluating autonomous testing should focus on orchestration, authentication flow handling and confirmed-exploit quality, not model branding alone, because those choices determine whether findings are useful or noisy.

By the numbers:

• Escape found 15 of 20 vulnerabilities on Duck Store, or 75.0%, in the benchmark.

Context

Agentic pentesting sits between traditional scanning and full human-led exploitation. The core governance problem is that autonomy claims can sound interchangeable even when tool behaviour is not, which makes procurement decisions hard to compare. For identity and access teams, the relevant question is not whether an agent can talk to an application, but whether it can manage authentication state, scope, and action sequencing safely enough to produce verified findings.

This benchmark uses a deliberately vulnerable modern web application to separate detection from confirmation. That matters because the practical value of an agentic pentesting tool depends on how it behaves under grey-box conditions, especially when it must infer API structure, maintain session context, and test access-control boundaries without losing control of scope. The benchmark is therefore less about one product and more about what actually governs trustworthy automation in security testing.

Key questions

Q: What does a low confirmed-detection rate mean for agentic pentesting tools?

A: It usually means the tool is not reliably turning reconnaissance into verified exploitation, so the output is not yet operationally trustworthy. Teams should treat low confirmed-detection results as a sign to inspect authentication handling, action sequencing and scope control before they rely on the tool for real assessments.

Q: Why do agentic pentesting tools with the same model perform so differently?

A: Because the model is only one component. The orchestration layer, including login flow handling, session persistence, tool-call sequencing and retry logic, often determines whether the agent reaches a valid exploit path or wanders into dead ends. That design layer is what security teams should evaluate first.

Q: What are the signs that an agentic pentesting tool is producing noisy results?

A: Look for a high volume of unconfirmed findings, repeated probing of irrelevant paths, and output that lacks reproducible proof of exploitation. If engineers must spend hours validating each finding, the tool is shifting work rather than reducing it, which makes the control ineffective in practice.

Q: Should teams trust autonomous pentesting claims without benchmark evidence?

A: No. Autonomous claims only become meaningful when the tool is tested against realistic applications, authentication states and access-control boundaries. Without benchmark evidence, teams cannot tell whether the agent is finding exploitable issues or simply generating plausible-looking output that increases triage load.

Technical breakdown

Why orchestration beats model choice in agentic pentesting

In agentic pentesting, the model is only one layer of the system. Orchestration determines how the agent plans actions, preserves session state, handles login flow, and decides whether to continue probing or stop after a partial clue. Two tools can use the same model and still produce very different results if one keeps authentication coherent while the other loses context or wastes steps. That is why benchmark outcomes often track the action loop more closely than the model card. For security teams, this means the control surface is the agent design, not the language model alone.

Practical implication: Evaluate the orchestration layer first, including session handling and tool-call structure, before comparing model families.

Why grey-box access changes what an agent can prove

Grey-box testing gives the agent the target URL, API documentation, and credentials, but not source code. That is enough to expose a large share of access-control and business-logic flaws if the tool can interpret the documentation and maintain state. It is also enough to surface false confidence if the agent flags suspicious behaviour without reproducing it. In this benchmark, confirmed exploitation was the threshold, which is a stronger standard than simple discovery. For practitioners, this mirrors the real operating requirement: findings only matter when they can be validated and actioned.

Practical implication: Demand confirmed-exploit evidence, not just suspicious output, when you assess agentic testing results.

How authentication handling drives finding quality

Authentication is not just an onboarding step for an agentic pentest tool. It shapes the entire exploration path, because an agent that cannot preserve login state, understand endpoint scope, or distinguish authenticated from unauthenticated routes will miss classes of broken access control. The strongest results in this benchmark came from tools that managed that state more deliberately, while weaker results reflected poor path selection and missed transitions between public and protected functionality. In identity terms, the issue is delegated access control for a non-human actor, where the agent’s own credentials and context become part of the test surface.

Practical implication: Test whether the tool can preserve and govern authenticated context before you trust it on protected application paths.

NHI Mgmt Group analysis

Orchestration, not model branding, is the decisive variable in agentic pentesting. This benchmark shows that the same underlying model can produce sharply different outcomes once the action loop, session handling and tool selection change. That shifts evaluation away from model marketing and toward operational design. For identity and security teams, the governance lesson is that autonomy is only as reliable as the state machine that surrounds it.

Confirmed exploitation is the only benchmark result that should influence tool choice. A high discovery count is not enough if the tool cannot prove the issue. The article’s scoring method reflects a more mature control philosophy: security work needs reproducible evidence, not speculative flags. That aligns with how detection and validation should be treated in any automated assurance workflow.

Agentic pentesting creates a delegated-identity problem that traditional scanner evaluation does not capture. The tool is not merely querying a target, it is acting with credentials, maintaining scope, and deciding when to continue. That makes authentication handling, session persistence and access boundaries part of the control plane. The practical conclusion is that teams should treat agentic security tools as governed non-human operators, not just smarter scanners.

Signal-to-noise is the real adoption threshold for autonomous testing. The article shows that a tool can find issues and still create too much triage burden to be operationally useful. False positives, weak reproduction and poor route selection all erode trust faster than raw coverage builds it. Practitioner takeaway: if a tool cannot reduce engineer effort, it is not yet mature enough for broad deployment.

From our research library:

• 67% of organisations still rely heavily on static credentials despite the risks they pose to agentic AI deployments, according to the 2026 Infrastructure Identity Survey.
• 53% of security leaders expect AI to run major portions of their infrastructure autonomously within the next three years, according to the 2026 Infrastructure Identity Survey.
• Read next: Agentic AI Identity Maturity Model

What this signals

Agentic pentesting governance needs a delegated-identity lens. Once a security tool can log in, retain context and act across authenticated paths, it is operating as a non-human actor with scoped authority. That means the programme has to govern session handling, route constraints and proof-of-exploit thresholds as part of access control, not just as testing convenience. The benchmark shows why those controls matter before teams scale autonomous testing.

Signal-to-noise determines whether security engineers will keep using these tools. The benchmark’s value is not just that some tools found more vulnerabilities, but that weaker tools created very different levels of triage burden. That should push buyers to assess confirmed findings per hour, false positives and state management as primary adoption criteria, because operational trust depends on whether the tool reduces work or adds it.

Static credentials still matter in agentic AI programmes: 67% of organisations still rely heavily on them despite the risks they pose to agentic AI deployments, according to the 2026 Infrastructure Identity Survey. That persistence is relevant here because every autonomous security workflow expands the number of non-human actors that need controlled authentication, scoped access and revocation discipline.

For practitioners

• Define acceptance criteria around confirmed exploitation Measure agentic pentesting tools by reproducible proof of concept, not by raw issue counts or marketing claims about autonomy.
• Test authentication state handling under grey-box conditions Verify whether the tool can log in, retain session context and distinguish public from protected routes without manual intervention.
• Compare orchestration behaviours, not just model names Assess how the action loop plans requests, retries failures and sequences tool calls before you compare any underlying model.
• Track false positives as a procurement metric Use the rate of non-existent findings to judge how much engineer time the tool will consume after deployment.

Key takeaways

• Agentic pentesting tools are not interchangeable just because they share a model or market label.
• Orchestration, authentication handling and reproducible exploitation determine whether the tool produces usable security evidence.
• Buyers should measure false positives and verified findings together, because automation that adds triage burden is not operationally mature.

Key terms

• Agentic Pentesting: An approach to penetration testing that uses AI-driven systems to support planning, execution, or interpretation of tests. The key issue is not automation by itself, but whether the environment provides enough context for the output to be accurate, prioritised, and operationally useful.
• Confirmed Exploit: A finding that has been demonstrated through reproducible exploitation rather than inferred from a scanner signature or heuristic. This is the strongest evidence that an issue is real because it shows the control failure in action, not just as a theoretical possibility.
• CI Orchestration Layer: A CI orchestration layer coordinates test assignment, execution state, retries, and reporting across infrastructure. It becomes necessary when static pipelines can no longer handle scale, parallelism, or changing machine availability without causing delays or confusing results.
• Gray Box Testing: A testing approach that gives the tester partial internal context such as credentials, documentation, or workflow information without full source code. It improves coverage and efficiency, but still leaves implementation details and some trust assumptions invisible.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security and agentic AI identity. It helps security practitioners design controls for non-human access and delegated authority across modern environments.