AI pentesting benchmarks show coverage gaps beyond false positives

At a glance

What this is: Aikido’s benchmark, run by Doyensec, showed a wider verified-finding count than XBOW at the same price point, with only a small overlap in results.

Why it matters: For security teams, the issue is whether AI-assisted testing meaningfully expands coverage and retest velocity without introducing procurement friction and operational drag.

By the numbers:

• Aikido found 49 verified vulnerabilities, compared with 31 for XBOW, across two open-source applications at the same $4,000 tier.
• XBOW’s engagement required 22+ support emails, three scan restarts, and two infrastructure outages.

Context

AI pentesting tools are being evaluated less as novelty features and more as operational security capabilities. In this benchmark, the core question is not whether the tools find issues at all, but whether they produce materially different coverage, turnaround, and effort in a real assessment workflow.

That matters to IAM, PAM, NHI, and broader security teams because automated testing only creates value when the surrounding process is dependable. If a tool needs heavy human coordination, repeated restarts, or lengthy procurement steps, the practical security outcome can fall behind the headline finding count.

Key questions

Q: What should teams do when an AI pentesting tool has good precision but thin coverage?

A: Treat precision as only one input and compare verified findings across the same targets before standardising on a tool. If the platform consistently misses real issues that another tool finds at the same price tier, the problem is not report quality but inadequate discovery breadth.

Q: Why do support overhead and scan restarts matter in AI pentesting?

A: Because they change how often a team can test, retest, and operationalise results. A tool that needs repeated human intervention may still find vulnerabilities, but it reduces the cadence and reliability of assurance, which is what security programmes ultimately need.

Q: What do security teams get wrong about AI pentesting vendor claims?

A: They often focus on feature breadth instead of operational proof. A broad claim of autonomy or omni-capability means little if the tool cannot demonstrate safe behaviour, accurate findings, and evidence of how it performed against real systems. Procurement should start with proof, not presentation.

Q: What does a benchmark with limited overlap between tools actually tell practitioners?

A: It usually means the products are exploring different parts of the attack surface, so one may be missing classes of issues the other finds. In that situation, buyers should ask which coverage profile matches their risk model, not which report looks cleaner.

Technical breakdown

Verified vulnerability coverage in AI pentesting

AI pentesting tools generate value only when they convert scan output into verified findings that a practitioner can action. In this benchmark, the meaningful metric is not raw output volume but the number of manually validated vulnerabilities across the same target set and price tier. That makes coverage a closer proxy for operational usefulness than model confidence or scan speed alone. When two systems find different issues on the same applications, overlap becomes a useful indicator of whether they are exploring the same attack surface or covering distinct paths. Practical implication: assess AI pentesting tools by validated coverage on representative targets, not by marketing claims about automation.

Practical implication: Measure tools against the same applications and require manually validated findings before treating coverage claims as decision-grade.

Why setup friction changes the security outcome

Security tooling is not only about what it detects, but how easily it enters the workflow. A tool that needs contracts, human back-and-forth, scan restarts, or outage recovery introduces delay and operator fatigue, which can reduce how often teams run assessments and how quickly they retest fixes. In practice, that friction becomes part of the control surface: the best detection engine still delivers less value if it is hard to operationalise. For AI-assisted testing, deployment overhead is a governance issue because it affects cadence, responsiveness, and adoption across programmes. Practical implication: treat onboarding effort and retest friction as security metrics, not only procurement details.

Practical implication: Score assessment tools on onboarding time, retest speed, and operational interruptions, because friction directly affects testing cadence.

False-positive rates do not capture coverage gaps

False positives matter, but they are only one dimension of tool quality. Two products can have similar false-positive and severity accuracy while still differing sharply in how many real issues they uncover. That is the central lesson of this benchmark: near-parity on noise does not mean parity on discovery breadth. For practitioners, this means tuning for precision alone can hide a coverage problem, especially in open-source applications where real vulnerabilities are expected and the bigger risk is missing them. Practical implication: evaluate precision and recall-like behaviour together, then weigh them against the operational cost of running the test repeatedly.

Practical implication: Do not let low false-positive rates substitute for breadth testing across multiple applications and retest cycles.

NHI Mgmt Group analysis

Coverage is the real control variable in AI-assisted testing. False-positive rates are useful, but they do not tell you whether the tool is seeing enough of the attack surface to matter. In this benchmark, the gap between 49 and 31 verified findings suggests that practitioners should focus on validated discovery breadth, not just output cleanliness. The practitioner conclusion is simple: if coverage is thin, the tool may be under-delivering even when the reports look tidy.

Operational friction is part of the security assessment. A testing platform that requires repeated human intervention, restarts, or extended vendor coordination changes the economics of continuous assurance. That matters across application security, cloud validation, and AI-enabled offensive testing because the control only works at the pace it can actually be run. The practitioner conclusion is to treat process overhead as a control weakness, not an inconvenience.

AI pentesting is moving from novelty to procurement discipline. Once the tools are judged on validated output at a fixed price point, buyers will increasingly compare coverage density, retest flexibility, and workflow burden. That raises the bar for any automation claim and pushes programmes toward evidence-based evaluation instead of feature-led selection. The practitioner conclusion is to demand benchmark conditions that resemble real usage, not demo environments.

Benchmarks must test the full operational path, not just the scan engine. The report shows that purchase, onboarding, execution, and retest experience all shape usable security value. That is especially relevant for teams that operationalise testing inside broader governance programmes, because the control needs to be repeatable to be useful. The practitioner conclusion is to evaluate the whole service path before accepting a coverage claim.

AI pentesting benchmark discipline now belongs in security governance. As these tools enter regular use, teams need consistent ways to compare validated findings, turnaround, and operator effort across vendors and internal use cases. The key governance question is not whether automation exists, but whether it measurably expands assurance without increasing operational burden. The practitioner conclusion is to formalise benchmark-style acceptance criteria for AI-assisted testing.

What this signals

Coverage density is becoming the buying criterion for AI-assisted testing. Security teams will increasingly judge these tools on how many real issues they uncover in representative applications, not on whether they can produce a neat report. That shifts evaluation toward benchmarked breadth, retest consistency, and repeatable operational use.

Procurement friction is now a security variable. When a test requires contracts, multiple support exchanges, or repeated restarts, it becomes harder to embed into continuous assurance. The result is that programme value depends as much on the operational path as on the scanner itself.

For practitioners

• Benchmark validated coverage on the same targets Run AI pentesting candidates against identical applications and require manually validated findings before comparing results. Use the same price tier and target class so coverage, not procurement packaging, drives the decision.
• Measure operational friction as a control metric Track setup time, support interactions, scan restarts, and retest turnaround alongside discovery results. If the process repeatedly needs human intervention, the tool is not yet operationally efficient enough for routine use.
• Separate precision from breadth in tool evaluation Review false-positive rate, severity accuracy, and total verified findings as distinct measures. A clean report with thin coverage can still leave meaningful vulnerabilities undiscovered.
• Test retest economics before standardising a tool Check how many retests are included, how fast they return, and whether results stay usable after application changes. Retest friction determines whether findings can be closed and rechecked at programme speed.

Key takeaways

• The benchmark points to a gap between noise reduction and discovery breadth in AI pentesting tools.
• Validated coverage, not just precision, should shape how teams compare automated offensive testing options.
• Operational friction, including onboarding and retest burden, can materially reduce the security value of otherwise capable tooling.

Key terms

• Verified vulnerability: A verified vulnerability is a weakness that has been manually confirmed rather than inferred by automation alone. In benchmark testing, this matters because it filters out noisy results and makes coverage comparisons more meaningful for remediation planning and risk reporting.
• Coverage Breadth: The range of real issues a security tool can discover across a target environment. Breadth matters because two tools can report similar accuracy while still uncovering very different sets of exploitable weaknesses, which changes their practical value for assessment and remediation.
• Retest Cadence: The frequency and speed at which a team can rerun an assessment after fixes or application changes. Retest cadence affects whether a security programme can close the loop efficiently, especially when findings are being validated continuously rather than in one-off projects.
• Operational Friction: The extra effort teams absorb when a control is difficult to deploy, maintain, or scale. In secrets management, operational friction shows up as manual workflow building, brittle integrations, duplicated admin work, and delayed governance tasks that reduce the effectiveness of the programme.

Deepen your knowledge

NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course. Explore nhimg.org for resources that connect identity governance to the broader security disciplines your programme depends on.