Topic starter
02/08/2026 11:22 am
TL;DR: Across two open-source applications at the same $4,000 tier, Doyensec’s benchmark found Aikido identified 49 verified vulnerabilities versus 31 for XBOW, according to Aikido. The practical issue is not only detection quality but operational friction, which now shapes how security teams evaluate AI-assisted testing.
Editorial analysis by NHI Mgmt Group, based on content published by Aikido: “Aikido vs XBOW - Independent benchmark report by Doyensec”.
By the numbers:
- Aikido found 49 verified vulnerabilities, compared with 31 for XBOW, across two open-source applications at the same $4,000 tier.
Key questions
Q: What should teams do when an AI pentesting tool has good precision but thin coverage?
A: Treat precision as only one input and compare verified findings across the same targets before standardising on a tool.
Q: Why do support overhead and scan restarts matter in AI pentesting?
A: Because they change how often a team can test, retest, and operationalise results.
Q: What do security teams get wrong about AI pentesting vendor claims?
A: They often focus on feature breadth instead of operational proof.
Practitioner guidance
- Benchmark validated coverage on the same targets Run AI pentesting candidates against identical applications and require manually validated findings before comparing results.
- Measure operational friction as a control metric Track setup time, support interactions, scan restarts, and retest turnaround alongside discovery results.
- Separate precision from breadth in tool evaluation Review false-positive rate, severity accuracy, and total verified findings as distinct measures.
Bottom line: The benchmark points to a gap between noise reduction and discovery breadth in AI pentesting tools.
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
02/08/2026 12:40 pm
Coverage is the real control variable in AI-assisted testing. False-positive rates are useful, but they do not tell you whether the tool is seeing enough of the attack surface to matter. In this benchmark, the gap between 49 and 31 verified findings suggests that practitioners should focus on validated discovery breadth, not just output cleanliness. The practitioner conclusion is simple: if coverage is thin, the tool may be under-delivering even when the reports look tidy.
A question worth separating out:
Q: What does a benchmark with limited overlap between tools actually tell practitioners?
A: It usually means the products are exploring different parts of the attack surface, so one may be missing classes of issues the other finds. In that situation, buyers should ask which coverage profile matches their risk model, not which report looks cleaner.
👉 Read our full editorial: AI pentesting benchmarks show coverage gaps beyond false positives