Notifications
Clear all
Topic starter
02/08/2026 11:33 am
TL;DR: An independent benchmark against XBOW found 49 verified vulnerabilities versus 31 on two test applications, a 58% gap at the same price tier, with nearly identical false positive rates and fewer than 10% overlapping findings according to Aikido research. The result reinforces that application context, not just scan speed, determines whether AI-assisted pentesting surfaces IDORs, authorization failures, and logic abuse paths.
NHIMG editorial — based on content published by Aikido: Aikido vs XBOW: 58% more vulnerabilities found in independent benchmark
By the numbers:
- Aikido surfaced 49 verified vulnerabilities, while XBOW found 31 on the same two applications.
- The benchmark showed a 58% gap in findings at the same price tier.
Questions worth separating out
Q: What breaks when AI pentesting tools cannot test multiple user roles?
A: They miss the access-control failures that only appear when one identity is compared with another.
Q: Why do application context and identity boundaries matter in pentesting?
A: Because the most important bugs are often not simple technical errors, but logic flaws that depend on who is asking and what they should be allowed to do.
Q: How do security teams know if AI pentesting is actually finding the right risks?
A: Look for validated findings in the classes that matter most to your application model, especially authorization issues, object-level access failures, and logic abuse.
Practitioner guidance
- Require role-aware test coverage Use at least two distinct identities for applications with role-based access control, delegated access, or multi-tenant workflows, and verify that the tool compares what each identity can reach.
- Validate findings before triage Treat verified findings as the primary signal and track false positives separately, so teams can judge whether a tool is improving risk visibility or just increasing noise.
- Measure coverage of authorization paths Ask vendors to show how they detect IDORs, broken object-level authorization, and privilege escalation paths across user roles rather than relying on a single scan summary.
What's in the full article
Aikido's full blog post covers the operational detail this post intentionally leaves for the source:
- The benchmark methodology Doyensec used to compare the two tools across two real applications and 442 candidates.
- The full findings table covering true positives, false positives, setup time, reporting time, retest limits, and scan stability.
- The application-level findings on Fider and Photoview, including the validation process behind the 49 and 31 verified vulnerabilities.
- The commentary on why multi-user testing and social login support matter for role-based applications.
👉 Read Aikido's benchmark comparison of AI pentesting coverage against XBOW →
AI pentesting benchmarks: what does broader application context change?
Explore further
02/08/2026 12:47 pm
AI pentesting only matters when it understands identity boundaries. The benchmark shows that a tool can be fast, precise, and still miss the most security-relevant failures if it cannot reason across roles and object ownership. That is the same pattern IAM teams see when application testing ignores authorization semantics. The practical conclusion is that coverage of identity-aware workflows matters more than output volume.
A question worth separating out:
Q: How should organisations balance scan speed with remediation value?
A: Speed matters only if the report arrives early enough for product teams to act and then retest without friction. If setup is slow, reporting is delayed, or retests are limited, the operational value drops quickly. Teams should measure time to findings, time to fix, and time to validation together, because that is what determines whether testing reduces exposure.
👉 Read our full editorial: AI pentesting benchmark shows context changes vulnerability discovery
