Topic starter
21/08/2026 4:42 pm
TL;DR: Frontier LLMs vary sharply in web application pentesting, with recall, precision, and cost moving very differently across models and test-time compute settings, according to Novee. The broader lesson is that cyber evals must measure end-to-end attack workflow and false positives, not just raw finding volume.
Editorial analysis by NHI Mgmt Group, based on content published by Novee: “PWNBench-v0.1: Evaluating frontier models for web application pentesting”.
Key questions
Q: What breaks when AI pentest evaluations measure recall without precision?
A: They reward noisy systems that report more candidate issues without proving they are useful, correct, or worth remediation.
Q: Why do web application pentest benchmarks need severity and false-positive scoring?
A: Because not every finding has the same operational value.
Q: How should security teams compare AI pentesting models across different compute settings?
A: They should treat reasoning effort and repeated runs as part of the evaluation, not as incidental tuning.
Practitioner guidance
- Define pentest success by validated findings Score AI-assisted offensive testing on confirmed issues, severity, and report usability instead of counting raw detections or candidate labels.
- Test precision separately from recall Run the same evaluation with false positives, duplicate findings, and low-severity noise measured independently so noisy models do not look better than they are.
- Evaluate multiple compute settings Compare default, higher reasoning effort, and repeated-run configurations because cost and output quality move together and can change the ranking materially.
Bottom line: Recall-only benchmarks can flatter AI pentest systems that generate noisy findings rather than usable security output.
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
21/08/2026 5:03 pm
Precision is the governance control that recall-only evals leave out. A benchmark that rewards sheer finding volume can overstate the usefulness of AI pentesting output, because false positives consume the same human review time as real issues. In practice, the security decision is not whether a model can produce many labels, but whether those labels are credible enough to drive remediation. For AI-enabled offensive testing, precision is the control signal that keeps the programme usable.
A few things that frame the scale:
- AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.
A question worth separating out:
Q: What does a greybox pentesting benchmark tell practitioners that CVE recall does not?
A: It shows whether a model can work through an authenticated attack workflow against a live application, not just recognise known vulnerability patterns. That makes it a much better proxy for real offensive testing, where discovery, validation, and reporting all matter.
👉 Read our full editorial: PWNBench shows cyber evals need precision, not recall alone