PWNBench shows cyber evals need precision, not recall alone

At a glance

What this is: Novee’s PWNBench-v0.1 finds that frontier LLMs differ materially on web app pentesting once precision, severity, and cost are measured together.

Why it matters: IAM and security teams evaluating AI-assisted offensive testing need metrics that reflect usable findings and false positives, not just recall, because noisy output can mask weak real-world attack performance.

By the numbers:

• Novee benchmarked 11 frontier LLMs on real-world web app pentesting.

Context

PWNBench-v0.1 is a benchmark for agentic web application pentesting, meaning a model has to work through a real attack workflow against a live application instead of simply naming likely bugs. That matters because recall alone does not show whether findings are actionable, correctly scoped, or worth the time and cost required to produce them.

For identity and access teams, the key governance issue is measurement quality. When AI systems are used as security testers, the control question becomes whether the harness reflects real authenticated attack scenarios, produces validated findings, and separates signal from noisy over-reporting.

This is not a model leaderboard in the abstract. It is an attempt to measure how cyber evaluation design changes conclusions about effectiveness, which is typical of the broader problem in AI security testing today.

Key questions

Q: What breaks when AI pentest evaluations measure recall without precision?

A: They reward noisy systems that report more candidate issues without proving they are useful, correct, or worth remediation. In practice, that leads to inflated confidence, extra analyst workload, and weaker procurement decisions because teams optimise for volume instead of validated security findings.

Q: Why do web application pentest benchmarks need severity and false-positive scoring?

A: Because not every finding has the same operational value. A benchmark that weights severity and false positives tells you whether a model can surface the issues humans would actually care about, rather than flooding reviewers with low-value output that looks productive but is hard to trust.

Q: How should security teams compare AI pentesting models across different compute settings?

A: They should treat reasoning effort and repeated runs as part of the evaluation, not as incidental tuning. A model may look weak at one setting and materially better at another, so comparison only makes sense when cost, recall, and precision are measured across the same operating envelope.

Q: What does a greybox pentesting benchmark tell practitioners that CVE recall does not?

A: It shows whether a model can work through an authenticated attack workflow against a live application, not just recognise known vulnerability patterns. That makes it a much better proxy for real offensive testing, where discovery, validation, and reporting all matter.

Technical breakdown

Why recall-only scoring misrepresents pentesting quality

Recall tells you how many issues a system finds, but it does not tell you how many of those findings are wrong, redundant, or too low-value to act on. In offensive testing, a model that reports aggressively can look strong on recall while producing a report a human would spend too long validating. That is why precision matters. A pentest harness should reward correct issue identification, severity discrimination, and useful reporting, not just the number of labels returned. End-to-end workflow scoring also matters because pentesting includes discovery, exploitation, validation, and reporting, not just enumeration.

Practical implication: Measure AI-assisted testing with precision and severity filters, not recall alone.

Why greybox web app testing is a harder benchmark than CVE recall

Greybox pentesting gives ordinary credentials and limited context, then asks the agent to find real issues in a running application. That is much closer to actual security work than tests that depend on memorised CVEs, source code access, or planted bugs in toy environments. The benchmark design also matters because it avoids hinting at vulnerabilities and uses maintained open-source applications with human-validated ground truth. This shifts the test from pattern matching to operational reasoning. It also makes the harness a control object in its own right, because the quality of the evaluation depends on what the model can observe and how evidence is scored.

Practical implication: Use greybox-style evaluations when you want results that map to live application risk.

How test-time compute changes the security signal

Test-time compute is not just a model setting, it is part of the evaluation method. Running the same model multiple times and merging findings can lift recall because different runs surface partially overlapping issues, but cost rises roughly linearly while recall improves sub-linearly. Reasoning effort can also improve both recall and precision, although the effect varies by model family. That means a single default sample from one API is a weak basis for procurement or programme decisions. In practice, compute is a lever that changes where a model sits on the recall-cost curve, so the benchmark has to capture that trade-off explicitly.

Practical implication: Benchmark models across multiple reasoning and run settings before making selection decisions.

NHI Mgmt Group analysis

Precision is the governance control that recall-only evals leave out. A benchmark that rewards sheer finding volume can overstate the usefulness of AI pentesting output, because false positives consume the same human review time as real issues. In practice, the security decision is not whether a model can produce many labels, but whether those labels are credible enough to drive remediation. For AI-enabled offensive testing, precision is the control signal that keeps the programme usable.

Attack-workflow evaluation is the right unit of measurement for web app pentesting. The article’s strongest methodological point is that the relevant task is discover, validate, and report, not just detect. That matters because web application pentesting depends on authenticated access, limited context, and human-like reasoning across multiple steps. A cyber eval that ignores that workflow can overfit to recall-friendly shortcuts and understate operational failure modes.

Test-time compute is part of model risk, not a neutral tuning knob. If more reasoning and more runs change both the quality and the cost profile, then procurement teams need to treat compute budget as a governance variable. The market implication is that “best model” is the wrong question; the real question is which operating point produces dependable findings within acceptable cost and review capacity. Programmes should evaluate the full cost-to-signal curve, not a single score.

Web application pentesting benchmarks should be built around validated findings, not synthetic completeness. Novee’s benchmark design, with expert-labelled ground truth and injected real-world vulnerabilities, points to a broader AI security lesson: the evaluation harness is part of the control plane. That is especially relevant where AI systems are used inside security operations or adversarial testing workflows. The practical conclusion is that teams should distrust evals that are easy to game and hard to operationalise.

Model selection for security testing will increasingly depend on the harness as much as the model. Once precision, severity, and repeated-run behaviour are measured together, rankings become more conditional and less absolute. That mirrors what many security teams already see in practice: a tool can look strong in a narrow test and weaker in production-like conditions. The right programme response is to benchmark against the actual task environment, then govern the result as an operational capability, not a vendor scorecard.

From our research library:

• AI-related credential leaks surged 81.5% year-over-year in 2025, with the surrounding AI infrastructure leaking 5x faster than core LLM providers, according to the State of Secrets Sprawl 2026.

What this signals

Precision-first evaluation is becoming the more reliable control for AI-assisted security testing. If a pentest assistant cannot separate credible findings from noise, it increases validation load and weakens trust in the workflow. That is a governance problem as much as a model problem, because security teams need outputs they can operationalise, not just score.

Attack workflows will become the dominant benchmark design for AI security tooling. The field is moving away from isolated detection tasks toward full-path testing that includes discovery, validation, and reporting. For practitioners, that means procurement and internal validation should mirror the real task path, especially where AI touches web apps, credentials, or authenticated attack scenarios.

For practitioners

• Define pentest success by validated findings Score AI-assisted offensive testing on confirmed issues, severity, and report usability instead of counting raw detections or candidate labels.
• Test precision separately from recall Run the same evaluation with false positives, duplicate findings, and low-severity noise measured independently so noisy models do not look better than they are.
• Evaluate multiple compute settings Compare default, higher reasoning effort, and repeated-run configurations because cost and output quality move together and can change the ranking materially.
• Use greybox scenarios that mirror production Benchmark against authenticated attack paths and real applications with human-validated ground truth, not toy targets or planted bugs that reward pattern recall.

Key takeaways

• Recall-only benchmarks can flatter AI pentest systems that generate noisy findings rather than usable security output.
• PWNBench’s design points to a better standard for cyber evals, one that measures validated issues, severity, and cost together.
• Security teams should judge AI-assisted pentesting by the full attack workflow and the human review burden it creates.

Key terms

• Greybox Pentesting: A testing method where the tester has some insider context, such as account access or documentation, but not full source-code visibility. It mirrors realistic authenticated attack scenarios and is useful for measuring how well tools perform against live applications.
• Precision: The share of reported findings that are actually real weaknesses. High precision means fewer false positives, but it does not guarantee that the scanner is finding all of the important problems, so it must be read alongside recall to understand practical effectiveness.
• Recall: The share of real weaknesses that the scanner successfully identifies. High recall reduces the chance of missed vulnerabilities, but if it is achieved by over-reporting, teams can become overwhelmed and stop trusting the results.
• Test-Time Compute: The amount of reasoning budget, repeated runs, or search effort applied while a model is responding. In practice, it changes both cost and quality, so it should be treated as an operational variable rather than a hidden implementation detail.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps practitioners connect identity controls to broader security decisions in programmes that now include AI-enabled tooling.