PWNBench shows cyber evals need precision, not recall alone

At a glance

What this is: Novee’s PWNBench-v0.1 benchmark compares 11 frontier LLMs on real web app pentesting and finds that precision and cost separate the strongest systems more than recall alone.

Why it matters: For IAM, NHI, and agentic AI teams, the results show why tool-using AI systems need governance that accounts for noisy findings, task scope, and evidence quality, not just model capability.

By the numbers:

• The benchmark evaluated 11 frontier LLMs from 8 labs on end-to-end web application pentesting.
• PWNBench-v0.1 includes more than 400 labels covering all OWASP Top 10 risks.
• Claude Opus 5 at k =3 reached 51% recall at a cost of $1400, while Kimi K3 at k =3 reached 42% recall at $209.
• DeepSeek-V4-Flash-0731, Grok 4.5, and Kimi K3 formed the low-end to mid-range of the efficient frontier in the all-severity comparison.

👉 Read Novee's PWNBench-v0.1 analysis of frontier model pentesting performance

Context

PWNBench-v0.1 sits in the growing area of AI security evaluation, but its real value is narrower and more practical: it tests whether frontier LLMs can support realistic, authenticated web application pentesting. The primary issue is not whether a model can generate security text, but whether it can find, validate, and report issues in a live application without flooding the operator with noise.

That matters because agentic AI systems are increasingly being asked to perform technical security work, and the governance problem is no longer only model quality. It becomes a question of control, evidence, and accountability around tool use, especially where the system is operating with credentials, documentation, and some degree of autonomy. In that sense, the article is a benchmark post, but its implications reach AI governance and NHI-adjacent control design.

The article’s starting position is typical of mature AI security research: the benchmark is built to narrow the gap between synthetic evals and real operator workflows, which is exactly the right direction for this class of work.

Key questions

Q: How should security teams evaluate AI pentesting tools for enterprise use?

A: Judge them on representative coverage, reproducible proof, and reporting clarity, not on a single benchmark score. A useful tool must handle authenticated flows, multiple services, and realistic business logic, then show what it tested and why a finding is credible. If it cannot do that consistently, it is a research aid, not an enterprise control.

Q: Why do AI pentesting agents need governance beyond model selection?

A: Because the model is only one part of the system. The harness, credentials, runtime budget, and evidence-handling process can change how the agent behaves and how trustworthy its findings are. Governance has to cover the full agentic workflow, especially when the system can access live targets or sensitive documentation.

Q: What do security teams get wrong about benchmark scores for agentic systems?

A: They often treat a benchmark result as a stable property of the system, when it is really a snapshot of behaviour under specific conditions. For agentic systems, environment, run order, and prior state all matter. A better approach is to compare repeated outcomes and ask whether the benchmark preserves real operational differences.

Q: How can organisations control risk when AI systems are given pentesting credentials?

A: Scope those credentials like any other non-human identity. Give the agent only the access needed for the test, log every action, and revoke credentials as soon as the session ends. If the system can explore live applications, its access should be time-bound, observable, and easy to terminate.

Technical breakdown

Why pentesting benchmarks need precision and recall

Pentesting benchmarks often overvalue recall because they count how many issues a system can name, even when many are false positives or duplicates. PWNBench-v0.1 instead scores both recall and precision, and adds severity weighting, which better reflects how a real analyst judges output. That matters in greybox web testing because the workflow is not just discovery. It is discovery, validation, and reporting against a known target, under time and cost constraints. A model that sprays plausible findings can look strong on paper while creating operational drag for the human operator.

Practical implication: Treat benchmark results as workflow signals, not raw capability claims, and require precision by severity before trusting an AI pentesting system.

How test-time compute changes agentic security performance

The benchmark shows that test-time compute is a meaningful lever: more reasoning effort and more parallel runs can move a model along the recall-cost curve. In practice, that means model performance is not fixed at inference time. It depends on the operating envelope the orchestrator gives it, including how many attempts it can make and how findings are merged. For security teams, this is a governance issue as much as a performance issue, because the same model can behave very differently under different runtime budgets and harness designs.

Practical implication: Set explicit compute budgets and reporting rules for AI security workflows so operators understand what the system can do under each runtime configuration.

Why harness design matters as much as the model

PWNBench-v0.1 makes a strong methodological point: the harness is part of the system being evaluated. A thin, single-model setup does not equal a production security stack, especially if the real environment uses routing, validation, and multi-step orchestration. That distinction matters across agentic AI and NHI governance because the identity and privilege model around the workflow can change the risk profile even when the underlying model stays the same. The benchmark is therefore testing not only model reasoning, but also the practical security value of the surrounding control plane.

Practical implication: Evaluate the model plus orchestration layer together, then map that combined system to the access controls and audit requirements it actually needs.

NHI Mgmt Group analysis

Precision is the more important control signal than raw recall. A pentesting system that finds many issues but emits too many false positives does not reduce operational risk effectively. This benchmark shows why security teams should measure usefulness, not just output volume. For AI security programmes, the real question is whether the system can support triage and validation without forcing analysts to relearn the target from scratch.

Test-time compute is becoming part of the security control surface. The article shows that reasoning effort and repeated runs materially change results, which means operating policy is now part of model behaviour. That is a governance issue for AI systems used in offensive or defensive testing, because runtime budgets influence both cost and confidence. Practitioners should treat these settings as policy decisions, not incidental tuning.

Agentic security work needs evaluated guardrails, not assumed competence. When an AI system has credentials, docs, and tools, it is no longer a static model evaluation problem. It becomes an operational control problem involving scope, evidence quality, and auditability. That intersects directly with NHI governance because the system’s credentials and tool access must be bounded like any other non-human identity. The practitioner conclusion is straightforward: if the agent can act, it must be governed as a runtime identity.

Benchmarking web pentesting is also a proxy for AI operations maturity. The article correctly focuses on end-to-end discover-exploit-report workflow rather than isolated exploitation. That shift mirrors where the market is heading. Buyers will increasingly ask whether AI security tools can produce defensible findings under repeatable controls, not just impressive lab scores. The implication is that security leaders need evaluation standards tied to real workflows, evidence handling, and audit trails.

What this signals

Agentic security evaluation is becoming a governance problem, not just a testing problem. As AI systems are asked to operate against live services, teams need to know which credentials they can use, what they can touch, and how their output is validated. That intersects directly with NHI control design, and the baseline for acceptable behaviour is getting tighter as agent autonomy expands.

The operational signal for practitioners is that benchmark performance must be translated into policy. If a system cannot produce low-noise, auditable findings, it will not scale cleanly into production workflows even if headline recall looks strong. Teams should anchor their review process to NIST AI Risk Management Framework and pair it with OWASP Agentic AI Top 10 where tool use and delegation are in scope.

For practitioners

• Define severity-weighted acceptance criteria Require pentesting or agentic security tools to report precision, recall, and severity together, with high-severity false positives tracked separately from low-severity noise.
• Cap test-time compute by use case Set different reasoning and multi-run budgets for discovery, validation, and reporting tasks so teams can compare outcomes without uncontrolled cost drift.
• Bind AI security tools to governed credentials Treat credentials used by web-testing agents as a managed NHI estate, with scoped access, audit logging, and revocation tied to the testing session.
• Validate the harness before trusting the score Review whether the benchmark environment matches your production workflow, including the target class, ground-truth quality, and whether the system uses a thin or routed stack.

Key takeaways

• Frontier model evaluation for pentesting is only useful when precision, severity, and cost are measured together.
• Agentic AI performance depends on the harness, runtime budget, and credential scope as much as on the model itself.
• When AI systems act against live targets, their access should be governed like a managed non-human identity.

Key terms

• Agentic workflow: An agentic workflow is a sequence of tasks executed by an AI agent with some level of tool access and decision authority. In security terms, the workflow matters because it can span multiple systems, identities, and permissions, which makes attribution and revocation harder than with ordinary automation.
• Severity-Weighted Benchmarking: A scoring approach that gives more importance to serious findings than to low-value noise. In security evaluation, this prevents models from appearing strong simply because they generate many alerts, and it better reflects how practitioners judge useful output.
• Test-Time Compute: The amount of reasoning budget, repeated runs, or search effort applied while a model is responding. In practice, it changes both cost and quality, so it should be treated as an operational variable rather than a hidden implementation detail.
• Greybox Pentesting: A testing method where the tester has some insider context, such as account access or documentation, but not full source-code visibility. It mirrors realistic authenticated attack scenarios and is useful for measuring how well tools perform against live applications.

What's in the full report

Novee's full blog post covers the operational detail this post intentionally leaves for the source:

• Per-model benchmark curves for recall, precision, and cost across all 11 frontier systems.
• The exact evaluation harness design and how Novee validated its ground-truth issue labels.
• Severity-level splits showing how high-severity findings change the ranking.
• The model-by-model discussion of where added reasoning effort improved or degraded performance.

👉 Novee's full post includes the benchmark methodology, model comparison charts, and severity-level breakdowns.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, secrets management, and agentic AI identity. It helps practitioners translate agent behaviour into controls that fit identity and security operations.