Pentesting agent evaluation still fails real-world realism tests

At a glance

What this is: This is Ethiack’s analysis of why current pentesting-agent evaluation methods miss real-world behaviour and how structured ground-truth, LLM judging, and matching can improve measurement.

Why it matters: IAM, NHI, and AI security teams should care because agentic security tooling needs evaluation that reflects noisy, multi-step decisions, not just isolated capability checks.

By the numbers:

• The dataset contains 108 expert-annotated vulnerabilities across three open-source targets.

Context

Pentesting agent evaluation is difficult because real engagements do not present a single bounded success condition. Security teams need to know whether an agent can explore, validate, prioritise, and avoid noise across messy targets, not just complete a contained test case.

That creates a governance problem for AI security programmes: the measurement method can distort what teams think the agent can do. When evaluation rewards the wrong behaviours, the resulting confidence is not transferable to production use cases or client-facing offensive testing.

Key questions

Q: How should security teams evaluate pentesting agents for real-world use?

A: They should evaluate agent behaviour under open-ended conditions, not just whether the system can complete a known task. Good evaluation measures exploration, validation, prioritisation, precision, recall, and duplicate handling, then compares findings against structured ground-truth that reflects the target’s real vulnerability surface.

Q: Why do pentesting benchmarks often overstate agent capability?

A: Because many benchmarks use bounded objectives such as flags or single exploit goals, which reward task completion rather than realistic offensive judgement. That hides whether the agent can navigate noisy targets, discard false leads, and validate findings in a way that maps to production security work.

Q: What breaks when ground-truth is too rigid in agent evaluation?

A: Valid findings can be rejected if the annotation assumes one exact discovery path or one exact wording. Overly rigid ground-truth also makes the score depend more on benchmark design than on the agent’s actual ability to find and explain vulnerabilities.

Q: Should teams trust a pentesting agent with high benchmark scores?

A: Not without checking what the benchmark actually rewards. If the scoring model does not test open-ended reasoning, finding quality, and duplicate suppression, a high score may only show that the agent learned the test format rather than performing useful security work.

Technical breakdown

Why flag-based benchmarks distort pentesting agent measurement

Many public pentesting benchmarks borrow from Capture-the-Flag design, where success is defined by reaching a known flag or achieving one explicit objective. That structure is convenient, but it compresses the real job of a pentesting agent into a single outcome and hides the harder parts of the workflow: target exploration, evidence validation, dead-end handling, and prioritisation under uncertainty. A benchmark can therefore look rigorous while still missing the behaviours that matter in live assessments. In practical terms, a high score on a narrow benchmark may only prove that the agent can solve the benchmark’s rules, not that it can operate effectively in open-ended security work.

Practical implication: Use benchmarks that measure exploration, validation, and finding quality, not only terminal task completion.

How structured ground-truth turns findings into reliable metrics

A realistic evaluation needs a ground-truth that is specific enough to distinguish genuine vulnerabilities from broader classes, but not so specific that it encodes the exact path the agent must take. Ethiack’s approach structures each vulnerability entry with fields such as name, description, and additional context, then compares agent findings against that set. The key technical step is matching reported findings to ground-truth entries with an LLM-as-a-judge layer, then resolving ambiguous many-to-many matches through maximum bipartite matching. That combination lets evaluators keep plausible matches without inflating the score through duplicates or overlapping reports.

Practical implication: Build evaluation datasets with enough structure to support consistent matching, then deduplicate findings before scoring.

Why repeated review matters in open-ended pentesting targets

Open targets are not closed answer keys. Even with manual testing and initial annotation, some vulnerabilities will remain undiscovered at first, and others will be described too broadly or too narrowly. That means the ground-truth itself becomes part of the measurement system and must be maintained over time. Periodic expert review of unmatched findings helps refine entries, add missed vulnerabilities, and recalibrate the judge when ambiguity appears. This is especially important for agentic testing because the model may surface valid weaknesses through unexpected paths that static annotations would not anticipate. Reliable measurement depends on treating ground-truth as a living artefact, not a one-time setup.

Practical implication: Plan periodic annotation review so your scoring model improves as new finding patterns appear.

NHI Mgmt Group analysis

Benchmark realism is now a governance issue, not a research nicety. If pentesting agents are evaluated only on constrained tasks, security leaders will overestimate how those agents behave in live environments. The measurement gap is especially relevant for AI security programmes that want to use agents operationally, because capability claims lose meaning when the evaluation does not preserve exploration, uncertainty, and verification. Practitioners should treat evaluation design as part of the control plane, not an afterthought.

Structured ground-truth is the missing control for agentic security measurement. The article’s strongest contribution is not a score, but a method for turning messy findings into defensible comparisons. That matters because AI-driven security systems create their own evidence problems, including duplicates, ambiguous matches, and stochastic outputs. The field needs measurement frameworks that can survive those conditions before teams can trust agentic tooling in production.

Finding-quality metrics will matter more than pass-fail benchmarks. Precision, recall, F1, and severity-aware scoring give a much better picture of whether an offensive agent can be useful in practice. For practitioners, this shifts procurement and internal validation away from headline capability and toward whether the system can produce actionable, non-duplicative results at scale.

Agentic offensive tooling needs lifecycle governance, not one-time validation. A pentesting agent is not stable in the way a static scanner is stable; its outputs depend on prompt design, model choice, target complexity, and judge calibration. That makes continuous review of the evaluation corpus part of responsible AI security operations. Teams should expect the benchmark to evolve alongside the agent, not remain frozen after initial approval.

From our research library:

• 67% of organisations still rely heavily on static credentials despite the risks they pose to agentic AI deployments, according to the 2026 Infrastructure Identity Survey.
• 53% of security leaders expect AI to run major portions of their infrastructure autonomously within the next three years, according to the 2026 Infrastructure Identity Survey.
• Read next: Agentic AI Identity Maturity Model

What this signals

Finding quality is becoming the real differentiator for agentic security tooling. A pentesting agent that produces many findings is not automatically more useful than one that produces fewer, cleaner results. Practitioners will need evaluation methods that reward valid discovery and penalise duplicate or noisy reporting, otherwise operational trust will be misplaced.

Ground-truth maintenance is part of the control model. In open-ended agent evaluations, the annotation set is never truly finished because new valid findings can appear after the first pass. That means teams should budget for periodic expert review, especially when agent outputs repeatedly expose ambiguity in the existing corpus.

AI security programmes should treat benchmark design as a governance decision. When the evaluation method is weak, capability claims become hard to compare and even harder to operationalise. The practical shift is from asking whether an agent can solve a toy challenge to asking whether the measurement system can support safe adoption in real offensive workflows.

For practitioners

• Define evaluation goals around real engagement behaviour Measure exploration, validation, prioritisation, and duplicate handling instead of only whether an agent reaches a predefined end state.
• Build structured ground-truth for each target Annotate vulnerabilities with enough detail to support matching, but avoid entries so narrow that they only reward one discovery path.
• Add a deduplication stage before scoring Resolve many-to-many finding matches with a bipartite matching step so repeated reports do not inflate true positives.
• Schedule periodic expert review of unmatched findings Revisit findings that do not match any current entry, then refine or expand the ground-truth where the agent appears to have found a real issue.
• Track precision as a deployment gate Use precision alongside recall and F1 so a noisy agent cannot pass validation simply by producing more findings.

Key takeaways

• Pentesting-agent evaluation fails when benchmarks optimise for closed tasks instead of realistic offensive workflows.
• The article shows that structured ground-truth, LLM judging, and bipartite matching are needed to score findings credibly.
• Security teams should treat evaluation design and ground-truth maintenance as part of agent governance, not as a one-off test step.

Key terms

• Benchmark Ground Truth: The verified set of issues a benchmark uses as its reference point for scoring. In offensive security evaluation, ground truth has to be maintained carefully because new validated findings may emerge over time, and static answer keys can undercount genuine discovery.
• LLM-as-a-judge: A control pattern where one language model evaluates another model's prompts, tool calls, or outputs against policy. It is not content moderation alone. In practice, it acts as a runtime decision layer that can allow, block, redact, or escalate based on semantic context and organisational rules.
• Maximum Bipartite Matching: Maximum bipartite matching is an algorithmic method for pairing items from two groups so that each item is matched at most once. In this context, it prevents repeated or overlapping findings from being counted multiple times against the same vulnerability, which keeps precision and recall calculations defensible.
• Pentesting Agent: A pentesting agent is an AI system designed to assist or perform offensive security tasks such as reconnaissance, validation, and vulnerability discovery. Unlike a static scanner, it must make decisions under uncertainty, adapt to noisy targets, and report evidence in a way that can be operationally trusted.

Deepen your knowledge

NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course. It is suited to teams that need a shared baseline for machine identity, access governance, and emerging AI-driven control models.