Pentesting agent evaluation still fails real-world realism tests

At a glance

What this is: This is Ethiack's analysis of why pentesting agent benchmarks miss real-world conditions and how a more realistic evaluation pipeline can measure useful offensive capability.

Why it matters: It matters because security teams evaluating agentic offensive tooling need scores that reflect noisy targets, duplicate findings, and validation quality, not just narrow lab-style task success.

👉 Read Ethiack's evaluation framework for real-world pentesting agents

Context

Pentesting agent evaluation fails when it assumes clean targets, single objectives, and obvious success conditions. Real assessments are messy: tools must explore, prioritise, validate, and distinguish real findings from noise, which makes benchmark design a governance problem as much as a technical one.

For IAM and NHI practitioners, the intersection is indirect but real. Agentic offensive systems are themselves software identities operating with access to tools, data, and evidence pipelines, so how they are evaluated affects whether organisations trust them in security operations, testing, and validation workflows.

Key questions

Q: What fails when pentesting agents are only scored on flag capture or task completion?

A: Flag capture and narrow task completion reward isolated success, not real offensive judgement. They miss exploration, prioritisation, duplicate findings, and validation quality, so a system can look strong in a lab while producing weak or misleading results against noisy targets. Real-world evaluation needs structured evidence, not a single binary outcome.

Q: How do you know if a pentesting agent evaluation is actually measuring useful performance?

A: A useful evaluation should show whether the agent can find valid vulnerabilities, validate them correctly, and avoid flooding the workflow with duplicates or false positives. Precision, recall, F1, severity weighting, and coverage together tell a more complete story than success or failure alone.

Q: What do security teams get wrong when comparing pentesting tools?

A: They often compare output volume, interface polish, or feature lists instead of asking how the platform validates findings and limits unsafe access. A useful comparison starts with the quality of evidence, the scope of inspection, and the strength of guardrails around sensitive code and credentials.

Q: How should teams compare agentic security tools before using them in production?

A: Teams should compare them on the quality of validated findings, not just raw activity or issue counts. The right question is whether the agent can produce repeatable, defensible results on realistic targets, with evaluation data that is maintained over time and scoring that removes duplicate inflation.

Technical breakdown

Why flag-based pentesting benchmarks mismeasure real capability

Most public pentesting benchmarks were designed for convenience, not operational realism. Capture-the-Flag style tasks measure whether a specific flag is found, while real pentesting depends on open-ended exploration, noisy target interaction, and repeated validation of findings. That creates a mismatch between benchmark success and usable security output. Even better variants, such as exploit-confirmation or vulnerable-versus-patched tests, still compress evaluation into a narrow objective and miss the cumulative decisions that matter in practice.

Practical implication: treat benchmark scores as a proxy for narrow task completion, not proof that an agent can operate safely in production-like assessments.

How structured ground truth and LLM judging improve evaluation quality

A realistic evaluation pipeline needs structured ground truth entries that are specific enough to match findings, but not so specific that they encode the path an agent must take. Once findings are reported, an LLM-as-a-judge can compare them with ground truth descriptions and preserve plausible many-to-many matches. That helps separate semantic similarity from exact duplication, which is important when agents produce overlapping reports or incomplete evidence. The result is a more flexible matching layer before final scoring.

Practical implication: define vulnerabilities in a way that supports semantic matching, then use a judge model to reduce false rejection of valid but differently phrased findings.

Why bipartite matching is needed after semantic matching

LLM-based matching is intentionally permissive, so it can produce more matches than can be counted as true positives. Bipartite matching resolves that ambiguity by enforcing a one-to-one relationship between findings and ground-truth entries. Ethiack uses maximum bipartite matching, which prevents duplicate reports from inflating precision or recall and gives a cleaner view of agent performance. This is the same general logic used when detection systems must avoid counting the same underlying issue multiple times.

Practical implication: separate semantic match detection from scoring resolution, or duplicate findings will distort the metrics you rely on for deployment decisions.

NHI Mgmt Group analysis

Real-world pentesting evaluation is now a governance problem, not just a benchmark design problem. When an offensive agent is judged only by whether it hits a flag, the measurement framework encourages brittle behaviour and overstates readiness. Ethiack's critique is that the field has confused convenience metrics with operational assurance. For security leaders, the consequence is simple: do not trust agentic testing claims unless the evaluation model reflects exploration, validation, and noise handling.

Ground-truth ambiguity is the named failure mode here: if the target model is incomplete, the score is incomplete. The article shows that realistic targets can contain unknown vulnerabilities, meaning false positives may actually be missed true positives. That is a governance issue for any AI-assisted security workflow, because the quality of the result depends on how the underlying evidence set is maintained. The practitioner takeaway is to treat evaluation artefacts as living control data, not static truth.

Structured matching is the only defensible way to compare agents at scale. The combination of structured vulnerability entries, LLM-as-a-judge correspondence, and bipartite matching creates a defensible chain from report to score. That matters because agentic systems often produce duplicate or partially overlapping findings, which traditional task scoring cannot handle. The field should expect evaluation to converge toward evidence-backed pipelines rather than single-number challenge scores.

Agentic security tooling will increasingly be evaluated like a detection system, not like a demo. The article points toward metrics such as precision, recall, F1, severity weighting, and coverage, which is the right direction for operational adoption. Once offensive agents are used in real programmes, their value depends on consistency, not just capability spikes. Teams should therefore demand evaluation methods that approximate production constraints before they accept agent output as reliable.

The same realism gap now affects adjacent identity-adjacent automation programmes. As security teams extend AI into testing, triage, and validation, the issue is no longer whether an agent can act, but whether it can be trusted to act under noisy conditions. That intersects with NHI governance because these systems consume credentials, tool access, and evidence pipelines like any other privileged service. Practitioners should assume that evaluation quality is part of access governance, not separate from it.

What this signals

The immediate signal for practitioners is that AI-assisted security testing is moving toward evidence governance. As offensive agents become more capable, the evaluation stack has to look more like a control system, with maintained ground truth, repeatable scoring, and reviewable outcomes rather than one-off challenge scores.

Evaluation debt: the more organisations rely on agentic security tools, the more they inherit a hidden backlog of unverified assumptions about what those tools can actually prove. That backlog becomes visible only when findings are disputed, duplicated, or impossible to reconcile with target reality. Teams should expect procurement and assurance questions to shift from capability demos to measurement integrity.

For practitioners

• Adopt evidence-based agent evaluation pipelines Score offensive agents using structured findings, semantic matching, and one-to-one resolution so duplicate reports do not inflate performance claims.
• Maintain ground-truth as a living control Review unmatched findings periodically, add missed vulnerabilities, and refine overly vague entries so the evaluation dataset stays credible as targets evolve.
• Track precision alongside recall Use precision, recall, and F1 together, because high recall with weak precision can make an agent unusable in real security workflows.
• Validate agent performance on realistic targets Test against noisy, open-ended applications instead of only flag-style exercises so the assessment reflects exploration, prioritisation, and validation behaviour.

Key takeaways

• Flag-based pentesting benchmarks understate the operational complexity of real assessments and can overrate agent readiness.
• Structured ground truth, semantic matching, and bipartite resolution are the core ingredients of a defensible evaluation pipeline.
• Security teams should demand precision-aware scoring and living evaluation data before treating agent output as trustworthy.

Key terms

• Benchmark Ground Truth: The verified set of issues a benchmark uses as its reference point for scoring. In offensive security evaluation, ground truth has to be maintained carefully because new validated findings may emerge over time, and static answer keys can undercount genuine discovery.
• LLM-as-a-judge: A control pattern where one language model evaluates another model's prompts, tool calls, or outputs against policy. It is not content moderation alone. In practice, it acts as a runtime decision layer that can allow, block, redact, or escalate based on semantic context and organisational rules.
• Maximum Bipartite Matching: Maximum bipartite matching is an algorithmic method for pairing items from two groups so that each item is matched at most once. In this context, it prevents repeated or overlapping findings from being counted multiple times against the same vulnerability, which keeps precision and recall calculations defensible.

What's in the full article

Ethiack's full blog post covers the operational detail this post intentionally leaves for the source:

• The full EthiBench protocol for structured ground truth creation and maintenance
• LLM-as-a-judge prompting and matching logic used to classify findings against vulnerabilities
• Maximum bipartite matching methodology for resolving duplicate or overlapping reports
• Benchmark setup details for the evaluated pentesting agents and target applications

👉 The full Ethiack post covers the scoring pipeline, target setup, and comparison methodology in more depth.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, secrets management, workload identity, and agentic AI identity. It helps security practitioners connect identity control to the broader operational risk introduced by autonomous tooling.