Topic starter
18/08/2026 2:25 pm
TL;DR: Pentesting-agent benchmarks still favour bounded task completion over the open-ended exploration, validation, and prioritisation used in real engagements, according to Ethiack. Evaluation design and ground-truth maintenance now shape whether scores reflect useful security work.
Editorial analysis by NHI Mgmt Group, based on content published by Ethiack: “Evaluating Pentesting Agents for the Real-World”.
Key questions
Q: Why do pentesting benchmarks often overstate agent capability?
A: Because many benchmarks use bounded objectives such as flags or single exploit goals, which reward task completion rather than realistic offensive judgement.
Q: What breaks when ground-truth is too rigid in agent evaluation?
A: Valid findings can be rejected if the annotation assumes one exact discovery path or one exact wording.
Q: How should security teams evaluate pentesting agents for real-world use?
A: They should evaluate agent behaviour under open-ended conditions, not just whether the system can complete a known task.
Practitioner guidance
- Define evaluation goals around real engagement behaviour Measure exploration, validation, prioritisation, and duplicate handling instead of only whether an agent reaches a predefined end state.
- Build structured ground-truth for each target Annotate vulnerabilities with enough detail to support matching, but avoid entries so narrow that they only reward one discovery path.
- Add a deduplication stage before scoring Resolve many-to-many finding matches with a bipartite matching step so repeated reports do not inflate true positives.
Bottom line: Pentesting-agent evaluation fails when benchmarks optimise for closed tasks instead of realistic offensive workflows.
Glossary terms: Pentesting Agent, Benchmark Ground Truth, LLM-as-a-judge, Maximum Bipartite Matching
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
This topic was modified 7 hours ago by NHI Mgmt Group
18/08/2026 3:15 pm
Benchmark realism is now a governance issue, not a research nicety. If pentesting agents are evaluated only on constrained tasks, security leaders will overestimate how those agents behave in live environments. The measurement gap is especially relevant for AI security programmes that want to use agents operationally, because capability claims lose meaning when the evaluation does not preserve exploration, uncertainty, and verification. Practitioners should treat evaluation design as part of the control plane, not an afterthought.
A few things that frame the scale:
- 67% of organisations still rely heavily on static credentials despite the risks they pose to agentic AI deployments, according to the 2026 Infrastructure Identity Survey.
- 53% of security leaders expect AI to run major portions of their infrastructure autonomously within the next three years, according to the 2026 Infrastructure Identity Survey.
A question worth separating out:
Q: Should teams trust a pentesting agent with high benchmark scores?
A: Not without checking what the benchmark actually rewards. If the scoring model does not test open-ended reasoning, finding quality, and duplicate suppression, a high score may only show that the agent learned the test format rather than performing useful security work.
👉 Read our full editorial: Pentesting agent evaluation still fails real-world realism tests