Topic starter
18/08/2026 2:25 pm
TL;DR: Most Pentesting Agent benchmarks still reward bounded task completion, not the open-ended exploration, validation, and prioritisation required in real engagements, according to Ethiack, which proposes EthiBench as a more realistic evaluation protocol. The result is a shift from flag-chasing to structured ground-truth, LLM-as-a-judge matching, and bipartite resolution for more reliable measurement.
Editorial analysis by NHI Mgmt Group, based on content published by Ethiack: “Evaluating Pentesting Agents for the Real-World”.
Key questions
Q: How should security teams evaluate pentesting agents for real-world use?
A: They should evaluate agent behaviour under open-ended conditions, not just whether the system can complete a known task.
Q: Why do pentesting benchmarks often overstate agent capability?
A: Because many benchmarks use bounded objectives such as flags or single exploit goals, which reward task completion rather than realistic offensive judgement.
Q: What breaks when ground-truth is too rigid in agent evaluation?
A: Valid findings can be rejected if the annotation assumes one exact discovery path or one exact wording.
Practitioner guidance
- Define evaluation goals around real engagement behaviour Measure exploration, validation, prioritisation, and duplicate handling instead of only whether an agent reaches a predefined end state.
- Build structured ground-truth for each target Annotate vulnerabilities with enough detail to support matching, but avoid entries so narrow that they only reward one discovery path.
- Add a deduplication stage before scoring Resolve many-to-many finding matches with a bipartite matching step so repeated reports do not inflate true positives.
Bottom line: Pentesting-agent evaluation fails when benchmarks optimise for closed tasks instead of realistic offensive workflows.
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
18/08/2026 3:15 pm
Benchmark realism is now a governance issue, not a research nicety. If pentesting agents are evaluated only on constrained tasks, security leaders will overestimate how those agents behave in live environments. The measurement gap is especially relevant for AI security programmes that want to use agents operationally, because capability claims lose meaning when the evaluation does not preserve exploration, uncertainty, and verification. Practitioners should treat evaluation design as part of the control plane, not an afterthought.
A few things that frame the scale:
- 67% of organisations still rely heavily on static credentials despite the risks they pose to agentic AI deployments, according to the 2026 Infrastructure Identity Survey.
- 53% of security leaders expect AI to run major portions of their infrastructure autonomously within the next three years, according to the 2026 Infrastructure Identity Survey.
A question worth separating out:
Q: Should teams trust a pentesting agent with high benchmark scores?
A: Not without checking what the benchmark actually rewards. If the scoring model does not test open-ended reasoning, finding quality, and duplicate suppression, a high score may only show that the agent learned the test format rather than performing useful security work.
👉 Read our full editorial: Pentesting agent evaluation still fails real-world realism tests