Notifications
Clear all
Topic starter
18/08/2026 2:25 pm
TL;DR: Most Pentesting Agent benchmarks still reward bounded task completion, not the open-ended exploration, validation, and prioritisation required in real engagements, according to Ethiack, which proposes EthiBench as a more realistic evaluation protocol. The result is a shift from flag-chasing to structured ground-truth, LLM-as-a-judge matching, and bipartite resolution for more reliable measurement.
NHIMG editorial — based on content published by Ethiack: Evaluating Pentesting Agents for the Real-World
Questions worth separating out
Q: What fails when pentesting agents are only scored on flag capture or task completion?
A: Flag capture and narrow task completion reward isolated success, not real offensive judgement.
Q: How do you know if a pentesting agent evaluation is actually measuring useful performance?
A: A useful evaluation should show whether the agent can find valid vulnerabilities, validate them correctly, and avoid flooding the workflow with duplicates or false positives.
Q: What do security teams get wrong when comparing pentesting tools?
A: They often compare output volume, interface polish, or feature lists instead of asking how the platform validates findings and limits unsafe access.
Practitioner guidance
- Adopt evidence-based agent evaluation pipelines Score offensive agents using structured findings, semantic matching, and one-to-one resolution so duplicate reports do not inflate performance claims.
- Maintain ground-truth as a living control Review unmatched findings periodically, add missed vulnerabilities, and refine overly vague entries so the evaluation dataset stays credible as targets evolve.
- Track precision alongside recall Use precision, recall, and F1 together, because high recall with weak precision can make an agent unusable in real security workflows.
What's in the full article
Ethiack's full blog post covers the operational detail this post intentionally leaves for the source:
- The full EthiBench protocol for structured ground truth creation and maintenance
- LLM-as-a-judge prompting and matching logic used to classify findings against vulnerabilities
- Maximum bipartite matching methodology for resolving duplicate or overlapping reports
- Benchmark setup details for the evaluated pentesting agents and target applications
👉 Read Ethiack's evaluation framework for real-world pentesting agents →
Pentesting agent evaluation: where current benchmarks fall short?
Explore further
18/08/2026 3:15 pm
Real-world pentesting evaluation is now a governance problem, not just a benchmark design problem. When an offensive agent is judged only by whether it hits a flag, the measurement framework encourages brittle behaviour and overstates readiness. Ethiack's critique is that the field has confused convenience metrics with operational assurance. For security leaders, the consequence is simple: do not trust agentic testing claims unless the evaluation model reflects exploration, validation, and noise handling.
A question worth separating out:
Q: How should teams compare agentic security tools before using them in production?
A: Teams should compare them on the quality of validated findings, not just raw activity or issue counts. The right question is whether the agent can produce repeatable, defensible results on realistic targets, with evaluation data that is maintained over time and scoring that removes duplicate inflation.
👉 Read our full editorial: Pentesting agent evaluation still fails real-world realism tests
