AI-assisted penetration testing tools and services can test more often and more widely than a yearly manual engagement, but an automated tester acting on production systems needs tighter controls than a human team, not looser ones. Products in this category range from assistants that help human testers plan and report, to platforms that select and run test steps with limited human input, to managed services that combine both. The questions that matter most to buyers are about control: how scope is enforced, how the tool is stopped, what happens to your data, how good the evidence is and who is accountable. This vendor-neutral buyer's guide helps security leaders, procurement teams and testing managers define requirements, evaluate products and services, and run a safe proof of concept. It does not cover testing techniques. For how the approach works, see the AI-Assisted and Autonomous Penetration Testing Guide.
Key takeaways
- Scope enforcement must be technical. Ask how the tool blocks targets outside the agreed scope, not only how scope is written in the contract.
- Safety controls are non-negotiable. Approval gates for higher-impact actions, rate limits, test windows and an immediate kill switch should all be demonstrable in the proof of concept.
- Evidence beats volume. Judge tools on reproducible, validated findings with clear evidence, and on false positive rates you measure yourself.
- Humans stay accountable. A named, qualified person should approve scope, review high-impact steps and sign off the report.
- Contracts carry the risk. Authorisation, liability, data handling, subcontracting and model training terms need the same care as the technical evaluation.
Define requirements
Agree these points internally before talking to suppliers:
- Purpose: continuous validation between manual tests, wider coverage of external assets, internal network or identity paths, web applications and APIs, or support for an in-house team. Purpose drives the operating model.
- Operating model: software you run, a hosted platform, or a managed service where the supplier's staff oversee the tool. Each changes who holds credentials and who is accountable.
- Environments in scope: production, staging, cloud tenants, SaaS, OT or third-party hosted systems. Third-party and cloud-hosted systems may need the provider's permission or must follow its testing policy.
- Rules of engagement: permitted targets, test windows, prohibited actions (for example denial of service, destructive changes or data exfiltration beyond proof), escalation contacts and stop conditions. NIST SP 800-115 is a useful reference for planning tests.
- Credentials and identities: which test accounts, API keys or agent identities the tool will use, how they are issued, stored, rotated and revoked. These are non-human identities and belong in your inventory.
- Compliance drivers: where results support penetration testing requirements (for example PCI DSS, or NIST SP 800-53 CA-8), check whether automated results are accepted by your assessor. See the Penetration Testing Compliance Requirements Guide.
- Data handling: what the tool may collect, where it is processed, how long it is kept and whether any of it can be used to train models.
Evaluation criteria
| Area | What good looks like | How to verify |
|---|---|---|
| Scope enforcement | Allow-list of targets enforced by the platform; out-of-scope targets blocked and logged even if discovered during testing | Seed an out-of-scope asset in the PoC and confirm it is not touched |
| Approval gates | Configurable human approval before higher-impact steps such as exploitation, credential use or lateral movement | Watch an approval request appear, be declined, and the step not run |
| Rate limits and windows | Per-target and global request limits; testing confined to agreed windows | Set a low limit and window and check logs for compliance |
| Kill switch | One action stops all activity at once and is available to your team, not only the supplier | Trigger it mid-test and measure time to full stop |
| Action logging | Complete, tamper-evident log of every action with timestamp, target, source and the reasoning or rule that triggered it | Export logs and reconcile against your own network and SIEM records |
| Evidence quality | Each finding reproducible, with request and response or equivalent proof, affected asset, impact and remediation | Have a senior tester re-validate a sample of findings |
| False positive handling | Findings validated before reporting; confidence levels shown; easy to dispute | Measure the false positive rate on your PoC findings |
| Data handling | Minimal collection, proof without bulk data copying, defined retention, deletion on request, no training on customer data without consent | Review data flow diagram, retention settings and contract terms |
| Credential handling | Test credentials held in a secrets store, scoped, rotated and revoked after testing | Inspect how credentials are entered, stored and removed |
| Human oversight | Named qualified reviewers for scope, high-impact steps and report sign-off | Ask for the oversight procedure and reviewer qualifications |
| Integration | Findings export to ticketing and vulnerability management; re-test on fix | Push PoC findings to your tracker and run a re-test |
| Security of the platform itself | Strong admin authentication, role separation, audit logs and supplier security assurance | Review assurance reports and admin controls |
Weight the table to your purpose. A team using the tool to support its own testers will weight evidence and integration; a team running it unattended against production will weight scope enforcement, approval gates and the kill switch highest.
Questions to ask vendors
- Scope: how is scope enforced in the platform, and what happens when the tool discovers an asset outside it?
- Autonomy: which actions can run without human approval by default, and can we require approval for every category?
- Stopping: how do we stop all activity immediately, and how long does it take to halt every running task?
- Impact: what safeguards prevent service disruption, account lockouts or data changes, and what is your record of production incidents caused by testing?
- Models: which AI models are used, where do they run, and is any of our data sent to a third-party model provider?
- Training: is any customer data, finding or log used to train or improve models, and can we opt out in the contract?
- Data: what data does the tool collect as proof, where is it stored, how long is it kept, and how is it deleted?
- Evidence: how are findings validated before they are reported, and what false positive rate do you measure?
- Oversight: who reviews high-impact steps and signs off reports, and what qualifications do they hold?
- Credentials: how do you store, scope and revoke the test credentials and keys we provide?
- Legal: how do you document our authorisation, and what liability and insurance cover applies if testing causes damage?
- Subcontracting: do any subcontractors or cloud providers process our data or run tests on your behalf?
- Assessor acceptance: have assessors accepted your reports for compliance testing, and what limits apply?
Red flags
- Scope described only as a contract term, with no technical enforcement you can see.
- No kill switch available to the customer, or a stop that only takes effect after running tasks complete.
- Default settings that allow high-impact actions without human approval.
- Vague answers about which models process your data or whether data is used for training.
- Findings counts presented as value, with no validation evidence or false positive measurement.
- Reports that cannot be reproduced by a human tester.
- Contracts that exclude liability for testing damage entirely or say nothing on authorisation.
- No audit log of tool actions that you can export and keep.
Proof of concept
- Authorise: sign rules of engagement and written authorisation naming systems, windows, prohibited actions, contacts and stop conditions; notify the SOC and system owners.
- Prepare a safe target set: use a staging environment or a small production slice, plus one seeded out-of-scope asset and a known set of planted issues so you can measure accuracy.
- Issue scoped credentials: create dedicated test identities and keys, store them in your secrets manager, and record them for revocation.
- Configure safety controls: set approval gates for all high-impact categories, low rate limits and the test window, and confirm your team can use the kill switch.
- Run and observe: let the SOC monitor as normal; compare the tool's action log with your own network and SIEM records.
- Test the stop: trigger the kill switch during an active test and record how long activity takes to cease.
- Validate findings: have a qualified tester re-check a sample; calculate detection of planted issues and the false positive rate.
- Close down: revoke all test credentials, confirm deletion of collected data per the agreed terms, and document lessons for the contract.
Agree success criteria before the PoC starts, for example no out-of-scope contact, full stop within a set time, a target share of planted issues found and a ceiling on false positives. For related tooling decisions, see the Breach and Attack Simulation Buyer's Guide and the AI Red Teaming and Testing Tools Buyer's Guide, which covers testing AI systems rather than testing with AI.
How NHI Mgmt Group can help
We provide independent requirements and evaluation support. Browse vendors in our products directory or contact us.
Related NHI Mgmt Group resources: AI-Assisted and Autonomous Penetration Testing Guide · Reading and Acting on a Penetration Test Report Guide · Penetration Testing Compliance Requirements Guide · Breach and Attack Simulation Buyer's Guide