BAS Buyer's Guide: Choosing a BAS Platform

Breach and attack simulation platforms are often bought on the strength of a coverage heat map, then left running a library of tests that nobody maps to their own threats, controls or detection backlog. The products sold under this label range from agent-based simulation of single ATT&CK techniques, through email, web and cloud control testing, to automated penetration testing that exploits real weaknesses, and purple team tooling that supports human-led exercises. This vendor-neutral guide helps security operations leads, detection engineers, control owners and procurement teams decide which of these they need, test coverage claims rather than accept them, judge safety in production, account for the privileged footprint the platform adds, compare pricing models and run a proof of value. For what BAS is and how to run a programme, see the Breach and Attack Simulation (BAS) Guide.

Key takeaways

Decide which question you are buying an answer to. BAS asks whether known techniques are prevented and detected; automated penetration testing asks whether a path to your critical assets exists; purple team tooling helps people find and fix detection gaps together. They are different purchases.
ATT&CK coverage is a claim, not a result. A technique ID on a test means the vendor wrote a test for it, not that the test exercises the behaviour your controls must catch. Verify a sample against your own EDR, SIEM and email logs.
Outcomes matter more than library size. Score each test as prevented, detected with an alert, logged only, or missed, and insist that the platform reads the result from your tools rather than inferring it.
Safety in production must be designed in. Benign payloads, controlled destinations, synthetic data, clean-up and a kill switch are requirements to test in the proof of value, not features to accept from a slide.
The platform is a high-value non-human identity. Agents that run code as SYSTEM or root on every endpoint, plus API keys into your SIEM and EDR consoles, make the BAS platform one of the most privileged systems you own.

Define requirements

Choosing a BAS platform starts with these decisions, written down before the first demo. They decide which category you are buying and stop the evaluation turning into a comparison of library counts.

The question and the owner: who will act on results. If the detection engineering team will consume them, prioritise SIEM and EDR result collection and detection-as-code workflow. If control owners will, prioritise per-control reporting and regression tests after change. See the Detection Engineering Guide.
Threat-informed scope: list the adversaries and behaviours that matter to you, from threat intelligence and from advisories. CISA's joint #StopRansomware advisories, such as the Medusa advisory AA25-071A, recommend "exercising, testing, and validating your organization's security program against the threat behaviors mapped to the MITRE ATT&CK Matrix for Enterprise framework", in six steps from selecting a technique to tuning people, processes and technologies. A platform should let you run that loop for any advisory within days of publication.
Control surfaces: endpoints (Windows, macOS, Linux, servers), email gateway, web proxy and DNS filtering, network egress and segmentation, data loss prevention, cloud control planes, identity providers and directories. Many vendors are strong on one surface only.
Where it may run: production endpoints, a pilot ring, a lab, or only test cloud accounts. CISA recommends "continually testing your security program, at scale, in a production environment", but your change control, OT safety limits and regulator expectations decide how far you go.
Relationship to people-led testing: BAS does not replace penetration testing or red teaming. The NCSC notes that "Assessing whether defined security controls are functioning is not a valuable use of penetration testing resources", which is the gap BAS fills, and that a penetration test shows you are not vulnerable to known issues "on the day of the test." See the Red Teaming Guide.
Constraints: where results, logs and endpoint telemetry may be stored, whether a hosted management server is acceptable, and who may approve new scenarios.

Size changes the answer. A small team with one EDR and one SIEM often gets more from open adversary emulation content run in a pilot ring, plus a quarterly purple team session, than from a licence it cannot staff. MITRE's Center for Threat-Informed Defense (CTID) publishes micro emulation plans that are "fully automated using compatible tools" and intended "to allow even those without a red team to run scenarios". A large enterprise with several security stacks, regions and change windows needs scheduling, role-based access, multi-tenant reporting and integration at scale, which is where a commercial platform earns its cost.

Capability areas and evaluation criteria

Products overlap at the edges, but each leads with one approach. Score like with like; comparing an agent-based simulation tool with an automated penetration testing tool on the same scorecard produces a meaningless BAS platform comparison.

Category	What it does	Operational risk	Buy it when
Agent-based BAS	Runs single techniques or chained scenarios on endpoints through an installed agent and checks prevention and detection	Low to moderate if payloads are benign and actions are reversed	You need breadth and trend data on known techniques and regression tests after control changes
Agentless channel testing	Sends test emails, downloads and outbound traffic through the real email gateway, proxy and firewall	Low; uses controlled destinations	Email and web controls are a priority and endpoint agents are restricted
Cloud and identity simulation	Performs API actions in test cloud accounts or a test directory, such as creating keys or changing logging, and checks cloud detection	Moderate; needs privileged test identities	Cloud detection coverage is unproven, for example T1685.002 Disable or Modify Cloud Log
Automated penetration testing	Discovers hosts, harvests and reuses credentials, and exploits real weaknesses to find attack paths	Higher; real exploitation and credential use	You need to know whether a path to critical assets exists between manual tests
Purple team tooling	Plans, records and tracks human-led exercises, maps each step to ATT&CK and to detection outcomes	Depends on the exercise	You already run red or purple team exercises and lose findings in spreadsheets

Treat automated penetration testing as a penetration test for change control and legal authorisation, following the planning, execution and reporting structure in NIST SP 800-115. Purple team tooling supports collaboration but does not replace it: the TIBER-EU Purple Teaming Guidance states that purple teaming "is not intended to replace the confidential active red teaming phase of a TIBER test", and sets a closure-phase exercise "no later than ten weeks after the end of the active red team testing phase." See the Purple Teaming Guide. If the real gap is prioritising exposures across sources rather than validating controls, see the Exposure Management Buyer's Guide.

Weight criteria to your requirements. An illustrative split for an agent-based BAS purchase: 25% result accuracy and integration, 20% scenario quality and threat relevance, 20% safety and platform security, 20% workflow and reporting, 15% commercial terms.

Criterion	What good looks like	Weak signs
Result collection	Reads alerts and log events from your EDR, SIEM and email tools through their APIs and matches them to each test step by host, time and marker	Results based only on whether the agent's action succeeded; "detected" with no alert ID
Outcome model	Prevented, detected with alert, logged only, missed, and test error recorded separately	A single pass or fail per test
Scenario quality	Tests reproduce the behaviour defenders detect, such as access to LSASS memory for T1003.001, not just a file named after a tool; procedure-level detail visible	Opaque tests; technique IDs without procedure descriptions
Threat relevance	New advisories turned into scenarios quickly; custom scenarios and imported open emulation plans supported	Library growth measured only by count
Surface coverage	Endpoint, email, web, network egress, DLP, cloud control plane and identity scenarios at the depth you need	One strong surface with thin add-ons
Safety controls	Benign payloads, controlled destinations, synthetic data, automatic clean-up, scheduling windows, per-scope kill switch	Clean-up is manual; no audit trail of what changed
Workflow	Findings routed to ticketing or detection backlog with owner, re-test on fix, trend by technique over time	Static PDF reports
Platform security	Signed agents, mutual authentication to the server, SSO with MFA, role separation for scenario authors and approvers, full audit log	Shared admin accounts; agents that accept unsigned instructions

Testing ATT&CK coverage claims

MITRE describes ATT&CK as a "knowledge base of adversarial techniques based on real-world observations", and the ATT&CK Navigator is used "to visualize defensive coverage". A vendor heat map shows which techniques it has tests for. It does not show that a test exercises the procedure your controls see, that your tools fired, or that an analyst would act. Check claims this way:

Sample, do not trust totals: pick 15 to 20 techniques that matter to you and ask for the procedure behind each test. One technique can have many procedures; a single PowerShell test does not cover T1059.001 in general.
Reconcile against your logs: for every "detected" result, find the alert or event in your SIEM or EDR yourself. For example, T1685.001 Disable or Modify Windows Event Log and T1685.005 Clear Windows Event Logs should leave evidence such as Security log event 1102 (audit log cleared) or a gap in forwarding.
Check the ATT&CK version: ATT&CK v20 reorganised defence evasion into the Defense Impairment tactic (TA0112) with techniques such as T1685 Disable or Modify Tools. A library still mapped to retired IDs has not been maintained.
Separate sub-techniques: T1003.001 LSASS Memory and T1003.006 DCSync need different telemetry; credit at parent technique level hides gaps.
Look for mapping discipline: CISA's Best Practices for MITRE ATT&CK Mapping covers "common analytical biases, mapping mistakes"; ask how the vendor reviews its own mappings.

Safety in production

Benign but realistic payloads: industry test files such as EICAR, or custom binaries that reproduce behaviour without destructive capability. For T1486 Data Encrypted for Impact, encryption must touch only sacrificial files the agent created.
Controlled destinations: simulated exfiltration (for example T1048 Exfiltration Over Alternative Protocol) and command and control go only to infrastructure the vendor or you control, with synthetic data.
Change control: scenarios approved before scheduling, with a second person for anything that touches identities, cloud configuration or availability.
Kill switch and clean-up: stop all agents in a scope within minutes, and verify reversal of registry values, scheduled tasks, files and cloud objects.
Authorised content: the CTID adversary emulation library states its plans are "only to be used with appropriate prior, explicit authorization"; the same applies to any vendor library.

Agent deployment and its NHI footprint

A BAS deployment adds non-human identities with more reach than most service accounts. Inventory them before purchase and govern them as high-risk NHIs. See the Service Account Security Guide.

Identity or credential	Why it is privileged	Minimum control
Endpoint agent service	Runs as SYSTEM or root and executes instructions from the management server on every enrolled host	Signed agents and instructions, mutual TLS, enrolment limited to approved scopes, alert on agents outside them
Management server and console accounts	Can schedule code execution across the estate	SSO with phishing-resistant MFA, separate author and approver roles, admin actions logged to your SIEM
SIEM and EDR API keys	Read alerts and telemetry; some integrations ask for write access to suppress or close alerts	Read-only scopes, vaulted and rotated, never write access to detection rules or alert status
Cloud simulation roles	Create keys, change logging or modify policies to test detection	Test accounts only, federated role assumption instead of static keys, session duration limits
Test user and service accounts	Used for credential, phishing and directory scenarios	Dedicated, tagged, excluded from real groups, monitored for use outside scheduled runs
Automated penetration testing credentials	Harvested or supplied credentials reused to move laterally	Engagement-scoped credentials revoked after each run; harvested secrets never stored in clear

Ask how the vendor's own operators access your tenant, and whether a compromise of the hosted service could push instructions to your agents. See the Secrets Management Guide for vaulting the integration keys.

Pricing models

BAS pricing is rarely published and the unit of charge shapes how much you test. Compare models at today's scale and at your three-year target.

Model	What is counted	Watch for
Per agent or endpoint	Hosts with an installed agent	Pressure to test from too few hosts to be representative
Per module or surface	Endpoint, email, web, cloud, DLP or automated penetration testing sold separately	Needing three modules to cover the controls in your requirements
Per run or scenario volume	Simulations, assessments or targets	Discouraging re-testing after a fix, which is the point of BAS
Enterprise or unlimited	Flat fee within an estate size band	Band thresholds, and premium content or integrations outside the fee
Delivered as a service	Platform operated by a provider	Who owns scenarios and results, and provider access to your agents

Framework mapping

Framework	Where BAS evidence fits
NIST SP 800-53 Rev. 5	CA-2 Control Assessments, CA-7 Continuous Monitoring, CA-8 Penetration Testing and CA-8(2) Red Team Exercises, SI-4 System Monitoring (detection outcomes), RA-5 for automated penetration testing findings
NIST SP 800-115	Planning, rules of engagement, execution and reporting for technical testing, including automated penetration testing
MITRE ATT&CK v20	Common language for scenarios and coverage, at sub-technique level
TIBER-EU Purple Teaming Guidance	Structure for collaborative exercises that purple team tooling records

Questions to ask vendors

Use these as the core of a BAS RFP and ask every shortlisted vendor the same set in writing. Ask for answers demonstrated in your environment during the proof of value, not on a demo tenant.

How do you decide a test was detected? Show the alert or event you read from our EDR and SIEM for one test, and how you match it to the test step.
For these 15 ATT&CK sub-techniques, what procedure does each test run, on which operating systems, and which ATT&CK version is your library mapped to?
How quickly do you add scenarios for a new CISA advisory, and can we import open emulation plans and write our own?
Which surfaces do you test natively: endpoint, email, web, network egress, DLP, cloud control plane, identity? Which need another module or partner?
What exactly can a destructive-looking test change, how is it reversed, and how do we stop every agent in a scope immediately?
What privileges do the agent, the cloud roles and the SIEM and EDR integrations need? Can they be read-only, federated and vaulted?
Are agents and instructions signed, and could a compromise of your hosted service run code on our endpoints? Who on your side can access our tenant?
For automated penetration testing: how are harvested credentials stored, which exploits are excluded by default, and how is the run bounded?
How do results flow into ticketing and our detection backlog, and is re-testing after a fix automatic?
What counts toward the licence, and what does running the full library weekly across our estate cost over three years?
Can we export all results and custom scenarios in an open format at contract end?

Red flags

"Detected" results that cannot be traced to a specific alert or event in your own tools.
Coverage percentages quoted against the whole ATT&CK matrix, or credited at parent technique level only.
Tests that drop a file named after an attacker tool and call it credential dumping, without reproducing the behaviour.
Integrations that ask for write access to your SIEM or EDR, for example to close or suppress alerts the platform caused.
Agents running with full privileges and accepting unsigned instructions, or a shared console admin account.
Automated penetration testing sold as "safe BAS" without separate authorisation, scope limits and credential handling.
No clean-up evidence after ransomware or persistence simulations.
Pricing that charges per run, so teams stop re-testing after fixes.

Proof of concept

Define success: pick two threat profiles that matter to you and 15 to 20 sub-techniques from them, plus one email and one cloud scenario. Agree measurable outcomes, such as every result reconciled to a log and two detection gaps closed and re-tested.
Baseline: record what your detection engineers believe is covered for those techniques today, so the proof of value shows real change, not a new dashboard.
Deploy to a pilot ring: representative endpoints and servers, test cloud accounts and dedicated test identities, under a written authorisation and change record. Log every permission requested by agents and integrations.
Run and reconcile: run the scenarios, then have your analysts independently find the alert or event for each "detected" result and confirm each "missed" result is genuinely missed.
Test safety: trigger the kill switch mid-run, verify clean-up of every change, and confirm simulated exfiltration reached only controlled destinations.
Close the loop: fix two gaps through your detection engineering process, re-run the same tests and check the result changes.
Roll back cleanly: uninstall agents, revoke integration keys and cloud roles, disable test accounts, confirm in your logs that access stopped, and get written confirmation that trial data has been deleted.

Carry these measures into production: share of priority sub-techniques with a reconciled detected or prevented result, time from a new advisory to a tested scenario, time from a missed result to a re-tested fix, regressions caught after control changes, and agent coverage of representative hosts.

Standards and references

How NHI Mgmt Group can help

We provide independent requirements, RFP and evaluation support. Browse vendors in our products directory or contact us.

Breach and Attack Simulation Buyer’s Guide: Choosing a BAS Platform