Breach and Attack Simulation (BAS) Guide

Most organisations cannot say with evidence which attacker techniques their security controls would actually stop or detect today, because controls drift between annual tests and nobody re-checks them. Breach and attack simulation (BAS) closes that gap by running safe, repeatable simulations of real adversary behaviour against production controls and measuring what was prevented, detected and missed. This guide explains how BAS differs from penetration testing and red teaming, how ATT&CK-mapped test libraries and adversary emulation work, how to validate EDR, email, web and DLP controls, and how to run simulations without creating new risk. It is for security operations, detection engineering and control owners, and for leaders deciding how BAS fits a continuous threat exposure management (CTEM) programme.

Key takeaways

• BAS answers a different question from a pen test. A penetration test asks "can someone get in?"; BAS asks "do the controls we already paid for still work against these specific techniques?" The NCSC says checking whether defined controls function "is not a valuable use of penetration testing resources."
• Map every test to ATT&CK. CISA's #StopRansomware advisories set out a six-step method: select a technique, align controls, test, analyse detection and prevention, repeat across technologies, and tune.
• Measure outcomes, not test counts. Record for each technique whether it was prevented, detected with an alert, only logged, or missed, and track the change over time.
• Safe by design is a requirement, not a feature. Simulations should use benign payloads, controlled destinations and test data, under written rules of engagement and change control.
• The BAS platform is a privileged non-human identity. Its agents, server and service accounts hold credentials that can run code on every endpoint; govern them like any other high-risk NHI.

What BAS is, and how it compares

BAS tools run predefined attack scenarios, usually through lightweight agents on endpoints and simulated traffic through email and web channels, and then check which controls responded. MITRE describes its open-source Caldera platform as "an adversary emulation platform designed to easily run autonomous breach-and-attack simulation exercises", which captures the idea: emulation of real adversary behaviour, automated and repeatable.

Decision guidance: use BAS when you need breadth and trend data on known techniques, and when you want to prove that a control change did not break coverage. Use penetration testing to find unknown weaknesses and satisfy scope-based assurance needs, and red teaming to test response under realistic pressure. Automated penetration testing sits between BAS and a manual pen test: it exploits real weaknesses, so treat it as a pen test for change control. Most mature programmes run all of them and feed red team findings into the BAS library as new scenarios.

How BAS works

• Agents: small programs installed on representative endpoints and servers that execute simulation steps and report results to the platform. In Caldera, agents receive "abilities" (single technique implementations) grouped into "adversary profiles" and run as "operations".
• Network and channel simulations: the platform sends test emails, downloads and outbound traffic through the real email gateway, web proxy and firewall to see what passes. Many controls never touch an endpoint agent.
• Cloud and SaaS simulations: API-driven tests that perform actions such as creating a key or changing a logging setting in a test account, then check whether cloud detection fired.
• Result collection: the platform records whether the step was blocked locally, and, through integrations with the SIEM or EDR console, whether an alert or log was produced.
• Libraries: scenarios are built from ATT&CK techniques, threat intelligence and advisories, and refreshed as new behaviours are reported.

Safe-by-design simulation

A simulation must behave enough like the real technique to trigger the same controls, without the harm. Common design patterns:

• Benign payloads: industry test files (such as the EICAR anti-malware test file) or custom binaries that reproduce behaviour but carry no destructive capability.
• Controlled destinations: simulated exfiltration and command and control go only to infrastructure the organisation or platform controls.
• Synthetic data: DLP tests use fake card numbers, fake personal data and canary documents, never real customer data.
• Reversible actions: anything the simulation changes, such as a registry value, scheduled task or file, is recorded and cleaned up; encryption simulations work on sacrificial test files only.
• No real credential use: credential access tests check whether the attempt is blocked or detected; they do not harvest or reuse real secrets.

ATT&CK-mapped test libraries and adversary emulation

ATT&CK gives BAS a shared vocabulary: each test maps to a technique ID, and each result maps back to coverage. CISA's joint advisories, including the updated Akira advisory AA24-109A, recommend "exercising, testing, and validating your organization's security program against the threat behaviors" mapped to ATT&CK, and "continually testing your security program, at scale, in a production environment." Their six steps are a practical BAS operating model:

1. Select an ATT&CK technique described in the advisory.
2. Align your security technologies against the technique.
3. Test your technologies against the technique.
4. Analyse your detection and prevention technologies' performance.
5. Repeat the process for all security technologies.
6. Tune people, processes and technologies based on the data.

Open frameworks

Emulation plans and test libraries are dual-use content. The CTID library states its material is "only to be used with appropriate prior, explicit authorization." Keep them inside the same change control and approval process as the BAS platform itself. See the Threat Intelligence Guide for choosing which adversaries matter to you.

Validating specific controls

Pick techniques per control, not just per adversary. The table shows examples using ATT&CK v20 IDs and what a good result looks like.

A finding is only useful when it names an owner. Route email gaps to the messaging team, EDR policy gaps to endpoint engineering and missing alerts to detection engineering. See the Detection Engineering Guide, the EDR Evasion and Defence Guide and the Email Security Guide.

Measuring detection and prevention coverage

Score each technique-and-control result on one scale so trends are comparable. A five-level scale works well: prevented, detected (alert raised), logged (telemetry present but no alert), partial (some steps seen), missed. Load the results into ATT&CK Navigator layers by date to show change.

Report coverage against a defined threat profile (for example the techniques in your top ten adversaries or recent advisories), not against all of ATT&CK. A 40 percent figure across the whole matrix means little; 90 percent across the techniques of the ransomware groups targeting your sector is a meaningful claim.

Continuous validation in CTEM

Continuous threat exposure management, a model introduced by Gartner, runs in five stages: scoping, discovery, prioritisation, validation and mobilisation. BAS is the main tool of the validation stage: it shows whether a discovered exposure is actually exploitable given the controls in place, and whether a fix worked.

• Scoping: pick the business services and threat profile to test, such as remote access and ransomware techniques.
• Discovery and prioritisation: vulnerability scanning, attack surface management and identity posture data produce candidate exposures.
• Validation: BAS and automated penetration testing confirm which exposures controls fail to stop.
• Mobilisation: findings become owned tickets, and a BAS retest closes the ticket only when it passes.

Trigger extra runs on events, not only on a schedule: a new joint advisory, a major EDR or email policy change, a merger, or a significant incident elsewhere in your sector.

Running simulations safely

BAS runs in production by design, so it needs the same discipline as any production change. NIST SP 800-115 covers planning, rules of engagement and reporting for technical testing, and the same structure applies.

• Written rules of engagement: scope (hosts, tenants, channels), excluded systems (safety-critical, OT, regulated production databases), permitted techniques, time windows and named approvers.
• Change control and SOC awareness: register scheduled runs, tag simulation traffic and accounts so analysts can confirm a test quickly, but do not suppress alerts, or you stop measuring the SOC.
• Kill switch and clean-up: a tested way to stop all agents at once, plus verification that every artefact was removed.
• Pilot rings: new scenarios run first in a lab, then on a small set of representative hosts, before estate-wide runs.
• Destructive techniques: encryption, wiping, account lockout and service stop simulations run only on sacrificial systems.
• Rollback: if a scenario disrupts a system, stop the run, restore from the clean-up record, and move the scenario back to lab status until fixed.

Avoid destinations and techniques that could affect third parties; simulated phishing and exfiltration must stay inside infrastructure you control. Remember that DORA threat-led penetration testing and TIBER-EU style red teaming in regulated financial services follow their own scoping and approval rules; BAS does not replace them.

Where identity fits

A BAS platform is, by design, a way to run code on many systems from one console. That makes its own identities some of the most sensitive non-human identities in the estate.

• Agent credentials: each agent authenticates to the server with a key or certificate. Issue unique credentials per agent, rotate them, and revoke them when a host is retired. See the Machine Identity, PKI and Certificate Lifecycle Guide.
• Agent privileges: agents often run as SYSTEM or root to simulate privileged techniques. Limit which hosts carry them, and prefer on-demand agents that are removed after a run over permanent ones.
• Server and console access: put the console behind SSO with phishing-resistant MFA, use role separation between scenario authors and run approvers, and log every operation. Change default accounts and secrets on open-source deployments before first use.
• Integration secrets: API keys for the EDR, SIEM, email and cloud tenants let the platform read alerts and sometimes change policy. Store them in a vault, scope them read-only where possible and rotate them. See the Secrets Management Guide.
• Test accounts: simulations that touch identity systems need dedicated, clearly named test users and service accounts with expiry dates and owners. See the Service Account Security Guide.
• Patch the platform: CVE-2025-27364, published 24 February 2025, was a critical (CVSS 3.1 10.0) remote code execution flaw in MITRE Caldera's agent-building API, affecting versions through 4.2.0 and 5.0.0 before the fix. An internet-reachable or unpatched BAS server is a ready-made command and control server for an attacker.

Identity controls themselves deserve simulation: test that legacy authentication is blocked, that sign-in from an unmanaged device to a sensitive application is refused, and that stolen-session detections fire. See the Red Teaming AI Agents for Identity Abuse for agent-specific identity testing.

Implementation roadmap

1. Define the threat profile: choose 5 to 10 adversaries or advisories relevant to your sector and extract their ATT&CK techniques into a Navigator layer.
2. Set governance first: rules of engagement, approvers, excluded systems, SOC notification and the identity controls for the platform listed above.
3. Start in a lab: run single-technique tests (Atomic Red Team or Caldera abilities) against a lab build that mirrors production EDR and logging, and check clean-up.
4. Pilot ring: deploy agents to 10 to 20 representative production hosts across workstation, server and cloud images, plus email and web channel tests, for four weeks.
5. Baseline: record prevention and detection rates per control and per technique before changing anything.
6. Fix and retest: open owned tickets, tune controls and rules, and close only on a passing retest.
7. Scale and schedule: expand coverage, schedule weekly runs of the core library and event-triggered runs for new advisories, and report monthly.

Small versus large organisations: a small team can get most of the value from open-source frameworks and CISA advisory techniques run monthly against a few representative hosts, with results in a spreadsheet and Navigator. A large organisation with several EDR, email and proxy configurations needs a platform that schedules runs, integrates with the SIEM and tracks regressions, and should staff a purple team function to turn results into detections. In both cases, buying a platform without an owner for findings produces dashboards, not improvement.

Framework mapping

Common failure patterns

• Counting tests instead of closing gaps: thousands of simulations run, the same "missed" results every month, no owner.
• Testing the lab, not production: the lab EDR policy is stricter than the real one, so results flatter coverage.
• Whitelisting the BAS platform: excluding agent processes or simulation IPs from EDR and proxy policy to avoid noise makes every result meaningless.
• Assuming universal controls: a control "everyone has" is never tested on the edge case. The Change Healthcare breach began with one remote access login without MFA, exactly the kind of exception a scoped validation run should find.
• Treating the platform as a tool, not an identity: agents running as SYSTEM everywhere, shared API keys and an internet-reachable console turn a validation tool into an attack path.
• Stale libraries: scenarios not refreshed after new advisories, so coverage looks high against last year's threats.

Practitioner checklist

• Define a threat profile of 5 to 10 relevant adversaries or advisories and map it to an ATT&CK Navigator layer.
• Approve written rules of engagement covering scope, exclusions, destructive techniques, windows and a kill switch.
• Govern the BAS platform as a privileged NHI: SSO and MFA on the console, unique agent credentials, vaulted integration keys, patched server.
• Use only benign payloads, synthetic data and controlled destinations.
• Test email, web, EDR, DLP and SIEM separately, and record prevented, detected, logged, partial or missed per technique.
• Never exclude simulation activity from the controls being tested.
• Baseline coverage before tuning, then track regressions after every major control change.
• Close findings only on a passing retest, with a named owner and due date.
• Run event-triggered simulations within days of new joint advisories relevant to your sector.
• Feed red team and incident findings into the scenario library.

Standards and references

• NIST: SP 800-115 Technical Guide to Information Security Testing and Assessment
• CISA and FBI: #StopRansomware: Akira Ransomware (AA24-109A), Validate Security Controls
• MITRE Center for Threat-Informed Defense: Adversary Emulation Library
• MITRE Center for Threat-Informed Defense: Attack Flow
• MITRE: Caldera documentation
• MITRE ATT&CK: ATT&CK Navigator
• NCSC: Penetration testing
• Atomic Red Team: library of tests mapped to MITRE ATT&CK

Related NHI Mgmt Group resources: Detection Engineering Guide · Threat Intelligence Guide · EDR Evasion and Defence Guide · SIEM Guide · Service Account Security Guide