Breach and Attack Simulation vs Penetration Testing: Continuous Validation or Point-in-Time Depth

Breach and attack simulation (BAS) is often sold as a cheaper, always-on replacement for penetration testing, but the two answer different questions, and cutting the wrong one leaves a gap that neither auditors nor attackers will overlook. BAS repeatedly checks whether your controls prevent and detect known attacker techniques. A penetration test sends skilled people to find and chain weaknesses nobody has catalogued yet, inside an agreed scope, and gives assurance that your vulnerability processes work. This guide compares them by what they validate, frequency, the human creativity gap, safe-payload limits, compliance acceptance and cost model, then shows how to design a combined programme inside continuous threat exposure management (CTEM), including how to split the budget. It is written for testing programme owners, security operations leads and CISOs deciding where the next pound of assurance spend should go. It covers programme design only, not attack techniques or tooling.

Key takeaways

• BAS validates controls, pen testing validates exposure. BAS asks "did the control fire against this known behaviour?" while a pen test asks "what can a capable person reach from here?".
• BAS can replace repetition, not judgement. It can take over regression checks, detection coverage tracking and post-change verification that once consumed pen test days, but not discovery of unknown flaws, business logic abuse or creative chaining.
• Regulators and schemes still expect people. PCI DSS, FedRAMP and DORA TLPT require tests by qualified, independent testers to a defined methodology, and FedRAMP says testing should not be limited to automated scanning.
• Safe payloads cap what BAS can prove. Simulations are designed not to cause harm, so they show that a control would or would not stop a technique, not that an attacker could reach your crown jewels.
• Design them as one loop. Pen test and red team findings become new BAS scenarios, and BAS trend data tells the next pen test where to look.

The short answer

Keep penetration testing for assurance and discovery: annual and change-driven tests of applications, external perimeter, internal network and cloud, done by qualified people. Add BAS when you have enough controls (EDR, email security, web proxies, SIEM detections, data loss prevention) that you need continuous proof they still work after every change. BAS can shrink the part of a pen test spent re-checking known control behaviour, which frees tester time for depth. It cannot shrink the parts that a regulation scopes or that depend on human reasoning. The Breach and Attack Simulation (BAS) Guide and the Penetration Testing Guide cover each discipline in depth; this page is about choosing and combining them.

What each actually validates

MITRE describes its open-source Caldera platform as automated adversary emulation built on the ATT&CK framework, used for red teaming, purple teaming and security validation. That is a fair picture of the BAS model: emulate known behaviour, repeatably. The NCSC describes penetration testing as a method for gaining assurance, comparable to an external audit of internal financial controls, and "not as a primary method for identifying vulnerabilities". Both statements matter here. BAS is not a vulnerability scanner, and a pen test is not a substitute for your own scanning and patching.

Threat model and architecture of each

BAS threat model: a known adversary using catalogued techniques that your controls are supposed to stop. The risk it addresses is drift: a policy change, agent failure, rule disabled or new exclusion that silently removes coverage.

BAS architecture in words: lightweight agents on representative endpoints and servers, plus simulated email, web and network traffic, run scenarios from a library. Results are correlated with what your controls and SIEM recorded, usually through API integrations, and summarised as prevented, detected or missed per technique and per control. Scheduling and change triggers keep it running continuously.

Pen test threat model: a capable, motivated person with time to understand your specific systems. The risk it addresses is the unknown weakness and the combination of individually minor issues into a serious path.

Pen test architecture in words: an agreed scope and rules of engagement, written authorisation, testers working to a named methodology with credentials or access as scoped, a report ranking findings by impact, and a retest. The CISA red team assessment AA24-326A shows what people find that libraries do not: initial access through an unpatched web server holding a web shell left from an earlier disclosure, then a poorly segmented DMZ, SSH private keys and cleartext domain credentials on a file share, and an insecure delegation setting that led to domain compromise. No single item was exotic; the chain was the finding.

The human creativity gap

BAS libraries grow, and some products now add attack path analysis, but a library run cannot do these things a skilled tester does:

• Business logic: spot that a refund endpoint trusts a client-side amount, or that an approval workflow can be skipped.
• Context: recognise that a harmless-looking share holds keys for production, or that a support tool can reset administrator passwords.
• Adaptation: change approach when a control blocks one route, the way a real attacker would.
• Chaining: combine low and medium findings into a critical path, as in AA24-326A.
• Judgement on impact: explain in business terms why a finding matters and what to fix first.

AI-assisted and automated pen testing tools narrow part of this gap for infrastructure paths. They exploit real weaknesses, so treat them as testing under change control rather than simulation. See the AI-Assisted and Autonomous Penetration Testing Guide.

Safe-payload limits

BAS is designed to run in production without causing damage. That design choice sets a ceiling on what it can prove.

For running simulations safely in production, see the safety section of the Breach and Attack Simulation (BAS) Guide and the Continuous Automated Red Teaming (CART) Guide.

Compliance acceptance

Compliance regimes name penetration testing explicitly and set conditions BAS does not meet on its own. Use BAS as supporting evidence for control effectiveness, not as the test of record.

For the TLPT schemes, see the Threat-Led Penetration Testing Guide; for PCI testing detail, see the PCI DSS Requirement 11 Guide.

Cost model and budget split

The two are priced differently, which is why the replacement question comes up.

• Pen testing: priced per engagement, usually by tester days, scaled by scope. Costs recur per test and per retest. Pen testing as a service (PTaaS) packages this as a subscription, but the work is still human hours.
• BAS: priced as an annual subscription, usually by agents, endpoints or modules. The hidden cost is people: someone must run scenarios, triage misses, tune controls and keep integrations working. Budget at least part of a detection engineer's time.

A practical way to split a fixed testing budget:

1. Ring-fence obligations first: fund every pen test or TLPT that a regulation, contract or certification requires, at the scope it requires.
2. Audit last year's pen test reports: mark findings that were control-behaviour checks (for example, "EDR did not block a known technique", "outbound traffic to a test domain was allowed"). Those days are candidates for BAS.
3. Fund BAS from that saving, plus operations time: if fewer than a fifth of findings were control checks, BAS will not pay for itself through pen test savings and should be justified by detection coverage alone.
4. Reinvest freed tester days in depth: application logic, cloud identity paths, high-value integrations, or a purple team exercise.
5. Review the split yearly against outcomes: new critical findings per test day, and detection coverage trend.

Smaller organisations with few detective controls should spend on pen testing and fixing first; BAS measures controls, and there is little value in measuring controls you have not deployed. Larger organisations with a SOC and many security tools get the most from BAS, because control drift across many tools is their real risk.

What BAS can replace in a pen test programme, and what it cannot

Designing a combined programme inside CTEM

CTEM runs scoping, discovery, prioritisation, validation and mobilisation as a repeating cycle. Both disciplines belong in the validation stage, with different jobs. See the Exposure Management and CTEM Guide.

• Scoping: pick a business scope per cycle, such as payments or the customer portal. That scope drives both the pen test statement of work and the BAS scenario set.
• Discovery and prioritisation: scanning, attack surface data and threat intelligence produce candidate exposures. Choose BAS scenarios for the techniques of the threat groups that matter to you.
• Validation, BAS: continuous control checks on the scope, with misses raised as detection or prevention gaps.
• Validation, pen test: periodic human testing focused where BAS and discovery show uncertainty: new systems, complex logic, identity paths.
• Mobilisation: one remediation queue for both, with owners and deadlines. Pen test findings get a retest; BAS misses get a re-run after the fix.
• Feedback: every pen test or red team technique that worked becomes a BAS scenario where safe, so a fixed weakness is checked every week from then on.

Worked scenario

A mid-sized insurer spends most of its testing budget on an annual external and internal pen test plus two application tests. Reviewing three years of reports shows that around a quarter of internal findings repeat each year and are control-behaviour issues: endpoint protection missing on some servers, outbound connections to test infrastructure allowed, and credential dumping not alerted.

1. It keeps both application tests and the external test unchanged, because they produce most new critical findings and its customers' contracts ask for them.
2. It shortens the internal test, rescoping it to identity and cloud paths, and funds a BAS subscription and part of a detection engineer from the saving.
3. BAS runs weekly against representative server and laptop builds; the first month shows that a third of servers miss endpoint telemetry, which matches the pen test history.
4. Fixes are verified by BAS within days instead of waiting a year for the next pen test.
5. The next internal pen test, now focused, finds a privilege path through a service account in the cloud management plane that BAS could not have seen. The technique is added to the BAS library once fixed.

The outcome is not fewer pen test days overall; it is tester time moved from re-finding known gaps to finding new ones.

Where identity fits

Both disciplines depend on identities. BAS agents and integrations run with service accounts and API keys that can read security telemetry and sometimes execute on many hosts; govern them as privileged non-human identities with rotation and least privilege. Pen testers receive test accounts and credentials that must be time-limited and removed afterwards. Identity attack paths are also where human testing adds most over simulation. See the Red Teaming AI Agents for Identity Abuse page for agent-specific testing.

Detection and measurement

• Log: every BAS run with scenario, target, result per control and SIEM correlation; every pen test finding with severity, root cause, owner, fix date and retest result.
• Metric 1, prevention and detection rate: share of BAS techniques prevented or detected per scope, tracked weekly.
• Metric 2, time to restore coverage: days from a BAS miss to a passing re-run.
• Metric 3, repeat finding rate: pen test findings that appeared in a previous report; it should fall once BAS is checking them.
• Metric 4, new critical findings per test day: a proxy for pen test value after rescoping.
• Metric 5, retest closure: share of exploitable pen test findings corrected and retested within the agreed timeframe.
• Metric 6, agent coverage: share of in-scope asset types with a BAS agent or equivalent test path.

Common failure patterns

• Cancelling the pen test after buying BAS: the next audit or customer questionnaire fails, and unknown flaws go unfound.
• Allowlisting the BAS agent: results look excellent and mean nothing.
• Running BAS without an owner: misses pile up on a dashboard; nobody tunes the controls.
• Treating an alert as a defence: the CISA AA24-326A organisation had alerts that were disabled, ignored or never reviewed.
• Pen testing the same scope every year: testers re-find known control gaps instead of exploring new risk.
• Separate remediation queues: BAS and pen test findings compete instead of being prioritised together.

Framework mapping

Practitioner checklist

• List every pen test and TLPT that a regulation, contract or certification requires and ring-fence its budget.
• Classify last year's pen test findings into control-behaviour checks and genuine discoveries.
• Buy BAS only when you have the controls and the people to act on misses.
• Never exclude the BAS agent from the controls it is testing.
• Map BAS agent coverage against the asset inventory, including legacy hosts.
• Rescope pen tests towards logic, identity and cloud paths once BAS covers control regression.
• Run one remediation queue with retest for pen test findings and re-run for BAS misses.
• Turn every fixed pen test or red team technique into a BAS scenario where safe.
• Govern BAS service accounts and tester credentials as privileged identities.
• Report detection rate, time to restore coverage and repeat finding rate to leadership quarterly.

Standards and references

• NIST: SP 800-115 Technical Guide to Information Security Testing and Assessment
• NCSC: Penetration testing
• FedRAMP: Penetration Test Guidance
• EUR-Lex: Commission Delegated Regulation (EU) 2025/1190 on threat-led penetration testing
• CISA: Enhancing Cyber Resilience, Insights from CISA Red Team Assessment (AA24-326A)
• MITRE: Caldera
• NIST: SP 800-53 Rev 5 Security and Privacy Controls

Related NHI Mgmt Group resources: Breach and Attack Simulation Buyer's Guide, Penetration Testing Provider Buyer's Guide, Purple Teaming Guide, How to Read a Penetration Test Report, Red Team vs Blue Team vs Purple Team.