Disaster Recovery Testing: Failover Drills and Test Types

A disaster recovery plan that has never been failed over is an assumption, and most of its faults only show up when a real site, region or platform is lost. Restore tests prove that data comes back; disaster recovery (DR) tests prove that whole services come back at the recovery site, in the right order, within the recovery time objective, with working identity, networking and DNS, and that they can be failed back afterwards. This guide covers the types of DR test, how to plan and run a failover drill safely, what to measure, and the evidence auditors and regulators expect. It is written for infrastructure, resilience and security teams who own recovery capabilities. For backup restore testing, see the Restore Testing and Recovery Assurance Guide; this page starts where that one stops.

Key takeaways

Test types form a ladder. Plan reviews and walkthroughs check documents, component and isolated failover tests check technology, and full failover and full interruption tests check the real thing; each rung costs more and proves more.
Isolated failover is the workhorse. Bringing recovery copies up in a network fenced off from production proves most of the plan with no customer impact, and cloud DR services build it in.
Identity and DNS fail first. Recovered services that cannot authenticate, resolve names or reach secrets are not recovered; test the identity plane and the people who sign in during the drill.
Failback is half the test. A plan that gets you to the recovery site but not home again leaves you running on the less resilient side indefinitely.
Evidence is the deliverable. Timed results against RTO and RPO, issues raised and closed, and sign-off are what CP-4, DORA Article 11 and ISO 22301 auditors ask for.

What a DR test proves that a restore test does not

A restore test answers "can we get this data back, intact, from the backup". A DR test answers "can we run this business service somewhere else, within the agreed time, and come back". The difference matters because most DR failures sit between the components: replication that runs but lags beyond the recovery point objective (RPO), application tiers that start in the wrong order, hard-coded IP addresses, certificates bound to production names, licence servers, firewall rules that were never mirrored, and staff who cannot sign in to the recovery environment.

NIST SP 800-53 Rev 5 control CP-4 Contingency Plan Testing requires organisations to test the plan to determine its effectiveness and readiness to execute, review the results and initiate corrective actions. Its enhancements make the DR-specific parts explicit: CP-4(2) tests at the alternate processing site, to familiarise staff with the facility and evaluate its capabilities, and CP-4(4) includes a full recovery and reconstitution of the system to a known state. CP-4(1) asks for coordination with the owners of related plans, which in practice means the business continuity, incident response and crisis communications teams. See the Business Continuity and Disaster Recovery Guide for the plan family.

DR test types compared

Test type	What happens	What it proves	Production impact	Suggested frequency
Plan review	Owners check the DR plan against current architecture, contacts and dependencies	The plan describes what actually exists	None	Quarterly or on change
Walkthrough (tabletop)	Team talks through a scenario step by step using the runbooks	Roles, decisions and sequence are understood	None	Twice a year
Simulation	Team performs recovery tasks against a simulated outage, with some real actions in non-production	People can execute the runbooks under time pressure	None to low	Yearly
Component failover	One tier fails over: a database replica promotion, a load balancer pool switch, a DNS change	Each mechanism works and is timed	Low if done in maintenance windows	Quarterly for critical tiers
Isolated (bubble) failover	Recovery copies of a whole service start in an isolated network at the DR site while production keeps running	The service starts, in order, at the DR site, and meets RTO and RPO	None if isolation holds	Yearly per critical service, more often if automated
Full failover (parallel or planned)	Live traffic moves to the DR site for a period, then fails back	The DR site carries real load, and failback works	Planned, short disruption	Yearly for the most critical services
Full interruption	The primary site or region is actually taken out of service	Recovery works without the primary at all	High: real risk of an outage	Rare; only with mature, proven DR

For the exercise design side (tabletop formats, facilitation and after-action reports) see the Tabletop Exercise Guide and the Cyber Incident Exercises Beyond Tabletop Guide.

Choosing the right test

Start with isolated failover: it proves most of what a full failover proves with no customer impact. Move to live failover once isolated drills pass cleanly at least twice.
Use full failover where the DR site must carry real load: capacity, performance under real traffic, external integrations, and partner allow lists are only proven by live traffic.
Reserve full interruption for services already proven: it adds evidence that nothing hidden depends on the primary, but an untested plan turns it into an incident.
Active-active services need different tests: there is no failover event, so test by draining a site or region and confirming the remaining capacity absorbs the load within the agreed degradation.
Small organisations: a yearly isolated failover of the few critical systems plus a twice-yearly walkthrough is realistic. Disaster recovery as a service providers usually include drill capability; confirm it in the contract. See the Disaster Recovery as a Service buyer's guide.
Large organisations: tier services by criticality, automate isolated drills for tier 1, and run one live failover per tier 1 service a year, rotating which services fail over together to find shared dependencies.

Planning a failover drill

A drill plan is a short document agreed in advance by the service owner, the DR team and change management. It should state:

Scope: the services and their dependencies (identity, DNS, secrets, message queues, third-party APIs), and what is out of scope.
Objectives: the RTO and RPO being tested, taken from the business impact analysis, plus any specific question (for example "can the payments tier start without the primary directory"). See the RTO and RPO Guide.
Scenario: what has been lost (a site, a region, a cloud account, a platform after ransomware) and what is assumed still to work.
Isolation design: how the recovered copies are kept away from production: separate virtual networks, no routes to production subnets, separate DNS, blocked outbound mail and payment integrations.
Success criteria: functional checks per service (synthetic transactions, a test login, a test order), data checks (last transaction time against RPO), and timings.
Abort criteria and rollback: conditions that stop the drill, who can call it, and the steps to return to the starting state.
Roles: drill lead, service owners, observers and timekeeper, scribe, and a named approver for any production change.

Running the drill: a phased approach

Pre-checks (day before): confirm replication health and lag, recovery capacity and quotas at the DR site, break glass credentials, and that runbooks match the current configuration.
Declare and start the clock: record the scenario start time. The RTO clock starts at declaration, not when engineers begin typing.
Fail over in dependency order: identity and DNS first, then data tiers, then application tiers, then front ends. Record the time each tier reports healthy.
Validate: run the functional and data checks, with business users where possible. Record the latest transaction present to measure the actual data loss against RPO.
Operate (live failover only): run on the DR site for a planned period, watching performance, error rates and capacity.
Fail back or clean up: for isolated drills, tear down the recovered copies; for live failovers, reverse replication, confirm it is in sync, and fail back, timing it too.
Debrief within a week: record every deviation from the runbook, assign each an owner and date, and update the plan.

Platform drill features

Cloud DR services build isolated testing in. These descriptions come from each vendor's documentation (platforms listed alphabetically; the list is illustrative, not a ranking):

AWS: the Well-Architected Reliability Pillar best practice REL13-BP03 "Test disaster recovery implementation to validate the implementation" calls for regular failover testing to confirm RTO and RPO are met, and points to AWS Elastic Disaster Recovery to launch drill instances without redirecting traffic.
Google Cloud: the Disaster recovery planning guide advises testing the plan regularly and noting issues, including users who sign in to the DR environment in regular DR tests, including the DR environment in penetration testing, and using infrastructure as code to make repeated testing easier.
Microsoft Azure: Azure Site Recovery's test failover runs against a chosen recovery point (for example Latest processed, Latest app-consistent or Latest) in a test virtual network, without affecting ongoing replication. Microsoft recommends an isolated test network that mirrors production subnets and addressing, and a Cleanup test failover step afterwards.

Where identity fits

Recovered workloads depend on identities that are easy to forget in a DR design: service accounts, managed identities, API keys and certificates, plus the directory and federation services that issue tokens. Common drill findings include service accounts that exist only in the primary directory, secrets vault replicas that were never configured, certificates issued to production host names, and workload identities scoped to resources in the primary region.

Test sign-in at the DR site: include the administrators and users who will sign in during a real event, and the break glass accounts. See the Break-Glass Account Guide.
Inventory machine credentials per service: list the service accounts, keys and certificates each recovered service needs, and confirm each is available and valid at the DR site.
Scenario where identity is the outage: after a directory compromise, recovery must start from a trusted identity source. See the Active Directory Forest Recovery Runbook Template.
Isolation and duplicates: recovered domain controllers or service accounts in a bubble network must never reach production; the Azure Site Recovery guidance warns that testing into the production network risks two machines with the same identity on one network.

Detection and measurement

Metric	How to measure	Good looks like
Achieved RTO per service	Declaration time to validated healthy, from drill timeline	At or under the BIA RTO
Achieved RPO per service	Last transaction present at the DR site versus the declared failure time	At or under the BIA RPO
Failback time	Start of failback to production validated	Recorded for every live failover
Drill coverage	Tier 1 services with a passed isolated or live failover in the last 12 months	100%
Runbook deviations	Steps that did not work as written per drill	Falling drill on drill
Finding closure	Drill findings closed by due date	Above 90%, none open past the next drill

Between drills, monitor replication lag against RPO, replication job failures, DR site quota and capacity headroom, and configuration drift between primary and DR (security groups, firewall rules, DNS records). Alert when lag exceeds a set share of the RPO, so a breach of the objective is seen before a disaster rather than during one.

Evidence and regulatory expectations

Source	Testing expectation
NIST SP 800-53 Rev 5 CP-4, CP-4(1), CP-4(2), CP-4(4)	Test the contingency plan, review results, take corrective action; coordinate with related plans; test at the alternate site; include full recovery and reconstitution
NIST SP 800-53 Rev 5 CP-7, CP-10	Alternate processing site and system recovery and reconstitution, which the drill exercises
NIST CSF 2.0 ID.IM-02, ID.IM-04, RC.RP-01	Improvements from tests and exercises; plans maintained and improved; recovery executed when initiated
DORA (Regulation (EU) 2022/2554) Article 11(6)	Financial entities test ICT business continuity and ICT response and recovery plans at least yearly and on substantive change to systems supporting critical or important functions, including cyber-attack scenarios and switchovers between primary infrastructure and redundant capacity
DORA Article 12(2)	Backup, restoration and recovery procedures tested periodically
ISO 22301:2019 clause 8.5	An exercise programme for business continuity strategies and solutions

A good evidence pack per drill holds the approved drill plan, the timed log, screenshots or exports of validation checks, the measured RTO and RPO, the list of findings with owners, and sign-off by the service owner. For financial entities, see the DORA Guide. Keep the Disaster Recovery Plan Template and drill evidence in the same controlled repository so plan versions and test results line up.

Common failure patterns

Testing only the easy tier: databases fail over every quarter but the full service, with its front end, queues and integrations, never has.
Isolation leaks: a recovered copy reaches production through a forgotten route or sends real emails or payments, turning the drill into an incident.
Clock starts late: measuring RTO from when engineers began, not from declaration, hides the time spent deciding and assembling the team.
No failback: the organisation runs for months on the DR site because failback was never planned or rehearsed.
Identity left out: recovered services cannot authenticate because service accounts, certificates or vault replicas exist only at the primary site.
Findings never closed: the same runbook errors appear drill after drill because nobody owns the fixes.

Practitioner checklist

Tier services by criticality and set a test type and frequency per tier, with RTO and RPO from the BIA.
Write a drill plan for each test with scope, isolation design, success criteria, abort criteria and roles.
Run at least one isolated failover per tier 1 service each year, and a live failover with failback for the most critical.
Start the RTO clock at declaration and time every tier and the failback.
Include identity, DNS, secrets and certificates in scope, and test real sign-ins at the DR site.
Monitor replication lag, DR capacity and configuration drift between drills.
Debrief within a week, assign owners to findings and track closure.
Keep an evidence pack per drill that maps to CP-4, DORA Article 11 or ISO 22301 as required.

Disaster Recovery Testing Guide: Failover Drills, Test Types and Evidence