A backup job that reports "success" proves only that data was written somewhere; it says nothing about whether the organisation can restore the right data, cleanly, in time, with the identities and keys it will have on the worst day. Recovery assurance is the discipline of proving that, repeatedly, with evidence. This guide covers the types of restore test and what each proves, how often to run them, how to validate that a restore is complete and clean, how to automate testing, and what evidence auditors and boards should see. It is written for infrastructure, backup and resilience teams, and for the risk and audit functions that rely on their results. How to make backups survive an attack (immutability, isolation, backup admin protection) is covered in the Backup Security and Immutable Backup Guide; this guide starts where that one ends.
Key takeaways
- Test the restore, not the backup. Integrity checks on backup files are useful, but only a restore into a working application shows that recovery works.
- Use a ladder of test types. Frequent automated file and VM restores, regular application-level restores, and at least annual full recovery of a critical service, including identity.
- Validate that the restore is clean and complete. Scan for malware and attacker persistence, check data counts and application function, and confirm the restore point predates the compromise.
- Test under realistic constraints. Assume the production directory, the usual admin accounts and some keys are unavailable, because in a ransomware event they often are.
- Evidence is the deliverable. A restore test without recorded scope, timings, results and remediation is not assurance for anyone except the person who ran it.
What the standards ask for
Every major framework treats testing as part of the backup control, not an optional extra.
| Source | Requirement | What it means in practice |
|---|---|---|
| NIST CSF 2.0 PR.DS-11 | Backups of data are created, protected, maintained and tested | Testing is part of the outcome, not a separate activity |
| NIST CSF 2.0 RC.RP-03 | Integrity of backups and other restoration assets is verified before use | Check before restoring in an incident, not only in tests |
| NIST CSF 2.0 RC.RP-05 | Integrity of restored assets is verified and normal operating status confirmed | Validation after the restore is required too |
| NIST SP 800-53 CP-9(1) | Test backup information to verify media reliability and information integrity | Scheduled integrity checks |
| NIST SP 800-53 CP-9(2) | Use a sample of backup information in restoring selected system functions as part of contingency plan testing | Sampled functional restores |
| NIST SP 800-53 CP-4 and CP-4(4) | Test the contingency plan, review results, take corrective action; include full recovery and reconstitution to a known state | Exercises with documented findings and at least one full recovery |
| CIS Controls v8.1 Safeguard 11.5 | Test backup recovery quarterly, or more frequently, for a sampling of in-scope enterprise assets (IG2 and IG3) | A quarterly floor, measured by tests run, working and failed restores, and time between tests |
| CISA #StopRansomware Guide | Maintain offline, encrypted backups and regularly test their availability and integrity in a disaster recovery scenario | Test as if production is gone |
| NCSC ransomware-resistant cloud backups | Restore testing on demand, non-destructive, able to detect corruption, part of regular monitoring | Requirements to put to backup and cloud suppliers |
For the CIS view in detail see the CIS Control 11 Data Recovery Guide.
The ladder of restore tests
No single test proves recovery. Each level proves something the level below does not, and costs more to run.
| Test type | What it proves | What it does not prove | Suggested frequency |
|---|---|---|---|
| Backup integrity check (checksums, catalogue verification) | Backup files are readable and unchanged | That the data inside is usable or clean | Every job or daily |
| File or object restore | Individual items can be found and restored with correct permissions | Application consistency | Weekly, automated, random sample |
| VM or instance boot test in an isolated lab | The machine boots and services start | That the application works with its dependencies | Weekly to monthly, automated |
| Database restore with consistency checks | Transactions are complete and the database mounts cleanly to the expected point in time | End-to-end business function | Monthly for critical databases |
| Application-level restore | A full application with its databases, configuration, secrets and integrations works for users | Recovery when shared services are also lost | Quarterly for critical applications |
| Identity and tier 0 recovery | Directory, authentication, PKI and privileged access can be rebuilt first | Business application recovery at scale | At least annually, in isolation |
| Full service or site recovery exercise | A critical business service can be restored in priority order within its recovery time objective | Every possible scenario | Annually per critical service |
| SaaS restore | Mailboxes, sites, records or repositories can be restored from the third-party or native backup | Tenant-wide loss | Quarterly sample |
Choosing frequency and scope
- Tier by business impact: use the service tiers from the business impact analysis. Tier 1 services get monthly application-level tests; lower tiers rely on automated samples and an annual test.
- Sample randomly: always testing the same server proves only that server. Random selection across the estate, as the AWS Backup option RANDOM_WITHIN_WINDOW does for recovery points, finds the jobs nobody looked at.
- Test old restore points too: in a ransomware case the clean point may be weeks old. Restore from the oldest retained point at least quarterly to prove long-term retention and media.
- After change: re-test after backup platform upgrades, storage migrations, encryption key rotation and major application releases.
Small vs large organisations: a small organisation can meet the bar with a monthly restore of a few files, a quarterly restore of its main systems into an isolated network and an annual tabletop plus restore of email and finance. A large organisation should automate the lower rungs, run application and tier 0 tests on a published calendar, and report results to the risk committee.
Clean restore validation
A restore that brings back the attacker is worse than no restore. NIST CSF 2.0 requires integrity checks both before restoring (RC.RP-03) and after (RC.RP-05). Build a validation checklist per application.
- Choose the restore point from evidence: pick a point before the earliest known attacker activity from the incident timeline, not simply the most recent backup.
- Scan before reconnecting: run malware scanning with current signatures and the incident's indicators against the restored data in the isolated environment.
- Hunt for persistence: check for unexpected local accounts, scheduled tasks, services, startup items, web shells and SSH keys that the attacker may have planted before the restore point.
- Check identities and secrets: restored systems carry the service account passwords, API keys and certificates that existed at backup time. Plan to rotate them on restore, because the attacker may have them. See the Backup Identity Isolation Guide.
- Verify completeness: compare row counts, file counts, last transaction timestamps and checksums against known values recorded at backup time.
- Verify function: run scripted smoke tests (log in, read and write a record, run a report, send a message through an integration) and get the business owner to sign off.
- Patch before production: a clean image from before the compromise still has the vulnerability that let the attacker in. Patch and harden before reconnecting.
The Codefinger S3 ransomware campaign shows why validation includes the access path: data encrypted with attacker-held SSE-C keys could only be recovered from a separate backup, and a backup reachable with the same stolen keys would have offered little protection. Testing should confirm that restores work using credentials and keys that are separate from production.
Testing under realistic conditions
Many restore tests succeed only because the production environment is still there. In a destructive attack, the directory, the backup server's own credentials, DNS and the key management service may be unavailable or untrusted.
- Isolated recovery environment: restore into a network with no route to production, so tests cannot be contaminated by, or contaminate, live systems. See the Cloud Recovery and Clean Room Guide.
- Break-glass credentials only: run at least one test a year using only the recovery credentials and out-of-band access path, without the corporate directory.
- Keys from escrow: decrypt backups with keys recovered from the documented escrow or out-of-band backup, as the NCSC recommends for cloud backups, not from the live key service.
- Identity first: recover directory services, authentication and PKI before applications that depend on them. See the Identity Recovery Guide.
- Dependencies and order: restore in the order of the recovery plan and record where hidden dependencies (DNS, NTP, licence servers, certificate authorities) blocked progress. The Minimum Viable Company and Critical Service Recovery Order Guide covers sequencing.
- Infrastructure as code and golden images: CISA recommends maintaining golden images and keeping offline copies of infrastructure as code templates; include rebuilding from them in tests rather than assuming they work.
Automating restore tests
Automation makes the lower rungs of the ladder cheap enough to run constantly, and produces consistent evidence.
| Capability | Example feature | What to configure |
|---|---|---|
| Scheduled restore of recovery points | AWS Backup restore testing plans | Cron schedule, recovery point selection (latest or random within a window), start window of 1 to 168 hours, selections per resource type |
| Validation window | AWS Backup retention before cleanup (1 to 168 hours) | Long enough for scripted validation; restored resources are tagged awsbackup-restore-test and deleted afterwards |
| Restore time compliance | AWS Backup Audit Manager control "Restore time for resources meet target" | Target restore time per resource type, reported for audit |
| VM boot and application checks in an isolated lab | Veeam SureBackup (virtual lab, application groups, SureBackup jobs) | Application groups for dependencies, verification tests, isolated lab network |
| Custom validation | Scripts or pipelines triggered after restore | Row counts, checksums, smoke tests, malware scan, result written to the evidence store |
- Least privilege for the test role: AWS Backup restore testing needs a role assumable by backup.amazonaws.com with restore permissions and access to the KMS keys used. Do not give the test role delete rights on vaults. See the AWS Backup Security Guide.
- Clean up: restored test copies are data too. Make sure automatic deletion works and alert when a test copy outlives its window.
- Do not let automation replace exercises: automated tests prove components; people, decisions, communications and order of recovery are only tested in exercises.
Building the programme in phases
- Baseline (month 1): list every backup policy and the systems it covers, match them to the service tiers in the business impact analysis, and record the recovery time and recovery point objectives each service has been promised. Gaps between what is backed up and what the business thinks is backed up usually appear here.
- First manual tests (months 1 to 2): restore one system from each tier by hand, timing every step. Expect failures: missing agents, expired credentials, keys nobody can find, undocumented dependencies. Each becomes a finding.
- Automate the lower rungs (months 2 to 4): introduce scheduled integrity checks and sampled restores, starting with a small pilot set of non-critical systems. Watch for impact on production storage and backup windows; if tests slow production or break replication, pause the schedule (the rollback is simply disabling the test plan or job) and move tests to a quieter window or a copy of the backup.
- Application and tier 0 tests (months 3 to 6): agree a calendar with application owners, build the isolated recovery environment, and run the first identity recovery test.
- Full exercise (by month 12): recover a critical service end to end under realistic constraints, with the incident, communications and business teams involved, then feed the findings into the recovery plan.
- Operate: monthly metrics, quarterly review of coverage and findings, and an annual review of objectives with the business.
Evidence for auditors and boards
CP-4 requires reviewing test results and initiating corrective actions, and the CIS 11.5 measures count tests run, working and failed restores and the time between tests. Capture, for every test:
- Scope: system, data set, restore point date and test type.
- Timings: start, data available, application working, and the recovery time and recovery point achieved against the objectives.
- Result: pass, partial or fail, with validation checks run and their output.
- Conditions: environment, credentials and key sources used, and whether production services were relied on.
- Findings and actions: each problem with an owner, a due date and the re-test that closed it.
- Sign-off: technical lead and business owner.
Report to the board by critical service, not by job count: for each tier 1 service, the date of its last successful full test, the recovery time achieved against target, and open findings.
Detection and measurement
- Coverage: share of protected systems that have had any restore test in the last quarter, and share of tier 1 services with an application-level test.
- Health signals: backups stopping, failed integrity checks, restore test failures and changes to retention or encryption policies, alerted as the NCSC recommends.
| Metric | Target |
|---|---|
| Restore test success rate (first attempt) | Above 95%, with every failure investigated |
| Tier 1 services with a full recovery test in the last 12 months | 100% |
| Recovery time achieved versus objective for tier 1 services | At or under objective |
| Maximum time between tests for any in-scope asset sample | Three months or less (CIS 11.5) |
| Oldest restore point successfully tested | Matches the retention period relied on in the ransomware plan |
| Open findings from restore tests past due date | Zero |
Framework mapping
| Activity | NIST CSF 2.0 | NIST SP 800-53 Rev. 5 | CIS Controls v8.1 | ISO/IEC 27001:2022 Annex A |
|---|---|---|---|---|
| Scheduled restore tests | PR.DS-11 | CP-9(1), CP-9(2) | 11.5 | A.8.13 Information backup |
| Pre-restore integrity checks | RC.RP-03 | CP-9(1) | 11.5 | A.8.13 Information backup |
| Post-restore validation | RC.RP-05 | CP-10 | 11.5 | A.5.30 ICT readiness for business continuity |
| Full recovery exercises | RC.RP-02, ID.IM-02 | CP-4, CP-4(4) | 11.1 | A.5.30 ICT readiness for business continuity |
| Protection of recovery components | PR.DS-11 | CP-10(6) | 11.4 | A.8.13 Information backup |
Common failure patterns
- Green dashboards, no restores: job success rates are reported for years while nobody restores anything larger than a file.
- Testing with production present: restores succeed because the live directory, DNS and key service are available, then fail in a real event when they are not.
- Unverified claims: teams accept that a copy exists without restoring it. In the JADEPUFFER agentic ransomware case, databases were dropped and the encryption key was never stored, so only a separate, restorable copy could have helped; the attacker's claim of a backup was itself unverified.
- Always the same server: the quarterly test restores the same easy VM, while the database with a broken backup chain is never chosen.
- Restoring the attacker: the latest backup is restored without checking the incident timeline, bringing back persistence and stolen credentials.
- Findings never closed: slow restores and missing dependencies are recorded and then forgotten until the next real incident.
Practitioner checklist
- Map every tier 1 service to a restore test type, frequency and owner on a published calendar.
- Automate integrity checks and random sampled restores, with results written to an evidence store.
- Run at least quarterly sampled restores across the estate, meeting CIS Safeguard 11.5.
- Restore from the oldest retained restore point at least quarterly.
- Run at least one annual full recovery of a critical service, starting with identity, in an isolated environment.
- Use only break-glass credentials and escrowed keys in at least one test a year.
- Apply a clean restore checklist: pre-compromise restore point, malware scan, persistence hunt, secret rotation, completeness and function checks.
- Record recovery time and recovery point achieved against objectives for every test.
- Track findings to closure and re-test, and report results by critical service to the board.
Standards and references
- NIST: SP 800-53 Rev. 5 (CP-4, CP-9, CP-10)
- NIST: Cybersecurity Framework 2.0
- NIST: SP 800-34 Rev. 1 Contingency Planning Guide for Federal Information Systems
- CIS: Control 11 Data Recovery
- CISA: #StopRansomware Guide
- NCSC: Principles for ransomware-resistant cloud backups
- AWS: AWS Backup restore testing
- NIST: SP 800-209 Security Guidelines for Storage Infrastructure
Related NHI Mgmt Group resources: Backup Security and Immutable Backup Guide · Ransomware Response and Recovery Guide · Business Continuity and Disaster Recovery Guide · Cyber Recovery Vault Guide