Join our Newsletter — 33% off our NHI Course
Home› Guides› Disaster Recovery Runbook Template: Per System Recovery Steps…
Template Resilience & Recovery

Disaster Recovery Runbook Template: Per System Recovery Steps and Checks

PDF for newsletter subscribers · free to read online

← All guides
By NHI Mgmt Group, edited by Lalit Choda Updated 14 November 2026 14 min read
On this page

A disaster recovery plan says which systems come back first and how fast, but on the night the engineer at the console needs a per system runbook that lists every step, every credential and every check, because nobody should be working out a restore sequence from memory during an outage. This template is that per system document: one copy for each application, platform or recovery group named in your Disaster Recovery Plan Template. It follows the NIST SP 800-34 Rev. 1 sequence of activation and notification, recovery and reconstitution, and adds the clean restore point and credential checks that ransomware recovery needs. It is written for system owners, platform engineers and the recovery lead who coordinates them. Replace every [bracketed] placeholder, keep a printed or offline copy, and run it in a test at least once a year.

How to use this template

  • One runbook per system or recovery group: the plan sets tiers, recovery time objectives (RTO), recovery point objectives (RPO) and restore order across the estate. This runbook takes those values as given and describes how one system is brought back. Where a system has no runbook, the plan's RTO for it is a guess.
  • Write for the deputy, not the expert: NIST SP 800-34 says recovery procedures should be straightforward step by step instructions and that no procedural step should be assumed or omitted. Test the runbook by having someone other than its author execute it.
  • Keep it available when systems are not: store the master in [runbook repository] and an offline copy in [location], including the access details in section 4. A runbook held only on the wiki that just went down is not a runbook.
  • Link, do not duplicate: point to vendor procedures, infrastructure as code (IaC) pipelines and scripts by name and version rather than pasting them in, so they cannot drift. Record the exact commands or console paths in section 6 only where no maintained script exists.
  • Specialised runbooks exist for some platforms: use the Active Directory Forest Recovery Runbook Template for the directory and the Secrets Manager Resilience Guide for vault promotion, and reference them from section 2 as dependencies.
  • Owner and review: [System owner] owns this runbook. Review every [12] months, after each test, and after any change to architecture, hosting, backup product or dependencies.

1. Runbook header

FieldValue
System or recovery group[Payments API and database]
Recovery tier (from DR plan)[Tier 1]
RTO / RPO[4 hours] / [15 minutes]
Last demonstrated recovery time and data loss[date of test, 3h 40m, 6 minutes]
Recovery strategy[Warm standby in secondary region; fallback rebuild from backups and IaC]
Runbook owner and deputy[Name, role] / [Name, role]
Data owner (signs off data validation)[Name, role]
Version and last tested[1.3, date, test type]

2. Scope, dependencies and prerequisites

2.1 Components in scope. List every component this runbook restores, with its location and identifier: [application servers or containers, database cluster, message queue, object storage buckets, load balancer, DNS records, scheduled jobs].

2.2 Upstream dependencies. The following must be running and verified before step 6.1 starts. Record which runbook restores each one.

DependencyWhy it is neededRestored byHow to confirm it is ready
Recovery network and DNSName resolution and routing for all components[Network runbook][Resolve test records from recovery subnet]
Directory and identity providerAdministrator sign-in and service authentication[AD forest recovery runbook; IdP runbook][Test sign-in with recovery admin account]
PKI and certificatesTLS for endpoints and mutual TLS between services[PKI runbook][Certificate chain validates; CRL reachable]
Secrets vaultDatabase passwords, API keys and encryption keys used at start-up[Vault runbook][Read test secret from recovery path]
Backup platformRestore source[Backup runbook][Catalogue browsable; restore job test passes]
External services[Payment processor API, email relay][Supplier][Health endpoint reachable from recovery site]

2.3 Downstream dependants. List the systems that will fail or queue work while this one is down, and who must be told when it returns: [list, with contact].

2.4 Non-human identities used by this system. List every service account, workload identity, API key and certificate the system needs, where its secret is held, and whether it must be rotated after a compromise. Services that fail at start-up because a credential was not restored or no longer matches are one of the most common causes of missed RTOs. See the Secrets Rotation Runbook Template.

3. Activation criteria and authority

3.1 This runbook is opened when [the DR plan is invoked for this system's recovery group], or when [the system is expected to exceed its RTO and the recovery lead authorises failover or rebuild].

3.2 Who may authorise: failover to [secondary region] by [recovery lead or deputy]; rebuild from backup by [recovery lead] with [incident lead] confirming the system is released for recovery where a cyber incident is involved.

3.3 Choose the path. Record the decision and time in the recovery log.

SituationPathNotes
Site, region or hardware failure; data and credentials trustedA. Failover to standbyFastest; check replication lag against RPO first
Data corrupted or deleted and the change has replicatedB. Restore data to a point before the corruptionStandby holds the same bad data; do not fail over
Ransomware or attacker with administrative accessC. Rebuild from IaC and golden images, restore data from a verified clean backupOnly after the incident lead releases the system; credentials rotated first
Single component failure inside a healthy systemNormal incident processThis runbook is not needed

4. Access needed

4.1 Before any step, confirm the recovery team can reach every item below. Hold the locations, not the secrets, in this document.

  • Administrative accounts: [recovery admin accounts or break-glass accounts], stored in [vault path or sealed offline store]. Follow the Break Glass Access Procedure Template for retrieval and logging.
  • Cloud and console access: [account or subscription IDs, recovery roles], with [phishing-resistant MFA tokens held by named people].
  • Encryption keys: [backup encryption key location, key custodians, HSM quorum if applicable].
  • Code and images: [IaC repository and tag, pipeline name, golden image IDs, offline copy location].
  • Supplier support: [vendor, contract number, priority line, account manager].
  • Licences: [licence keys or entitlement portal access needed to run the product at the recovery site].

5. Pre-recovery checks

5.1 Outage assessment. Record what is affected, the suspected cause, the time of last known good state, and the estimated time to restore at the primary site. If the primary will be back inside the RTO, consider waiting rather than failing over.

5.2 Select the restore point. For path B or C, record the backup or snapshot chosen, its timestamp, and why it is believed clean. CISA advises checking for dropper malware before rebuilding from backups and restoring only from offline or protected copies. Confirm with [security operations] that the restore point predates the earliest known attacker activity, and that the copy's integrity check passes. See the Backup Security and Immutable Backup Guide.

5.3 Preserve evidence. Where a cyber incident is involved, confirm with the incident lead that images, memory and logs needed for investigation have been captured before anything is wiped or rebuilt.

5.4 Isolate the recovery environment. For path C, recover into [isolated recovery network or clean room] and connect only systems verified clean. See the Cloud Recovery and Clean Room Guide.

5.5 Go / no-go. [Recovery lead] confirms dependencies in section 2.2 are ready, access in section 4 works, and the path is agreed. Record the time: the step timings in section 6 are measured from here.

6. Recovery procedure

6.1 Execute in order. Each step names an owner, an expected duration, where the exact instructions are, and the evidence to record. If a step exceeds its expected duration by [50%], escalate under section 10.

#StepOwnerExpected timeInstructionsEvidence
6.1Provision infrastructure (network, compute, storage) in recovery location[Platform engineer][30 min][IaC pipeline name, tag]Pipeline run ID
6.2Deploy operating system and base configuration from golden image[Platform engineer][20 min][Image ID; hardening baseline version]Image ID, baseline scan result
6.3Restore or promote the database to the selected point[DBA][60 min][Restore job or promotion procedure reference]Restore point timestamp, job log
6.4Rotate and inject credentials (database, service accounts, API keys) from the vault[Identity or platform engineer][20 min][Rotation runbook reference]Rotation ticket, secret versions
6.5Deploy the application at the approved release version[Application engineer][20 min][Deployment pipeline, release tag]Release tag, deployment log
6.6Restore configuration not held in IaC (feature flags, scheduler jobs, integration settings)[Application engineer][15 min][Configuration export location]Configuration diff against last export
6.7Restore certificates and bind TLS to endpoints[Platform engineer][10 min][Certificate source, binding steps]Certificate serial and expiry
6.8Run technical health checks with traffic still blocked[Application engineer][15 min][Health check script or URL list]Health check output
6.9Switch traffic: update DNS or load balancer to recovery endpoints[Network engineer][10 min plus DNS TTL][DNS records, TTL values, load balancer pool]Change record, resolution test
6.10Re-enable outbound integrations and scheduled jobs in the agreed order[Application engineer][15 min][Integration list with order]Job run log

6.2 Order matters. NIST SP 800-34 gives the example of restoring and verifying an application server's operating system before recovering the application and its data. Do not start step 6.5 until 6.3 and 6.4 are complete; an application that starts against a half-restored database or with stale credentials can write bad data or lock accounts.

6.3 Credentials after compromise. On path C, CISA advises resetting passwords for affected accounts only once the environment is clean and rebuilt, otherwise new credentials can be stolen again. Step 6.4 therefore runs inside the clean environment, and the old credentials are revoked, not just replaced.

7. Validation and acceptance

7.1 NIST SP 800-34 describes two checks before reconstitution is declared: validation data testing (data is complete and current to the last available backup) and validation functionality testing (the system works and is ready for normal operation). Complete both and record the results.

CheckMethodPass criterionSigned off by
Data completeness[Row counts, latest transaction ID, checksum comparison against source of truth]Data loss within RPO; gap documented[Data owner]
Data reconciliation[Reconcile with upstream or downstream records, for example payment processor settlement file]No unexplained differences[Data owner]
Functional test[Scripted user journeys, smoke test suite]All critical journeys pass[Application owner]
Security controls[Logging to SIEM, endpoint agent running, MFA enforced, no default or recovery credentials left active]All controls reporting[Security operations]
Monitoring[Alerts, dashboards, synthetic checks pointed at recovery endpoints]Alerts firing on test event[Operations]
Performance[Load test or observed latency][Within agreed degraded service level][Application owner]

7.2 Declare service restored. [Recovery lead] declares the system restored and records the time. Actual recovery time is measured from the declaration of disaster in the DR plan to this point.

8. Reconstitution and failback

8.1 Decide when to fail back. Running at the recovery site may continue for [period]. Failback is a planned change, scheduled at low demand, with its own go / no-go.

8.2 Resynchronise data from the recovery site to the primary and verify it with the same checks as section 7. Where NIST SP 800-34's concurrent processing is used, run both until the primary is confirmed correct.

8.3 Switch back traffic, then decommission temporary resources, remove temporary firewall rules and accounts, and revoke recovery credentials issued during the event.

8.4 Back up the restored system. Take a full backup soon after reconstitution and store it with other backups, as NIST SP 800-34 recommends, so the next restore does not start from a pre-incident copy.

9. Rollback and abort

9.1 Abort criteria. Stop and return to the last stable state if [restored data fails validation twice], [security operations finds indicators of compromise in the restored system], or [the recovery will clearly exceed the plan's maximum tolerable downtime and another path is faster].

9.2 Rollback steps. [Block traffic to recovery endpoints; revert DNS to previous values; preserve the failed environment for analysis; select an earlier restore point or a different path; restart from section 5.]

9.3 Never delete a failed recovery environment until [incident lead] confirms it is not needed as evidence.

10. Communication and escalation

10.1 Status updates go to [recovery bridge, out-of-band channel] every [30 minutes] and at each step marked as a checkpoint: after 5.5, 6.3, 6.8 and 7.2.

10.2 Escalate to [recovery lead] when a step overruns by [50%], a dependency is not ready, or a supplier does not respond within [1 hour]. Escalate to [CIO or DR plan owner] when the RTO is at risk.

10.3 Notify downstream dependants (section 2.3) and the service desk when service returns, with any data loss window users must re-enter.

11. Records and post-recovery actions

  • Recovery log: every step with start and end times, owner, decisions and evidence references, kept by [scribe].
  • Actual versus target: recovery time and data loss recorded against RTO and RPO, and copied to the DR plan's recovery objectives register.
  • Runbook defects: each step that was wrong, missing or unclear raised as an action with an owner and date.
  • After-action review within [10 working days], feeding the DR plan and this runbook.
  • Formal close: [recovery lead] confirms all temporary access is removed and the runbook is deactivated.

Implementation guidance

Rolling out runbooks across the estate

  1. Start with tier 0 and tier 1. Write runbooks first for foundation services (network, DNS, identity, PKI, vault, backup platform) and the most critical business systems. Everything else depends on the first group.
  2. Draft from the last real change. The most accurate source for steps is the most recent deployment or rebuild record; walk it with the engineer who did it.
  3. Tabletop, then technical test. Walk the runbook with owner and deputy, then execute it in a test environment, timing every step. The Disaster Recovery Testing Guide covers test types and schedules.
  4. Have the deputy run the test. Steps that only the author can perform reveal missing detail.
  5. Automate the stable parts. Replace manual steps with pipelines over time, but keep the runbook as the control document; see the Runbook Automation Security Guide for securing those pipelines and their credentials.
  6. Tie updates to change management. Add "DR runbook updated?" to the change template for any change to hosting, dependencies or credentials.

Measuring whether runbooks work

MetricTarget
Tier 0 and tier 1 systems with a runbook tested in the last 12 months[100%]
Demonstrated recovery time within RTO in the last test[100% of tier 1]
Steps that failed or needed undocumented action in the last testFalling test on test
Runbooks reviewed after relevant changes[95%] within [30 days] of change
Recovery tests executed by the deputy rather than the author[At least one per year per runbook]

Framework mapping

FrameworkReferenceRunbook sections
NIST SP 800-53 Rev. 5CP-2 Contingency Plan; CP-4 Contingency Plan TestingWhole runbook; Implementation guidance
NIST SP 800-53 Rev. 5CP-9 System Backup; CP-10 System Recovery and Reconstitution5, 6, 8
NIST CSF 2.0PR.DS-11; RC.RP-03Backups protected and tested; integrity verified before restore (5.2)
NIST CSF 2.0RC.RP-02; RC.RP-05; RC.RP-06Recovery actions performed (6); restored assets verified (7); end of recovery declared (11)
ISO/IEC 27001:2022Annex A 5.30 ICT readiness for business continuity; 8.13 Information backupWhole runbook; 5.2

Common failure patterns

  • Credentials missing at start-up: the database comes back but the application cannot authenticate because a service account password or API key was rotated after the backup or held only in the failed environment.
  • Failing over to corrupted data: replication carries deletion or encryption to the standby; section 3.3 path B exists for this case.
  • Access details held online only: vault paths, licence keys and supplier numbers stored on systems that are down.
  • Timings never measured: the runbook's expected times are estimates that have never been tested, so the plan's RTOs are not evidenced.
  • No rollback: a failed restore leaves the team with neither the old nor the new environment working.

Sources

Related NHI Mgmt Group resources: Disaster Recovery Plan Template · SOC Runbook Template · Incident Response Playbook vs Runbook · Secrets Manager Resilience Guide · Business Continuity and Disaster Recovery Guide

PDF for subscribers

Take this guide with you

Every section, table and checklist in one PDF, with links to the sources and related guides. Subscribe to our weekly NHI & AI Identity Journal and we’ll email you a download link.

Free weekly newsletter

Subscribe to the NHI & AI Identity Journal

The latest on NHI and Agentic AI security – articles, research, breaches, news and events every week.

Bonus 33% off our NHI Course when you subscribe.

NHIMG Editorial Note
Researched and written by the NHI Mgmt Group editorial team using AI tools, under the editorial direction of Lalit Choda. Facts are drawn from the sources listed on this page and checked by our automated quality process. Last updated 14 November 2026. Spotted an error? Use our contact page.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM and Agentic AI security. nhimg.org

PDF Downloads for Newsletter Subscribers

Get the guide

Enter your details and we’ll email you a download link.

The full guide is free to read online. The PDF is a thank you for subscribing.

Fill in all fields marked * and tick the PDF box to continue.