Most industrial sites can restore a file server faster than they can restore the PLC logic, HMI projects and engineering workstation that actually run the plant. OT recovery depends on artefacts that IT backup tools rarely see: controller programs, device configurations, firmware versions, licence files, project archives and the exact software versions needed to open them. This guide explains what to back up across Purdue levels 0 to 3, how to build and maintain golden images, how to keep copies offline and trustworthy, and how to prove restoration works without disturbing production. It is written for control system engineers, OT security leads and site reliability or maintenance managers who own recovery.
Key takeaways
- Back up the means to restore, not just the data. A PLC program is useless without the engineering software version, licence and drivers needed to download it, so the engineering workstation image is part of the backup.
- Golden images are versioned engineering records. Each image needs an owner, a build sheet, a hash and a tested restore date, and must be refreshed after every approved change.
- Offline means offline. At least one copy of controller logic and workstation images must be unreachable from the domain and from OT backup service accounts, so ransomware and wipers cannot reach it.
- Integrity before restore. Compare running controller logic with the stored master and verify backup hashes before using them, otherwise you may restore the attacker's change.
- Test on the bench, not on the line. Restore drills use spare controllers, virtual machines or a test cell, and CISA's performance goals expect OT backups to be tested at least annually.
What an OT backup must contain
CISA's Cross-Sector Cybersecurity Performance Goals (goal 2.R, System Backups) state that for OT the stored information should include configurations, roles, PLC logic, engineering drawings and tools, that backups are stored apart from the source systems, and that they are made and tested at least once a year. In practice most sites need a far tighter cadence for anything that changes. The table sets out the inventory.
| Level / asset | Artefacts to capture | Typical method | Trigger |
|---|---|---|---|
| Level 1 controllers (PLC, RTU, safety controller) | Program and project file, tag database, hardware configuration, firmware version, network settings, CPU protection settings | Upload through vendor engineering software, or controller change management tool | After every approved change, and on a schedule to detect drift |
| Level 0 and 1 field devices (drives, relays, smart instruments, I/O modules) | Parameter sets, relay settings files, firmware version | Vendor configuration tools, relay setting management software | After commissioning, maintenance or replacement |
| Level 2 HMI and SCADA servers | Full system image, application projects, historian configuration, alarm databases, licences | Image based backup to OT backup server, plus project export | Weekly image, project export after change |
| Engineering workstations | Full golden image with exact engineering software versions, patches, drivers and licence dongle details | Offline image capture | After any software or patch change |
| Level 3 site services (historian, domain controllers in OT, jump hosts, patch servers) | System images, AD system state, configuration and secrets inventory | Image based backup with immutable copy | Daily or weekly depending on rate of change |
| Network and security devices (switches, firewalls, data diodes) | Running configurations, rule bases, firmware versions | Configuration management export | After every change |
| Documentation | Network drawings, P&IDs, cause and effect matrices, IP and tag lists, vendor contacts, manual operating procedures | Printed and offline electronic copies | After every change, reviewed annually |
- Licences and keys: record licence server details, dongle serial numbers, and activation procedures. Many restores fail because licences are bound to hardware that no longer exists.
- Installers: keep verified installation media for every engineering tool version in use, with the vendor's published hash where available. Vendors withdraw old versions.
- Credentials needed for restore: controller passwords, OPC UA certificates and project passwords belong in a sealed, offline vault, not in a spreadsheet next to the backups.
Golden images for OT hosts
A golden image is a known-good, hardened, documented build of an OT host that can be redeployed to new or wiped hardware. For OT it is often the only realistic route back for HMIs and engineering workstations, because reinstalling vendor software by hand can take days and may not reproduce the validated configuration. See the Server Build and Golden Image Hardening Guide for the IT build practices this extends.
- Define the build sheet: operating system version, patch level, vendor application versions, drivers, hardening settings, local accounts and allowlisting policy, all agreed with the system vendor or integrator.
- Build from verified media: check installer hashes against the vendor's publication before installing.
- Harden and scan offline: apply the site baseline, then scan the image with up-to-date antimalware before it is sealed.
- Capture and hash: capture the image, record a SHA-256 hash, the build date, the builder and the approver.
- Store in at least two places: the OT backup server and an offline copy (removable drive in a safe, or offline vault).
- Test restore: deploy to spare hardware or a virtual machine and confirm the application opens projects and can communicate with a test controller.
- Retire old images deliberately: keep the previous version until the new one has passed a restore test, then mark it superseded.
Hardware matters. Older HMIs and workstations may only run on specific chipsets or need specific serial cards. Keep spare hardware for critical hosts, or validate that the image runs on current hardware or under virtualisation that the vendor supports.
Protecting backups from ransomware and wipers
Destructive ICS malware targets the recovery path. ATT&CK for ICS records Data Destruction (T0809) for families such as Industroyer and INCONTROLLER, and IT ransomware routinely seeks out backup servers first. ATT&CK's Data Backup mitigation (M0953) is only effective if the copies survive.
| Protection | What it stops | OT implementation | Control reference |
|---|---|---|---|
| Offline copy | Encryption or deletion by anything on the network | Removable media rotated into a safe; offline vault with scheduled connection windows | NIST SP 800-53 CP-9(3) Separate Storage for Critical Information |
| Immutable or write once storage | Deletion by a compromised backup admin | Object lock or hardened repository on the OT backup server | NIST CSF 2.0 PR.DS-11 |
| Separate backup identities | Domain admin compromise reaching backups | Backup server not joined to IT domain, dedicated accounts, MFA on the console | NIST SP 800-53 CP-9, AC controls |
| Integrity hashing | Restoring tampered logic or images | Hash at capture, verify before restore, compare logic with running controllers | NIST SP 800-53 SI-7; NIST CSF 2.0 RC.RP-03 |
| Encryption at rest | Theft of project files and credentials | Encrypt offline media; keys held separately | NIST SP 800-53 CP-9(8) Cryptographic Protection |
| Off-site copy | Site loss through fire, flood or full compromise | Copy at another site or a secured vendor escrow | NIST SP 800-53 CP-6 Alternate Storage Site |
Backup service accounts are non-human identities with wide reach: they read every HMI and server and often write to every repository. Keep them local to the OT backup domain, restrict where they can log on, and rotate their credentials. See the Backup Identity Isolation Guide and the Backup Security and Immutable Backup Guide.
Controller logic integrity and change detection
Controller backups double as a detection control. Stuxnet and Triton both changed controller programs (ATT&CK Modify Program T0889 and Program Download T0843), and PLC-Blaster showed code can spread between controllers. Comparing the running program with the approved master copy is the simplest way to find unauthorised change.
- Master copy: one authoritative, version-controlled project per controller, updated only through change control (NIST SP 800-53 CM-3).
- Scheduled comparison: upload and compare running logic, or use the controller's program checksum or signature where the platform provides one, on a weekly or daily schedule for critical units.
- Firmware tracking: record firmware versions, because ATT&CK lists System Firmware modification (T1693.001) among Triton's behaviours. Verify firmware against vendor hashes during maintenance.
- Safe upload practice: uploads read from controllers and are generally low risk, but confirm with the vendor that upload does not affect scan time on older CPUs, and schedule them outside critical operations.
- Alert on difference: a mismatch with no change ticket is handled as a security incident, not as housekeeping.
See the PLC Security Guide and the Configuration Drift and Baseline Enforcement Guide.
Choosing a backup approach
The right tooling depends on fleet size, vendor mix and staff. All three approaches can work, but each fails in a different way.
| Approach | Best fit | Strengths | Weaknesses |
|---|---|---|---|
| Manual uploads by engineers | Small sites, under roughly 20 controllers | No new software in OT; engineers understand what they capture | Depends on discipline; drift goes unnoticed; no central hashing |
| Vendor specific change management tools | Single vendor sites | Native project formats, version comparison, audit trail | Gaps for other vendors and field devices |
| Multi-vendor OT change and backup platform | Large or multi-site estates | Scheduled uploads, comparison and reporting across vendors | Server becomes high value target with credentials to every controller |
| IT image backup extended into OT | Level 2 and 3 Windows and Linux hosts | Mature, immutable options, fast restore | Agents may not be vendor approved; does not capture controller logic |
- Small organisations: a disciplined manual process with a monthly upload, offline media and a documented restore test is better than an unmanaged platform.
- Large organisations: centralise, but treat the backup and change management server as a crown jewel: its own zone, MFA, allowlisting and monitored service accounts.
- Vendor approval: check that backup agents or upload frequency are supported by the system vendor, especially on safety or validated systems.
Recovery planning and restoration order
NIST SP 800-82 Rev. 3, the Guide to Operational Technology Security, treats recovery within the safety and reliability constraints of the process. Recovery must therefore be planned with operations, not only by IT. The CISA CPGs also expect IT and OT incident response plans to be drilled at least annually (goal 2.S). A typical restoration order:
- Make the process safe: rely on safety systems and manual operation. CISA lists maintaining the ability to operate OT systems manually as a primary OT mitigation.
- Establish a clean build environment: an isolated recovery network, verified media and a clean engineering workstation from the offline golden image.
- Verify backups: check hashes and compare stored logic with known-good versions (NIST CSF 2.0 RC.RP-03).
- Restore identity and core services: OT domain controllers or local accounts, time sources and licence servers.
- Restore SCADA and HMI: servers first, then operator stations, validated against the alarm and display database.
- Validate controllers: compare running logic and firmware to masters; download only where a difference is confirmed and approved.
- Reconnect in stages: reconnect conduits one at a time with monitoring in place, and keep IT connections closed until the cause is understood.
See the OT Incident Response Guide and the OT Ransomware Response Playbook for the response steps around recovery.
Testing restores without disturbing production
- Bench restores: restore controller projects to spare CPUs of the same model and firmware, and confirm the program runs and matches.
- Virtual restores: restore HMI and server images into an isolated virtual environment and confirm the application starts, licences activate and projects open.
- Timed drills: measure time to restore an engineering workstation from offline media to a working state, including licences.
- Tabletop plus practical: combine a tabletop on decision points with a practical restore of one system per quarter. See the Cyber Incident Exercises Beyond Tabletop Guide.
- Roll back the test: restored test hosts stay isolated and are wiped after the drill; never reconnect a test restore to production networks.
Detection and measurement
- Monitor: backup job failures, deletion or retention changes on repositories, logons by backup service accounts outside their schedule, controller logic mismatches, firmware changes, and connections to the offline vault outside its window.
- Coverage: percentage of controllers and field devices with a current backup (target 100 percent of critical assets).
- Freshness: percentage of backups taken after the latest approved change.
- Restore success: percentage of restore tests passing first time, per asset class.
- Time to recover: measured time to restore an engineering workstation and a critical HMI from offline media.
- Integrity: number of unexplained logic or firmware mismatches found, and time to resolve each.
Framework mapping
| Framework | Reference | Relevance |
|---|---|---|
| NIST CSF 2.0 | PR.DS-11, PR.PS-01, RC.RP-02, RC.RP-03 | Backups created, protected and tested; configuration management; integrity verified before restoration |
| NIST SP 800-53 Rev. 5 | CP-9, CP-9(1), CP-9(3), CP-10, CP-4, CM-2, CM-3, SI-7 | System backup, testing, separate storage, reconstitution, baselines, integrity |
| CISA CPGs | 2.R System Backups, 2.S Incident Response Plans, 5.A Incident Planning and Preparedness | OT backups include PLC logic and tools; tested at least annually |
| MITRE ATT&CK for ICS | M0953 Data Backup; T0809, T0889, T0843, T1693.001 | Mitigation and the behaviours it counters |
| NIST SP 800-82 Rev. 3 | OT security countermeasures | Recovery within safety and reliability constraints |
Common failure patterns
- Backups on the domain they protect: the OT backup server joined to the IT domain is encrypted along with everything else.
- Logic without tools: project files are safe, but the matching engineering software version or licence is gone, so nothing can be downloaded.
- Stale masters: the stored program predates the last three changes, so a restore reintroduces old faults or removes safety modifications.
- Never-tested images: images that fail on current hardware are discovered during the incident, not before.
- Whole-business shutdown: manufacturing can stop for weeks when IT and OT recovery are coupled, as the Jaguar Land Rover Cyberattack showed with a phased restart after more than five weeks.
- Precautionary shutdowns: without confidence in OT recovery, operators shut down even when OT is not hit, as with the Colonial Pipeline Ransomware Attack.
Practitioner checklist
- Inventory every controller, field device, HMI, server and workstation that needs a backup, with an owner.
- Capture controller logic, configurations and firmware versions after every approved change.
- Build, hash and document golden images for every HMI and engineering workstation type.
- Keep installers, licences and restore credentials offline with the images.
- Hold at least one offline and one off-site copy, unreachable by domain or backup service accounts.
- Compare running logic with masters on a schedule and treat unexplained differences as incidents.
- Run a bench or virtual restore test for each asset class at least annually, critical systems quarterly.
- Write and drill the restoration order with operations, including manual operation.
- Report coverage, freshness and restore success metrics to the site leadership.
Standards and references
- NIST: SP 800-82 Rev. 3 Guide to Operational Technology (OT) Security
- CISA: Cross-Sector Cybersecurity Performance Goals
- CISA: Primary Mitigations to Reduce Cyber Threats to Operational Technology
- NIST: SP 800-53 Rev. 5
- NIST: Cybersecurity Framework 2.0
- MITRE ATT&CK: Data Backup (M0953)
- CISA: APT Cyber Tools Targeting ICS/SCADA Devices (AA22-103A)
Related NHI Mgmt Group resources: OT Incident Response Guide · OT Ransomware Response Playbook · Cyber Recovery Vault Guide · OT Security Guide