OT Incident Response Guide: Containment, Safety and Recovery

An IT incident response plan applied unchanged to a plant can turn a cyber incident into a safety incident, because the instinct to isolate, wipe and reimage ignores the physical process the systems control. Operational technology (OT) incident response has to keep people, equipment and the environment safe first, keep or restore the process second, and only then pursue the attacker. It depends on process engineers and operators as much as on security analysts, and on the ability to run the plant by hand while systems are untrusted. This guide covers how OT response differs, decision rights, preparation (playbooks, golden images, logic backups, spares), detection sources, containment choices including when not to isolate, forensics on OT devices at a high level, recovery and validation with operations, and reporting. It is written for OT security leads, site and engineering managers, and the IT incident responders who will be pulled into a plant incident. For programme-level OT controls see the OT Security Guide; for the general response lifecycle see the Incident Response Guide.

Key takeaways

• Safety decides containment. Every containment action is tested against one question first: does this remove view, control or protection that operators need to keep the process safe?
• Process engineers sit in the response cell. The joint government Principles of OT cyber security say the involvement of a process engineer is important during investigation, containment and recovery; security staff are not expected to understand the physical system in depth.
• Manual operation is a response capability, not an afterthought. CISA, FBI, EPA and DOE list practising manual operation as one of five primary OT mitigations; if it has not been rehearsed, it is not an option on the day.
• Recovery depends on trusted, offline known-good state. Controller logic, firmware versions, HMI and engineering workstation images and configuration must be backed up offline, hashed and stored outside the reach of corporate IT accounts.
• Restart is an engineering sign-off. A system is not recovered until operations confirm logic, setpoints, alarms and safety functions match the approved baseline and the process has run stably under heightened monitoring.

How OT incident response differs from IT

NIST SP 800-82 Rev. 3 is blunt about priorities: in its comparison of IT and OT, "Any security measure that impairs safety is unacceptable." It also notes that OT outages are planned days or weeks in advance and that many updates need vendor testing. Response plans must be built around those facts rather than around the IT assumption that a host can be taken offline and rebuilt within hours.

NIST also describes graceful degradation: a process should move in stages from normal automated operation, to emergency operation with more operator involvement, to manual operation with no automation. A good OT response plan maps each stage to named systems and triggers, so the operations team knows which step it is taking and why.

Roles and decision rights

SP 800-82 Rev. 3 recommends that the OT security team includes, at minimum, IT staff, a control engineer, a control system operator, a network and system security expert, a management representative and a physical security member, and that it consults the control system vendor or integrator. The same mix is needed in the response cell. Agree in writing, before an incident, who may do what:

• Operations or shift manager: owns the process. Decides on shutdown, transfer to manual or local control, and restart. Security advises but does not override.
• OT incident lead: coordinates investigation and containment, keeps the timeline and links the plant to the corporate CSIRT.
• Process or control engineer: interprets process data, judges whether an anomaly is cyber or mechanical, and validates logic and setpoints during recovery.
• Safety lead: confirms the status of safety instrumented systems and whether any response action affects them.
• IT incident manager: owns the IT side of the IT to OT boundary and the decision to sever enterprise links from the IT end.
• Vendor and integrator contacts: pre-agreed call-out terms, named engineers and remote access rules that apply during an incident.
• Legal, communications and regulatory reporting: as for any incident, with sector regulators and ISACs added.

The Principles of OT cyber security add a question many plans skip: is the organisation prepared to send staff to a site knowing that a bad actor has been, or is currently, on the network? Decide in advance what protective steps make that acceptable, such as operating locally with HMIs disconnected from untrusted networks.

Preparation: playbooks, golden images and recovery kits

Most of the difference between a short OT incident and a long one is decided before it starts. The joint principles expect OT-specific incident response plans and playbooks to be integrated with the wider plan, and state that all plans, playbooks and third-party packs must be regularly exercised.

OT playbooks to write first

Recovery kit and known-good state

• Controller logic and configuration: current project files for every PLC, RTU, DCS controller and safety controller, exported after each approved change, hashed and stored offline. Record firmware versions alongside.
• Golden images: HMI, historian, engineering workstation and jump host images with licence keys and installation media. MITRE ATT&CK for ICS mitigation Data Backup (M0953) refers to managing gold-copy backup images and configurations for key systems and keeping backup systems separate from the corporate network.
• Spare parts and standby systems: spare controllers, I/O modules, network switches and HMI hardware for critical cells, plus hot standby platforms where data destruction would halt the process.
• Offline documentation: network drawings, zone and conduit diagrams, IP plans, cause and effect matrices and manual operating procedures, in printed or offline form.
• Out-of-band communications: radios or phones that do not depend on the corporate network or identity provider.
• Manual operating procedures: written, current and rehearsed, with staffing levels for sustained manual operation.

For backup architecture and immutability, see the Backup Security and Immutable Backup Guide.

Detection sources in OT

Many controllers log little, so detection relies on network visibility, integrity checks and operators noticing that the process looks wrong. SP 800-82 Rev. 3 lists the kinds of event to plan for: loss of view, loss of control, unauthorised changes to commands or alarm thresholds, inaccurate information sent to operators and interference with safety systems. The table maps common sources to MITRE ATT&CK for ICS v20 techniques. Note that v20 renumbered several ICS techniques into the T16xx and T17xx range, for example Block Operational Technology Message (T1691), Unauthorized Message (T1692) and Modify Firmware (T1693), so update detection mappings written against older versions.

Give operators a simple route to report "this does not look right" to the OT incident lead, and treat it as a detection source with the same weight as an alert. For alert pipelines and correlation see the SIEM Guide.

Containment: isolate, island or keep running

Containment in OT is a choice between risks. Cutting a link can stop an attacker and also stop the data a plant needs to run, and pulling an HMI off the network can leave operators blind. Use a decision table agreed with operations in advance.

When not to isolate: do not cut a network path before checking whether it carries safety system communications, controller to controller interlocks, time synchronisation or the only view of a running process. Do not power off controllers or HMIs to "stop" malware without engineering advice. The Principles of OT cyber security note that critical OT systems should not rely on IT systems to operate; if yours do, record each dependency now so that severing the conduit is a known quantity. The Colonial Pipeline attack shows the other side: when an operator cannot prove the boundary holds, it may shut down the process even though the ransomware is on IT systems.

Where identity fits

Shared engineering accounts, vendor accounts and jump host credentials are the paths attackers use to cross into OT, and a domain trust between IT and OT can carry an IT compromise straight to the plant. The joint principles warn that "a more critical environment should never be administered from a less critical environment" and flag the risk when OT backups are managed by privileged corporate IT accounts. During containment, disable remote and vendor accounts at the DMZ, rotate jump host and service account credentials, and check that backup systems cannot be reached with compromised IT credentials. See the OT and ICS Identity and Access Guide.

Forensics on OT devices

OT forensics is about reconstructing what changed in the process as much as what ran on a computer. CISA publishes a recommended practice on creating cyber forensics plans for control systems; the plan should be written before it is needed, because evidence on controllers is easily lost. The joint principles point out that the usual fault fix, re-flashing or loading a known good configuration, can wipe volatile memory that may have helped an investigation.

• Engineering workstations, HMIs and servers: standard Windows or Linux acquisition (memory, disk, logs) when they can be spared; take memory before powering off.
• Controllers: upload the running program and configuration with the vendor's engineering tool, read-only where possible, and compare against the offline baseline. Record firmware version, mode and key switch position. Have the vendor present for unfamiliar devices.
• Historian and alarm logs: export the time window around the incident; these show the process effect and help separate cyber cause from equipment failure.
• Network evidence: keep passive captures and monitoring records. Do not run active scans or IT forensic agents against Level 0 to Level 2 devices during an incident.
• Chain of custody: label removed hardware such as failed modules or replaced workstations, and keep them for analysis rather than returning them under warranty.
• Clocks: controller and HMI clocks often drift; record offsets so the timeline holds together.

For evidence handling, imaging and timeline methods, see the Digital Forensics (DFIR) Guide.

Recovery and validation with operations

The joint principles ask three uncomfortable questions: is restoring from backup acceptable, can backups be trusted if an attacker has been present for some time, and is there a way to validate that the critical OT system is safe after recovery? They also note that there is often no timely large-scale method to verify that a system has been returned to a safe state. Recovery plans have to answer those questions for each critical system.

1. Choose the restore point: select the last backup that predates the earliest known attacker activity, and verify its hash against the record made when it was taken.
2. Rebuild Level 3 and Level 2 hosts: restore engineering workstations, HMIs and historians from golden images onto clean or wiped hardware, patched and hardened, before connecting them to controllers.
3. Restore controller logic and firmware: load verified project files and firmware in a planned window, then compare online and offline programs so they match.
4. Re-establish identities: new credentials for engineering, vendor, service and jump host accounts; remove accounts created during the incident.
5. Functional testing with operations: loop checks, interlock and alarm tests, and safety function proof tests as the safety lead requires.
6. Staged restart: bring units back one at a time from manual to automatic control, with engineers watching process values against expected ranges.
7. Reconnect last: restore the IT to OT conduit only after IT is declared clean, with tightened rules and heightened monitoring for an agreed period.
8. Operations sign-off: the operations manager, not security, declares the unit recovered.

Where power loss is possible, the principles refer to systems being black-start compliant, "meaning they will not hinder a restart after a complete loss of electricity". Test that the authentication, licence servers and time services needed for restart work when everything starts cold. For the ransom decision and clean-room rebuilds, see the Ransomware Response and Recovery Guide. Even a working decryptor leaves the validation problem above unsolved, so OT recovery plans should assume a rebuild from known-good state.

Reporting and external coordination

Name one person responsible for each clock, and pre-draft the notification fields. See the NIS2 Directive Guide.

Building the capability: a phased approach

1. Agree decision rights: write down who can order isolation, manual operation, shutdown and restart at each site, and get operations to sign it.
2. Map dependencies: list what each critical OT system needs from IT (authentication, DNS, time, licences, file shares, remote support) and what happens when the conduit is cut.
3. Build the recovery kit: offline logic backups with hashes, golden images, spares and printed procedures. Test a restore of one controller and one HMI in a lab or outage.
4. Write five or six playbooks from the table above, each with a containment decision table.
5. Exercise: start with a tabletop at one site with operators and engineers, then a functional exercise that runs a unit in manual mode and severs the conduit during a planned outage. Keep the change rolled back by default: restore the conduit and automation at the end of the window and record any step that did not work.
6. Train and connect detection: give IT responders basic OT training and operators basic incident reporting training, and route OT monitoring and operator reports to the OT incident lead and the SOC with clear escalation.
7. Review after each exercise or incident and update playbooks, baselines and the dependency map.

Decision guidance

• Small sites (a water utility or single plant with a handful of PLCs): focus on offline logic backups, manual operating procedures, a one-page decision table, vendor call-out and a CISA or national CERT contact. A retainer with OT experience matters more than in-house tooling.
• Large multi-site operators: a central OT incident function with site leads, an OT SOC or OT-trained analysts inside the main SOC who can read process context, standard playbooks, a common backup and golden image service, passive monitoring at each site and exercises that include IT and OT together.
• In-house or external: keep decision rights and process knowledge in house; buy specialist ICS forensics and surge support.

Detection and measurement

• Backup coverage: percentage of critical controllers with a hashed offline backup taken after the last approved change.
• Restore proof: time to restore one controller and one HMI from backup in the last test, against target.
• Manual operation readiness: units whose manual procedures were rehearsed in the last 12 months.
• Dependency clarity: critical OT systems with a documented answer to "can it run if the IT conduit is cut?"
• Integrity alerting: share of program downloads and mode changes matched to a change ticket within one working day.
• Exercise follow-through: corrective actions from OT exercises closed by their due date.

Common failure patterns

• IT playbooks applied to the plant: responders isolate or reimage HMIs and lose the operators' view of a running process.
• Hidden IT dependencies: the plant cannot run when the conduit is cut because of licence servers, domain authentication or scheduling systems. The Jaguar Land Rover cyberattack showed how long production can stop when a cyber incident reaches manufacturing systems.
• Backups that cannot be trusted: logic backups missing, out of date or stored on systems administered with corporate IT accounts that the attacker controls.
• Evidence wiped by the fix: controllers re-flashed and workstations rebuilt before anyone captures program state or memory.
• Manual operation never rehearsed: procedures exist but staffing, local instrumentation or radios do not support them for more than a shift.
• Restart without validation: units returned to automatic control before setpoints, alarms and safety functions are checked against the baseline.

Framework mapping

Practitioner checklist

• Document who may order isolation, manual operation, shutdown and restart at each site, signed by operations.
• Put a process engineer and safety lead in the response cell for every OT incident.
• Keep hashed, offline backups of every critical controller program and firmware version, taken after each approved change.
• Store golden images for HMIs, historians, engineering workstations and jump hosts away from corporate IT administration.
• Record every IT dependency of each critical OT system and test running with the conduit cut.
• Write a containment decision table that states when not to isolate.
• Capture controller program state and workstation memory before re-flashing or rebuilding.
• Require functional and safety tests and operations sign-off before restart.
• Pre-assign owners for CISA, ISAC, NIS2 or other reporting clocks.
• Exercise manual operation and the OT playbooks with operators at least yearly.

Standards and references

• NIST: SP 800-82 Rev. 3, Guide to Operational Technology (OT) Security
• ASD's ACSC, CISA, NSA, FBI, NCSC-UK and partners: Principles of operational technology cyber security
• CISA, FBI, EPA and DOE: Primary Mitigations to Reduce Cyber Threats to Operational Technology
• CISA: ICS Recommended Practices (including incident response and cyber forensics plans for control systems)
• MITRE: ATT&CK for ICS Matrix
• MITRE: ATT&CK for ICS Mitigation M0953, Data Backup
• EUR-Lex: Directive (EU) 2022/2555 (NIS2)
• CISA: Report a Cyber Issue

Related NHI Mgmt Group resources: OT Security Guide · Incident Response Guide · Ransomware Response and Recovery Guide · OT and ICS Identity and Access Guide · Business Continuity and Disaster Recovery Guide