Network Rebuild After Compromise Playbook: Restoring Trusted Firewalls, Switches and VPNs

When a firewall, VPN gateway, router or switch has been compromised, patching it is not recovery: the attacker may hold its configuration secrets, have changed its authentication, or left persistence that survives an upgrade or even a factory reset. This playbook template covers the rebuild of network infrastructure after a confirmed or strongly suspected compromise of an edge or core device: deciding scope, containing without losing evidence, establishing a trusted image and configuration, rebuilding, rotating every secret the device held, and returning to service in stages. It is written for network, security operations and incident response teams and is built from CISA Emergency Directive 24-01 (Ivanti), vendor recovery guidance, the December 2024 joint communications infrastructure guidance and NIST SP 800-184. Investigation of whether a device is compromised is covered in the Edge Device Compromise Investigation Playbook, and ongoing device protection in the Edge Device Security Guide; this page starts once you have decided to rebuild.

How to use this template

• Adapt the placeholders: replace [Organisation], [Network Lead], [IR Lead] and the other bracketed items with your names, contacts and thresholds before an incident, and store the completed copy where it is reachable when the production network is down (printed and on the out-of-band network).
• One device class per annex: add an annex per vendor and platform with the exact image verification, factory reset and configuration export steps from that vendor's documentation.
• Use with your incident response plan: this playbook is the recovery part of a wider response. NIST SP 800-61 Rev. 3 (April 2025) frames incident response within CSF 2.0; NIST SP 800-184 (December 2016) covers recovery planning and playbooks.
• Record decisions: every numbered decision point below should be logged with time, owner and rationale.
• Exercise it: rehearse at least annually on a lab or spare device, timing each section.

1. Scope and triggers

1.1 This playbook applies to [Organisation] firewalls, VPN concentrators and SSL VPN gateways, routers, switches, wireless controllers, load balancers and their management systems.

1.2 Invoke it when any of the following is true:

• A vulnerability affecting the device is listed in the CISA Known Exploited Vulnerabilities catalog and indicators of compromise are found, or the vendor or a government agency advises that affected devices be treated as compromised (as CISA ED 24-01 did for Ivanti Connect Secure, CVE-2023-46805 and CVE-2024-21887).
• Integrity checks, unexpected files, unknown local accounts, added SSH keys or changed AAA servers are found on a device.
• Device configurations have been exfiltrated, or credentials stored on the device are seen in use elsewhere.
• [IR Lead] decides that trust in a device cannot be re-established by patching alone.

1.3 Scope decision: list every device of the same model, firmware line and exposure as the confirmed device, every device that shares credentials or AAA secrets with it, and every device reachable from it on management protocols. Default to rebuilding the whole peer group, not just the device with indicators.

2. Roles and contacts

2.1 Communicate over channels that do not depend on the compromised network or identity system: [out-of-band chat or phone bridge].

3. Detection and triage

1. Confirm the trigger and record the device inventory in scope (model, serial, firmware, location, exposure, role).
2. Check the vendor advisory and the CISA KEV entry for the vulnerability; note whether the vendor or CISA warns of persistence that survives patching or reset. CISA's Supplemental Direction V2 for ED 24-01 warned that rootkit-level persistence may survive a factory reset; Fortinet documented a symlink that could persist read-only file access on FortiGate devices after upgrade.
3. Decide the rebuild model (decision point D1): rebuild in place with a verified image, replace with new hardware, or replace with a different platform. Prefer new or vendor-replaced hardware where the vendor cannot assure that a factory reset removes persistence, or where the device is end of life (see the Edge Device End of Life Replacement Playbook).
4. Estimate business impact and agree a maintenance approach (standby device, temporary bypass with compensating controls, or outage).
5. Open an evidence log and start the timeline.

4. Containment

4.1 Preserve before you change. Before resetting or disconnecting, and where it is safe, capture: running and startup configuration, local account and key lists, logs held on the device, a file system or memory image if the vendor or forensic team supports it, and any integrity tool output. ED 24-01 required forensic images and continued threat hunting on connected systems.

4.2 Isolate. Remove the device from the production path or restrict it to essential traffic, and cut its management access to everything except the forensic and rebuild workstations on the out-of-band network. Block its outbound management protocols so it cannot be used to reach other devices.

4.3 Cut the attacker's credentials now, not after the rebuild. Disable or reset accounts the device used to authenticate to directories and AAA servers, and monitor those services for use of the old credentials. CISA directed agencies to monitor authentication and identity services that were exposed and to keep auditing privileged accounts.

4.4 Contain laterally. Hunt on systems the device could reach: directory servers it bound to, management servers, jump hosts and peer devices. In the Salt Typhoon intrusions, Cisco Talos reported pivoting from one telecom's device to another's and redirecting AAA servers; see the Salt Typhoon Telecom Intrusions analysis.

4.5 Record the containment state and residual risk for the incident commander (decision point D2: proceed to eradication).

5. Eradication

5.1 Establish a trusted image.

• Download the target firmware from the vendor's official portal over a trusted path, to a workstation that has not touched the compromised device.
• Verify the published hash and, where supported, the vendor signature; use the platform's secure boot and image signature verification so the device refuses unsigned images.
• Choose a fixed, supported version; ED 24-01 required upgrade to a supported version as part of the rebuild.

5.2 Establish a trusted configuration. Do not reimport the compromised configuration blindly. Fortinet's guidance after the symlink issue was to treat all configuration as potentially compromised.

• Start from the last known-good configuration from your backup system, taken before the earliest suspected compromise date, or from your configuration standard.
• Diff it against the compromised configuration: remove unknown accounts, SSH keys, ACL entries, SNMP communities, AAA servers, tunnels, NAT rules, scheduled scripts and logging changes.
• Replace every secret in the configuration (see 5.4) before it is loaded.

5.3 Reset or replace the hardware. Follow the vendor's factory reset and recovery procedure (annex per platform), including clearing any secondary partitions and boot loaders the vendor documents. For new hardware, verify it on receipt against the order, inspect firmware version and run the vendor's integrity checks before connecting it.

5.4 Rotate every secret the device held or exposed. ED 24-01 listed what to revoke and reissue for Ivanti gateways; use it as the baseline for any edge device:

See the Leaked Credential Response Playbook for the rotation procedure and the Ivanti Connect Secure Exploitation analysis for why appliance-held machine credentials must all be assumed stolen.

6. Recovery

1. Build offline: load the verified image and the cleaned configuration with new secrets while the device is connected only to the out-of-band management network.
2. Harden before exposure: management only from the management network, no internet-facing management, unused services disabled, logging to the central collector. See the Out-of-Band Management Network Design Guide.
3. Verify integrity: run vendor integrity tools and compare the running configuration with the approved build. Do not rely on one check: CISA noted that Ivanti's integrity checker could miss compromise because attackers minimised traces.
4. Return in stages (decision point D3): pilot with a limited user group or traffic subset, then full service. Keep the old device powered off and preserved, not reconnected.
5. Heightened monitoring for [30] days: alert on new local accounts, SSH key changes, AAA server changes, configuration changes outside change windows, outbound connections from the device to unexpected destinations, and authentication with rotated credentials.
6. Patch cadence: apply further fixes quickly; ED 24-01 required future Ivanti updates within 48 hours of release. Set [Organisation]'s target for internet-facing devices at [48 hours].

7. Communication and notification

• Internal: staff told of the expected outage and of any forced password resets, through channels that do not depend on the affected systems.
• Regulators and authorities: [Legal] assesses notification duties under applicable law, sector rules and contracts, with deadlines recorded at triage. Report to [national CSIRT or agency] where required or useful; federal agencies under ED 24-01 reported to CISA.
• Customers and partners: notify where their data passed through the device, where VPN or site-to-site credentials shared with them are being rotated, or where contracts require it.
• Vendor: share indicators and request any undisclosed persistence guidance.
• Recovery status: report progress to stakeholders as each stage completes (NIST CSF 2.0 RC.CO-03).

This template is not legal advice; confirm notification obligations with counsel.

8. Post-incident review

1. Hold the review within [10] working days of return to service.
2. Establish the root cause: unpatched known exploited vulnerability, exposed management interface, weak or reused credentials, missing logging.
3. Measure against the playbook: time to contain, time to verified image, time to full secret rotation, time to return to service.
4. Record gaps: missing configuration backups, secrets without owners, devices without out-of-band access, absent integrity tooling.
5. Update this playbook, the vendor annexes and the device standard; schedule an exercise to test the changes.

Implementation guidance

Prepare before the incident. The rebuild is only fast if these exist:

• A configuration backup system that keeps dated versions for at least [12 months], stored off the device and off the production network, so a pre-compromise baseline exists (NIST CSF 2.0 RC.RP-03 asks that backup integrity be verified before restoration).
• An inventory of every secret each device holds, with owner and rotation procedure.
• Vendor annexes with image download, hash and signature verification, factory reset and recovery steps, checked yearly.
• Spare or standby hardware for critical edge devices, or a contract for rapid replacement.
• Out-of-band management and console access to all edge and core devices.

Framework mapping.

Firmware trust. NIST SP 800-193 (May 2018) frames platform firmware resiliency as protection against unauthorised changes, detection of changes that occur, and rapid, secure recovery. Prefer platforms with hardware roots of trust and signed boot, and record in each vendor annex what the platform can and cannot attest. See the Firmware and Boot Security Guide.

Metrics for exercises and real incidents: time from trigger to isolation; time to a verified image; percentage of device-held secrets rotated within [24 hours]; time to return to service; number of devices in the peer group rebuilt versus only patched.

Common mistakes to avoid: patching and declaring recovery; reimporting the compromised configuration with its secrets; resetting the device before capturing evidence; rotating the device's passwords but not the AAA shared secrets and service account it used; reconnecting the old hardware "temporarily"; and rebuilding one device while its identical peers stay untouched.

Related NHI Mgmt Group resources: Edge Device Security Guide · Edge Device Compromise Investigation Playbook · Network Device Management Plane Hardening Guide · Incident Response Guide