Mass Endpoint Recovery After an Outage or Wiper

When thousands of endpoints stop booting at once, the fix is usually known within hours but applying it to every device takes days, and the gap is almost entirely logistics. Microsoft estimated that the July 2024 faulty security agent update affected 8.5 million Windows devices, and most of them needed someone to boot recovery media, find a BitLocker recovery key and remove a file by hand. A destructive attack through a management console or wiper malware creates the same queue, except the devices have to be rebuilt rather than repaired. This guide is the operational method for that queue: getting recovery keys out at scale, choosing boot media, sequencing recovery waves and staffing the work. It is for endpoint engineering, service desk leads and incident commanders running Windows fleets managed through Active Directory, Microsoft Entra ID or Intune. The lessons from the incidents themselves are in the Major IT Outage Lessons Guide.

Key takeaways

Decide repair or rebuild first. A bad driver or content file can be removed in place; a wiped or untrusted device must be reimaged. The choice drives every later step, so make it per device class in the first hour.
Recovery keys are the bottleneck. Each encrypted device shows a key ID on its recovery screen; your team needs a fast, logged way to turn that ID into a 48-digit password when the directory, the portal or the technician's own laptop may be down.
Boot media is a supply chain. USB sticks, PXE responders and WinRE each suit different devices; build, test and count them before deploying, and remove temporary PXE and firewall changes afterwards.
Recover in waves, not in queue order. Identity and key escrow first, then the tools your recovery depends on, then critical services and the people who run them, then everyone else by business priority.
Clean up the secrets you spent. Every recovery key read out and every local administrator password used during the surge is exposed; rotate them and reconcile the key access log against tickets.

Repair in place or rebuild: the first decision

Mass outages fall into two families. In a faulty update the disk, user data and device identity are intact; one component prevents boot, and removing or rolling it back restores the device. In a destructive event, whether wiper malware (MITRE ATT&CK T1561 Disk Wipe, T1485 Data Destruction) or a legitimate management console used to issue wipe commands, the operating system or data is gone or cannot be trusted. Attackers may also target recovery itself (T1490 Inhibit System Recovery), so recovery partitions, restore points and shadow copies may not exist.

Situation	Recovery approach	Key needed?	Typical hands-on time per device
Faulty driver or content file, device boot loops	Boot to WinPE, WinRE or safe mode and remove or roll back the file	Usually, for BitLocker volumes	Minutes, once media and key are in hand
Faulty quality update, recovery partition healthy	WinRE uninstall of the latest update, point-in-time restore, or quick machine recovery where enabled	Often, when WinRE asks to unlock the drive	Minutes, sometimes hands-off
Wiped by management console, device still enrolled or re-enrolable	Reprovision through Windows Autopilot or your imaging service	No, the disk is reset	Mostly automated, user time for sign-in and data
Wiper malware or suspected compromise of the device	Full reimage from a known-good image, then restore data from clean backups	No for the reimage; yes to recover data first if forensics needs the disk	Longer: imaging plus data restore and checks
Virtual machines and cloud PCs	Attach the disk to a repair VM, restore a snapshot, or use the platform's point-in-time restore	Depends on the disk encryption in use	Scriptable at scale

Two rules apply before any recovery starts. First, confirm the cause: a vendor statement is good evidence, but a crash signature that does not match it, or wipe commands in the management audit log, turns the event into a security incident handled through the Wiper Attack Response Playbook. Second, never rebuild through a management plane that may still be compromised. For the step-by-step incident template for a vendor defect, use the Security Tool Faulty Update Outage Playbook; this guide covers the recovery engine behind it.

Getting BitLocker recovery keys out at scale

Microsoft lists the triggers for BitLocker recovery, and several of them occur in a mass recovery: booting from PXE, changes to the boot manager, firmware upgrades and changes to the TPM. The recovery screen shows a key identifier (the ms-FVE-RecoveryGuid value) so the right 48-digit recovery password can be found; the job is to match that ID to the password quickly and safely.

Where the keys are, and how to read them in bulk

Escrow location	Retrieval routes	Permission or role	Audit trail
Microsoft Entra ID (Entra joined and hybrid devices)	Entra admin centre, Intune admin centre (Devices, device, Recovery keys, Show Recovery Key), PowerShell, Microsoft Graph `informationProtection/bitlocker/recoveryKeys`	Graph: BitlockerKey.ReadBasic.All (metadata) or BitlockerKey.Read.All; delegated callers must own the device or hold a role such as Helpdesk administrator, Cloud device administrator or Intune service administrator. Directory permission `microsoft.directory/bitlockerKeys/key/read`	Reading the key (Graph `$select=key` or Show Recovery Key) writes an Entra audit event in the KeyManagement category; self-service reads appear as Read BitLocker key
Active Directory Domain Services	Recovery objects (`ms-FVE-RecoveryInformation`) under each computer object, read with directory tools or PowerShell	Delegated read on the recovery objects, normally restricted to domain administrators and named help desk groups	Directory service access auditing, if enabled on those objects
Microsoft account (unmanaged devices)	The user signs in on another device	The user only	Not visible to the organisation

The Graph design matters at scale: by default the API returns only key metadata, and the 48-digit key is returned only when the caller explicitly selects it, which is what generates the audit record. That gives you two workable patterns.

Lookup desk (preferred): a small team of key custodians holds the read role, technicians phone or message the key ID and device name, and the custodian reads that single key. Every read is logged against a ticket. It keeps keys off shared spreadsheets.
Pre-staged export (when the directory or cloud console may be unavailable): a bulk export of key IDs and passwords for the affected population, created by a dedicated app registration with BitlockerKey.Read.All under change control, encrypted and handed to site leads on offline media. It is faster in the field and far riskier: treat the file as a credential store, number every copy and destroy it when the incident closes.
User self-service: users of Entra joined devices can retrieve keys for devices registered to them from another device, which Microsoft pointed to in its July 2024 remediation article. It only works when users have a second device and can sign in.
Tier 0 servers: keys for domain controllers and other servers that hold the directory itself cannot depend on that directory. Keep an offline, access-controlled copy and test it, as the Emergency Identity Operations When the Primary IdP Is Unavailable guide recommends for other break-glass material.

Escrow gaps are discovered too late in most incidents. Microsoft notes that AD DS keeps old recovery objects until the computer object is deleted, so a device may show several keys and only the one matching the displayed ID will work. Devices encrypted before escrow was enforced may have no key at all; they go straight to the rebuild queue. The BitLocker policy setting that blocks encryption until recovery information is stored in the directory prevents new gaps; see the BitLocker guide for the policy set.

Boot media: USB, PXE, WinRE and quick machine recovery

A device that cannot boot needs another environment to run the fix. The options differ in who can use them, whether a recovery key is needed and how many devices one person can serve.

Option	Best for	Requirements and limits	Risk to manage
USB WinPE media running the fix automatically	Field and desk-side recovery of laptops and desktops	Microsoft's July 2024 tool built media on a 64-bit Windows machine with the Windows ADK, using a 1 GB to 32 GB USB drive formatted FAT32; the WinPE route needed no local administrator rights but could need the recovery key	USB boot blocked by firmware policy on hardened devices; lost sticks carrying scripts; driver gaps for keyboards on some models
USB or PXE safe mode route	Devices where keys are scarce	Microsoft noted the PXE safe mode option was less likely to require the BitLocker recovery key, but a local administrator must sign in and run the steps	Needs local admin credentials per device; use Windows LAPS rather than a shared password
Temporary PXE responder	Desks of wired devices on one site, and devices that cannot boot USB	Microsoft's tool opened UDP 67, 68, 69, 547 and 4011 on the PXE server and needed devices on the same subnet and wired, or IP helpers for other subnets	A rogue or forgotten boot service on the production network; remove the firewall rules and network boot entries afterwards
WinRE on the device	Rolling back a quality update or running startup repair without media	WinRE lives in a recovery partition (`winre.wim`) and starts automatically after two consecutive failed boots; tools include automatic repair and point-in-time restore	Disabled or damaged recovery partitions on older builds; wipers may destroy it
Quick machine recovery	Windows 11 and Windows Server 2025 fleets where Microsoft publishes a fix through Windows Update	Builds on Startup Repair and a network-connected WinRE; wired or WPA/WPA2 password Wi-Fi only; cloud remediation is disabled by default on Enterprise, Education, managed Pro and Server 2025	Best effort: only helps when a remediation exists; decide deliberately whether to enable it in your policy
Hypervisor and cloud tooling	Virtual desktops, servers and cloud PCs	Attach a recovery ISO as a virtual DVD, repair the disk from a helper VM, or restore a snapshot from before the faulty release	Snapshot restores lose data written since; restore order matters for domain controllers

Treat a temporary PXE service as production infrastructure for its lifetime: run it on an isolated segment where you can, bind it to the recovery VLAN and take it down the moment the queue is empty. Network boot is unauthenticated by design. The Bare-Metal Provisioning and PXE Boot Security Guide covers DHCP snooping and boot image integrity in depth.

Test the fix on several devices of each hardware model before producing media in bulk, as Microsoft advised for its own tool, and number every stick.

Recovery waves and prioritisation

Set waves in advance, linked to the service tiers in your business impact analysis, so the devices that unblock everyone else are not stuck behind the loudest users.

Wave 0, recovery enablers (first hours): domain controllers and identity provider connectors, key escrow and LAPS retrieval paths, management servers, the privileged access tooling and the endpoint engineers' own devices. Without these, nothing else scales.
Wave 1, life, safety and regulated operations: clinical, safety, plant and trading systems, plus the kiosks, point-of-sale and shared devices that front them. Shared devices lack a primary user, so send technicians.
Wave 2, critical people: the incident team, service desk, communications, executives and staff who run critical processes, identified by role group rather than by name on the day.
Wave 3, general workforce by site: office-based devices in walk-up recovery hubs, where one technician with a key custodian on call can process devices in parallel.
Wave 4, remote and hard cases: home workers, devices without escrowed keys, damaged media and unknown hardware, routed to courier swap, rebuild or replacement stock.

Each wave has an exit check: device boots, signs in, security agent reports healthy and management check-in resumes.

Staffing and throughput

Recovery time is technicians multiplied by devices per hour, minus time lost waiting for keys. Size it before the incident with your own measured repair times. The table is a planning example, not a benchmark.

Input	Example value	What changes it
Affected devices	6,000	Update ring design: devices on later rings never receive the fault
Hands-on minutes per device with key ready	10	Automated media, fewer reboots, keyboard drivers on the media
Minutes waiting for a key	5 to 20	Lookup desk size, pre-staged export, user self-service
Devices handled in parallel per technician at a hub	3 to 5	Bench space, power, media count
Technicians available	Service desk plus trained volunteers	Pre-trained "recovery reserve" from other IT and business teams

Recovery reserve: train staff outside the service desk to run the media in an hour-long session, with a printed one-page card. They handle the mechanical steps; trained staff handle exceptions.
Key custodians: one custodian can serve several technicians if requests arrive with device name and key ID already typed.
Remote workers: self-service keys and a guided call, or courier swap from pre-built spares.
Tracking: one register outside the affected systems recording device, wave, status and whether a key was read, which drives reporting and later rotation.

Rebuilding after a wiper or console-driven wipe

Rebuilding is mostly automated, if the rebuild services survived and can be trusted. In the Stryker cyberattack, attackers used a hijacked administrator account to issue Intune wipe commands to tens of thousands of devices: the endpoint management console was both the weapon and the rebuild tool.

Regain the management plane: remove attacker accounts and app registrations, reset privileged credentials, and turn on Intune Multi Admin Approval for device actions so wipe, retire and delete need a second administrator. Microsoft notes an administrator cannot approve their own request, and requests expire after three days.
Check the build chain: Autopilot profiles, enrolment restrictions, application packages, scripts and the base image. A rebuild that installs a tampered package recreates the compromise. See the Windows Autopilot Security Guide.
Choose the reprovisioning route: Autopilot Reset removes personal files, apps and settings while keeping the device's Entra and Intune connection, but it needs a working WinRE and is not supported for hybrid joined devices, which need a full wipe. Devices with destroyed partitions need media or network reimaging.
Restore data from clean sources: the NCSC advises connecting backups only to known clean devices before recovery and scanning backups for malware before restoring files.
Reissue identity material: device certificates, Wi-Fi and VPN profiles and user credentials must be valid for the rebuilt device and revoked for anything the attacker could have copied.

Where identity fits

Mass recovery hands out privileged secrets faster than at any other time. Recovery passwords, local administrator passwords and app registrations with BitLocker read rights are all non-human or shared credentials with fleet-wide reach.

Local administrator: safe mode routes need a local administrator. Windows LAPS backs up a unique, rotated password per device to Entra ID or AD, never both, and lets authorised administrators retrieve it to recover a device. See the Local Administrator Password Solution (LAPS) Guide.
Bulk-read app registrations: any service principal with BitlockerKey.Read.All can read every key in the tenant. Create it for the incident under approval, scope its use to named operators and disable it afterwards.
Rotation after use: Intune's BitLocker key rotation remote action works on Windows 10 1909 or later and Windows 11 once client-driven recovery password rotation is enabled and keys are saved to Entra ID. Rotate every key read during the incident.
Emergency accounts: if the identity provider is also down, use tested break-glass accounts, as described in the Break-Glass Account Guide.

Implementing the capability before you need it

Measure escrow coverage: report every encrypted device without a retrievable key in Entra ID or AD and fix it; enforce the policy that blocks encryption until the key is stored.
Write the wave plan: map devices to waves through groups or tags you can query on the day, including shared and kiosk devices.
Build and test media: one generic WinPE image with storage and keyboard drivers for each hardware model, a documented temporary PXE build kept switched off, and a decision on quick machine recovery for Windows 11.
Set up the key lookup desk: named custodians, the read role assigned through just-in-time elevation, a request format and the offline tier 0 key copy.
Exercise at scale: recover a lab of 50 devices against the clock, then a pilot site, and compare measured rates with the throughput plan. Roll back exercise changes (PXE services, firewall rules, temporary roles) the same day, as you would in a real incident.

Detection and measurement

During the incident, watch management check-ins per wave, security agent health, recovery key read events in the Entra KeyManagement audit category against open tickets, and any wipe, retire or script deployment actions in the management console. Afterwards, track readiness with a small set of measures.

Escrow coverage: percentage of encrypted devices with a retrievable key (target 100 percent).
Measured recovery rate: devices per technician hour from the last exercise.
Time to wave 0 complete: hours from declaration to identity, escrow and management restored.
Unmatched key reads: key reads with no matching ticket (target zero).
Post-incident rotation: percentage of disclosed recovery passwords rotated within an agreed window.
Media readiness: hardware models with tested recovery media.

Framework mapping

Activity	NIST SP 800-53 Rev. 5	NIST CSF 2.0
Recovery waves, plan and roles	CP-2 Contingency Plan	RC.RP-01
Exercising recovery at scale	CP-4 Contingency Plan Testing	PR.IR-03
Rebuild from known-good images and clean backups	CP-9 System Backup, CP-10 System Recovery and Reconstitution, CM-2 Baseline Configuration	PR.DS-11, RC.RP-03, RC.RP-05
Escrowed recovery keys and disk encryption	SC-28 Protection of Information at Rest	PR.DS-11
Restricted, logged key and admin password access	AC-6 Least Privilege	PR.AA-05
Status reporting during recovery	IR-4 Incident Handling	RC.CO-03

Threats that drive the rebuild path map to MITRE ATT&CK T1561 Disk Wipe (including T1561.002 Disk Structure Wipe), T1485 Data Destruction and T1490 Inhibit System Recovery; the Wiper Malware Guide covers prevention and detection.

Common failure patterns

Keys stored only on the devices that are down: keys printed into a document on a file share or a password manager that runs on the broken laptops.
The key desk becomes the queue: two administrators reading keys for six hundred technicians. Size custodians to technicians, and prepare self-service for users.
Shared local administrator passwords: a single password used for safe mode recovery across thousands of devices and spoken aloud at every desk.
Forgotten recovery infrastructure: temporary PXE responders, open firewall ports and bulk-read app registrations still live months later.
Rebuilding through a compromised console: reimaging into a tenant where the attacker's account or app registration is still present.

Practitioner checklist

Report BitLocker escrow coverage monthly and block encryption until the key is stored.
Keep an offline, access-controlled copy of recovery keys for tier 0 servers and test it.
Define recovery waves in advance and tag devices so each wave can be queried.
Build and test WinPE media for every hardware model and document a temporary PXE build.
Decide whether to enable quick machine recovery for Windows 11 and record the decision.
Staff a key lookup desk with named custodians and a ticket-linked request format.
Use Windows LAPS for local administrator recovery, never a shared password.
Turn on Multi Admin Approval for device wipe, retire and delete actions in Intune.
Rotate every recovery key read during an incident and reconcile reads against tickets.
Exercise mass recovery on a lab and a pilot site and compare rates with the plan.

Mass Endpoint Recovery Guide: Restoring Thousands of Devices After a Bad Update or Wiper