A faulty update to a kernel-level security agent can crash thousands of endpoints in minutes, and no attacker is needed for it to become a major incident. On 19 July 2024 a CrowdStrike Falcon content update (Channel File 291), released at 04:09 UTC and reverted at 05:27 UTC, crashed Windows hosts running sensor 7.11 and above that were online in that window. The reverted file did not help machines already stuck in a boot loop: each one needed hands-on repair, often behind a BitLocker recovery prompt. This template gives IT operations, endpoint engineering and incident response teams a playbook for a vendor update that takes down the fleet, whichever EDR, antivirus or other privileged agent caused it. The wider supplier lesson is covered in the Third-Party Concentration Risk Guide; this page is the response.
How to use this template
- Replace placeholders: [Organisation], [EDR product], [MDM], [key escrow] and the bracketed thresholds with your own values. Keep the section numbers so people can say "we are in section 6".
- Prepare offline: this is the one incident where your laptops, jump hosts and wiki may all be down at once. Print the recovery runbook, the BitLocker key retrieval steps and the contact list, and keep a copy on devices that do not run the affected agent.
- Treat it as an incident: run it through your incident process with an incident commander, even though the cause is a supplier defect. Recovery touches privileged keys and accounts at scale, which attackers will try to exploit.
- Do not rule out an attack too early: a vendor announcement is strong evidence, but confirm the crash signature matches the vendor's description before closing the security question. If agents are being killed rather than crashing, use the EDR Tampering and BYOVD Response Playbook.
1. Scope and triggers
This playbook covers all [Organisation] endpoints, servers and cloud virtual machines running third-party agents with kernel or boot-level privilege: EDR and antivirus, disk encryption, data loss prevention, VPN and network filter drivers, and device management agents. Invoke it on any of the following:
- Crash cluster: [10] or more devices, or any [3] servers, crashing or boot looping within [30] minutes, with stop codes or crash dumps pointing at the same third-party driver (in July 2024 Microsoft listed stop errors 0x50 and 0x7E and the faulting driver csagent.sys).
- Vendor alert: a tech alert or status page entry from a security tool supplier describing a defective update, even before local impact appears.
- Health drop: agent check-ins fall sharply across a platform (for example only Windows) right after a sensor or content release.
- Service impact: critical services unavailable because their Windows hosts will not boot, including identity, line-of-business and point-of-sale systems.
Out of scope: crashes caused by an operating system patch (use the Emergency Out-of-Band Patching Playbook and your change rollback process) and deliberate agent tampering.
2. Roles and contacts
| Role | Responsibilities | Named lead / deputy |
|---|---|---|
| Incident commander | Declares the incident, sets recovery priority order, approves emergency changes and key release | [Name] / [Name] |
| Endpoint engineering | Confirms the faulty component, builds the fix (scripts, boot media), pauses further updates | [Name] |
| Service desk and field teams | Hands-on recovery of devices, BitLocker key entry, user guidance | [Name], [site leads] |
| Key custodian | Controlled, logged retrieval of BitLocker recovery keys at scale | [Name] (security) |
| Cloud and server team | Disk detach and repair or snapshot restore for virtual machines | [Name] |
| Identity team | Break-glass access if domain controllers or the identity provider are affected; local admin password retrieval | [Name] |
| Communications | Staff, customer and regulator messages; phishing warnings | [Name] |
| Vendor liaison | Official support channel, technical alerts, fix verification | [Support route, contract ID] |
3. Detection and triage
- Confirm the common cause. Compare crash dumps or stop codes from three or more devices. Check the vendor's official status page or support portal, not social media, for a matching alert.
- Size the blast radius. Pull the agent inventory: which platforms, sensor versions and update policies received the update, and which devices were online during the release window. In 2024 only hosts online during the 78-minute window were affected.
- Classify devices into recovery tiers: still healthy, crashed but reachable through a management interface, crashed and needing hands-on repair, and encrypted devices needing a recovery key.
- Rank services. Map crashed hosts to business services from the CMDB and the business continuity plan, and set the recovery order (section 6).
- Check for an attack in parallel. One analyst confirms that the faulty file, its hash and the vendor fix match the vendor's description, and that no unexpected change was made to the agent's management console.
| Finding | Severity | Next action |
|---|---|---|
| Vendor alert, no local crashes, update not yet received | Low | Pause updates (section 4), monitor |
| Workstation crashes, servers healthy | High | Recovery by site, service desk surge |
| Servers, domain controllers or identity systems down | Critical | Major incident, business continuity plan, break-glass access |
| Crash signature does not match vendor description | Critical | Treat as possible attack, invoke security incident response |
4. Containment
- Stop the spread: pause further sensor and content updates in the [EDR console] for groups not yet affected, if the vendor offers that control. Since the 2024 incident CrowdStrike states customers can choose where and when Rapid Response Content is deployed; record what your own product allows.
- Keep healthy devices online: tell staff not to restart working machines until the vendor confirms the faulty file is withdrawn.
- Freeze other change: stop unrelated deployments so new failures are not confused with this one.
- Do not uninstall security tools estate-wide as a reflex. Removing protection from healthy devices trades an outage for an exposure. If a group of servers must run without the agent temporarily, the incident commander records the decision and compensating controls (network segmentation, increased log monitoring).
- Harden against opportunists: warn all staff that only [official channel] will send fix instructions. In July 2024 CrowdStrike's chief executive warned that adversaries would try to exploit the event, and its intelligence team reported fake update lures and a fake recovery manual that delivered an information stealer (T1566 Phishing).
5. Eradication
- Remove the faulty component using only the vendor's published fix. For the 2024 incident Microsoft KB5042421 described booting into Safe Mode or the Windows Recovery Environment and deleting files matching C-00000291*.sys from %WINDIR%\System32\drivers\CrowdStrike, then restarting. Several reboots recovered some machines because the agent fetched the reverted file before crashing.
- Automate where possible: Microsoft released a USB recovery tool for this case; build and test equivalent boot media for [EDR product] in advance.
- Confirm the vendor fix is in place on every recovered device: the faulty file is gone and the agent reports the corrected version.
- Validate integrity of any tool you run: scripts and boot images come only from the vendor or your own signed repository, with hashes recorded.
6. Recovery
- Restore in priority order: identity (domain controllers, identity provider connectors, privileged access tooling), then key escrow and management servers, then critical business services, then workstations by role.
- Retrieve BitLocker recovery keys under control. Microsoft documents that keys for Entra joined devices are stored in Entra ID and retrievable from the Entra or Intune admin centres, PowerShell or Microsoft Graph, and that keys for domain joined devices sit in AD DS as ms-FVE-RecoveryInformation objects. Users can self-serve keys for their own Entra devices. Plan for the case where AD DS itself is down: keep an offline, access-controlled export of tier 0 server keys.
- Recover cloud virtual machines: snapshot the OS disk, attach it to a healthy repair VM, remove the faulty file, then reattach; or restore a snapshot taken before the release time, accepting the data loss since then.
- Rotate exposed keys: a recovery password read out over the phone or typed into a shared screen is exposed. After recovery, rotate the BitLocker recovery password on every device where a key was disclosed, through [MDM] key rotation or manage-bde.
- Re-enable protection: confirm every recovered device has a healthy security agent and has resumed telemetry before closing its ticket.
- Track completion daily: devices recovered, devices outstanding by site, services restored. CrowdStrike reported about 99 percent of Windows sensors back online by 29 July 2024, ten days after the release.
Where identity fits
Mass recovery hands out privileged secrets fast: BitLocker recovery passwords, local administrator passwords and break-glass credentials. Log every key retrieval, restrict retrieval to named roles, and review the log after the incident for retrievals that do not match a recovery ticket. Use Windows LAPS rather than shared local admin passwords (see the Windows LAPS Guide), and keep tested emergency accounts for when the identity provider is down (see the Break-Glass Account Guide). The security agent's own console credentials and API clients are non-human identities with fleet-wide reach: confirm none were changed during the chaos.
7. Communication and notification
- Staff: what has happened, what not to do (do not restart, do not follow emailed fix links), and where to queue for help. Send updates at a fixed interval, for example every [2] hours.
- Executives and board: services affected, recovery estimate by tier, customer impact, and the fact that this is a supplier defect, not a breach, if that is confirmed.
- Customers and partners: service status and expected restoration, through channels that do not depend on the affected systems.
- Regulators: legal assesses whether the outage is a reportable operational or ICT incident under [DORA, NIS2, sector rules]; availability incidents can be reportable even without a data breach.
- Vendor: open a priority case through the official support portal, record case numbers, and request the root cause analysis.
8. Post-incident review
- Timeline: vendor release, first crash, detection, update pause, first recovery, 50 percent and 99 percent recovered.
- Update policy: do sensor and content updates go out in rings (pilot, early, broad) and can you hold content updates as well as agent versions? CrowdStrike's root cause analysis committed to staged deployment of content with rollback and customer control.
- Recovery readiness: were keys reachable, was boot media ready, could field teams reach remote sites, and did shared local passwords slow things down?
- Supplier risk: update the supplier risk record, contract terms on update testing and notification, and the concentration risk register. See the EDR and XDR Buyer's Guide for questions to ask about update controls.
- Exercise gaps: add a "security agent bricks the fleet" scenario to the business continuity exercise programme.
Implementation guidance
- Set update rings now: put [5] percent of devices, including IT staff and a sample of each hardware model, on the earliest ring; keep tier 0 servers on a later ring where the product allows.
- Prove key escrow: run a monthly report of devices without an escrowed BitLocker key and fix them; enforce "do not enable BitLocker until recovery information is stored" policies.
- Build and test recovery media for each agent with a boot-level driver, and rehearse the cloud VM disk repair on a test instance.
- Print the runbook and contact list for every site and store them with the boot media.
- Rehearse once a year with a tabletop where the identity servers and the IT team's own laptops are affected.
| Metric | Target example |
|---|---|
| Devices with an escrowed, retrievable BitLocker recovery key | 100 percent |
| Security agents with content and sensor updates in staged rings | All products that support it |
| Time from first crash cluster to update pause | Under [30] minutes |
| Tier 0 services restored | Within [4] hours |
| Recovery key retrievals without a matching ticket | Zero |
Framework mapping
| Framework | Reference |
|---|---|
| NIST CSF 2.0 | GV.SC-07, GV.SC-08, RS.MA-01, RS.MA-03, RS.MI-01, RC.RP-01, RC.RP-02, RC.RP-05, RC.CO-03, RS.CO-02, ID.IM-04 |
| NIST SP 800-53 Rev. 5 | CM-3, CM-4, SA-9, SR-6, CP-2, CP-10, IR-4, SC-28 |
| MITRE ATT&CK v20 | T1566 Phishing (opportunistic lures during recovery) |
Standards and references
- NIST: SP 800-61 Rev. 3, Incident Response Recommendations and Considerations for Cybersecurity Risk Management
- CrowdStrike: Channel File 291 Incident Root Cause Analysis
- CrowdStrike: Technical details of the Falcon update for Windows hosts
- Microsoft Support: KB5042421, CrowdStrike issue impacting Windows endpoints
- Microsoft Learn: BitLocker recovery overview
- CrowdStrike: Falcon content update remediation and guidance hub
- NIST: Cybersecurity Framework 2.0
Related NHI Mgmt Group resources: Third-Party Concentration Risk Guide · Business Continuity and Disaster Recovery Guide · Disk Encryption Guide: BitLocker and FileVault · EDR Guide