When an attacker controls your virtualisation management plane, they control every workload running on it, including the domain controllers, databases and security tools you would normally use to investigate. The compromise may start with a stolen administrator login to the central management server, a host root password reused across a cluster, a directory group mapped to full virtualisation admin rights, an exposed management interface, or an attacker who already owns the domain and pivots down to the hosts. This playbook is vendor-neutral and covers bare-metal hypervisor clusters with a central management server, on premises or hosted. It deals with management-plane compromise short of mass encryption: if virtual machine disks are already being encrypted, switch to the ESXi Ransomware Playbook and use this page as the credential and rebuild workstream. It is written for incident responders, virtualisation and infrastructure teams and security leads. Hardening is covered in the Hypervisor and Virtualisation Security Guide and is not repeated here.
Key takeaways
- Investigate from outside the platform. Your SIEM collectors, jump hosts and even the domain may run as virtual machines on the compromised cluster; move responders to clean devices and out-of-band channels first.
- Every guest is in scope, not just the hosts. Management access can clone, snapshot, export or run commands inside any virtual machine, so a guest with no alerts of its own may still have been read or changed.
- Rotate in layers, top down. Management server admins and identity source, then host root and service accounts, then the credentials of every integration that talks to the platform, then secrets held inside affected guests.
- Rebuild hosts, do not clean them. Hypervisor hosts have little room for endpoint tooling; reinstall from verified media, restore configuration from a known-good record and rejoin one host at a time.
- Snapshots and clones are evidence and exposure at once. Preserve them for forensics, but treat any copy the attacker could reach as a possible data theft path.
What the management plane controls
Scope the incident by what each component lets an attacker do across the estate, not only by where the first login happened.
| Component | What an attacker gains | First containment action |
|---|---|---|
| Central management server and its admin roles | Create, clone, snapshot, export, power off and delete any virtual machine; change permissions and alarms | Disable suspect accounts, end sessions, restrict console access to a clean admin network |
| Identity source linked to the management server (directory, single sign-on) | Any user added to the mapped admin group becomes a virtualisation administrator | Freeze membership of mapped groups; consider switching to local emergency accounts |
| Host root and local accounts, host shell and remote shell services | Direct control of the host outside management server logging (ATT&CK T1059.012 Hypervisor CLI, T1021.004 SSH) | Disable remote shell services, rotate root, enable lockdown-style restrictions |
| Guest operations interfaces (agents inside guests) | Run programs and copy files inside guests from the management plane, bypassing guest network controls | Remove guest operation privileges from all roles that do not need them |
| Datastores and storage fabric | Copy virtual disks offline, read memory files of suspended machines, delete or overwrite disks (T1485 Data Destruction) | Restrict datastore browse and file transfer rights; check storage array snapshots |
| Management network | Reach host management interfaces, out-of-band controllers and storage management | Block all paths except from a responder jump host |
| Integrations: backup, monitoring, automation, orchestration | Stored platform credentials with broad rights, often never rotated | Inventory every integration account; plan rotation at source |
| Hardware out-of-band controllers | Remote console, virtual media and firmware access beneath the hypervisor | Rotate controller credentials; check firmware and virtual media history |
NIST SP 800-125A Rev. 1 frames the hypervisor's baseline job as mediating access to physical resources and keeping resident virtual machines isolated. A management-plane compromise means both guarantees are suspended until you prove otherwise.
How the compromise shows up
| Signal | Where you see it | What it can mean |
|---|---|---|
| Logons to the management server from unusual hosts, at unusual times, or by accounts that never use it | Management server events, identity provider sign-in logs | Stolen admin credentials (T1078 Valid Accounts) |
| New local admin accounts, new role assignments, changes to the identity source or mapped admin group | Management server permission events, directory change logs | Persistence (T1098 Account Manipulation) |
| Remote shell enabled on hosts, host firewall rules changed, lockdown disabled | Host configuration events, syslog | Direct host access outside central logging |
| Unexpected clones, snapshots or exports of domain controllers, identity servers or databases | Task and event history, datastore file changes | Offline credential theft from a copied disk or memory file (T1003 OS Credential Dumping) |
| Guest operation calls in high volume, from unexpected accounts or out of hours | Management server API logs | Commands run inside guests from the hypervisor layer; the MITRE ATT&CK detection guidance for guest operations abuse calls out exactly these three signals |
| Virtual machines that are not in your inventory, or registered directly on a host | Host inventory compared with management server inventory and CMDB | Hidden attacker workloads (T1564.006 Run Virtual Instance) |
| Syslog forwarding changed or stopped, alarms disabled, audit retention shortened | Missing log volume in the SIEM | Defence impairment (T1685 Disable or Modify Tools) |
| Snapshots deleted, mass power-off tasks scheduled | Task history, storage snapshots | Preparation for impact (T1490 Inhibit System Recovery, T1529 System Shutdown/Reboot) |
Triage: the first hour
- Are responders working from inside the blast radius? List which security, logging, jump host, identity and communications systems run as guests on the affected cluster. Anything that does is untrusted for coordination. CISA's #StopRansomware Guide advises coordinating isolation out of band, for example by phone, so the attacker does not learn they have been detected.
- Which identities were used? Management server accounts, host root, directory accounts in mapped groups, integration service accounts and API tokens active in the suspect window.
- Is encryption or destruction under way? If disks are being encrypted or deleted, move straight to the ransomware playbook.
- Which guests were touched? Clones, snapshots, exports, console sessions and guest operations, with tier 0 systems (domain controllers, PKI, identity providers, secrets managers) checked first.
- Do the attacker's rights extend beyond virtualisation? A stolen directory admin or cloud credential found in a copied guest makes this a domain or cloud incident as well.
Roles and decision points
| Role | Owns | Key decision |
|---|---|---|
| Incident lead | Timeline, scope, link to the wider incident | Whether this stays a virtualisation incident or becomes a domain-wide one |
| Virtualisation owner | Host and management server state, configuration records, rebuild | Which hosts are rebuilt now, and in what order, without stopping critical workloads |
| Identity team | Directory groups, admin accounts, MFA, integration identities | Whether to cut the management server off from the directory and run on local emergency accounts |
| Forensics lead | Log export, host images, guest disk and memory captures | Which snapshots and copies to preserve, and the earliest attacker activity |
| Application and data owners | Guests and data on the platform | Accepting downtime for migration or rebuild of affected guests |
| Data protection officer and legal | Breach assessment and notification | Whether copied disks or memory files exposed personal data |
Containment by priority
- Move response off the platform: use clean laptops, a responder jump host outside the cluster and out-of-band chat and phone bridges.
- Cut attacker access to the management plane: disable suspect accounts, end active sessions, revoke API tokens, and restrict management server and host management interfaces to the responder jump host at the network layer.
- Freeze the identity link: lock membership of every directory group mapped to virtualisation roles. If the directory itself is suspect, switch the management server to named local emergency accounts. See the Break-Glass Account Guide.
- Close direct host paths: disable remote shell and host shell services, rotate root on each host from the console, and turn on the platform's host lockdown restrictions so changes go through the management server only.
- Strip high-risk privileges: remove guest operations, datastore file transfer, clone, export and delete rights from every role that is not essential during the incident.
- Protect recovery: confirm backups and storage array snapshots of tier 0 guests sit outside the attacker's reach, and place holds on them. See the backup platform compromise playbook if the backup system's platform credentials were exposed.
- Hunt for hidden workloads: compare each host's local inventory with the management server and CMDB, and isolate unknown virtual machines without deleting them.
CISA advises powering devices down only if they cannot be disconnected, because power loss destroys volatile evidence. For hosts, prefer isolating the management interface and migrating trusted guests away over shutting the host off.
Credential rotation
Rotate in an order that removes the attacker's highest leverage first and does not lock responders out. The CISA guide recommends full password resets once the environment is cleaned, with remote access and single sign-on resources disabled during containment; for a hypervisor incident, rotate the management layer during containment and repeat after rebuild.
| Order | Credential | Notes |
|---|---|---|
| 1 | Management server administrator and emergency local accounts | New values held in the privileged access vault, phishing-resistant MFA for named responders |
| 2 | Directory accounts in mapped admin groups and the identity source binding account | Remove stale members rather than just resetting them |
| 3 | Host root and local host accounts, per host, unique values | Shared root passwords across a cluster turn one host into all of them |
| 4 | Out-of-band hardware controller accounts | Often forgotten, and they sit beneath the hypervisor |
| 5 | Integration accounts: backup, monitoring, automation, orchestration, storage | Rotate at source and update each consumer; record which ones broke |
| 6 | Management server certificates and tokens | Reissue if the server's private keys were reachable |
| 7 | Secrets inside guests that were cloned, exported or accessed | Directory secrets on a copied domain controller mean a domain-level rotation, including the Kerberos service account key, run twice |
Integration accounts are non-human identities with standing, broad rights and are the most common gap after a rebuild. See the guide to detecting interactive logons by service accounts for monitoring their reuse afterwards.
Eradication and host rebuild
- Preserve first: export management server events, tasks and permission history, host logs and configuration, and image any host you plan to examine. For ESXi hosts, the ESXi forensics guide lists the host-level artefacts to collect.
- Rebuild the management server from verified installation media on a clean network segment, restore only configuration you have reviewed (inventory, networks, roles), and re-create roles from a written least-privilege design rather than restoring old permissions.
- Evacuate and reinstall hosts one at a time: migrate trusted guests off, reinstall the hypervisor from vendor-verified media at the current patch level, check firmware and out-of-band controller versions, apply the hardening baseline, then rejoin with new credentials.
- Remove persistence: unknown virtual machines, unauthorised scheduled tasks, startup scripts, added packages or modules on hosts, and changed syslog or alarm settings.
- Handle touched guests by tier: tier 0 guests that were cloned or accessed are treated as compromised and rebuilt or restored from a verified point; other guests are scanned and their credentials rotated.
Recovery and reconnection
- Verify before restore: NIST CSF 2.0 RC.RP-03 requires the integrity of backups and restoration assets to be verified before use; restore tier 0 guests into an isolated network first. See the Cloud Recovery and Clean Room Guide.
- Reconnect integrations one by one with their new credentials and the smallest role that works.
- Restore the identity link last, to a dedicated admin group with no nested groups, after the directory is confirmed clean.
- Run heightened monitoring for 30 days on admin logons, permission changes, remote shell enablement and clone or export tasks.
Evidence to preserve and notification
- Management plane: event and task history, permission changes, API and session logs, identity source configuration.
- Hosts: system, authentication and shell logs, configuration exports, the list of installed packages and running virtual machines per host.
- Guests: snapshots and clones created in the window (preserve, hash and restrict access), memory captures of suspect guests.
- Surrounding systems: directory logs for mapped groups, network flow records for the management segment, storage array logs.
- Notification: under UK GDPR, the ICO guidance says a notifiable personal data breach must be reported without undue delay and not later than 72 hours after becoming aware of it, and every breach must be recorded whether reported or not. A copied database or file server disk counts as possible exposure. Check sector rules (for example NIS obligations) for your organisation.
Decision guidance
| Situation | Choose | Trade-off |
|---|---|---|
| Only management server accounts were used, no host shell activity | Rebuild the management server, rotate, keep hosts running under monitoring, rebuild hosts in a planned window | Faster, but relies on host logs being complete |
| Host shell access or unknown packages on any host | Rebuild every host in the cluster | Longer outage window, but removes the risk of implants below the guests |
| Tier 0 guests cloned or exported | Treat as a domain compromise and run the Active Directory domain compromise playbook in parallel | Largest effort, but a copied domain controller disk gives the attacker every password hash |
| Small estate, two or three hosts, no spare capacity | Restore critical guests to temporary hosted capacity, then rebuild all hosts at once | Cost of temporary capacity against a long, risky host-by-host rotation |
| Large estate, many clusters with a shared identity source | Assume all clusters are in scope until permission history shows otherwise | More work up front; a shared admin group crosses cluster boundaries |
Detection and measurement
- Time to responder isolation: minutes from declaration until responders work only from systems outside the cluster.
- Rotation coverage: percentage of credentials in the rotation table rotated and confirmed, target 100% before reconnection.
- Integration accounts with documented owner and minimum role: target 100% after rebuild.
- Hosts with remote shell disabled and lockdown on: continuous check, alert on change.
- Inventory drift: virtual machines on hosts that are absent from the management server or CMDB, target zero.
- Alert coverage: detections for admin role change, guest operations use, clone or export of tier 0 guests, and syslog forwarding change, each tested at least quarterly.
Framework mapping
| Activity | NIST CSF 2.0 | NIST SP 800-53 Rev. 5 | ATT&CK v20 |
|---|---|---|---|
| Incident handling and coordination | RS.MA-01 | IR-4 Incident Handling | T1078 Valid Accounts |
| Admin roles and least privilege | PR.AA-05 | AC-6 Least Privilege, AC-5 Separation of Duties | T1098 Account Manipulation |
| Credential rotation | PR.AA-05 | IA-5 Authenticator Management | T1003 OS Credential Dumping |
| Host configuration baseline | PR.PS-01 | CM-3 Configuration Change Control | T1059.012 Hypervisor CLI, T1564.006 Run Virtual Instance |
| Management network isolation | PR.PS-01 | SC-7 Boundary Protection | T1021.004 SSH |
| Monitoring and log protection | DE.CM-09 | AU-9 Protection of Audit Information | T1685 Disable or Modify Tools |
| Verified recovery | RC.RP-01, RC.RP-03 | CP-10 System Recovery and Reconstitution | T1490 Inhibit System Recovery |
Common failure patterns
- Investigating with tools hosted on the compromised cluster: the attacker watches the response through the SIEM or jump host they already control.
- Resetting the management server password and stopping there: host root, hardware controllers and integration accounts still work for the attacker.
- Restoring old permissions with the configuration: the backdoor role or admin group membership returns with the rebuilt server.
- Ignoring cloned domain controllers: the hypervisor incident is closed while the attacker keeps every domain hash from a copied disk.
- Deleting unknown virtual machines on sight: evidence of the attacker's tooling and timeline goes with them.
- A single shared root password across hosts: rotation is skipped because it breaks scripts, and the next intrusion starts there.
Practitioner checklist
- Move responders and coordination to systems and channels outside the affected cluster.
- Disable suspect accounts, revoke tokens and restrict management interfaces to a responder jump host.
- Freeze directory groups mapped to virtualisation roles and prepare local emergency accounts.
- Disable remote host shells and remove guest operations, clone, export and datastore transfer rights not needed during the incident.
- Export management, host and directory logs and preserve suspect snapshots and clones before changes.
- Rotate in order: management admins, mapped directory accounts, host root, hardware controllers, integrations, certificates, guest secrets.
- Rebuild the management server and hosts from verified media with roles re-created from a least-privilege design.
- Run domain compromise response in parallel if any tier 0 guest was cloned, exported or accessed.
- Assess personal data exposure from copied disks against the 72-hour reporting rule.
Standards and references
- NIST: SP 800-125A Rev. 1 Security Recommendations for Server-based Hypervisor Platforms
- NIST: SP 800-61 Rev. 3 Incident Response Recommendations and Considerations
- NIST: Cybersecurity Framework 2.0
- NIST: SP 800-53 Rev. 5
- CISA: #StopRansomware Guide
- MITRE ATT&CK: Hypervisor CLI
- NCSC: Incident management
- ICO: Personal data breaches, a guide
Related NHI Mgmt Group resources: Hypervisor and Virtualisation Security Guide · Hypervisor and virtualisation platform health check · ESXi Ransomware Playbook · VMware ESXi Forensics