VMware ESXi Forensics: Investigating Hypervisor Compromise

When an ESXi host is compromised there is no EDR agent, no Windows event log and often no persistent disk log, so investigators must know exactly which hypervisor artefacts exist, where they live and how fast they disappear. ESXi investigations fall into two shapes: loud ransomware that powers off and encrypts virtual machines, and quiet espionage where an actor installs a backdoored vSphere Installation Bundle (VIB) and uses the hypervisor to run commands inside guests. This guide covers both, from first signs and triage questions to evidence collection from logs, the image profile, VMFS datastores and per-VM files, then containment, eradication, recovery and notification. It is written for incident responders, forensic analysts and virtualisation administrators. Log forwarding and detection rules are covered in the ESXi Logs Detection Guide and host settings in the VMware ESXi Hardening Guide; this page is about investigating.

Key takeaways

• Preserve before you reboot. ESXi keeps many logs on an in-memory file system and backdoors may run only from memory, so a reboot or reinstall can destroy the evidence you most need.
• The image profile tells the truth about VIBs. /var/db/esximg/profile and esxupdate.log record every VIB installed and whether --force was used; compare with esxcli software vib signature verify.
• Each VM's vmware.log records hypervisor to guest activity. Filtering for GuestOps shows commands and file transfers pushed into guests that the guest's own logs never saw.
• Flat disks often survive ransomware. CISA and the FBI noted that ESXiArgs encrypted VM configuration files but not flat files, which is why recovery by rebuilding configuration was possible.
• Most intrusions arrive through identity. Scope the investigation to vCenter, Active Directory groups and the help desk, not just the host, and rotate every credential the host could see.

How the incident shows up

Triage questions

1. Is encryption still running? If VMs are still being powered off or files renamed, containment comes before any collection.
2. Which hosts, and are they managed by vCenter? vCenter compromise means every attached host is in scope.
3. How did the intruder authenticate? Local root, an Active Directory group mapped to ESXi admin, vCenter credentials, or an exposed service such as SLP. Check auth.log for method (password or public key) and source IP.
4. Are logs forwarded, and since when? If not, the host's in-memory logs are the only copy and must be collected before any reboot.
5. Is there persistence? Check VIBs, /etc/rc.local.d/ scripts (T1037.004 RC Scripts), new local accounts and SSH authorized_keys.
6. Did the intruder touch guests? Review each VM's vmware.log for GuestOps and look for exported or snapshotted disks of domain controllers.
7. Are backups intact? Backup servers and their credentials are often reached from the same admin accounts.

Evidence to preserve

Collect in order of volatility: running state and memory first, then in-memory logs, then on-disk configuration, then datastores. Record every command you run, with time and the account used, and hash each artefact as you take it. See the Evidence Handling and Chain of Custody Guide.

• vm-support: Broadcom documents three routes: run it locally (it writes a .tgz to the accessible location with most free space unless you pass -w), stream it with ssh root@<host> vm-support -s, or download it from https://<host>/cgi-bin/vm-support.cgi. Write it to a datastore you are not investigating, and treat the bundle as sensitive.
• Remote scanning: Mandiant mounted the ESXi root over SSHFS from a Linux system and scanned it with YARA. It warns that compressed .vgz VIB payloads and binaries running only from memory will not be found by a file system scan.
• Guest side: also collect guest artefacts. In the VIRTUALPITA case, commands ran as children of vmtoolsd.exe and left TS_*.tmp and .cab files in C:\Windows\Temp. See the Memory Forensics Guide.
• Datastores and VMFS: copying VM folders off a datastore preserves the files a recovery or guest investigation needs, but not deleted or slack space. Where the case may go to court or the family is unknown, image the backing LUN or local disk at block level from the storage array or with the host powered down, after volatile collection is complete. Record datastore UUIDs and paths so copies can be matched to vmware.log references.
• Clock integrity: Mandiant found system clock changes in vmkwarning.log around attacker actions and VIB install dates set years in the past (T1070.006 Timestomp). Build the timeline from forwarded logs and vCenter where possible, not from host timestamps alone.

Containment priorities

1. Stop the spread, not the host. Isolate management networks at the switch or firewall, and block the host's management interfaces from everything except a forensic workstation. Avoid powering off or rebooting hosts until volatile evidence is collected, unless encryption is actively spreading.
2. Cut the identity path. Disable the accounts and AD groups used, remove any AD group mapped to ESXi administration until reviewed, and revoke vCenter sessions. See the Help Desk Social Engineering Incident Playbook where access began with a call.
3. Disable exposed services. Stop SSH and the ESXi Shell after collection, and disable SLP where it is enabled, as CISA advised for ESXiArgs.
4. Protect backups. Disconnect backup repositories from the compromised admin plane and confirm immutable copies exist.
5. Preserve the encrypted datastores. Do not reformat or reuse them until imaging or recovery decisions are made.

Eradication

A host with a malicious VIB or modified startup scripts should be treated as untrusted. Removing a single file is not enough, because an attacker who had root on the hypervisor could have changed anything it controls.

• Rebuild, do not clean: reinstall ESXi from vendor media on a current supported release, then reapply configuration from a known-good source rather than from the compromised host's backup.
• Verify VIBs: run esxcli software vib signature verify on every host in the cluster. Google's analysis cautions that a CommunitySupported VIB can still show verification Succeeded if its payload was not modified, so compare installed VIBs against the expected list for the build, and use VMware's detection script referenced in KB 89619.
• Enforce acceptance level and Secure Boot so unsigned VIBs cannot be installed again (NIST SP 800-53 CM-14 Signed Components, SI-7 integrity).
• Rotate secrets: root passwords on every host, vCenter SSO and service accounts, any AD account in mapped admin groups, backup software credentials and SSH keys. See the SSH Key Compromise Response Playbook.
• Treat guests as suspect where GuestOps activity or vmtoolsd child processes appear, and run a guest-level investigation.

Recovery

Recovery choices depend on what was encrypted. CISA and the FBI's ESXiArgs guidance listed VM file types the ransomware targeted (including vmdk, vmx, vmsd, vmsn, vswp, nvram and vmem) and stated that it did not encrypt flat files, so their recovery script created new configuration files for each VM and the VM could be re-registered. Other families encrypt disks fully, so confirm per family.

Before reconnecting any host, CISA advised updating ESXi, disabling SLP and making sure the hypervisor is not exposed to the internet. Confirm log forwarding is working first, so a return of the intruder is visible. See the ESXi Ransomware Playbook for the role-based runbook.

Notification triggers

Hypervisor compromise usually means every guest's data was within reach, so assess notification on the guests' data, not the host's. Common triggers include GDPR Article 33 (notify the supervisory authority without undue delay and, where feasible, within 72 hours of becoming aware of a personal data breach), NIS2 Article 23 for essential and important entities (early warning within 24 hours of becoming aware of a significant incident, incident notification within 72 hours), and for US public companies Form 8-K Item 1.05 within four business days of determining an incident is material. In the US, CISA and the FBI ask victims to report ransomware to a local FBI field office or CISA ([email protected]). Record the time awareness began. See the NIS2 Directive Guide and the SEC Cybersecurity Disclosure Rules Guide.

Lessons from incidents

• Identity first, hypervisor second. In the MGM Resorts breach, ALPHV claimed to have deployed ransomware to more than 100 ESXi hypervisors after a help desk call led to identity provider administrator access (claims MGM did not confirm). The ESXi investigation is only complete when the identity path is traced.
• Exposure plus unpatched services. ESXiArgs hit internet-reachable hosts, and CISA's advice centred on patching, disabling SLP and removing internet exposure.
• Logs the attacker cannot reach. VIRTUALPITA and VIRTUALPIE could stop vmsyslogd, so evidence held off the host in a SIEM was more reliable than anything on it.

Measuring readiness

• Percentage of hosts forwarding logs with audit records enabled, and days of retained history.
• Time to collect a vm-support bundle and volatile state from a host in an exercise.
• Percentage of hosts with Secure Boot and acceptance level enforcement, and with a passing VIB baseline comparison.
• Number of AD groups mapped to ESXi admin, and time to disable them in an exercise.
• Recovery time for a VM from immutable backup to a rebuilt host.

Framework mapping

Practitioner checklist

• Do not reboot or reinstall a suspect host until volatile state, in-memory logs and backdoor process memory are collected.
• Collect a vm-support bundle to a datastore outside the investigation scope and hash it.
• Export /var/db/esximg/profile, esxupdate.log and esxcli software vib signature verify output for every host in the cluster.
• Pull every affected VM's vmware.log and filter for GuestOps.
• Correlate auth.log and shell.log with vCenter events, AD group changes and help desk tickets to find the entry point.
• Build the timeline from forwarded logs, checking vmkwarning.log for clock changes.
• Rebuild compromised hosts from vendor media with Secure Boot and acceptance level enforcement, never clean them in place.
• Rotate host root, vCenter, AD admin group, backup and SSH credentials.
• Assess GDPR, NIS2 and SEC notification on the guests' data and record when awareness began.

Standards and references

• CISA and FBI: ESXiArgs Ransomware Virtual Machine Recovery Guidance
• Broadcom: Location of ESXi Log Files
• Broadcom: Collecting Diagnostic Information for ESX/ESXi Using vm-support
• Google Cloud (Mandiant): Investigating Novel Malware Persistence Within ESXi Hypervisors
• Google Cloud (Mandiant): ESXi Hypervisors Detection and Hardening
• MITRE ATT&CK: ESXi Administration Command
• MITRE ATT&CK: Hypervisor CLI
• NIST: SP 800-53 Rev. 5

Related NHI Mgmt Group resources: ESXi Logs Detection Guide · Hypervisor and Virtualisation Security Guide · Digital Forensics (DFIR) Guide · Scattered Spider Threat Actor Profile