A malware sandbox that is badly isolated, badly configured or misread can leak samples, tip off attackers or produce a clean verdict for malware that simply waited out the analysis. Dynamic analysis sandboxes run a suspicious file or URL in an instrumented virtual machine and record what it does: processes, file and registry changes, network traffic and memory. This guide covers what open-source sandboxes (CAPE and the Cuckoo family) and commercial services do, how defenders use them, how to deploy and operate one safely, how to read results with evasion in mind, and when to choose each option. It is for SOC analysts, incident responders and detection engineers. Reverse engineering is covered in the Malware Analysis Guide for Defenders; sandboxing for AI agents is a different problem, covered in the AI Agent Sandboxing and Code Execution Guide.
Key takeaways
- A sandbox report is evidence, not a verdict. "Nothing happened" often means the sample detected the environment, needed user interaction or was waiting for a command, mapped in ATT&CK as T1497 Virtualization/Sandbox Evasion.
- Isolation is the first design decision. Keep the analysis network physically or logically separate from production and management networks, and choose per-analysis routing deliberately: none, drop, simulated services, or a dedicated "dirty" internet line.
- Know your project's status. CAPE (CAPEv2) is actively maintained; Cuckoo Sandbox 2.x is marked unmaintained by its own repository; Cuckoo3 from CERT-EE says it is not yet production ready.
- Public sandboxes are sharing decisions. Uploading a file to a public or community service can expose its contents and tell the attacker you found it; check hashes first and decide by data sensitivity.
- The output feeds detection. Extracted configurations, unpacked payloads, PCAPs and behaviours become YARA, Suricata and Sigma content and blocklist entries, which is where most of the value is realised.
What the tool does
CAPE's documentation describes a sandbox as a way "to execute malicious files in an isolated environment whilst instrumenting their dynamic behaviour and collecting forensic artefacts". In practice, a sandbox host schedules a sample onto a clean guest virtual machine restored from a snapshot, an in-guest agent and monitor execute and instrument it, and the host processes the results into a report before reverting the guest.
| Option | Status and licence (checked 27 October 2026) | Distinctive capabilities | Fits |
|---|---|---|---|
| CAPE (CAPEv2) | Actively developed; GPL v3; derived from Cuckoo v1, ported to Python 3 in 2019 | API-hooking monitor (capemon), automated unpacking, YARA classification of unpacked payloads, malware configuration extraction, YARA-programmable debugger for anti-evasion, AMSI payload capture, interactive desktop | SOC and CTI teams that want deep Windows malware output and can run Linux and KVM infrastructure |
| Cuckoo Sandbox 2.x | Repository states it is unmaintained; GPL v3 | The original open-source architecture many tools descend from | Legacy installations only; plan migration |
| Cuckoo3 (CERT-EE) | Active, EUPL 1.2; its README says it is not production ready and requires Ubuntu with Python 3.10 and Windows guests | Rewrite with VMCloak-based guest preparation and a quickstart installer | Labs and evaluation |
| Commercial sandbox services and appliances | Vendor-supported; licensing per submission, seat or appliance | Typically managed guests, wider file-type and OS coverage, anti-evasion features, integrations with email, EDR and SOAR, threat intelligence enrichment | Teams without capacity to run infrastructure, or needing scale and support |
CAPE's README lists the core capabilities it inherited from Cuckoo: behavioural instrumentation based on API hooking, capture of files created, modified and deleted, network capture in PCAP format, classification by behavioural and network signatures, screenshots and full memory dumps. It adds automated unpacking (capture of payloads from process injection, hollowing and in-memory decompression), configuration extraction through its own parser framework or RATDecoders, DC3-MWCP, MalDuck and MaCo, and classification through YARA on unpacked payloads, Suricata on network captures and behavioural signatures on the API trace. Note one warning from the Cuckoo3 project: avoid the old cuckoosandbox.org related sites, which it describes as neglected or compromised, and take software only from the maintained repositories.
Defender use cases
- Alert triage: detonate an email attachment, downloaded file or quarantined EDR artefact to decide whether an alert is a real infection and what it did. See the Phishing Incident Response Playbook.
- Indicator extraction: collect contacted domains, IPs, URLs, dropped file hashes and mutexes for scoping and blocking. Treat contacted infrastructure with care: malware often contacts legitimate sites to test connectivity.
- Configuration extraction: recover command-and-control addresses, campaign IDs and encryption keys from families with parsers, which is often more reliable than behaviour, because the sample does not have to fully run.
- Detection engineering: turn behaviours into Sigma rules, unpacked payloads into YARA rules and PCAPs into Suricata signatures, then test those rules against the sandbox output. See the YARA Rules Guide and the Suricata IDS Guide.
- Inline prevention: email and web gateways that hold files for detonation before delivery, which NIST SP 800-53 Rev. 5 describes as SC-44 Detonation Chambers.
- Incident scoping: confirm persistence mechanisms and credential theft behaviour so responders know which accounts and hosts to check. See the Malware Outbreak Playbook.
Deployment and safe use
Architecture and isolation
- Dedicated host: run the sandbox on dedicated hardware or an isolated cloud account, never on a shared analyst workstation or a hypervisor that also runs production workloads. CAPE's documentation uses KVM as its default machinery.
- Separate interfaces: CAPE's routing documentation describes a management interface for the web interface and SSH and a separate "dirty line" for analysis traffic. Put management in its own subnet, reachable only from analyst networks through MFA-protected access.
- Per-analysis routing: CAPE offers none (the default, which does not need the Rooter component), drop, internet, InetSim (simulated services on the host), Tor, tun, VPN and SOCKS routing. Choose per sample, not once for the whole platform.
- No path to production: analysis guests must never reach internal DNS, file shares, identity providers or cloud APIs. Test this with a deliberate connectivity check from a guest before going live.
| Routing choice | What you learn | Risk | Use when |
|---|---|---|---|
| None or drop | Host behaviour only, attempted connections | Lowest; staged malware may do nothing | Default for sensitive or unknown samples |
| Simulated services (InetSim) | Protocol behaviour, requested URLs and domains | Low; sample may detect fake responses | Indicator extraction without contacting attackers |
| Anonymised egress (VPN, Tor, SOCKS) | Live C2 interaction, second-stage downloads | Attacker sees a connection; possible participation in attacks | Approved investigations needing live behaviour |
| Direct internet (dirty line) | Full live behaviour | Highest; traffic attributable to the line, legal and abuse exposure | Rarely, with written approval and egress rate limits |
Guest design
CAPE's documentation calls the guest virtual machine "probably the most critical" part of a deployment. Decide the operating system, language, patch level and installed software (Office, PDF reader, browsers, runtimes) to match the estate you defend, because exploits and macro malware depend on versions. Add realistic usage traces such as documents, browsing history and recent files, and reduce obvious virtualisation artefacts. Snapshot the clean guest, and refresh it on a schedule so it does not drift years behind production. CAPE's 2026 release notes support all Windows 10 versions and Windows 11 23H2.
Operating procedure
- Hash first: look up the SHA-256 in internal history and threat intelligence before detonating; a known verdict saves time and avoids exposure.
- Classify sensitivity: a document that may contain customer or internal data stays on an internal sandbox. Uploading to a public service is a disclosure decision; record who approved it.
- Choose routing and timeout: start with no egress or simulated services and a default timeout; rerun with longer timeouts or approved egress only when the first run is inconclusive.
- Detonate and review: read process tree, injected processes, network attempts, dropped files and signature hits together; do not rely on a single score.
- Handle artefacts safely: store samples and dumps encrypted or in password-protected archives, on storage excluded from production backup sync, with access limited to analysts.
- Feed detections: publish indicators to the TIP or SIEM with context and an expiry, and write or update rules.
- Revert and verify: confirm the guest returned to its clean snapshot; rebuild the template if anything unexpected persisted.
Pilot on known-bad test samples and benign files before production use, and keep the previous host image and guest snapshot so an upgrade can be rolled back. Changes to routing or guest images should go through change control like any other security tool.
Where identity fits
Sandboxes accumulate privileged non-human identities: API keys used by the email gateway or SOAR to submit files, service accounts for result storage, keys for threat intelligence lookups and VPN credentials for egress. Scope each to submission or retrieval only, keep them in a secrets manager, rotate them and never place real credentials inside guest images, because malware in the guest will steal them. Seed decoy credentials instead if you want to observe credential theft. See the Secrets Management Guide.
Detection and hardening value
Reading results with evasion in mind
| Evasion behaviour | ATT&CK | What it looks like in a report | Defender response |
|---|---|---|---|
| Hardware, registry or process checks for virtualisation | T1497.001 System Checks | Queries for VM drivers, MAC prefixes, disk size, then early exit | Harden guest artefacts; rerun on a different guest profile or bare metal |
| Waiting for clicks, scrolling or recent documents | T1497.002 User Activity Based Checks | Little activity, calls that poll cursor position | Enable human interaction simulation or use an interactive session |
| Long sleeps or time checks | T1497.003 Time Based Checks | Large sleep calls, run ends with no behaviour | Longer timeout; CAPE debugger countermeasures for known families |
| Debugger and hook detection | T1622 Debugger Evasion | Integrity checks on API stubs, abnormal termination | Compare with static analysis; escalate to a reverse engineer |
| Staged delivery | T1204.002 Malicious File (initial step) | Downloader contacts a URL and stops when there is no egress | Use simulated services or approved egress to capture the next stage |
CAPE addresses some of this directly: its debugger can be programmed through YARA signatures to detect evasive code during detonation and change control flow so the sample runs fully, and it offers options such as active unpacking (unpacker=2, off by default because it may affect detonation quality) and breakpoints (bp0 to bp3) for analysts.
Measuring value
- Time to verdict: median minutes from submission to analyst decision.
- Inconclusive rate: share of runs with no meaningful behaviour, tracked by file type; a rising rate signals evasion or stale guests.
- Detection yield: rules and blocklist entries created from sandbox output per month, and how many later fired on real activity.
- Guest freshness: days since each guest template was patched and re-snapshotted.
- Exposure decisions: public uploads with recorded approval, target 100 per cent.
Monitor the sandbox itself: outbound traffic from the analysis network that bypasses the configured route, logins to the management interface and changes to routing configuration should all alert in the SIEM.
Alternatives
| Need | Better choice than a full sandbox | Why |
|---|---|---|
| Fast verdict on known files | Hash reputation and internal history | No execution, no exposure |
| Understanding obfuscated or evasive code | Static analysis and reverse engineering | Sees code paths that never ran; see the Ghidra for Malware Analysis Guide |
| Behaviour on a real endpoint | EDR telemetry from the affected host | Real environment, no evasion of a lab |
| Scale and support without infrastructure | Commercial or private cloud sandbox service | Managed guests and integrations, at the cost of sending samples to a provider |
| Running untrusted code from AI agents | Agent execution sandboxes and microVMs | Different threat model: containment of a workload, not observation of malware |
Small teams: a commercial or private cloud service plus hash lookups usually beats running CAPE, which needs Linux, KVM and guest maintenance skills. Large SOCs and CTI teams: an internal CAPE deployment for sensitive samples and configuration extraction, with a commercial service for scale and coverage, is a common pattern. In both cases, map the capability to NIST CSF 2.0 DE.AE-02 (adverse events are analysed), NIST SP 800-53 Rev. 5 SC-44 Detonation Chambers, SI-3 Malicious Code Protection and IR-4 Incident Handling, MITRE D3FEND D3-DA Dynamic Analysis and D3-EFA Emulated File Analysis, and ISO/IEC 27001:2022 Annex A 8.7 protection against malware.
Practitioner checklist
- Confirm the maintenance status and licence of the sandbox you run, and migrate off Cuckoo 2.x.
- Separate management and analysis interfaces, and prove guests cannot reach internal DNS, shares or identity providers.
- Default to no egress or simulated services; require written approval for live internet routing.
- Look up hashes before detonating and record approval for any public upload.
- Patch and re-snapshot guest templates on a schedule that matches your estate.
- Rerun inconclusive samples with longer timeouts, interaction simulation or a different guest profile.
- Keep only decoy credentials in guests, and scope, vault and rotate the sandbox's own API keys.
- Turn every confirmed sample into indicators with expiry and at least one YARA, Suricata or Sigma rule.
Standards and references
- CAPE: CAPEv2 malware configuration and payload extraction sandbox
- CAPE: CAPEv2 documentation
- CERT-EE: Cuckoo3
- Cuckoo Foundation: Cuckoo Sandbox 2.x repository
- MITRE ATT&CK: Virtualization/Sandbox Evasion
- MITRE D3FEND: Dynamic Analysis
- NIST: SP 800-53 Rev. 5 (SC-44 Detonation Chambers)
- NIST: Cybersecurity Framework 2.0
Related NHI Mgmt Group resources: Malware Analysis Guide for Defenders · YARA Rules Guide · Beaconing and C2 Detection Guide · Malware Outbreak Playbook