When a suspicious file or memory region turns up during an incident, the response stalls until someone can say what it does, how it persists, what it talks to and what it stole. Malware analysis for defenders is the disciplined process that answers those questions fast enough to drive containment, using triage, static analysis, controlled dynamic analysis and, only where needed, reverse engineering. This guide sets out a tiered analysis method, a safe lab and handling model, how samples try to defeat analysis, and how to turn findings into indicators, detections and hardening. It is written for SOC analysts, incident responders and detection engineers who analyse samples as part of response rather than as full-time reverse engineers. For deep code analysis, see the Ghidra for Malware Analysis Guide; this page covers the overall method.
Key takeaways
- Analyse to answer response questions. Start from what the incident needs (scope, persistence, C2, data at risk, credentials exposed) and stop when those are answered with enough confidence.
- Work in tiers. Triage and reputation in minutes, static properties in under an hour, sandbox and controlled dynamic analysis next, and reverse engineering only for the questions the earlier tiers cannot answer.
- Isolation and handling come first. A dedicated, isolated lab, hashed and password-protected sample storage and rules on what may be uploaded to public services protect both the organisation and the investigation.
- Assume the sample resists analysis. Packing, obfuscation and sandbox checks are normal; a clean sandbox report is not a verdict.
- Output is detection and action, not a report alone. Every analysis should produce behavioural detections, YARA or Sigma rules, prioritised indicators and a credential rotation list.
How the attack works
For the analyst, the "attack" is the sample's attempt to run, persist and hide while you study it. Understanding the common lifecycle tells you what to look for at each tier.
| Stage | What malware typically does | What the analyst looks for | Related ATT&CK technique |
|---|---|---|---|
| Delivery and execution | Arrives as a document, archive, script, installer or loader that the user or a service runs | File type versus extension, parent process, embedded scripts or macros | T1204.002 Malicious File |
| Staging | Downloads further stages or tools | Embedded URLs, download functions, second-stage hashes | T1105 Ingress Tool Transfer |
| Concealment | Packs, encrypts or encodes code and strings; loads code reflectively or injects into other processes | High entropy sections, few imports, unbacked executable memory | T1027 Obfuscated Files or Information, T1620 Reflective Code Loading, T1055 Process Injection |
| Analysis evasion | Checks for virtual machines, debuggers, sandbox artefacts, user activity or delays execution | Environment checks in strings or behaviour; activity that only starts after long sleeps | T1497 Virtualization/Sandbox Evasion |
| Persistence | Creates services, scheduled tasks or Run keys | Registry and service changes during detonation | T1543.003 Windows Service, T1053.005 Scheduled Task, T1547.001 Registry Run Keys |
| Credential access | Reads browser stores, LSASS memory or token caches | Access to credential files and processes | T1003.001 LSASS Memory |
| Command and control | Beacons over HTTPS or other protocols, often to cloud or compromised infrastructure | Domains, IP addresses, URI patterns, certificate and timing characteristics | T1071.001 Web Protocols |
Families and delivery chains are covered in the Malware Guide, the Malware Loaders and Droppers Guide and the Fileless Malware Guide.
Where it shows up
Samples reach analysts from several sources, and each changes what can be learned.
- EDR quarantine and alerts: the file plus process tree and command line; the richest starting point.
- Email security and user reports: attachments and links, often before execution. See the Phishing Report Button and User Reported Message Triage Guide.
- Disk and triage collections: files found by timeline or artefact review, with execution evidence from prefetch, Amcache and event logs.
- Memory images: injected code and unpacked payloads that never touch disk; see the Volatility 3 Guide.
- Network captures and proxies: downloaded payloads and C2 traffic.
- Software supply chain: malicious packages and build artefacts, where the "sample" is a dependency.
The tiered method
| Tier | Activities | Typical time | Output |
|---|---|---|---|
| 1. Triage | Hash (SHA-256), file type identification, reputation lookup by hash in internal and approved threat intelligence sources, check whether it is signed and by whom | Minutes | Known or unknown; initial severity |
| 2. Static properties | Strings, imports, sections and entropy, embedded resources, compile and signature metadata, YARA matches; for documents and scripts, extract and deobfuscate embedded code | Under an hour | Capabilities hypothesis, candidate indicators |
| 3. Automated dynamic | Detonation in a sandbox with realistic configuration: process, file, registry and network activity, dropped files, memory dumps | Minutes to an hour | Behaviour report, network indicators, dropped payloads |
| 4. Controlled manual dynamic | Run in an instrumented, isolated VM with monitoring tools and a simulated network; observe persistence and C2 handling | Hours | Confirmed behaviours, configuration and protocol details |
| 5. Reverse engineering | Disassembly and decompilation to recover configuration, encryption, kill switches or capabilities not seen in execution | Hours to days | Config extractors, high-fidelity rules, capability list |
Most incident samples are resolved at tiers 1 to 3. Escalate to tiers 4 and 5 when the sample is novel, targeted, evasive, or when the response depends on details such as a decryption routine or the full list of C2 servers. Sandbox selection and configuration are covered in the Malware Sandbox Guide.
Impact
The quality and speed of analysis directly change the cost of the incident.
- Scope: indicators and behaviours from analysis decide which other hosts are hunted. Missing a second C2 domain or persistence method leaves footholds after "eradication".
- Credentials: knowing that a sample is an infostealer, or dumped LSASS, turns a host clean-up into an identity incident covering every account, token and API key used on the machine. See the Infostealer Threat Guide.
- Data: exfiltration capability and destinations inform notification decisions.
- Exposure from poor handling: uploading a targeted sample or an internal document to a public service can reveal the investigation to the attacker and disclose confidential data embedded in the file.
Where identity fits
Treat every confirmed credential-access capability as an identity event: list human and non-human identities whose secrets were present on the host (service accounts, cached domain credentials, cloud CLI tokens, browser sessions, API keys in configuration files), then rotate and review their recent use. See the Credential Dumping Guide and the Secrets Management Guide.
Detection
Analysis exists to improve detection. Convert findings into durable detections in order of resilience: behaviours first, tooling and code patterns second, infrastructure last.
| Finding | Detection artefact | Durability |
|---|---|---|
| Process behaviour (parent and child chain, injection, persistence method) | EDR or Sigma rule on the behaviour; see the Sigma Rules Guide | High: survives recompilation and infrastructure changes |
| Unique code, strings or structure | YARA rule for files and memory; see the YARA Rules Guide | Medium to high, depending on what the rule anchors on |
| Network protocol traits (URI pattern, header order, beacon timing) | Network or proxy detection | Medium |
| Domains, IP addresses, file hashes | Blocklists and IOC sweeps; see the IOC Scanners Guide | Low: change quickly, but useful for immediate scoping |
Reporting indicators
- Context with every indicator: type, role (payload, C2, staging), first and last seen, confidence and the analysis tier that produced it.
- Machine-readable sharing: express indicators and behaviours in STIX 2.1 for platforms and partners. See the STIX and TAXII Guide and the Indicators of Compromise vs Indicators of Attack Guide.
- ATT&CK mapping: map observed behaviours to technique IDs so coverage gaps are visible to detection engineering.
- Audience split: a one-paragraph verdict and actions for incident command, and a technical annex for the SOC and threat intelligence. See the Threat Intelligence Report Template.
Measuring the capability
- Time to verdict: minutes from sample receipt to a triage verdict and severity (target under 30 minutes for tier 1 and 2).
- Escalation rate: share of samples needing tier 4 or 5; a high rate suggests weak sandbox configuration.
- Detection yield: behavioural detections or YARA rules created per analysed family.
- Retro-hunt hit rate: share of new rules that find additional affected hosts.
- Handling exceptions: count of samples or documents shared outside approved channels (target zero).
Prevention and hardening
Building a safe lab
- Isolate: analysis VMs on a separate network segment or host with no route to production, no shared folders or clipboard by default, and snapshots to revert after every run.
- Control the network: default to a simulated internet; allow real egress only through a controlled, attributable connection when the investigation needs live C2 behaviour, with approval.
- Use dedicated identities: no corporate accounts, tokens or VPN profiles in the lab; lab credentials should be unique and monitored.
- Store samples safely: keep samples in password-protected archives named by SHA-256, in a restricted repository, with access logged.
- Define sharing rules: state which samples may be submitted to public sandboxes and reputation services (for example commodity malware) and which may not (targeted samples, files containing internal data); hash lookups are usually safe where uploads are not.
- Test and rehearse: run benign test files through the full workflow quarterly and confirm tooling, snapshots and isolation.
Feeding findings back
- Endpoint: add behaviour blocks, attack surface reduction rules or application control for the observed execution chain. See the CIS Control 10 Malware Defenses Guide.
- Email and web: block the delivery file types and URL patterns seen in the chain.
- Identity: rotate exposed credentials and reduce standing privileges on hosts where credential theft was possible.
- Response: update the Malware Outbreak Playbook with containment steps specific to the family.
Framework mapping
| Framework | Reference | How malware analysis supports it |
|---|---|---|
| NIST SP 800-53 Rev. 5 | SI-3 Malicious Code Protection; IR-4 Incident Handling; SI-4 System Monitoring | Analysis improves malicious code protection and drives incident handling decisions |
| NIST CSF 2.0 | DE.AE-02, RS.AN-03 | Adverse events are analysed; root cause is established |
| MITRE D3FEND | D3-DA Dynamic Analysis, D3-FA File Analysis, D3-EFA Emulated File Analysis, D3-FHRA File Hash Reputation Analysis | Defensive techniques that correspond to the analysis tiers |
| NIST SP 800-83 Rev. 1 | Malware incident prevention and handling | Overall prevention and response process for malware incidents |
Binary formats and packers
Static triage starts with the file format, because the headers tell an analyst what platform the sample targets, where code starts and whether the file looks normal. This section covers recognition only; unpacking a live sample belongs in an isolated lab.
- PE (Windows): Microsoft's PE format specification describes an MS-DOS stub at the front of the image, with the file offset of the "PE\0\0" signature at 0x3c. A COFF header follows (machine type, number of sections, characteristics), then an optional header that is required for images, whose magic value distinguishes PE32 (0x10b) from PE32+ (0x20b) and which holds the entry point and data directories such as the import table. A section table of 40-byte entries follows.
- ELF (Linux and Unix): the file starts with 0x7f followed by "ELF". The elf(5) manual page describes two views: the section header table, which locates sections such as .text, .data and the symbol tables used in linking, and the program header table, whose entries describe segments the system needs to prepare the program for execution.
- Mach-O (macOS): Apple's loader.h defines the 64-bit magic 0xfeedfacf. Load commands directly follow the header, and each LC_SEGMENT_64 command describes a segment such as __TEXT, __DATA or __LINKEDIT, followed by its sections, for example __text.
Sections versus segments: sections are the linker's view of a file; segments are the loader's view of memory. In ELF and Mach-O, segments contain one or more sections. In PE, each section header records both a size on disk (SizeOfRawData) and a size in memory (VirtualSize), and the specification says the remainder is zero-filled when the virtual size is larger.
Packers: UPX is a free executable compressor for Windows, macOS and Linux programs that can also list, test and unpack the files it produced. MITRE ATT&CK tracks packing as T1027.002 Software Packing, naming UPX and MPRESS and noting custom packers that leave fewer known artefacts. Analysts suspect packing when they see:
- very few imports, often little beyond loader functions;
- a section whose virtual size is far larger than its raw size, or raw size near zero;
- high entropy in code or data sections and few readable strings;
- non-standard or packer-named sections, and an entry point outside the normal code section;
- at runtime, memory allocation followed by execution from non-image memory, which ATT&CK lists as a behavioural detection.
For tooling, see the Ghidra for Malware Analysis Guide.
Practitioner checklist
- Write down the response questions before analysing any sample.
- Apply the five-tier method and record which tier produced each finding.
- Maintain an isolated lab with snapshots, simulated networking and no corporate credentials.
- Store samples by SHA-256 in password-protected archives with logged access.
- Publish clear rules on what may be submitted to public services.
- Produce at least one behavioural detection or YARA rule per new family and retro-hunt with it.
- Share indicators with context and confidence, in STIX where possible, mapped to ATT&CK.
- Turn any credential-access capability into an identity rotation task.
- Track time to verdict, escalation rate and detection yield.
Standards and references
- NIST: SP 800-83 Rev. 1 Guide to Malware Incident Prevention and Handling for Desktops and Laptops
- NIST: SP 800-86 Guide to Integrating Forensic Techniques into Incident Response
- NIST: SP 800-53 Rev. 5
- NIST: Cybersecurity Framework 2.0
- MITRE ATT&CK: Virtualization/Sandbox Evasion
- MITRE D3FEND: Dynamic Analysis
- YARA: Documentation
- Microsoft Learn: PE Format
- Linux man-pages: elf(5), format of ELF files
- Apple open source (XNU): mach-o/loader.h
- UPX: the Ultimate Packer for eXecutables
- MITRE ATT&CK: T1027.002 Software Packing
Related NHI Mgmt Group resources: Ghidra for Malware Analysis Guide · Malware Sandbox Guide · YARA Rules Guide · Malware Analyst Career Guide · Volatility 3 Guide