Files that cross the network without touching a mail gateway or a managed endpoint are invisible to file scanners, unless a sensor rebuilds them from the packets. MITRE D3FEND names this countermeasure File Carving (D3-FC) and defines it as "Identifying and extracting files from network application protocols through the use of network stream reassembly software." It sits under the Detect tactic as a child of Network Traffic Analysis (D3-NTA), and its one digital artefact is File Transfer Network Traffic. File carving does not decide whether a file is bad: it is the feed that hands files to hashing, rules and sandboxes. This card covers how carving works, what D3FEND relates it to, how two widely used open source sensors implement it, how to check it and where it breaks. It is for network detection engineers and SOC leads; packet capture basics are in the Network Monitoring and Packet Capture Guide.
Key takeaways
- It is a feed, not a verdict. D3FEND says a carved file "is then processed with standard File Analysis Techniques", so value depends on what you do with each file next.
- Reassembly is the hard part. D3FEND calls the process error prone: packets arrive out of order, fragmented, duplicated or retransmitted, and the software must rebuild a well formed file from that.
- Encryption limits what you can carve. Files inside TLS are only reachable where traffic is decrypted or the sensor sees it in clear, so carving coverage follows your inspection points.
- No direct ATT&CK mapping. D3FEND lists no hand-mapped technique for D3-FC; two inferred relations come through its artefact, T1071.002 File Transfer Protocols and T1570 Lateral Tool Transfer.
- Depth limits silently truncate files. Sensors cap how much of each stream they reassemble, and Suricata's documentation warns that files reaching the limit may be truncated.
D3-FC at a glance
| Attribute | Value |
|---|---|
| D3FEND ID | D3-FC (d3f:FileCarving) |
| Tactic | Detect |
| Parent technique | Network Traffic Analysis (D3-NTA) |
| Digital artefact | File Transfer Network Traffic (D3FEND relation: analyzes) |
| Example protocols named by D3FEND | HTTP, SMTP, FTP, HTTP/2, and TLS/HTTP/Dropbox |
| Files of interest named by D3FEND | Executable, archive and document file formats |
| D3FEND reference | FireEye patent US20130036472A1, Computer Worm Defense System and Method: network data copied by a tap, analysed by a heuristic module, and replayed to a virtual machine |
| Main limitation | Encrypted traffic, imperfect packet capture and file type confusion |
How it works
D3FEND describes three steps, and real sensors add a fourth.
- Stream reassembly: software rebuilds a directional byte stream from captured packets, either in real time or by writing packets to disk and processing them in bulk.
- Protocol parsing: an application protocol parser finds where file content begins and ends inside the stream: an HTTP response body, an SMTP MIME part, an SMB write.
- File identification: pattern matching decides whether the content is a file of interest by its type, which sensors detect from content rather than trusting the name.
- Hand-off: the file, or only its hash and metadata, goes to analysis: hash reputation, content rules, emulation or sandbox detonation.
The FireEye patent D3FEND cites shows the full pattern: a network tap copies traffic, a heuristic module flags suspicious data and the transmission is replayed into a virtual machine to see what it does. The downstream analysis techniques are covered in D3-DA Dynamic Analysis, D3-FH File Hashing and D3-FCR File Content Rules.
ATT&CK techniques D3FEND relates to D3-FC
Relationship type: inferred only. D3FEND has no direct ATT&CK mapping for D3-FC. Its site lists two relations it labels "inferred and experimental", derived because D3-FC analyses File Transfer Network Traffic and these techniques produce sub-types of it (D3FEND knowledge graph 1.6.0, checked November 2026; IDs checked against ATT&CK v20).
| ATT&CK ID | Technique | D3FEND relation (inferred via artefact) | What carving can surface |
|---|---|---|---|
| T1071.002 | File Transfer Protocols | produces Outbound Internet File Transfer Traffic | Files moved over FTP, SMB or similar protocols used as a command and control channel |
| T1570 | Lateral Tool Transfer | produces Intranet File Transfer Traffic | Tools copied between internal hosts, often over SMB, where an east-west sensor sees it |
In NHI Mgmt Group's view, not D3FEND data, carving at the internet edge is also the main network-side control for files downloaded into the estate, which ATT&CK describes as T1105 Ingress Tool Transfer. Defensive detail is in MITRE ATT&CK T1570 Lateral Tool Transfer, MITRE ATT&CK T1071 Application Layer Protocol and T1105 Ingress Tool Transfer Defensive Guide.
Where it is implemented
Network detection and response products, network sandboxes and full packet capture platforms all carve files. Two open source sensors document the mechanics openly, so they are described here as examples, alphabetically.
| Aspect | Suricata | Zeek |
|---|---|---|
| Protocols documented for file handling | HTTP, SMTP, FTP, NFS, SMB and HTTP2 | FTP, HTTP, IRC, Kerberos, MIME, RDP, SMTP and SSL/TLS/DTLS |
| How extraction is triggered | Rules with the filestore keyword; matching by fileext, filemagic and checksum lists (filemd5, filesha1, filesha256) | Script calls Files::add_analyzer with Files::ANALYZER_EXTRACT, usually from a file_new handler; policy/frameworks/files/extract-all-files.zeek is a starting point |
| Type identification | filemagic matching on the detected type | Zeek's own content signatures, reported in the file_sniff event |
| Metadata log | EVE fileinfo records, with force-magic and force-hash options | files.log with size, timing and MIME type for every file seen |
| Storage | File-store version 2: files named by SHA256 in subdirectories 00 to ff, deduplicated | Files written under FileExtract::prefix, normally ./extract_files/ |
| Depth setting | file-store.stream-depth (documented default 1 MB, 0 for unlimited) and HTTP body limits | Extraction analyzer options set in script |
A full packet capture platform such as the one in the Arkime Full Packet Capture Guide lets you carve after the event from stored packets, which is the bulk processing mode D3FEND mentions. Product selection for commercial sensors is covered in the NDR Buyer's Guide.
Implementation steps
- Pick sensor positions. Place sensors where files cross in clear text or after decryption: behind the TLS inspection point at the internet edge, on mail relay segments, and east-west on server and file share segments where lateral transfer would pass.
- Decide what to carve. Start with executables, scripts, archives and Office and PDF documents, and with SMB to and from user segments. Carving every file type everywhere fills disks without adding detections.
- Set depth limits deliberately. Raise stream and body depth for the file types you care about, accept the CPU and memory cost, and record the limit so analysts know what a truncated file means.
- Log metadata for everything, store content selectively. Hash and log every file seen; store full content only for types of interest or files that match rules or hash lists. Treat stored files as hazardous and possibly sensitive data, with access control and a retention period.
- Wire the hand-off. Send hashes to reputation lookup and carved files to content rules, emulation or a sandbox, then send verdicts to the SIEM keyed to the connection so analysts can pivot to the hosts involved.
- Pilot and roll back. Enable carving on one sensor first, watch packet drop and CPU, and keep the previous configuration so you can revert if the sensor starts dropping traffic.
How to verify it works
- Known-file test: from an approved test host, download a benign file of each type of interest (including the EICAR test file) over each monitored protocol and confirm the sensor logs the file, the hash matches the original and the downstream verdict arrives.
- Truncation test: transfer a file larger than the configured depth and confirm how the sensor marks it, so analysts recognise partial files.
- Health metrics: watch capture loss and reassembly errors; a sensor dropping packets carves broken files without saying so.
- Coverage check: compare files seen by the sensor with files seen by the mail gateway or proxy for the same period to find blind segments.
| Metric | Example target (NHI Mgmt Group suggestion, not a standard) | Why it matters |
|---|---|---|
| Test files carved with matching hash, per protocol | 100% | Proves reassembly and parsing work end to end |
| Sensor capture loss | Near zero at peak | Dropped packets mean corrupt or missing files |
| Carved files of interest sent to analysis | 100% of stored files | Carving without analysis detects nothing |
| Share of files truncated by depth limits | Tracked per protocol | Shows whether limits hide large payloads |
| Segments with a sensor that can see clear text | Rising, gaps documented | Coverage of where files actually move |
Limits, bypasses and failure patterns
- Encrypted transport: HTTPS, SMB encryption and encrypted archives hide content. Carving coverage depends on decryption points; see M1020 SSL/TLS Inspection.
- File type confusion: D3FEND notes file type identification is difficult and can be exploited by adversaries, for example by renaming, nesting or wrapping content.
- Imperfect capture: out of order, fragmented and retransmitted packets, and capture loss on busy links, produce corrupt files that analysis rejects or misjudges.
- Depth limits: payloads beyond the configured depth are truncated, and a truncated executable rarely detonates.
- Sensitive data in storage: carved files include business documents and personal data, so an unmanaged carve directory becomes a data protection problem.
- No downstream analysis: files stored and never analysed give evidence after an incident but no detection.
Framework mapping
| Framework | Reference | Relevance |
|---|---|---|
| NIST SP 800-53 Rev 5 | SI-4 System Monitoring | Network monitoring that carving extends to file content |
| NIST SP 800-53 Rev 5 | SI-3 Malicious Code Protection | Carved files feed malicious code detection at the network level |
| NIST CSF 2.0 | DE.CM-01 | Networks and network services are monitored to find potentially adverse events |
| MITRE ATT&CK mitigation | M1031 Network Intrusion Prevention | Inline sensors that act on carved file verdicts; see M1031 Network Intrusion Prevention |
| MITRE D3FEND | D3-NTA, D3-FC | Parent and technique |
Practitioner checklist
- Place sensors where files are visible in clear text, including behind TLS inspection.
- Carve executables, scripts, archives and documents first; log hashes for everything.
- Set and record stream and body depth limits per protocol.
- Send every stored file to hashing, rules or detonation, and verdicts to the SIEM.
- Protect and expire the carved file store as sensitive data.
- Monitor capture loss and reassembly errors on every sensor.
- Run known-file tests per protocol on a schedule and compare hashes.
Standards and references
- MITRE D3FEND: File Carving (D3-FC)
- NIST: SP 800-53 Rev. 5 Security and Privacy Controls (SI-4 System Monitoring)
- MITRE ATT&CK: T1570 Lateral Tool Transfer
- MITRE ATT&CK: T1071.002 File Transfer Protocols
- Suricata documentation: File Extraction
- Zeek documentation: File Analysis Framework
Related NHI Mgmt Group resources: Network Forensics Guide · NDR Guide · IDS and IPS Guide · MITRE D3FEND Guide