Every meaningful action a process takes, from opening a file to writing into another process's memory, ends up as a request to the kernel, and that request stream is one of the hardest things for malicious code to hide. MITRE D3FEND catalogues the defensive use of that stream as D3-SCA System Call Analysis, a Detect tactic countermeasure that sits under Process Analysis (D3-PA). D3FEND defines it as analysing system calls to determine whether a process is exhibiting unauthorised behaviour. This card explains how the countermeasure works on Windows and Linux, which ATT&CK techniques D3FEND relates it to, how to deploy it without drowning in telemetry, and how to prove it fires. It is written for detection engineers, EDR owners and Linux platform teams.
Key takeaways
- It watches the kernel boundary. D3-SCA analyses the system call artefact: the calls a process makes to the operating system, their arguments and their order.
- Sequences beat single calls. D3FEND's own examples are behaviour chains, such as fetch a URL, write a new file, execute that file, rather than one suspicious API.
- Collection method decides what you can trust. User-mode hooks can be removed by the code being watched; kernel sensors (ETW providers, eBPF, kernel audit) are harder to blind but cost more to run.
- Mappings are broad. D3FEND relates D3-SCA to process injection, signed binary proxy execution, UAC bypass, AppInit and AppCert DLLs and many discovery techniques, all inferred through the System Call artefact.
- Verification is mandatory. Benign test harnesses, CAR analytics and counts of rule hits per host show whether the sensor and the logic both work.
D3-SCA at a glance
| Field | Value |
|---|---|
| D3FEND ID | D3-SCA System Call Analysis |
| Tactic | Detect |
| Parent technique | D3-PA Process Analysis |
| Child technique | D3-FCA File Creation Analysis (properties of file create system call invocations) |
| Digital artefact analysed | System Call (synonym: syscall), the programmatic way a program requests a kernel service |
| Typical data sources | EDR kernel sensors, Windows ETW and Sysmon (process access, remote thread, image load), Linux audit subsystem, eBPF based runtime sensors, seccomp in LOG mode |
| Effort | Medium to high: sensor deployment is easy, tuning and baselining are the real cost |
| Main limitation | Volume and evasion: hooks can be bypassed, padding calls can disguise sequences, and models age |
How it works
D3FEND describes two stages. First, gather: kernel APIs are used to hook a process's system call invocations. On Linux, ptrace lets one process trace and alter another's system calls (debuggers such as GDB use it, and strace builds on it to print each call). Production sensors record the same information to local or remote stores. Second, analyse: each record, with context such as time and the process tree, goes to an analysis module that compares it against a catalogue of known good or known bad patterns, either as exact sequences or by statistical deviation.
D3FEND gives three examples of sequences that rise to user activity worth alerting on:
- Privilege escalation bursts: many calls in a short window, including several sequences that could be used to escalate privileges.
- Download and run: fetching content from a URL, writing it to a new file and then executing that same file.
- Ransomware loop: a loop or many threads that read a file, encrypt it, write a similarly named output and delete the original.
Collection options by platform
| Platform | Mechanism | What it gives you | Watch out for |
|---|---|---|---|
| Windows | Sysmon Event ID 8 CreateRemoteThread | Source and target process plus StartAddress, StartModule and StartFunction of the new thread | StartModule and StartFunction are inferred and can be empty when the address is outside loaded modules |
| Windows | Sysmon Event ID 10 ProcessAccess | One process opening another, the precursor to memory reads such as LSASS access | Microsoft advises filters that remove expected accesses, as diagnostic tools generate heavy volume |
| Windows | Sysmon Event ID 7 Image loaded, Event ID 25 ProcessTampering | DLL loads and process image changes | Event 7 is off by default (enable with -l) and is very noisy unfiltered |
| Windows | EDR kernel and ETW sensors | Wider API coverage than Sysmon, often with in-line blocking | User-mode hook sensors can be unhooked by the monitored process |
| Linux | Kernel audit (auditctl -a always,exit -F arch=b64 -S ...) | Per-syscall records with a searchable key (-k) | Every syscall rule is evaluated for every call, so rule count drives overhead |
| Linux | eBPF runtime sensors (for example the CNCF Falco project's modern eBPF probe) | Streaming syscall events with container and process context | Needs BTF and BPF ring buffer support; the sensor itself needs privileged capabilities |
| Linux | seccomp filter with SECCOMP_RET_LOG | Log of calls a workload makes, useful for building an allow list | Logged only if log is in /proc/sys/kernel/seccomp/actions_logged; ALLOW cannot be logged |
See the Sysmon Guide, the Linux Logging and auditd for Defenders Guide and the eBPF Security Guide for sensor setup detail.
Where identity fits
Credential theft from memory is a system call story: a process opening LSASS with read access, or creating a remote thread inside it, is how service account and machine credentials leave a Windows host. Three of the CAR analytics D3FEND cites for D3-SCA are exactly this (CAR-2020-05-001 MiniDump of LSASS, CAR-2021-05-011 Create Remote Thread into LSASS, CAR-2019-08-001 Credential Dumping via Windows Task Manager). On Linux and in containers, a workload process reading a mounted token or secret file and then opening an outbound connection is a sequence worth a rule, because that is how a stolen workload identity is typically used.
ATT&CK techniques D3FEND relates to D3-SCA
Relationship type: inferred, not a direct mapping. D3FEND does not map ATT&CK techniques to D3-SCA directly; it infers each relationship because the technique and the countermeasure involve the same digital artefact (System Call). An inferred relationship shows relevance, not coverage.
The table shows a selection of the techniques on the D3FEND D3-SCA page, each checked against ATT&CK v20 and grouped by ATT&CK v20 tactic. The evidence column is an NHI Mgmt Group illustration of what to look for, not D3FEND content.
| ATT&CK ID | Name | Example system call evidence (NHI Mgmt Group illustration) |
|---|---|---|
| Stealth and privilege escalation: process injection | ||
| T1055.001 | Dynamic-link Library Injection | VirtualAllocEx, WriteProcessMemory, then CreateRemoteThread starting at LoadLibrary (CAR-2013-10-002) |
| T1055.003 | Thread Execution Hijacking | Suspend, context change and resume of a thread in another process |
| T1055.004 | Asynchronous Procedure Call | Remote memory write followed by APC queueing to a foreign thread |
| T1055.005 | Thread Local Storage | Remote write that alters TLS callbacks before thread start |
| T1055.008 | Ptrace System Calls | ptrace attach to an unrelated process outside debugging workflows |
| T1055.013 | Process Doppelgänging | Transacted file operations followed by section and process creation |
| T1055.014 | VDSO Hijacking | Memory mapping changes affecting the vDSO of another process |
| T1134.004 | Parent PID Spoofing | Process creation with an explicit parent attribute that differs from the creator |
| Stealth: proxy execution, masquerading and evasion | ||
| T1218.001, .002, .003, .005, .011, .013 | Compiled HTML File, Control Panel, CMSTP, Mshta, Rundll32, Mavinject | Signed binary followed by network, file write or remote thread calls it does not normally make |
| T1036.005 | Match Legitimate Resource Name or Location | System binary name executing from an unexpected path |
| T1497.003 | Time Based Checks | Sleep and timing calls clustered before payload behaviour |
| Privilege escalation and persistence | ||
| T1548.002 | Bypass User Account Control | Elevated process spawned by an auto-elevating binary with a modified environment |
| T1548.004 | Elevated Execution with Prompt | macOS authorisation API calls that prompt for administrator credentials |
| T1546.009, T1546.010 | AppCert DLLs, AppInit DLLs | Unexpected DLL loads into many processes at creation |
| T1505.001 | SQL Stored Procedures | Database server process spawning shells or writing files |
| Discovery, collection and credential access | ||
| T1518.001 | Security Software Discovery | Enumeration of processes, services and drivers of security tools |
| T1074.001 | Local Data Staging | Many reads followed by writes into one archive or directory |
| T1555.003 | Credentials from Web Browsers | Non-browser process opening browser credential stores |
D3FEND also relates D3-SCA to T1106 Native API, T1047 Windows Management Instrumentation, T1053 Scheduled Task/Job, T1140 Deobfuscate/Decode Files or Information, T1220 XSL Script Processing and a set of discovery techniques, including T1007, T1016, T1018, T1033, T1049, T1057 and T1082. For the defensive detail behind the main ones, see the T1055 Process Injection Defensive Guide, the T1218 System Binary Proxy Execution Defensive Guide, MITRE ATT&CK T1548 Abuse Elevation Control Mechanism and the MITRE ATT&CK T1134 Access Token Manipulation guide.
Choosing an approach
| Approach | Best when | Trade-off |
|---|---|---|
| Signature sequences (known bad chains) | You need high-confidence alerts for injection and credential access | Misses novel chains; attackers vary call order |
| Allow list per workload (seccomp LOG then enforce) | Containers and services with a stable, narrow call set | Breaks on upgrades unless profiles are regenerated; kernel docs stress seccomp alone is not a sandbox |
| Statistical or model based | Large fleets with good baselines and a data science function | D3FEND notes models trained on old data may lag new threats |
| Vendor EDR behavioural engine | Most Windows and mixed estates | Logic is opaque; you must test it rather than read it |
Small organisations: rely on EDR behavioural detection plus a filtered Sysmon config for events 8 and 10, and keep Linux audit rules to a short, high value set. Large organisations: add per-workload syscall profiles in Kubernetes, stream eBPF events to the SIEM and own a detection-as-code library with tests. See the seccomp Guide and the EDR Guide.
Implementation steps
- Pick the questions first. Start with three: who opens LSASS, who creates remote threads, which signed binaries make network or write calls. Each maps to rows in the table above.
- Deploy the sensor in audit mode. On Windows, enable Sysmon events 8 and 10 with
onmatch="exclude"filters for known diagnostic and security tools (exclude rules win over include rules). On Linux, add audit rules with explicit-F arch=b64and keys, combining calls that share action and fields in one rule as the auditctl manual recommends, and check that no shipped-a never,taskrule silences them. - Baseline for two to four weeks. Record which processes legitimately inject or open others: CAR-2013-10-002 notes csrss.exe and host security software as normal sources of remote threads.
- Write sequence rules. Express detections as chains with a time window and the process tree, not single events, and tag them with the ATT&CK ID.
- Pilot on a ring. Alert only to the detection team for a pilot group; measure false positives per rule per day.
- Move to blocking only where proven. Promote to EDR prevention or seccomp enforce mode (for example
SECCOMP_RET_ERRNOorSECCOMP_RET_KILL_PROCESS) only for rules with a clean pilot. - Keep rollback simple. Version sensor configs and seccomp profiles in source control so a bad rule is a single revert, and return enforced profiles to LOG first rather than removing them.
How to verify it works
- Sensor health: every in-scope host reports system call events within the expected interval; alert on silence.
- Safe emulation: run benign, authorised test harnesses that perform a remote thread creation into a test process you own, and a process access to a dummy target, then confirm the rule fires with the right ATT&CK tag.
- Analytic parity: compare your rules with the CAR analytics D3FEND cites (CAR-2013-10-002, CAR-2020-05-001, CAR-2021-05-011, CAR-2019-08-001) and record gaps.
- Tamper test: confirm that stopping or unloading the sensor raises its own alert. See the EDR Evasion and Defence Guide.
| Metric | Target signal |
|---|---|
| Host coverage | Percentage of in-scope hosts sending syscall telemetry in the last 24 hours |
| Rule hit rate | Alerts per rule per 1,000 hosts per day, trending down after tuning |
| True positive ratio | Share of D3-SCA alerts confirmed as malicious or policy breaking |
| Emulation pass rate | Share of scheduled tests that produced the expected alert |
| Dropped events | Sensor-reported drops per host; spikes mean you are blind during load |
Limits, bypasses and failure patterns
- Padding the sequence: D3FEND notes duplicative or extraneous calls can be added to defeat analytics. Use windows and ordering tolerance rather than strict adjacency.
- Removing the hooks: malware can replace hooking instructions so calls go unmonitored. Prefer kernel-side collection and watch for the unhooking itself. See the Bring Your Own Vulnerable Driver and EDR Killers Guide.
- Stale models: a model built from old training data may not detect new behaviour; retrain on a schedule and keep signature rules alongside.
- Over-broad exclusions: excluding a whole directory or a common binary to cut noise creates a hiding place. See the EDR Tuning and Exclusions Management Guide.
- Unfiltered noise: enabling Sysmon event 7 or audit rules on every call without filters floods the pipeline, and analysts stop looking.
- Architecture gaps on Linux: rules without
arch=or only for b64 miss 32-bit calls, because syscall numbers differ between the two.
Framework mapping
| Framework | Reference | Relevance |
|---|---|---|
| NIST SP 800-53 Rev 5 | SI-4 System Monitoring | Monitoring for attacks and unauthorised behaviour on hosts |
| NIST SP 800-53 Rev 5 | AU-12 Audit Record Generation | Generating the syscall audit records |
| NIST CSF 2.0 | DE.CM-09 | Monitoring software and runtime environments for adverse events |
| NIST CSF 2.0 | DE.AE-02 | Analysing events to understand associated activity |
| MITRE D3FEND | D3-PA (parent), D3-FCA (child) | Placement in the D3FEND tree |
Related countermeasures
Other children of Process Analysis complement D3-SCA: D3-PSA Process Spawn Analysis, D3-SEA Script Execution Analysis, D3-FAPA File Access Pattern Analysis, D3-PSMD Process Self-Modification Detection, D3-PCSV Process Code Segment Verification, D3-IBCA Indirect Branch Call Analysis, D3-SSC Shadow Stack Comparisons and D3-DQSA Database Query String Analysis. Its own child, D3-FCA File Creation Analysis, narrows the analysis to file create calls. For the framework as a whole, see the MITRE D3FEND Guide.
Practitioner checklist
- Confirm which sensor collects system call or API events on each platform and whether it works in kernel or user mode.
- Enable Sysmon events 8 and 10 (or EDR equivalents) with exclude filters for documented legitimate sources.
- Write Linux audit rules with explicit architecture, keys and combined syscalls, and remove any blanket never rule.
- Build at least one sequence rule each for LSASS access, remote thread creation and signed binary proxy behaviour.
- Profile container workloads with seccomp LOG before enforcing allow lists.
- Run safe emulation tests monthly and track the pass rate.
- Alert on sensor silence, dropped events and sensor tampering.
- Review exclusions quarterly and remove any without an owner.
Standards and references
- MITRE D3FEND: System Call Analysis (D3-SCA)
- MITRE CAR: CAR-2013-10-002 DLL Injection via Load Library
- Microsoft Learn: Sysmon
- Linux kernel documentation: Seccomp BPF (SECure COMPuting with filters)
- Linux man-pages: auditctl(8)
- The Falco Project: Kernel event source
Related NHI Mgmt Group resources: MITRE D3FEND Guide · T1055 Process Injection Defensive Guide · Sysmon Guide · seccomp Guide · EDR Guide