Linux Forensics Guide: Artefacts, Collection Order and Investigation Workflow

Linux runs most cloud workloads, web servers and network appliances, yet many incident responders are far less fluent in its artefacts than in Windows event logs and the registry. Evidence on a Linux host is spread across systemd journals, auditd records, text logs whose paths differ by distribution, shell history that is written only when a shell exits, cron and systemd units, SSH key files and the live state in /proc, and much of it disappears on reboot or is easy for an intruder to alter. This guide is for DFIR practitioners, SOC analysts and platform engineers who need to investigate a compromised Linux server or virtual machine. It covers collection order, the artefacts that answer each investigative question, persistence locations, anti-forensics, tooling and how to prepare hosts so the evidence exists. General DFIR method is in the Digital Forensics (DFIR) Guide; this guide is Linux specific.

Key takeaways

• Collect volatile state before anything else. Running processes, network connections, deleted-but-running binaries and memory are lost on reboot, and many Linux implants live only in memory or in /tmp and /dev/shm.
• Know which logs the host actually had. Journald may be volatile, auditd may not be installed and text log paths differ between Debian and Red Hat families, so establish the logging baseline before drawing conclusions from absence.
• Persistence has a short list of homes. Cron, systemd units and timers, SSH authorized_keys, shell start-up files, PAM modules, preload libraries and kernel modules cover most real cases.
• Shell history is weak evidence. It is written on exit, has no timestamps unless configured and is trivially cleared; corroborate with auditd EXECVE records and process accounting.
• Machine credentials on the host are part of the scope. SSH private keys, cloud credentials, tokens in environment files and service account secrets must be inventoried and rotated, not just the user passwords.

Collection order and preserving evidence

NIST SP 800-86, Guide to Integrating Forensic Techniques into Incident Response (Final, 2006), frames forensics from an IT rather than law enforcement view and remains the baseline reference for integrating collection into incident response. On Linux, the practical order is from most to least volatile:

1. Record the context: hostname, time and time zone, clock offset from a trusted source, uptime and who is logged in. Note every command you run with a timestamp.
2. Memory: acquire RAM before running large tool sets. LiME is a loadable kernel module that writes memory to disk or over TCP; its documentation recommends the lime or padded formats over raw, and warns that the digest and compression options disturb more of the target's memory. The module must match the running kernel, so prebuild modules for your standard kernels. On cloud VMs, a provider snapshot that includes memory, where available, avoids loading a module. See the Memory Forensics Guide and the Volatility 3 Guide.
3. Live state: process list with full command lines and parent relationships, open network sockets with owning processes, loaded kernel modules, mounts, logged-in users, and for each suspicious process its /proc/<pid>/exe, cmdline, environ and maps. A process whose exe link ends in "(deleted)" is running from a binary that has been removed from disk; copy it from /proc while it is still running.
4. Triage collection: logs, configuration, user artefacts and a file system timeline bodyfile.
5. Disk: snapshot or image the volume. In the cloud, snapshot the disk and analyse a copy attached to a forensic instance, never the original.

Hash everything you collect, write output to external or network storage rather than the suspect disk, and record chain of custody. Collection on a live host always changes it; the aim is to change it as little as possible and document what you did. Note that some journalctl operations such as vacuum, rotate and flush change journal state, so run analysis against copies.

Artefacts by investigative question

Systemd journal and auditd

• Journald storage: journals are persistent only if /var/log/journal exists or Storage=persistent is set; otherwise they live in /run/log/journal and are lost on reboot. Check this first, because "no journal entries before boot" may simply mean volatile storage.
• Reading journals offline: journalctl --directory=, --file=, --root= or --image= read copied journals or mounted images. Use --utc for cross-host timelines, --list-boots and -b to scope boots, field matches such as _SYSTEMD_UNIT=sshd.service, _COMM= and _UID=, and -o json or -o export to keep every field.
• Journal integrity: --verify checks internal consistency, and with Forward Secure Sealing (--setup-keys run before an incident, verified with --verify-key=) can show whether sealed entries were altered. --header helps spot out-of-order entries from a changed clock.
• auditd: on RHEL the audit log is /var/log/audit/audit.log, with each event split into records (SYSCALL, EXECVE, PATH, CWD, PROCTITLE) that share a timestamp and event ID. Use ausearch -i to interpret values, -m USER_LOGIN or -k <key> to filter, -ts for time, and aureport for summaries. The value of auditd in an investigation depends entirely on the rules that were loaded; record them from /etc/audit/rules.d/.
• Audit tampering: look for auditd stopped, rules removed or auditctl -e 0 in records and journals; ATT&CK v20 lists execution of auditctl and stopping auditd as log sources for this behaviour. Clearing logs maps to T1685.006 Clear Linux or Mac System Logs.

For configuring auditd before an incident, see the Linux Logging and auditd for Defenders Guide.

Persistence locations to check

The most reliable way to separate malicious from normal is package verification: files under system paths that are not owned by any package, or whose hash differs from the package database, deserve review first. Compare with configuration management or a known-good image where you have one.

Anti-forensics and how to see through it

• Cleared history: an empty or truncated .bash_history, HISTFILE unset (Bash then saves nothing) or history symlinked to /dev/null are themselves indicators (T1070.003). Recover command evidence from auditd EXECVE, sudo logs and memory.
• Edited text logs: gaps in syslog timestamps, a missing rotation file or a log shorter than its rotated predecessor. Off-host log copies are the only reliable answer.
• Timestomping: modify and access times set to match neighbouring system files (T1070.006). Inode change time and ext4 birth time are harder to alter and often betray it.
• Deleted binaries: malware that deletes itself after launch still runs from memory; recover it from /proc/<pid>/exe on the live host.
• Rootkits: user-space preload rootkits hide files and processes from normal tools. Compare live tool output with memory analysis and with an offline view of the disk.
• Memory-only payloads in tmpfs: /dev/shm and /run vanish on reboot; collect them in the live phase.

Tooling

Containers add their own layers: overlay file systems, short-lived processes and logs held by the runtime. See the Container and Kubernetes Forensics Guide.

Where identity fits

Linux servers hold the machine credentials that turn one compromised host into many. Inventory and treat as exposed: SSH private keys in ~/.ssh/ and their entries in other hosts' authorized_keys, cloud CLI credential files and instance metadata access, tokens and passwords in .env files and systemd EnvironmentFile entries, Kubernetes service account tokens, database connection strings and CI runner tokens. Check auth logs on other hosts for the same keys being used (T1021.004), and query the cloud provider's audit log for the instance role being used from outside the instance. See the SSH Key Management Guide.

A practical investigation workflow

1. Scope and contain without destroying evidence: isolate the host at the network or security group level rather than powering it off.
2. Acquire volatile data: memory, then live state and suspicious process binaries from /proc.
3. Run triage collection: UAC or an equivalent to external storage, and snapshot the disk.
4. Establish the logging baseline: journald persistence, auditd presence and rules, log rotation and off-host forwarding.
5. Find the entry point: authentication logs, web and application logs, and exposed service versions against known exploited vulnerabilities.
6. Build a timeline: merge logs, journal, audit and file system times in UTC.
7. Enumerate persistence: walk the persistence table and verify packages.
8. Trace credentials and lateral movement: keys and secrets on the host, and their use elsewhere.
9. Eradicate and recover: rebuild from a known-good image rather than cleaning in place, rotate every exposed credential and monitor for reuse.

Preparing Linux hosts before an incident

• Persistent journals and forwarding: create /var/log/journal or set persistent storage, and forward logs off-host.
• auditd with useful rules: start from the sample rule sets shipped with the audit package (Red Hat ships them in /usr/share/audit/sample-rules/, and notes they are not exhaustive), keep audit logs on a separate mount and set log handling to keep logs.
• Shell history with timestamps: set HISTTIMEFORMAT and histappend globally; treat it as a convenience, not a control.
• Forward Secure Sealing: set up journal sealing keys before you need them.
• Prebuilt collection kits: LiME modules for your kernels, UAC and symbol tables stored ready to use.
• Time synchronisation: consistent, monitored time sources so timelines from different hosts align.

Decision guidance

• Live response or snapshot only: for suspected memory-resident malware, rootkits or active intruders, live response first. For a single cloud VM with persistent logs off-host and no sign of in-memory implants, a disk snapshot may be enough.
• Clean or rebuild: rebuild. In-place cleaning of a Linux host with root-level compromise cannot prove a clean state, as appliance integrity tools failing in the Ivanti Connect Secure exploitation illustrated.
• Small teams: standardise on one triage collector and a written runbook, and make sure logs leave the host.
• Large estates: use fleet tooling to collect the same artefacts across all hosts sharing an image or credential, because Linux compromises often spread through shared SSH keys.
• Appliances: many network and security appliances run Linux but restrict shell access; collection must use the vendor's supported methods. See the Edge Device Compromise Investigation Playbook.

Measuring readiness

• Persistent journal coverage: percentage of Linux hosts with persistent journals and off-host forwarding.
• auditd coverage: percentage of servers running auditd with the standard rule set loaded.
• Collection kit currency: percentage of production kernels with a prebuilt memory module and symbol table.
• Time to triage: minutes from decision to a complete triage package for one host.
• Credential inventory: percentage of hosts whose SSH keys and secrets are inventoried, so rotation scope is known.

Common failure patterns

• Rebooting first: memory, tmpfs content, volatile journals and deleted running binaries are lost.
• Trusting tools on the suspect host: a rootkit or modified binaries can lie; use trusted static tools and cross-check with memory and offline disk analysis.
• Reading absence as innocence: no auditd records may mean no rules, and no journal history may mean volatile storage.
• Ignoring performance anomalies: the XZ Utils backdoor was found because SSH logins used unexpected CPU and were slower than usual, not through logs or signatures.
• Forgetting machine credentials: rebuilding the host while leaving its SSH keys and cloud credentials valid lets the intruder walk back in.
• Mixed time zones: syslog in local time, journals and cloud logs in UTC; timelines go wrong unless everything is normalised.

Framework mapping

Practitioner checklist

• Isolate suspected Linux hosts at the network layer instead of shutting them down.
• Acquire memory with a prebuilt module before running broad collection.
• Capture processes, sockets, modules and binaries of suspicious processes from /proc.
• Run a triage collector to external storage, hash the output and snapshot the disk.
• Record whether journald was persistent, which auditd rules were loaded and where logs were forwarded.
• Walk every persistence location and verify system files against the package database.
• Build a single UTC timeline from logs, journal, audit and file system data.
• Inventory and rotate SSH keys, cloud credentials and secrets found on the host.
• Rebuild from a known-good image and monitor for reuse of old credentials.

Standards and references

• NIST: SP 800-86 Guide to Integrating Forensic Techniques into Incident Response
• systemd: journalctl manual
• Red Hat: RHEL 9 Security Hardening, Auditing the system
• Linux man-pages: utmp(5)
• Linux man-pages: ld.so(8)
• GNU: Bash Reference Manual, Bash History Facilities
• UAC project: Unix-like Artifacts Collector
• LiME project: Linux Memory Extractor

Related NHI Mgmt Group resources: Linux Detection and Response Guide · Digital Forensics (DFIR) Guide · Linux Server Hardening Guide · Cloud Forensics Guide · Memory Forensics Guide