Linux Detection and Response Guide: auditd, eBPF and Linux EDR

Most Linux servers record logins and little else, so an intruder with a shell can run tools, add persistence and hide a kernel module without leaving anything a defender would see. Detection on Linux depends on choosing the right telemetry (the Linux audit framework, eBPF sensors, journald and authentication logs), getting it off the host before root can erase it, and knowing which techniques each source cannot see. This guide goes deeper than hardening: auditd rule design, eBPF sensors and the verifier, the io_uring blind spot, rootkit and persistence detection, hunting and incident response. It is written for detection engineers, Linux platform owners and incident responders. For the baseline configuration that comes first, see the Linux Server Hardening Guide.

Key takeaways

• Linux telemetry is opt-in. Process execution, module loads, preload files and unit changes are recorded only if you add auditd rules or an eBPF sensor.
• auditd and eBPF are complementary. The Linux audit framework gives a lockable trail tied to the login user (auid); eBPF sensors add richer process and network context.
• Know the blind spots. io_uring, LD_PRELOAD hooks and kernel module rootkits can hide activity from tools on the same host, so use cross-checks and off-host records.
• Protect the sensor and the log path. Lock the audit configuration with -e 2, forward records in near real time and alert when a host goes quiet, because stopping the collector is the attacker's first move (ATT&CK T1685 Disable or Modify Tools).
• Plan response before the incident. Decide how to capture memory on a locked-down kernel, and rebuild rather than clean a host with kernel-level compromise.

Linux telemetry sources and what each can see

Every Linux EDR agent, open-source sensor or home-grown pipeline draws on the same few sources. Their limits drive every later design decision.

The international event logging guidance led by ASD's ACSC with CISA, the FBI, NSA and the UK NCSC aims at "a baseline for event logging" that catches living-off-the-land techniques. On Linux that baseline is auditd plus authentication logs on every server, with eBPF telemetry where you need process and network context.

The Linux audit framework and auditd rules

Red Hat is explicit that audit "does not provide additional security to your system": it records, it does not block. The kernel evaluates rules and auditd writes records and feeds plugins (in RHEL 9 the audisp dispatcher is built into auditd, configured in /etc/audit/plugins.d/).

How rules are evaluated

• Rule lists: auditctl(8) defines the task, exit, user, exclude, filesystem and io_uring lists. Detection rules belong on exit (-a always,exit) with arch=b64 before -S, plus a b32 twin where 32-bit binaries run.
• Cost: the manual warns that "Syscall rules get evaluated for each syscall for every program" and that "Too many syscall rules will hurt performance". Group system calls, filter with -F fields and prefer path rules.
• The default never,task rule: many distributions ship -a never,task, which the manual says "causes every new process to skip all audit rule processing". Check for it before concluding your rules are failing.
• Watches: the -w syntax is "deprecated due to poor system performance"; write path= or dir= rules with perm=wa instead.
• Attribution and keys: auid is "The original ID the user logged in with", so root commands run through sudo still point to the named user; filter auid>=1000 and auid!=unset for human activity. Give every rule a key (-k) so searches find events by intent.

Detection-focused auditd rules

The compliance sample files in /usr/share/audit/sample-rules (30-stig, 30-pci-dss-v31, 30-ospp-v42, 30-nispom) are a starting point, but Red Hat warns they "are not exhaustive nor up to date". Add rules that map to attacker behaviour:

auditd.conf for a detection pipeline

• Lock the rules: end with -e 2 so the configuration "can't be changed" until reboot, itself a detectable event.
• Failure mode: -f sets 0 silent, 1 printk (default) or 2 panic; the manual notes "Secure environments will probably want to set this to 2". Most servers keep 1 and alert on lost events.
• Backlog and queue: raise -b above the kernel default of 64 and q_depth (default 2000) on busy hosts, with overflow_action = syslog so drops are visible.
• Disk handling: Red Hat recommends a separate mount point for /var/log/audit, max_log_file_action = keep_logs, space_left_action = email or exec, and disk_full_action = halt or single on systems where audit is mandatory.
• Format and transport: log_format = ENRICHED resolves UIDs, GIDs, syscalls, architecture and socket addresses so records are readable off-host. auditd's own TCP transport is clear text by default; use its Kerberos or TLS options, or forward through an encrypted log shipper.

eBPF sensors, the verifier and eBPF as attack surface

eBPF lets a privileged process load small programs into the kernel and attach them to events: the bpf(2) manual lists program types including kprobe, tracepoint and socket filter, with maps shared between kernel and user space. Many Linux EDR agents and eBPF runtime security tools use it for eBPF monitoring of process, file and network activity, filtering in the kernel before events reach user space. eBPF observability has largely replaced kernel module sensors.

• The eBPF verifier: before loading, the kernel's verifier works in two steps. The kernel documentation describes a DAG check "to disallow loops and other CFG validation", then a pass that "simulates execution of every insn and observes the state change of registers and stack". It rejects uninitialised reads, out-of-bounds access and unchecked map pointers, so a buggy eBPF sensor is far less likely to crash a server than a buggy kernel module.
• Privilege: since Linux 5.8, CAP_BPF and CAP_PERFMON separate eBPF loading from CAP_SYS_ADMIN. Set kernel.unprivileged_bpf_disabled to 1 (fixed until reboot) or 2 (adjustable).
• eBPF as an attack tool: the hooks that make eBPF detection possible let an attacker with root attach programs that hide processes or intercept credentials. Treat any eBPF program not owned by an approved agent as suspicious: collect the kernel's BPF audit records for program load and unload, and inventory loaded programs and their owning processes.
• Kernel lockdown: the lockdown LSM (enabled when lsm= includes lockdown) restricts BPF and kprobes along with /dev/mem, /dev/kcore and unsigned modules. It can block your own sensor, so test it with the agent first.

Choosing a sensor approach

Kubernetes runtime sensors are covered in the Kubernetes Security Guide. When buying a Linux EDR, ask which kernels the sensor supports, how it reports its own health and whether it detects eBPF programs loaded by other processes.

journald and the Linux logging pipeline

• Persistence: journald.conf Storage= defaults to persistent in the default namespace on current systemd. Older releases defaulted to auto, which stays in memory unless /var/log/journal exists; confirm logs survive a reboot.
• Rate limiting: the default of 10,000 messages in 30 seconds can drop messages during a noisy attack. Raise RateLimitBurst for critical services rather than disabling the limit.
• Forward Secure Sealing: Seal= with a key from journalctl --setup-keys "may be used to protect journal files from unnoticed alteration". It reveals tampering; it does not stop deletion.
• Retention and forwarding: SystemMaxUse defaults to 10 per cent of the file system, capped at 4G. Keep the look-back centrally: ship the journal, auth, sudo and audit records off-host over TLS within seconds, on hosts with monitored time synchronisation. See the Log Management and Retention Guide.

The io_uring blind spot

io_uring gets its name from "ring buffers which are shared between user space and kernel space": a program queues batches of I/O requests and reads results back, and with SQPOLL a kernel thread polls the queue so no system call is needed per operation. File and network operations submitted this way bypass the individual system calls that syscall-based audit rules and many sensors watch.

• Disable it where unused: kernel.io_uring_disabled = 2 blocks creation for all processes; 1 blocks unprivileged processes except members of io_uring_group. Set it at boot, since existing instances keep working.
• Audit it where used: auditctl has an io_uring rule list (operations named with -S) for hosts that need it.
• Test your sensor with a benign io_uring program to see what it records.

Detecting Linux rootkits and persistence

ATT&CK T1014 Rootkit sits under the Stealth tactic in v20, and its page states the technique "cannot be easily mitigated with preventive controls". Linux rootkit detection therefore relies on catching the install, on cross-view checks, and on records that left the host before the rootkit loaded. The same logic applies to Linux persistence generally.

User-space rootkits and LD_PRELOAD

The ld.so(8) manual describes LD_PRELOAD as shared objects "loaded before all others" that can "selectively override functions", and /etc/ld.so.preload as having "a system-wide effect", normally used "only as an emergency remedy". ATT&CK T1574.006 Dynamic Linker Hijacking notes that hooking functions such as execve and readdir lets malware scrub its own artefacts from tools like ls. Secure-execution mode limits preloading into setuid programs, but not into ordinary processes.

• /etc/ld.so.preload should not exist on most servers. Alert on its creation and on any non-empty content.
• Watch environment injection: LD_PRELOAD exported in shell profiles, /etc/environment or systemd unit Environment= lines, and execve records whose environment carries it.
• Check libraries loaded by long-running daemons such as sshd against the package database. The XZ Utils backdoor showed hostile code reaching sshd through a trusted library dependency rather than a preload file.

Loadable kernel module and eBPF rootkits

T1547.006 describes a loadable kernel module rootkit as running at "the highest operating system privilege (Ring 0)", hiding files, processes and network activity. Once loaded, anything the host reports about itself is suspect.

• Prevent: enforce module signature checking with Secure Boot and lockdown, and on fixed-function servers set kernel.modules_disabled = 1 after boot (it cannot be reverted).
• Record the load: the auditd modules rule fires before the module can hide itself. ATT&CK's analytic also suggests correlating kernel header installs, compiler use and .ko downloads.
• Read the taint flags: /proc/sys/kernel/tainted is a bitfield; bit 12 (O) marks an out-of-tree module, bit 13 (E) an unsigned module and bit 1 (F) a forced load. The kernel documentation notes the taint persists after the module is unloaded, so an unexplained non-zero value warrants investigation.
• Cross-view checks: compare what the host reports (processes, modules, listening ports) with off-host views such as flow records, hypervisor introspection or an offline disk scan. Discrepancies are the classic rootkit signal.

Other persistence to cover

• Cron (T1053.003): ATT&CK recommends reviewing cron changes and restricting use with /etc/cron.allow and /etc/cron.deny. Alert on direct edits to spool files and jobs that run from /tmp or /dev/shm.
• systemd services (T1543.002): new or changed units and drop-ins, especially ExecStart paths in writable directories, plus user-level units and timers.

Linux threat hunting

Hunting fills the gaps between alerts. Run these as scheduled queries over the central store:

1. Processes executed from /tmp, /var/tmp, /dev/shm or deleted binaries, grouped by parent and auid.
2. Interactive shells spawned by web servers, databases or other service accounts.
3. New outbound connections from servers that normally only accept inbound traffic.
4. Module loads, BPF program loads and taint changes outside patch windows.
5. Cron, systemd and preload changes not linked to a configuration management run.
6. Hosts whose audit or journal volume dropped sharply: a silent host is a finding.

Build each hunt that finds something into a detection. See the Threat Hunting Guide.

Linux incident response and forensics

NIST SP 800-86 presents forensics "from an IT view, not a law enforcement view", which fits most Linux forensics work: preserve enough to understand scope, then restore service safely.

1. Contain without powering off: isolate the host at the network or security group level; memory, connections and processes vanish at shutdown.
2. Collect volatile data first: memory, processes with command lines and open files, connections, loaded modules and BPF programs, mounts and the taint value, using trusted static tools from read-only media because a user-space rootkit can hook the host's binaries.
3. Linux memory forensics: under lockdown /dev/mem and /dev/kcore are restricted and only signed modules load, so acquisition needs a signed acquisition module, a hypervisor or cloud snapshot, or a crash dump mechanism prepared in advance. Test it per kernel version; analysis also needs symbols matching that exact kernel.
4. Preserve disks and logs: snapshot volumes, export central audit and journal copies and verify journal seals.
5. Scope by identity: trace auid, sudo records and SSH key fingerprints to every account, key and reachable host.
6. Eradicate by rebuilding from a trusted image when there is evidence of a kernel-level rootkit, and rotate every secret the host held.

See the Digital Forensics (DFIR) Guide for evidence handling.

Where identity fits

Most Linux intrusions run on valid accounts: service accounts with shells, shared SSH keys and tokens on disk. auid is only useful if each person has a named account. Alert on interactive sessions by service accounts and treat SSH keys found during response as credentials to revoke. See the SSH Key Management Guide.

Rolling out Linux detection and response

1. Inventory: hosts by distribution, kernel version, role and owner; kernel versions decide which eBPF sensors will run.
2. Baseline logging everywhere: persistent journald, auth and sudo logs and a core auditd rule set, forwarded centrally with heartbeat alerts.
3. Pilot sensors on a non-production ring: measure CPU and event volume and run benign test cases per rule (module load, preload file, cron job, unit file).
4. Tune in detect-only mode for two to four weeks, suppressing known configuration management and patching accounts.
5. Roll out in rings, internet-facing and bastion hosts first. Rolling back a rule set locked with -e 2 needs a reboot, so schedule changes.
6. Enforce protections (unprivileged_bpf_disabled, io_uring_disabled, module signing, lockdown) once sensors are proven compatible.

Small versus large estates: a small team can run auditd with a vetted rule set, a managed log service and the hunts above. A large fleet needs an eBPF sensor, per-role tuning, kernel compatibility testing in the patch pipeline and a detection backlog. See the Detection Engineering Guide.

Detection and measurement

• Telemetry coverage: Linux hosts sending audit and journal records in the last hour (target 100 per cent of production).
• Sensor health: hosts with the agent stopped, rules unlocked (enabled not equal to 2) or lost-event counters rising.
• Kernel integrity: hosts with unexplained taint, or with io_uring or unprivileged eBPF enabled without an exception.
• Detection test pass rate: persistence test cases (module, preload, cron, systemd, eBPF) that alerted in the last quarterly test.
• Time to triage: median minutes from a high-severity Linux alert to an analyst decision.

Common failure patterns

• Logs only on the host: root deletes or edits them; a journal seal shows tampering but restores nothing.
• Compliance rule files loaded wholesale that slow servers until someone disables auditd.
• Assuming the agent sees everything: no test of io_uring, eBPF loads or preload hooks against the deployed sensor.
• Silent failure: a stopped auditd looks like a quiet week when nobody alerts on missing events.
• Cleaning instead of rebuilding a host with a kernel rootkit, which leaves persistence in place.

Framework mapping

Practitioner checklist

• Remove any default never,task rule, load a keyed detection rule set ending in -e 2 and make dropped events visible.
• Make journald persistent and sealed, forward all Linux logs off-host over TLS, and alert when a host goes quiet.
• Deploy an eBPF sensor or Linux EDR agent where process and network context is needed, after kernel compatibility testing.
• Set kernel.unprivileged_bpf_disabled, and io_uring_disabled = 2 on hosts that do not use io_uring.
• Enforce module signing with Secure Boot, and set modules_disabled on fixed-function servers.
• Alert on /etc/ld.so.preload, module and BPF program loads, unexpected taint, and new cron jobs or systemd units.
• Prepare and test memory acquisition for each kernel version in use before you need it.
• Rebuild hosts with confirmed kernel-level compromise and rotate every credential they held.

Standards and references

• Linux Audit (man7.org): auditctl(8)
• Linux kernel documentation: eBPF verifier
• Linux kernel documentation: Documentation for /proc/sys/kernel/
• Linux man-pages (man7.org): ld.so(8)
• systemd: journald.conf
• Red Hat: Auditing the system (RHEL 9 Security hardening)
• MITRE ATT&CK: T1014 Rootkit
• NIST: SP 800-86, Guide to Integrating Forensic Techniques into Incident Response

Related NHI Mgmt Group resources: Linux Server Hardening Guide · EDR Evasion and Defence Guide · Incident Response Guide · SIEM Guide