Linux Logging and auditd for Defenders Guide: journald, syslog and Audit Rules

Most Linux servers log enough to troubleshoot a failed service but not enough to reconstruct an intrusion. Default journald and syslog configurations record daemon messages and authentication events, yet they say nothing about who read /etc/shadow, which process added a key to authorized_keys or what a compromised service account executed at 03:00. The Linux Audit system (auditd) closes that gap, but only if its rules are written deliberately, its buffers survive load and its records leave the host before an attacker can touch them. This guide covers journald, rsyslog and the /var/log layout, the auditd architecture, practical rule sets for credential files, sudo, SSH keys and cron, performance and volume control, shipping to a SIEM, tamper protection and where eBPF-based sensors fit. It is for detection engineers, Linux administrators and responders.

Key takeaways

• Three layers, three jobs: journald and rsyslog capture what services say about themselves; auditd captures what the kernel saw processes do. Defenders need both, correlated by host, time and login UID.
• Use the login UID: auditd's auid survives sudo and su, so filters such as -F auid>=1000 -F auid!=unset attribute actions to the human or service that logged in, not to root.
• Write rules from questions, not from catalogues: start from the upstream sample rule files, then keep only rules that answer a detection question, tag each with a key and measure its volume.
• Logs on the host are evidence the attacker can edit: forward in near real time, lock the audit configuration with -e 2 once stable and alert on gaps as loudly as on events.
• Prefer syscall rules with path filters over legacy watches: the auditctl and audit.rules man pages mark -w as deprecated, and always set arch so the kernel does not evaluate every syscall.

The Linux logging stack: journald, rsyslog and /var/log

On a modern systemd distribution, messages from services, the kernel and PAM go first to systemd-journald. Depending on the distribution, rsyslog may then read from the journal and write classic text files under /var/log. The audit subsystem is separate: the kernel sends audit records to auditd, which writes /var/log/audit/audit.log. See the Linux Fundamentals for Defenders Guide for users, groups, PAM and systemd basics that this guide assumes.

journald settings that matter to defenders

• Storage=: volatile keeps logs only in memory under /run/log/journal and loses them at reboot; persistent writes to /var/log/journal; auto behaves as persistent only if that directory exists. The journald.conf man page gives persistent as the default in the default namespace, but check what your image actually ships, because some cloud images still create no persistent directory.
• SystemMaxUse= and MaxRetentionSec=: journal disk use defaults to 10% of the filesystem, capped at 4G, and time-based deletion is off (MaxRetentionSec=0). Size, not age, will decide what you lose, so set retention centrally in the SIEM rather than trusting the host.
• RateLimitIntervalSec= and RateLimitBurst=: the default allows 10,000 messages per 30 seconds per service. A noisy or deliberately chatty service can push security messages out of that window; rate-limit messages themselves are worth alerting on.
• Seal=: Forward Secure Sealing protects persistent journal files from undetected alteration once a key pair exists (journalctl --setup-keys). The verification key must be stored off the host, and journalctl --verify checks integrity.
• ForwardToSyslog=: off by default; rsyslog normally reads the journal through its own input module instead. Confirm which path your distribution uses before you assume text files contain everything.
• Read access: the system journal is readable by root and members of systemd-journal, adm or wheel. Review membership of those groups: log readers see whatever applications log by mistake, secrets included.

Where the files are by distribution family

Recent Debian releases can install without rsyslog, so auth.log may not exist and journalctl is the source. Collect from the journal and audit records, not from file names.

Application-level settings worth changing

• sshd: LogLevel defaults to INFO and SyslogFacility to AUTH; many hardening baselines raise LogLevel to VERBOSE to record more detail about key-based logins. Test the volume first.
• sudo: sudo logs to syslog by default. log_input and log_output (both off by default) record session I/O, and from sudo 1.9 log_servers sends event and I/O logs to sudo_logsrvd with TLS certificate verification (log_server_verify) on by default. log_subcmds (1.9.8 and later) records child processes a sudo command starts.

How auditd works

The Linux Audit system has a kernel component and user-space tools maintained by the linux-audit project. The kernel evaluates rules on filter lists and emits matching records; auditd writes them to disk; dispatcher plugins (configured in /etc/audit/plugins.d/, built into auditd on RHEL 9) send them to other consumers in real time.

• Record anatomy: one event is several records sharing a serial number in msg=audit(timestamp:serial), for example SYSCALL, EXECVE, CWD, PATH and PROCTITLE. Parsers that treat each line as an event lose the command line and file path.
• Enriched format: log_format = ENRICHED resolves UIDs, GIDs, syscall names and architecture when the record is written. The ausearch man page notes that interpreting raw logs on another machine uses that machine's accounts and can mislead, so ENRICHED is the sensible choice when logs are analysed centrally.
• Controlling the daemon: Red Hat's documentation says to use the service command to stop or restart auditd so that the auid of the operator is recorded; systemctl is limited to enable and status.
• Login UID: auid is set at login by PAM and inherited by every child process. An auid of unset (4294967295) means a daemon started at boot. Filtering on auid>=1000 -F auid!=unset (check UID_MIN in /etc/login.defs) separates interactive and service-account activity from system noise.

Writing audit rules

The audit.rules man page describes three rule types: control rules (delete all, backlog, failure mode, rate), file system rules and system call rules. The syscall form is -a always,exit -F arch=b64 -S <syscalls> -F <field>=<value> -k <key>. Key strings are limited to 31 bytes; use a short, consistent naming scheme because detections and dashboards will be built on them.

• Watches versus syscall path rules: -w /etc/passwd -p wa -k identity still works but is deprecated. The equivalent -a always,exit -F arch=b64 -F path=/etc/passwd -F perm=wa -F key=identity lets you choose a single inode (path=) or a tree (dir=). Directory rules are recursive but stop at mount points.
• Always set arch: without -F arch=, every syscall is evaluated. On 64-bit hosts write a b64 rule and, unless 32-bit execution is disabled, a matching b32 rule, because syscall numbers differ between ABIs. Upstream's 21-no32bit.rules instead logs any 32-bit syscall as suspicious.
• Combine syscalls: several syscalls in one rule are checked with a single mask comparison, so -S chmod,fchmod,fchmodat is cheaper than three rules.
• Per-architecture differences: the upstream README notes that aarch64 has no open syscall; rules copied from x86_64 must be pruned or augenrules --load fails.
• File layout: upstream groups rule files by number: 10 kernel and auditctl configuration, 20 overrides, 30 main rules, 40 optional, 50 server-specific, 70 local, 90 finalise. augenrules concatenates them in natural sort order and moves the last -D first, -b second, -f third and -e last.

Red Hat ships the upstream samples in /usr/share/audit/sample-rules/ (30-stig, 30-pci-dss-v31, 30-nispom, 30-ospp-v42 and others) and warns that they are neither exhaustive nor guarantees of compliance. Treat them as a parts bin.

Example rule sets for high-value targets

The rules below follow the upstream 30-stig.rules conventions. Only the b64 line is shown for brevity; add the b32 twin where 32-bit syscalls are possible. Each maps to an ATT&CK technique.

• Per-user authorized_keys: a dir=/home rule with perm=wa is noisy. A practical alternative is to move keys to a root-owned location with AuthorizedKeysFile /etc/ssh/authorized_keys/%u and watch that one directory. See the Linux Server Hardening Guide.
• Service accounts: the auid filter misses daemons started at boot, which is exactly where a compromised web or CI service runs. Add targeted execve rules by -F uid= for high-risk service accounts (web server, CI runner, backup agent) so a shell spawned by www-data is visible. See the Service Account Security Guide.
• Testing: in a test VM, run auditctl -l to confirm they parsed, trigger each one (for example touch /etc/cron.d/test) and confirm ausearch -k cron -i -ts recent returns it.

Performance, volume and backlog

Every syscall passes through the matching engine, so the audit.rules man page advises syscall rules only where needed. Execve auditing of all users is the single largest volume driver on busy build servers and container hosts.

• Watch for loss: auditctl -s shows lost and backlog counters; a rising lost counter is a detection gap.
• Exclude noise precisely: exclude monitoring agents by exe= and auid rather than deleting whole rules.

Shipping logs to a SIEM

1. Choose the transport: an audit dispatcher plugin (audisp-remote to a central auditd), rsyslog forwarding over TCP with TLS, or a log-shipping agent that tails audit.log and reads the journal. Avoid UDP syslog for security logs: it drops silently.
2. Keep events whole: the shipper or SIEM parser must reassemble multi-record audit events by serial number and decode hex-encoded PROCTITLE and EXECVE arguments.
3. Normalise identity fields: map auid, uid, euid, exe, comm, key and hostname to your SIEM schema so one query covers every host. ENRICHED format helps.
4. Queue on the host: configure disk-assisted queues so a SIEM outage does not drop events, and alert on queue growth.
5. Detect silence: build a heartbeat rule per host ("no audit events in 15 minutes") alongside content detections.
6. Write detections against keys: the key field is the stable contract between rule authors and detection engineers. See the SIEM Guide, the Sigma Rules Guide and the Log Management and Retention Guide.

Tamper protection

• Lock the rules: -e 2 in 99-finalize.rules (commented out upstream) makes the configuration immutable until reboot, so an attacker with root must reboot the host to change rules, which is itself visible. Enable it only after rules are stable, because every change then needs a reboot.
• Get logs off the host fast: a root attacker can stop daemons or edit files. Near-real-time forwarding to a store the host cannot write back to is the control that holds (NIST SP 800-53 AU-9(2) store on separate systems).
• Audit the auditors: rules on /etc/audit/, auditctl execution and log deletion (table above), plus SIEM alerts on CONFIG_CHANGE and DAEMON_END record types and on the heartbeat gap.
• Seal the journal: enable Forward Secure Sealing and keep the verification key off the host; run journalctl --verify during investigations.
• Restrict readers and writers: keep log_group = root for audit logs unless a dedicated, monitored group needs read access, and review adm and systemd-journal membership.
• Do not rely on shell history: attackers clear it (T1070.003 Clear Command History). execve auditing records commands regardless.

eBPF-based sensors and when to choose them

eBPF runs verified, sandboxed programs in the kernel attached to syscalls, tracepoints, kprobes and uprobes. Many modern Linux EDR agents and open-source runtime security tools use it to collect process, file and network events with lower overhead than broad audit rules and with container context (namespaces, cgroups, pod names) that auditd does not natively provide.

Small estates and regulated servers usually start with auditd plus forwarding. Large container platforms typically add an eBPF sensor for runtime detection while keeping a minimal auditd set (identity, sudo, audit config, module load) for compliance. See the Linux Detection and Response Guide and the EDR Guide.

Rolling out Linux audit logging

1. Inventory and baseline: list distributions, kernels and architectures; record whether journald is persistent, whether rsyslog is present and whether auditd is installed and running.
2. Fix the foundations: persistent journald, TLS forwarding, ENRICHED audit format, rotation and disk actions agreed with service owners.
3. Deploy a minimal rule set to a pilot ring: 10-base-config, identity, sudo, SSH, cron, module load and audit-config rules on a handful of hosts per role. Measure events per host per hour and the lost counter for two weeks.
4. Add execve selectively: interactive users and high-risk service accounts first, with excludes for known agents. Re-measure.
5. Build and test detections: one detection per key, tested by triggering the rule in a lab and confirming the alert in the SIEM.
6. Roll out by ring, then lock: expand via configuration management, then enable -e 2 once rules have been stable for a release cycle.
7. Rollback: keep the previous rules.d set in configuration management. Without -e 2, augenrules --load restores it immediately; with it, schedule a reboot.

Measuring it

• Percentage of Linux hosts sending audit and journal data to the SIEM in the last 24 hours (target near 100%).
• Audit lost records per host per day (target zero).
• Median forwarding delay from event to SIEM (target under five minutes).
• Percentage of audit keys with at least one tested detection.
• Hosts with immutable audit configuration (enabled 2 in auditctl -s).

Framework mapping

Common failure patterns

• auditd installed, no rules loaded: auditctl -l returns "No rules" and only hard-wired login and PAM events appear. Check every image.
• Volatile journal on cloud images: logs vanish at every reboot or autoscaling event, often the exact moment an investigator needs them.
• Everything audited, nothing reviewed: a full execve rule on every host overwhelms the SIEM budget, the team disables it, and coverage falls to zero. Volume budgets per key avoid this.
• Daemons invisible because of auid filters: a web shell running as the web server user has auid=unset and is excluded by interactive-user rules. Service accounts need their own rules.
• Behavioural anomalies ignored: the XZ Utils backdoor was found because SSH logins used unexpected CPU, not because a signature fired. Performance and authentication-path anomalies on sshd deserve investigation, and package-change logs help scope exposure.
• Logs left only on the compromised host: without forwarding, an attacker with root clears /var/log and stops auditd, and the timeline is gone.

Practitioner checklist

• Confirm journald Storage=persistent and a populated /var/log/journal on every Linux image.
• Install and enable auditd with log_format = ENRICHED, rotation and agreed disk-full actions.
• Deploy rules via /etc/audit/rules.d/ and augenrules, starting from 10-base-config and selected upstream samples, pruned per architecture.
• Cover account databases, sudoers, SSH configuration and keys, cron and systemd units, kernel modules and the audit configuration itself, each with a key.
• Add execve auditing for interactive users and named high-risk service accounts, with measured excludes.
• Forward audit and journal data over TLS with on-host queuing; alert on host silence and on rising lost counts.
• Enable -e 2 once rules are stable and enable journal sealing with the verification key stored off-host.
• Write and test one SIEM detection per audit key, mapped to ATT&CK.
• Review membership of adm, systemd-journal and wheel quarterly.

Standards and references

• Linux man-pages: auditctl(8)
• Linux man-pages: audit.rules(7)
• Linux man-pages: auditd.conf(5)
• Linux Audit project: audit-userspace sample rules
• Linux man-pages: journald.conf(5)
• Red Hat: Auditing the system (RHEL 9 Security Hardening)
• NIST: SP 800-53 Rev. 5
• MITRE ATT&CK: Disable or Modify Linux Audit System Log

Related NHI Mgmt Group resources: Linux Detection and Response Guide · Linux Server Hardening Guide · SIEM Guide · Linux Fundamentals for Defenders Guide · Detection Engineering Guide