Most Linux intrusions are logged as ordinary logins and sudo commands, and the evidence sits unread in auth.log, the journal and the audit log. This guide gives concrete detections for three identity problems on Linux servers: SSH password guessing and spraying, logins with keys nobody can account for, and sudo misuse. It explains which line or audit record each detection reads, which fields to extract, how the rules break on newer OpenSSH releases, and how to tune them so the alerts are worth answering. It assumes logs are already collected and forwarded; for journald, rsyslog and auditd set-up see the Linux Logging and auditd for Defenders Guide. It is written for detection engineers, SOC analysts and Linux platform owners.
Key takeaways
- Parse three sources, not one. sshd and sudo write text lines to the AUTH facility, while PAM, sshd and sudo also write USER_AUTH, USER_LOGIN and USER_CMD records to the audit log; each catches what the other misses.
- Aggregate across the fleet. Per-host thresholds miss spraying that tries one password per server; count failures by source and by username across every host.
- The key fingerprint is the identity. An "Accepted publickey" line carries the key type and SHA256 fingerprint, so compare it with your key inventory instead of trusting the username.
- Success after failure beats failure volume. Internet-facing hosts see constant guessing; the alert that matters is a successful login from a source that just failed, or a first-seen source for a privileged or service account.
- Sudo denials are reconnaissance. "user NOT in sudoers", "command not allowed" and incorrect password bursts are rare in normal operations and cheap to alert on.
Where the evidence lives
sshd logs to syslog facility AUTH by default, and sudo also defaults to facility auth, according to the sshd_config and sudoers manual pages. On Debian and Ubuntu that lands in /var/log/auth.log, on the Red Hat family in /var/log/secure, and on journal-only systems in the systemd journal, where the program name is in the SYSLOG_IDENTIFIER field. The audit log (/var/log/audit/audit.log) holds a parallel, structured record set written through libaudit and PAM.
| Source | What it records | Fields to extract | Weakness |
|---|---|---|---|
| sshd text log (AUTH) | Accepted, Failed, Invalid user, maximum authentication attempts exceeded, session start | result, method, user, valid or invalid user, source IP, port, key type and fingerprint, certificate key ID, serial and CA fingerprint | Free text; message formats and process names change between releases |
| sudo text log (auth) | Allowed commands and denials | invoking user, TTY, PWD, run-as USER, COMMAND, error text | Root can delete local copies; a shell launched through sudo hides what follows |
| Audit log, PAM records | USER_AUTH (1100), USER_ACCT (1101), CRED_ACQ (1103), USER_START (1105), USER_END (1106), USER_LOGIN (1112) | acct, exe, hostname, addr, terminal, res, auid, ses | Needs auditd running and shipped; volume on busy bastions |
| Audit log, command records | USER_CMD (1123) from sudo; execve and file watches from your own rules | auid, uid, cmd, cwd, terminal, res | execve rules are expensive; scope them |
| Audit log, lockout | ANOM_LOGIN_FAILURES (2100) when a failed login limit is reached | acct, addr | Only where the PAM lockout module is configured to emit it |
The record numbers above are taken from the linux-audit userspace header (audit-records.h), and the field meanings from the linux-audit field dictionary: acct is the account name, addr the remote address, res the result, auid the login user ID that survives su and sudo, and ses the login session ID. Upstream OpenSSH, when built with Linux audit support, writes a USER_LOGIN record with op=login and res=failed for each failed authentication method and for invalid users, and a successful USER_LOGIN when the session opens. Distribution builds may add further records, so check a sample from your own build before writing rules.
Reading sshd lines correctly
Detections fail most often because the parser misreads the line. The OpenSSH source (auth.c) builds every authentication result from one format: result word, method, an optional "invalid user" marker, the user, source address and port, then optional key details. Points that matter for rules:
- Result words: "Accepted", "Failed", "Partial" (one method of a multi-method requirement such as key plus password passed) and "Postponed". Count Partial as progress, not success.
- Invalid users: a name that does not exist produces both an "Invalid user NAME from IP port N" line and a "Failed ... for invalid user NAME" line. Count one attempt, not two.
- Key details: in current OpenSSH the "Accepted publickey" line ends with the key type and fingerprint (FingerprintHash, SHA256 by default). Certificate logins add the certificate key ID, serial number and the CA key fingerprint, which tells you which CA issued it.
- Log level: successful logins, password failures, failures for invalid users and failures once half of MaxAuthTries (default 6) are reached are logged at INFO. Other failed public key attempts are logged at VERBOSE, so at the default LogLevel INFO a client cycling through many keys is partly invisible. Set LogLevel VERBOSE on servers that matter.
- Per-connection limit: "maximum authentication attempts exceeded for USER from IP" marks a connection that used up MaxAuthTries; it is a strong single-line brute force signal.
- Process name change: OpenSSH 9.8 split the server into a listener, sshd, and a per-session binary, sshd-session. The release notes warn that some messages are now tagged as coming from "sshd-session" rather than "sshd". A rule that filters on program name sshd will silently lose events after an upgrade; match both.
Detecting brute force, spraying and credential stuffing
These map to ATT&CK T1110 Brute Force and its sub-techniques T1110.001 Password Guessing, T1110.003 Password Spraying and T1110.004 Credential Stuffing. Treat raw failure counts on an internet-facing host as background noise for dashboards, not alerts, and alert on the patterns below. See the MITRE ATT&CK T1110 Brute Force Guide for the technique itself.
| Detection | Logic (described) | Starting threshold | ATT&CK |
|---|---|---|---|
| Guessing against one account | Failed password or keyboard-interactive for the same valid user from one source | 10 or more in 10 minutes on internal hosts; dashboard only on exposed hosts | T1110.001 |
| Spraying | One source, many distinct valid usernames, one or two attempts each, across one or many hosts | 5 or more distinct valid users in 30 minutes | T1110.003 |
| Distributed spraying | One username failing from many sources across the fleet | 10 or more distinct sources in 1 hour for a named or privileged account | T1110.003 |
| Credential stuffing | Failures for real user names that match your naming convention, from sources with no history | Any burst of 3 or more distinct real usernames from one new source | T1110.004 |
| Success after failure | Accepted for USER from SOURCE within 30 minutes of 5 or more failures from the same source, or for the same user | Always alert | T1110 then T1078.003 |
| Password login where keys are policy | Accepted password or keyboard-interactive on a host that should be key or certificate only | Always alert | T1078 |
| Lockout reached | ANOM_LOGIN_FAILURES and RESP_ACCT_LOCK audit records written by pam_faillock | Always log; alert for privileged accounts | T1110 |
Tuning brute force rules
- Split valid from invalid users: guesses at "admin", "test" and "oracle" that do not exist are noise; failures against real accounts are signal. The "invalid user" marker makes the split free.
- Suppress known scanners: vulnerability scanners and configuration management tools fail authentication by design. Allowlist them by source and account, with an owner and review date, never by account alone.
- Handle NAT and bastions: many users behind one egress address look like spraying. Key the rule on source plus username, and give bastion and VPN egress ranges their own thresholds. sshd's PerSourcePenalties (on by default since OpenSSH 9.8) has an exemption list for the same reason.
- Account for server-side throttling: PerSourcePenalties refuses offending sources for a period that accumulates up to a default maximum of 10 minutes, and pam_faillock locks accounts after a configured number of failures within fail_interval. Both reduce the number of log lines an attack produces, so lower thresholds on hosts where they are active.
- Enrich, then decide: add source country, ASN and whether the source has ever succeeded before. A first-ever success from a hosting provider ASN is worth more than any failure count.
Detecting unexpected key logins
A planted or stolen key produces a perfectly valid "Accepted publickey" line. The only way to see it is to treat the key fingerprint as an identity and compare it with what should exist. This maps to T1098.004 SSH Authorized Keys (persistence through a planted key), T1552.004 Private Keys (a stolen key reused) and T1021.004 SSH (lateral movement).
- Build the reference list: export fingerprints of every approved key and the CA fingerprints you trust, with owner and expected hosts. See How to Find ssh authorized keys across a server fleet.
- Extract on ingest: parse user, source, key type and fingerprint (or CA fingerprint and key ID for certificates) from every Accepted publickey line.
- Join and score: alert when the fingerprint is not in the reference list, when a known fingerprint is used on a host or account outside its expected set, or when a certificate is signed by a CA you do not trust for that host group.
- Watch the files: pair the login rule with auditd watches or file integrity monitoring on authorized_keys files, sshd_config and sshd_config.d. A new key followed within hours by a first login with that fingerprint is high confidence.
| Signal | Why it matters | Tuning note |
|---|---|---|
| Fingerprint never seen before on this account | New key, or a stolen key used for the first time here | Suppress for 24 hours after an approved change ticket adds the key |
| One fingerprint used by several human accounts | Shared or copied private key | Should be zero; treat as a finding even when benign |
| Service account key used from a new source or with an interactive TTY | Automation keys should run from fixed hosts without a terminal | Session type comes from USER_LOGIN terminal or the session start line |
| Key login to root | Bypasses per-person accountability (auid is root) | Should be blocked by PermitRootLogin; alert on any occurrence |
| Certificate with unexpected key ID or very old serial | Leaked certificate, or a CA issuing outside policy | Compare with CA issuance logs |
Where you use AuthorizedKeysCommand to fetch keys centrally, log that lookup too: a successful login with no matching lookup record means a local authorized_keys file was used. For response steps once a key is suspect, see the SSH Key Compromise Response Playbook.
Detecting sudo abuse
sudo logs allowed and denied commands by default (log_allowed and log_denied are on). An accepted command line takes the form "username : TTY=... ; PWD=... ; USER=runasuser ; COMMAND=...", and denials carry fixed phrases. sudo built with Linux audit support also writes a USER_CMD record, and PAM writes USER_AUTH and USER_ACCT records for the password check. The relevant technique is T1548.003 Sudo and Sudo Caching. For rule design and hardening, see the sudo and sudoers Security Guide.
| Detection | Evidence | Severity guidance |
|---|---|---|
| Account not allowed to use sudo tries it | "user NOT in sudoers" or "user NOT authorized on host" | Alert on first occurrence for service accounts; for people, alert on 2 or more in an hour |
| Probing allowed commands | "command not allowed", often several in a row from an account that rarely uses sudo | Medium; high if it follows a new SSH source for the same account |
| Password guessing at sudo | "3 incorrect password attempts" (passwd_tries default is 3), repeated | High on servers; someone has a session but not the password |
| Escape to a shell | COMMAND is a shell or interpreter, or sudo -i or -s, by an account whose rules do not intend a root shell | High; follow with execve records under the same auid |
| Service account using sudo interactively | sudo line with a real TTY for an account that should only run fixed commands from automation | High |
| sudoers changed outside configuration management | auditd watch or FIM on /etc/sudoers and /etc/sudoers.d | High; compare with change records |
| First sudo use by an account | No sudo history in the previous 90 days | Low on its own; raise with any other signal |
The audit field auid is what makes sudo detections reliable: it is set at login and inherited through su and sudo, so commands run as root still carry the original user. Filter out records where auid is unset (4294967295), which come from daemons rather than logged-in users. Commands run inside a root shell started with sudo do not produce further sudo lines; only execve audit rules scoped to euid=0 and a real auid, or an EDR sensor, see them. The Linux Detection and Response Guide covers those rules.
Chaining signals into sequences
Single events produce the noise; sequences produce the incidents. Correlate on host, source address, user and the audit session ID (ses), which ties a login to everything done in that session.
- Guess, get in, escalate: failures from a source, then Accepted from the same source, then a sudo denial or shell escape in the same ses within an hour.
- Plant and return: an authorized_keys write by a session, then a later Accepted publickey with a new fingerprint from a different source.
- Hop through the estate: an Accepted login whose source is another internal server, for an account that normally connects from a bastion or workstation range, followed by the same pattern on a third host.
- Cover tracks: a privileged session followed by a gap in that host's log stream, a change to /etc/audit or rsyslog configuration, or auditd stopping. Map to T1685 Disable or Modify Tools and alert on the gap itself, since local logs can be removed by the account the attacker now holds.
Where identity fits
These rules only work if a login means one person or one workload. Shared accounts such as "deploy" or "admin" make every success ambiguous, and keys copied between people make the fingerprint join meaningless. Give each person a named account, give automation its own service accounts with keys bound to fixed source addresses, and prefer short-lived SSH certificates so the key ID names the requester. See the SSH Key Management Guide and the Linux Service Accounts and Keytab Management Guide.
Rolling out the detections
- Check the inputs: confirm every server ships AUTH facility logs and the audit log, set LogLevel VERBOSE on sshd, and pull one sample of each line and record type from each OS and OpenSSH version you run.
- Normalise: write one parser that accepts both sshd and sshd-session, maps auth.log, secure and journal sources to the same fields, and extracts key fingerprints.
- Run silent for two weeks: deploy rules as reports only, measure volume per rule and per host, and build the scanner, bastion and NAT allowlists from what you see.
- Promote high-confidence rules first: success after failure, password login on key-only hosts, unknown fingerprint on privileged accounts, sudoers changes and service account sudo with a TTY.
- Add fleet-wide aggregation: distributed spraying and fingerprint reuse need a central search; schedule them as correlation searches.
- Write the triage notes: each alert names the questions to ask (is the source known, is the key in the inventory, what ran in the session) and the containment step.
- Rollback: keep each rule versioned in your detection repository; if a rule floods the queue, return it to report-only rather than deleting it, and fix the threshold or allowlist. The Sigma Rules Guide covers portable rule formats.
Measuring it
- Log coverage: percentage of Linux servers with AUTH logs and audit logs arriving centrally in the last 24 hours; target 100 percent for internet-facing and privileged hosts.
- Fingerprint attribution: percentage of Accepted publickey logins whose fingerprint maps to a named owner; trend upwards towards 100.
- Parser health: count of sshd or sshd-session lines that fail to parse per day; any rise after an OS upgrade needs a fix before it becomes a blind spot.
- Alert precision: share of alerts per rule confirmed as true positives or policy violations; rules under 20 percent need tuning.
- Time to triage: median minutes from a success-after-failure alert to a decision.
- Sudo denial rate: denials per 1,000 sudo invocations; a steady rise usually means rules are out of step with work, a spike means someone is probing.
Framework mapping
| Framework | Reference | How these detections support it |
|---|---|---|
| NIST SP 800-53 Rev. 5 | AC-7 Unsuccessful Logon Attempts | Lockout events and failed login monitoring |
| NIST SP 800-53 Rev. 5 | AU-2 Event Logging, AU-6 Audit Record Review, Analysis, and Reporting | Defined authentication and sudo events, reviewed through correlation |
| NIST SP 800-53 Rev. 5 | SI-4 System Monitoring; AC-6 Least Privilege | Alerting on sudo misuse and privileged sessions |
| NIST CSF 2.0 | PR.PS-04, DE.CM-03, DE.AE-02 | Log records generated and available; personnel activity monitored; adverse events analysed |
| MITRE ATT&CK v20 | T1110 (.001, .003, .004), T1078.003, T1098.004, T1552.004, T1021.004, T1548.003, T1685 | Technique coverage of the rules above |
Common failure patterns
- Per-host thresholds only: a sprayer trying one password against 500 servers never crosses a per-host limit. Aggregate centrally.
- Parser pinned to "sshd": after an upgrade to OpenSSH 9.8 or later some lines come from sshd-session and the rule goes quiet without an error.
- LogLevel left at INFO: repeated failed key attempts below the MaxAuthTries halfway mark are not logged, so key guessing and stale automation look identical to silence.
- Trusting the username: "Accepted publickey for deploy" says nothing about who holds the key. Without the fingerprint join, a stolen key is a normal login.
- Alerting on every failure: exposed hosts generate thousands a day, the queue is muted, and the one success after failure is missed.
- Local logs as the only copy: an attacker with root edits auth.log and the audit log first. Forward in near real time and alert on gaps.
Practitioner checklist
- Ship AUTH facility logs and audit logs from every Linux server to a central store, and alert when a host stops sending.
- Set LogLevel VERBOSE on sshd and confirm that Accepted publickey lines carry fingerprints on every OS version you run.
- Parse both sshd and sshd-session program names and normalise auth.log, secure and journal fields.
- Alert on success after failure, password logins on key-only hosts and first-seen sources for privileged and service accounts.
- Run fleet-wide spraying and distributed spraying searches keyed on source and username.
- Join every key login to an owned fingerprint inventory and alert on unknown or shared fingerprints.
- Watch authorized_keys, sshd_config and sudoers files with auditd or FIM and correlate with later logins.
- Alert on sudo denials, incorrect password bursts, shell escapes and service account sudo with a TTY, using auid to keep the original user.
- Pilot every rule in report-only mode, document allowlists with owners, and track precision per rule.
Standards and references
- OpenBSD: sshd_config(5) manual page
- OpenSSH: Release notes for OpenSSH 9.8
- OpenSSH: openssh-portable auth.c (authentication logging)
- Sudo project: sudoers(5) manual
- Linux Audit: audit-documentation and field dictionary
- Linux-PAM: pam_faillock module
- Red Hat: RHEL 9 Security hardening, Auditing the system
- MITRE ATT&CK: T1110 Brute Force
Related NHI Mgmt Group resources: Linux Logging and auditd for Defenders Guide · OpenSSH Server Hardening Guide · sudo and sudoers Security Guide · SSH Key Management Guide · Root, sudo and superuser accounts on Linux and Unix