When a Linux server behaves oddly, the first hour decides whether you end up with evidence or with a wiped crime scene. This guide is a short, read-only triage checklist for one suspect host: who can log in, which SSH keys are trusted, what runs on a schedule, what is running now, what it is talking to, and which system files have changed. It is written for system administrators and on-call engineers who need to answer "was this server hacked?" quickly and safely, and to know when to stop and hand over. Full evidence collection and analysis are in the Linux Forensics Guide; estate-wide hunting is in the Compromise Assessment Guide.
Key takeaways
- Look, do not fix. Deleting a strange cron entry or killing a process destroys evidence and warns the intruder; record first, contain second, clean last.
- Capture volatile state first. Processes, network connections and logged-in sessions disappear on reboot, so record them before anything else.
- Persistence has a short list of homes. Accounts, SSH authorised keys, cron, systemd units and timers, shell start-up files and the dynamic linker preload file cover most cases.
- The host can lie to you. A rootkit or altered binaries can hide processes and files, so treat clean results from the host itself as weak evidence and compare with outside views.
- Escalate early. Any confirmed unknown key, account, binary change or outbound connection to an unknown address is enough to open an incident and bring in forensics.
Before you touch the host
- Decide who is in charge: open an incident ticket and note the time, your name and why you suspect compromise. If your organisation has an incident process, follow it; the Incident Response Guide covers the roles.
- Do not reboot or patch yet: a reboot loses memory, network state and anything running only from tmpfs. If the business needs the service back, snapshot first (VM or cloud volume snapshot) and consider memory capture; see the Memory Forensics Guide.
- Use a separate, trusted account: log in with a named account rather than a shared root password, and assume anything you type may be visible to an intruder who is still present.
- Write output somewhere else: save what you collect to an external location or your terminal log, not to the suspect disk, and record a hash of each file you copy off.
- Check outside views in parallel: cloud provider audit logs, firewall and flow logs, and your SIEM show what the host cannot hide. The Linux Logging and auditd for Defenders Guide covers what should already be forwarded.
The first-hour checklist
Work through these areas in order: volatile state first, then persistence, then integrity. Every check below is read-only. The ATT&CK column names the technique each check is looking for.
| Area | What to look at | Red flags | ATT&CK |
|---|---|---|---|
| Sessions and logins | Who is logged in now; recent login history from wtmp and failed logins from btmp; the SSH daemon's entries in the journal or auth log | Logins at odd hours, from unfamiliar addresses, or for service accounts that never log in interactively | T1078.003 Local Accounts |
| Accounts | /etc/passwd, /etc/shadow and /etc/group: new users, any account other than root with UID 0, members of sudo or wheel, files in /etc/sudoers.d | Unknown users, a second UID 0 account, a system account given a login shell, new sudo rules | T1136.001 Local Account |
| SSH keys | Every user's ~/.ssh/authorized_keys and authorized_keys2 (the OpenSSH default locations), root's included, and any AuthorizedKeysFile or AuthorizedKeysCommand setting in sshd_config | Keys nobody can name, keys with recent modification times, a changed AuthorizedKeysFile path | T1098.004 SSH Authorized Keys |
| Scheduled tasks | /etc/crontab, /etc/cron.d/, the cron.hourly/daily/weekly directories and per-user crontabs under /var/spool/cron/; systemd timers | Jobs that download and run content, base64 strings, jobs running from /tmp or /dev/shm | T1053.003 Cron |
| Services | systemd unit files in /etc/systemd/system/ and per-user unit directories; enabled services; legacy rc scripts | Recently created units, units whose ExecStart points to hidden or temporary paths, innocuous names copying real services | T1543.002 Systemd Service; T1037.004 RC Scripts |
| Shell start-up files | /etc/profile, /etc/profile.d/, /etc/bash.bashrc and users' .bashrc, .bash_profile and .profile | Added lines that run network commands or set aliases over common tools | T1546.004 Unix Shell Configuration Modification |
| Preloaded libraries | Whether /etc/ld.so.preload exists and what it lists; LD_PRELOAD in service environments | Any entry you did not put there: the ld.so manual describes the file as loading libraries into every program and calls this usually undesirable | T1574.006 Dynamic Linker Hijacking |
| Processes | Full process list with parent, user, start time and command line; executables whose file has been deleted; processes running from /tmp, /var/tmp or /dev/shm | High CPU from unknown names (often mining), a shell spawned by a web server user, binaries marked deleted | T1496.001 Compute Hijacking |
| Network | Listening TCP and UDP sockets with owning process and established outbound connections (for example ss with its listening, TCP, UDP, numeric and process options) | Unknown listeners, outbound connections to unfamiliar hosts or mining pool ports, processes with sockets that should have none | Check against flow logs |
| Web content | Recently modified files under web roots and upload directories | Scripts in upload folders, files owned by the web server user that should be deploy-owned | T1505.003 Web Shell |
| Recent and changed binaries | Package verification (rpm verify or debsums), recently modified files in /bin, /sbin, /usr/bin, /usr/lib, and new SUID files | Changed digests for ps, ss, netstat, ls, sshd or login; new SUID binaries | T1014 Rootkit |
Reading the package verification results
Package managers keep a record of what each installed file should look like, which makes them a quick first integrity check.
| Tool | What it compares | How to read it |
|---|---|---|
| rpm verify (RHEL, Fedora, SUSE family) | Each installed file against the rpm database: size, digest, permissions, type, owner, group, modification time, capabilities | Nine result characters per file; a dot is a pass. S is size, 5 is digest, M is mode, U and G are owner and group, T is time, P is capabilities. A "c" marks a configuration file, where changes are often expected |
| debsums (Debian, Ubuntu) | Installed files against the MD5 lists in /var/lib/dpkg/info/*.md5sums | The changed option lists only altered files; the all option includes configuration files, which are normally excluded |
Both read their reference data from the same disk an intruder may control, and the debsums manual itself says it is of limited use as a security tool and points to integrity tools that can run from safe media. A changed digest on a core binary is a strong signal; a clean result is not proof. A file integrity baseline taken before the incident is far better: see the Linux File Integrity Monitoring Guide.
Deciding what you found
| Finding | Likely meaning | Next step |
|---|---|---|
| Nothing unusual, and outside logs agree | Probably not compromised, or the intruder left no host traces | Record the checks, keep monitoring, and review why the alert fired |
| Unknown SSH key or account | Persistence or credential misuse | Open an incident; follow the SSH Key Compromise Response Playbook |
| Mining process or cron job fetching scripts | Commodity intrusion, often through an exposed service or weak credential | Contain on the network, preserve evidence, find the entry point before cleaning |
| Web shell or scripts in upload folders | Application-layer entry | Follow the Web Shell Response Playbook |
| Changed core binaries, ld.so.preload entry, or host and network views disagree | Possible rootkit; the host's own tools cannot be trusted | Stop live work, snapshot and image, analyse offline; see the Rootkits and Bootkits Guide |
Clean or rebuild
For a confirmed root-level compromise, rebuild from a known-good image rather than cleaning in place, because you cannot prove every persistence point was found. Cleaning in place is only defensible for a contained, well-understood finding at user level, with forensics agreement.
Containment and escalation, in order
- Record: save the volatile state and persistence findings with timestamps and hashes.
- Preserve: take a VM or volume snapshot, and memory if you have the tooling.
- Contain on the network: restrict the host with a security group, firewall or VLAN change rather than powering it off; the How to Isolate a Compromised Endpoint guide sets out options.
- Rotate what the host held: SSH keys, API keys, database passwords, cloud credentials and tokens stored on or used by the server, from a clean machine.
- Hand over: pass the notes and snapshot to forensics; check whether the same keys, accounts or indicators appear on other servers.
- Recover: rebuild, restore data from a backup taken before the earliest suspicious activity, and monitor closely.
Where identity fits
Most Linux intrusions start or spread with a credential: a reused password, a stolen private key, or a cloud key in a configuration file. Treat every secret on the host as exposed, including keys in environment files, deploy scripts and crontabs, and check where those keys can log in elsewhere. The How to Find ssh authorized keys across a server fleet and How to Find credentials in cron jobs and scheduled tasks guides cover the follow-up.
Measuring triage readiness
- Time from alert to volatile state captured: minutes, not hours.
- Servers with forwarded authentication logs: percentage of the Linux estate sending SSH and sudo events off-host.
- Servers with an integrity baseline: percentage covered by file integrity monitoring.
- Authorised keys with a named owner: percentage of keys mapped to a person or service.
- Triages that preserved evidence: share of incidents where a snapshot existed before any change.
Common failure patterns
- Cleaning first: deleting the obvious cron job removes the clue to how the intruder got in, and they return.
- Rebooting to "see if it goes away": memory, connections and tmpfs payloads are lost.
- Trusting the host's own tools: a hidden process will not show in a tampered process list; compare with network and cloud logs.
- Checking only root: service accounts and the web server user carry authorised keys and crontabs too.
- Forgetting the secrets: the server is rebuilt but the keys it held are never rotated.
Practitioner checklist
- Open an incident record and note time, operator and reason before touching the host.
- Do not reboot, patch or delete anything during triage.
- Capture sessions, processes and network connections first, saved off-host.
- Review accounts, sudo rules and every user's authorised keys.
- Review cron locations, systemd units and timers, shell start-up files and ld.so.preload.
- Run package verification and look for new SUID files and recently changed binaries.
- Compare host findings with firewall, flow, cloud and SIEM logs.
- Snapshot, contain on the network, rotate exposed secrets and hand over to forensics.
Standards and references
- NCSC: Incident management
- MITRE ATT&CK: T1098.004 SSH Authorized Keys
- MITRE ATT&CK: T1053.003 Cron
- MITRE ATT&CK: T1574.006 Dynamic Linker Hijacking
- Linux man-pages: ld.so
- Linux man-pages: crontab(5)
- OpenBSD: sshd_config manual
- Linux man-pages: rpm
Related NHI Mgmt Group resources: Linux Forensics Guide, Compromise Assessment Guide, Linux Detection and Response Guide, SSH Key Compromise Response Playbook.