Every Linux server carries dozens of accounts that no person logs into, plus Kerberos keytabs that hold long-term keys, and almost none of them appear in an identity inventory. System users created by packages, application accounts created by hand, the host keytab written at domain join and service keytabs copied between servers are all non-human identities. A keytab is equivalent to a password that never expires unless someone rotates it, and a service account with a real shell and an SSH key is an interactive login waiting to be used. This guide covers how to find these identities, how to stop them being used interactively, how to create, protect and rotate keytabs for MIT Kerberos, FreeIPA and Active Directory, and what to monitor. It is for Linux administrators, identity engineers and security teams who own server estates.
Key takeaways
- Treat keytabs as passwords. A keytab holds a principal's long-term keys; anyone who can read it can authenticate as that service, request tickets and, for a host keytab, impersonate the machine.
- System accounts should not be able to log in. A
nologinshell, a locked password and noauthorized_keysfile are the baseline; check all three, because each blocks a different route. - Inventory from several sources.
/etc/passwdalone misses directory-backed service accounts, keytabs, SSH keys, sudo rules and systemd units that define what the account actually does. - Rotate keys on a schedule, in two steps. Change the key, keep the previous version until issued tickets expire and directory replication completes, then remove the old key.
- One principal per service per host. Shared keytabs copied across servers turn one compromised box into access everywhere and make rotation a coordinated outage.
What counts as a Linux service account
| Type | Where it is defined | Typical example | Main risk |
|---|---|---|---|
| Package system user | /etc/passwd, created by useradd -r or a sysusers.d file | postgres, nginx, chrony | Usually safe if shell is nologin; owns data and config files |
| Hand-made application account | /etc/passwd with a regular UID | appsvc, deploy, batch | Often has a bash shell, a password, SSH keys and broad sudo |
| systemd dynamic user | Unit file with DynamicUser=yes | Self-contained daemons | Transient UID; ownership and access defined only in the unit |
| Directory service account | Active Directory, FreeIPA or LDAP, resolved through SSSD | [email protected] | Password or keytab may work on many hosts; managed outside Linux teams |
| Host (machine) principal | /etc/krb5.keytab written at domain join | host/web01.example.com, WEB01$ | Impersonation of the machine to the domain |
| Service principal keytab | Separate keytab file per service | HTTP/web01.example.com, nfs/, Hadoop service principals | Copied keytabs, weak encryption types, never rotated |
- UID ranges:
useradd -rallocates fromSYS_UID_MINtoSYS_UID_MAXin/etc/login.defs(defaults 101 toUID_MINminus 1), sets no password ageing and creates no home directory unless-mis given. Regular users default to 1000 to 60000. A service account in the regular range is a sign it was created by hand. - sysusers.d: declarative system users. A
uline creates the user and group with an invalid password; the default shell is/usr/sbin/nologinand the default home is/. Since systemd 257, au!line also locks the account against non-password authentication such as SSH, and is the recommended form. - Dynamic users:
DynamicUser=yesgives a service a UID from 61184 to 65519 only while it runs. See the systemd Service Hardening Guide. - Directory accounts: SSSD resolves users from Active Directory, FreeIPA or LDAP, caches them, and can enforce host-based access control and sudo rules from the directory. See the SSSD and Linux Active Directory Integration Guide.
Building the inventory
No single file tells you what a service account can do. Combine these sources per host and load them into your NHI inventory with an owner for each entry. The Service Account Inventory Template gives the fields.
| Question | Where to look | Command or file |
|---|---|---|
| Which accounts exist | Local and directory user databases | getent passwd (includes SSSD users when enumeration is enabled), /etc/passwd |
| Can it log in with a password | Shadow file | passwd -S <user>; a hash beginning ! or * is locked or invalid |
| Does it have a shell | Shell field | Anything other than /usr/sbin/nologin or /bin/false |
| Can it log in with SSH keys | Home directories and AuthorizedKeysFile | ~/.ssh/authorized_keys, any AuthorizedKeysCommand |
| What can it run as root | sudo policy | sudo -l -U <user>, /etc/sudoers.d/, directory sudo rules |
| What runs as it | Services and schedules | User= in unit files, crontab -l -u <user>, /etc/cron.d/, timers |
| Which keytabs exist | File system | Find files containing keytab data (the MIT format starts with bytes 0x05 0x02), plus default_keytab_name in krb5.conf |
| What each keytab contains | The keytab itself | klist -kte <file> shows principal, key version number (kvno), encryption type and timestamp |
The timestamp column in klist -kte is the most useful age signal you have: a keytab whose newest entry is years old has never been rotated.
Stopping interactive use of service accounts
A service account that a person can log into becomes a shared, unattributable admin account. Close each route separately.
- Shell: set
/usr/sbin/nologinwithusermod -s /usr/sbin/nologin <user>. This stops interactive shells but not every service: SFTP subsystems, port forwarding and some PAM services may still work. - Password: lock with
usermod -Lorpasswd -l. Locking the password does not block SSH key authentication; that needsusermod -e 1(account expiry) or au!sysusers entry, which locks non-password logins too. - SSH: remove
authorized_keysfor accounts that do not need inbound SSH. Where automation must log in, restrict keys withfrom=,command=andrestrictoptions, and add aDenyGroupsorAllowGroupsrule insshd_configso new service accounts are denied by default. Adding keys is persistence technique T1098.004 SSH Authorized Keys. See the SSH Key Management Guide. - su and sudo: admins who need to act as the service should use
sudo -u <svc> <command>with a logged rule, notsuto a shell. See the sudo and sudoers Security Guide. - Directory accounts: deny interactive logon for service accounts centrally (SSSD access filters, FreeIPA HBAC rules or AD GPO-based access control) so the account works only for the protocols it needs.
Keytabs: what they hold and how they are created
A keytab ("key table") stores long-term keys for one or more principals. Servers use it to accept Kerberos authentication, and clients can use it to obtain initial credentials without a password prompt. Each entry records a timestamp, the principal name, the key version number, the encryption type and the key itself. MIT Kerberos looks for the default keytab in the KRB5_KTNAME environment variable, then default_keytab_name in [libdefaults], then the compiled-in default (usually /etc/krb5.keytab). A separate client keytab can be set with KRB5_CLIENT_KTNAME or default_client_keytab_name.
| Realm type | Create | Notes |
|---|---|---|
| MIT Kerberos KDC | kadmin ktadd -e aes256-cts-hmac-sha1-96:normal ... | Run kadmin on the host that will hold the keytab so the key never crosses the network; ktremove removes entries |
| FreeIPA | ipa-getkeytab for service principals; ipa-client-install writes the host keytab | Retrieving a keytab this way generates new keys by default, which invalidates copies on other hosts |
| Active Directory, host | realm join or adcli join | Creates the computer account and writes /etc/krb5.keytab; --service-name adds extra service principals |
| Active Directory, service account | ktpass /princ ... /mapuser ... /crypto AES256-SHA1 /pass +rndpass /out ... | Maps the principal to an AD user; one account per service instance; always set /crypto explicitly |
- Encryption types: generate AES keys only (
aes256-cts-hmac-sha1-96, andaes128-cts-hmac-sha1-96where needed). Microsoft notes that DES is not supported by default and RC4-HMAC is weaker than AES; a keytab or account still carrying RC4 keys keeps the service open to cheaper offline cracking of tickets (T1558.003 Kerberoasting). See the Kerberos Security Guide. - Passwords on command lines:
ktpass /pass <password>leaks into shell history and process listings. Use/pass *to prompt, or a random password, because no human needs to know a service account password. - Single-entry keytabs: for a service that only needs to act as a client, MIT recommends a keytab with a single entry for that identity, readable by the service and referenced through
KRB5_CLIENT_KTNAME.
Protecting keytabs on the host
- Ownership and mode: MIT's guidance is that keytabs are stored on local disk and readable only by root. For a service keytab the service user must read it, so use owner
root, group set to the service group and mode0640, or owner the service user and0600. Never place keytabs in home directories, application repositories or shared NFS paths. - Privilege separation with gssproxy: gssproxy lets a service such as a web server accept GSSAPI security contexts using keytab entries it cannot read directly; the daemon holds the keytab and does the cryptography. Kernel clients for NFS and CIFS use the same upcall model. This removes the keytab from the application's reach entirely.
- systemd credentials: where gssproxy does not fit, load the keytab into the service with
LoadCredential=so it sits in a per-unit, non-inherited directory under/run/credentials/. - Transfer: never send a keytab across the network in the clear or by email. Extract it on the target host, or move it through a secrets manager with access logging. See the Secrets Management Guide.
- Backups and images: keytabs inside VM templates, container images or backups are copies you cannot rotate in place. Exclude them from golden images and join each clone individually.
- File monitoring: add auditd read watches on keytab paths (
-w /etc/krb5.keytab -p r -k keytab_read) and alert on reads by unexpected processes. Reading credential files is T1552.001 Credentials In Files.
Rotating keytabs and machine passwords
Rotation is always two steps, because tickets already issued were encrypted with the old key and directory replicas may not yet have the new one.
- Record the current state:
klist -kteon every host that holds the principal. If more than one host holds it, rotation will break the others: fix that first by issuing per-host principals. - Change the key: MIT
k5srvutil changereplaces the principal's keys in the KDC with new random keys and updates the keytab; it fails if the local kvno does not match the database. For AD computer accounts,adcli updatechanges the machine password, writes new keys and removes older ones, deliberately keeping the previous key while AD replicates. By default it renews only if the password is older than 30 days, adjustable with--computer-password-lifetime. SSSD can run machine account renewal automatically for AD-joined hosts. - Test: obtain a ticket with the new key (
kinit -kt <keytab> <principal>) and authenticate to the service from a client. - Remove old keys later:
k5srvutil deloldremoves keys that are not the latest version. Run it after tickets issued with the old key have expired (the ticket lifetime, typically 10 hours), not immediately. - Roll back if needed: while the previous kvno is still in the keytab, services keep accepting old tickets; if the new key fails, restore the keytab file from the pre-change copy held in your secrets store and rotate again once fixed.
For AD service accounts mapped with ktpass, rotation means a new password, a new kvno and redistribution of the keytab. Where a Linux service needs an AD identity and you control both sides, prefer a computer-account-based principal that adcli update can rotate automatically over a user account with a static password.
Choosing an approach
- Small estate, few Kerberos services: per-host principals, AES-only keytabs, quarterly
k5srvutil changeor automaticadcli update, and a spreadsheet inventory with owners. - Large estate or Hadoop-style clusters with hundreds of service principals: automate keytab issue and rotation through the identity platform or a secrets manager, generate principals per host and service, and alert on keytabs older than policy. See the Hadoop Security Guide for cluster specifics.
- Containers and short-lived workloads: avoid baking keytabs into images; inject them at start, or replace Kerberos with workload identity where the target supports it. See the Workload Identity Federation Guide.
Detection and measurement
| Signal | Source | Indicates |
|---|---|---|
| Interactive session for a service account | sshd and PAM logs in journald or /var/log/secure, auth.log | Shared use or compromise (T1078.003 Local Accounts) |
New local account or new authorized_keys file | auditd watches on /etc/passwd, /etc/shadow, home .ssh directories | Persistence (T1136.001, T1098.004) |
| Keytab read by an unexpected process | auditd read watch with key | Credential theft (T1552.001) |
| Service principal used from a host that should not hold it | KDC logs (MIT krb5kdc log, AD event 4768 and 4769 with client address) | Copied keytab or ticket reuse (T1550.003 Pass the Ticket) |
| RC4 tickets for service accounts | AD event 4769 ticket encryption type 0x17 | Accounts still carrying RC4 keys, Kerberoasting exposure |
Metrics
- Owned accounts: percentage of service accounts and keytabs with a named owner in the inventory.
- Interactive-capable service accounts: count with a real shell, an unlocked password or an
authorized_keysfile (target zero without an exception). - Keytab age: percentage of keytabs whose newest key is within the rotation policy, from
klist -ktetimestamps. - Shared principals: number of principals present in keytabs on more than one host.
- Weak encryption: number of keytab entries with RC4 or DES encryption types (target zero).
Common failure patterns
- Service accounts with bash, a password and sudo ALL: created during a project, used by several engineers, never reviewed. These become the attacker's preferred foothold because activity blends in.
- Credentials in home directories: keytabs, SSH private keys and
.git-credentialsleft under a service account's home are exactly what host-level malware collects. In the PyTorch torchtriton Supply Chain Attack 2022 the malicious package read/etc/passwd, the SSH key directory and the first 1,000 files in the home directory and sent them out over DNS. - One keytab copied to every node: convenient at build time, impossible to rotate without a coordinated outage, and one compromised node exposes the principal everywhere.
- Keytabs in images and backups: a VM template with
/etc/krb5.keytabproduces clones that share a machine identity. - Rotation that deletes the old key at once: clients holding valid tickets fail and the team concludes rotation is unsafe, so it is never done again.
- Locking the password and assuming the account is disabled: SSH keys still work. Check shell, password, expiry and keys together.
Framework mapping
| Framework | Reference | Relevance |
|---|---|---|
| NIST SP 800-53 Rev. 5 | AC-2 Account Management; IA-4 Identifier Management; IA-5 Authenticator Management; AC-6 Least Privilege | Inventory and ownership of service accounts; keytab protection and rotation; no interactive rights |
| NIST CSF 2.0 | PR.AA-01; PR.AA-05; ID.AM-08 | Identities and credentials for services managed; least privilege; lifecycle management |
| CIS Controls v8.1 | Control 5 Account Management; Control 6 Access Control Management | Service account inventory and review; restricting what accounts can do |
| ISO/IEC 27001:2022 Annex A | A.5.16 Identity management; A.5.17 Authentication information; A.8.2 Privileged access rights | Lifecycle of non-human identities and their secrets |
| MITRE ATT&CK | T1552.001; T1552.004; T1558.003; T1550.003; T1078.003; T1098.004; T1136.001 | Theft and abuse of keytabs, keys and local service accounts |
Practitioner checklist
- Build a per-host inventory combining
getent passwd, shadow status, shells,authorized_keys, sudo rules, unitUser=settings, crontabs and every keytab found withklist -kte. - Assign an owner to every service account and keytab; disable anything without one after a notice period.
- Set
nologin, lock passwords and remove or restrict SSH keys for all service accounts; useu!in sysusers.d where available. - Issue one principal per service per host and stop copying keytabs between servers or into images.
- Regenerate keytabs with AES-only encryption types and remove RC4 and DES keys.
- Set keytab mode to
0600or0640with root or the service as owner, or move keytab use behind gssproxy. - Rotate on a schedule with
k5srvutil changeoradcli update, and rundeloldonly after ticket lifetimes have passed. - Add auditd watches on keytab paths,
/etc/passwdand.sshdirectories, and alert on interactive logins by service accounts. - Track the five metrics above monthly and report exceptions with expiry dates.
Standards and references
- MIT Kerberos: Keytab file explained
- MIT Kerberos: Application servers
- MIT Kerberos: k5srvutil
- Microsoft Learn: ktpass
- realmd project: adcli, tool for performing actions on an Active Directory domain
- systemd: sysusers.d, declarative allocation of system users and groups
- SSSD: Introduction
- NIST: SP 800-53 Rev. 5
Related NHI Mgmt Group resources: Service Account Security Guide · Kerberos Security Guide · Hadoop Security Guide · Unix and Linux Privileged Access Guide · Non-Human Identity (NHI) Guide