Most SSH estates still run on static public keys copied into authorized_keys files and host keys pinned by trust on first use, which means access never expires and nobody can say who can reach what. OpenSSH has supported its own certificate format for years: a certificate authority (CA) signs a user or host key together with a list of principals, a validity window and a set of restrictions, and servers and clients trust the CA instead of thousands of individual keys. This guide goes deeper than our overview of SSH key management: it explains the protocol layers that matter for trust, the certificate fields and options, how to design and protect the CA, how to issue short-lived certificates to people and workloads, and how to revoke, monitor and roll back. It is written for platform, infrastructure and identity engineers who run Linux and network estates at any size.
Key takeaways
- Trust the CA, not the key. With
TrustedUserCAKeyson servers and@cert-authoritylines in known_hosts on clients, one CA public key replaces per-key distribution in both directions. - Short validity is the control. A certificate valid for hours removes most of the revocation problem and turns SSH access into just-in-time access tied to a fresh SSO and MFA event.
- Principals carry the authorisation. The certificate names the accounts or roles it is valid for, and
AuthorizedPrincipalsFileorAuthorizedPrincipalsCommandmaps them to local accounts, so access is defined centrally. - The CA key is now a crown jewel. Whoever holds the user CA private key can log in anywhere that trusts it, so keep it in an HSM, cloud KMS or PKCS#11 token and gate signing behind an audited service.
- Host certificates end TOFU prompts. Signing host keys removes the "unknown host" question that trains users to type yes, and makes forced host key rotation, as GitHub faced in 2023, far less disruptive.
How SSH establishes trust
RFC 4251 describes SSH as three layers. The transport layer (RFC 4253) authenticates the server with its host key and provides confidentiality and integrity. The user authentication layer (RFC 4252) authenticates the client user, most often with a public key. The connection layer (RFC 4254) multiplexes channels for shells, commands, port forwarding and agent forwarding. Certificates change the first two layers: they change how the client decides a host key is genuine and how the server decides a user key is acceptable.
RFC 4251 section 4.1 sets out two host key trust models. In the local database model every client maps host names to expected host keys, which needs no infrastructure but becomes hard to maintain. In the CA model a trusted CA certifies host keys, so clients only need the CA key, at the price of placing significant trust in the central infrastructure. The RFC is blunt about skipping verification on first connection: the protocol allows it, but it is "NOT RECOMMENDED", because it leaves the connection open to active man-in-the-middle attack. OpenSSH certificates are the practical way to run the CA model.
| Trust decision | Static key model | Certificate model |
|---|---|---|
| Client trusts server | Host key cached in ~/.ssh/known_hosts on first use, or distributed by configuration management | @cert-authority *.example.internal ssh-ed25519 ... line in known_hosts; server presents HostCertificate |
| Server trusts user | Each public key appended to the account's authorized_keys | Server lists the user CA in TrustedUserCAKeys; certificate principal must match the account or the principals mapping |
| Access expiry | None by default; optional expiry-time= per key line | Validity window inside every certificate |
| Restrictions | Per-line options such as restrict, from=, command= | Critical options and extensions signed into the certificate |
| Revocation | Delete the key from every file where it was copied | Let it expire, or add serial or key ID to a KRL referenced by RevokedKeys |
| Attribution | Key fingerprint, often shared or unlabelled | Key ID and serial written to the sshd log at login |
The OpenSSH certificate format
OpenSSH certificates are not X.509. They are a simpler OpenSSH format: a public key plus signed metadata, produced by ssh-keygen -s and written next to the input key as *-cert.pub. The fields an operator controls map directly to ssh-keygen flags.
| Field | ssh-keygen flag | What to put in it |
|---|---|---|
| Certificate type | -h for host, default user | Use separate CAs for user and host certificates |
| Key ID | -I identity | The human or workload identity and a request reference; it is logged by sshd and can be revoked |
| Serial | -z serial | A unique, non-zero 64-bit number from the CA's database, so a KRL can revoke it in very little space |
| Principals | -n a,b | Role names or account names; wildcards are supported for host certificates, not for user certificates |
| Validity | -V start:end | For users, a start a few minutes in the past and an end measured in hours, for example -5m:+8h |
| Options and extensions | -O | Start from clear and add only the permit-* extensions the role needs |
| CA signature | -s ca_key, with -D (PKCS#11) or -U (ssh-agent) | Ed25519, ECDSA or RSA with rsa-sha2-512 (the default for RSA CAs); never ssh-rsa SHA-1 |
Restrictions you can sign into a user certificate
- clear: removes the default permissions, which otherwise allow agent forwarding, port forwarding, PTY allocation, user rc and X11 forwarding.
- permit-pty, permit-port-forwarding, permit-agent-forwarding, permit-X11-forwarding, permit-user-rc: add back only what the role needs. An automation certificate rarely needs any of them.
- force-command=: runs a fixed command whatever the client asks for, which suits backup, deployment and file transfer jobs.
- source-address=: limits use to listed CIDR ranges, such as the CI runner subnet or the bastion addresses.
- verify-required and no-touch-required: apply only to FIDO-backed
ecdsa-skanded25519-skkeys;verify-requireddemands a PIN or other user verification on the authenticator. - Custom names:
critical:nameandextension:nameadd organisation options. Unknown critical options cause the certificate to be refused, so use domain-suffixed names and test before rollout.
Two behaviours changed in OpenSSH 10.3, according to the project's release notes: an empty principals list no longer matches any principal, and wildcard handling was made consistent (allowed for host certificates, not for user certificates). sshd already rejects user certificates without principals when trusted through TrustedUserCAKeys. Never issue a certificate without principals, whatever version you run.
Designing the CA
An SSH CA is a single signing key with no chain, so design is about separation, custody and issuance rather than hierarchy.
- Separate user and host CAs: a compromise or rotation of one should not touch the other, and the people who can approve host certificates are usually not the people who approve user access.
- Separate by environment or trust zone: a production user CA that is not trusted by development hosts, and vice versa, limits the blast radius of an issuance mistake. Very large estates add CAs per region or business unit.
- Key custody: ssh-keygen can sign with a CA key held in a PKCS#11 token (
-D) or loaded in ssh-agent (-U), passing only the public half. Hold production CA keys in an HSM or cloud KMS, never as a file on an engineer's laptop. See the HSM Deployment and Operations Guide. - Algorithm: Ed25519 is the ssh-keygen default and is accepted by default in
CASignatureAlgorithms. OpenSSH 10.6 enabled a hybrid post-quantum signature algorithm,ssh-mldsa44-ed25519, which is in the default CA signature list; older clients and servers will not understand it, so treat it as a planned migration, not a default for a mixed estate. - Two CA keys at once:
TrustedUserCAKeysand known_hosts can list several CA keys. Publish the next CA key alongside the current one well before you need it, so rotation is a configuration change, not an outage. - Issuance service: people should never run ssh-keygen against the CA themselves. Put signing behind a service that authenticates the requester through SSO and MFA (or a workload identity), applies policy, chooses principals and validity, assigns the serial and key ID, and writes an audit record.
| Design choice | Small organisation (tens of servers) | Large organisation (thousands of servers) |
|---|---|---|
| CA count | One user CA and one host CA | Per environment or zone, plus a dedicated CA for automation |
| Key custody | Hardware token or cloud KMS with signing restricted to one service account | Network HSM or KMS, dual control for CA key generation and rotation |
| Issuance | Managed SSH CA feature of an identity or PAM product, or a small signing service | Central certificate service integrated with IdP, PAM and CI identity |
| User certificate lifetime | A working day | One to eight hours; minutes for privileged roles |
| Principal mapping | Account names in principals, default sshd matching | AuthorizedPrincipalsCommand backed by the directory or PAM policy |
Configuring servers and clients
The sshd and ssh settings below are the ones that decide whether certificate trust is safe. Defaults are taken from the OpenBSD manual pages.
| Setting | Where | Recommended use |
|---|---|---|
TrustedUserCAKeys | sshd_config | Root-owned file listing the user CA public keys this host trusts |
AuthorizedPrincipalsFile | sshd_config | Per-account list of accepted principals (default none, meaning the account name must be in the certificate) |
AuthorizedPrincipalsCommand | sshd_config | Absolute path, root-owned, not group or world writable; requires AuthorizedPrincipalsCommandUser or sshd will not start |
RevokedKeys | sshd_config | Path to a KRL; if it is unreadable, public key authentication is refused for all users, so monitor its distribution |
HostCertificate | sshd_config | Host certificate matching a configured HostKey |
AuthorizedKeysFile | sshd_config | Set to none (or a root-controlled path) once certificates are proven, so users cannot add their own keys |
CASignatureAlgorithms | Both | Leave the default or narrow it; it excludes SHA-1 ssh-rsa |
ExposeAuthInfo | sshd_config | Default no; enable where session recording needs the certificate details in SSH_USER_AUTH |
StrictHostKeyChecking | ssh_config | Default ask; yes for automation once host certificates cover every target |
RevokedHostKeys | ssh_config | Host KRL; if missing or unreadable, host authentication is refused for all hosts |
ForwardAgent | ssh_config | Keep the default no; anyone who can bypass file permissions on the remote host can use a forwarded agent |
Two smaller points cause many incidents. First, a key line marked cert-authority in a user's own authorized_keys file also makes that CA trusted for that account, so lock down AuthorizedKeysFile or users can bring their own CA. Second, UpdateHostKeys only learns new keys for hosts trusted through a plain key, not a certificate, so it is a transition aid, not a substitute for host certificates.
Issuing certificates to people and workloads
People
- Login flow: the engineer authenticates to the issuance service with SSO and phishing-resistant MFA, requests a role, and receives a certificate for their local key, ideally a FIDO-backed
ed25519-skkey withverify-required. - Agent lifetime: from OpenSSH 10.1, ssh-add sets the agent's lifetime for a certificate to five minutes after the certificate expires, so expired certificates do not linger in the agent.
- Privileged roles: issue root-equivalent principals only through a PAM approval or break-glass workflow, with lifetimes in minutes. See the JIT Access Guide and the Break-Glass Account Guide.
Workloads and automation
Automation is where static SSH keys do the most damage: deploy keys, backup keys and Ansible keys that nobody owns and that never expire. Certificates make them non-human identities with an owner, a scope and an expiry.
- Bootstrap from a workload identity: a CI job or VM proves who it is with its platform identity (an OIDC token from the CI system or the cloud instance identity) and exchanges it for a certificate valid for the length of the job.
- Constrain hard:
clear, aforce-commandwhere the job runs a fixed task, andsource-addresslimited to the runner network. - Name the owner in the key ID: for example the pipeline, repository and run ID, so a log line leads straight to the job that used it.
- No private key at rest: generate the key pair in the job, sign it, use it and discard both. See the NHI Authentication Guide.
Hosts
Sign host keys at provisioning time from a trusted identity (the provisioning system or cloud instance identity document), with principals set to the host's DNS names and validity long enough to survive outages of the issuance service, often weeks to months, renewed automatically. Wildcard principals are permitted for host certificates but widen what a stolen host key can impersonate, so prefer exact names.
Revocation, rotation and rollback
Short validity is the primary revocation mechanism. For the cases that cannot wait, OpenSSH uses Key Revocation Lists: compact binary files built with ssh-keygen -k. A KRL specification can revoke by serial (single values or ranges), by id (key ID), by key, or by sha1 or sha256 hash (hash support needs OpenSSH 7.9 or later). Revoking by serial can take as little as one bit per certificate, and -u merges new entries into an existing KRL. ssh-keygen -Q -f krl tests keys against a KRL and returns a non-zero exit status if any is revoked, which is useful in a pipeline that checks the KRL before publishing it.
- Revoke a user certificate: add its serial to the KRL, publish the KRL to every host that sets
RevokedKeys, and confirm with-Qon a sample host. - Revoke a user: remove the person from the IdP group that the issuance service checks, revoke their outstanding serials or key ID, and let the rest expire.
- Rotate a CA: generate the new CA key in the HSM, add its public key to
TrustedUserCAKeysor known_hosts beside the old one, switch issuance, wait for the longest certificate lifetime to pass, then remove the old CA key everywhere. - Respond to CA compromise: remove the CA public key from trust everywhere (configuration management push), issue from the standby CA, and treat every login since the suspected compromise as suspicious. This is why a standby CA and a fast configuration push matter.
- Roll back: keep the old authorized_keys path disabled rather than deleted during migration, so a failed issuance service can be bypassed by re-enabling it for named break-glass accounts only.
Phased rollout
- Discover: inventory existing authorized_keys entries, host keys and known_hosts pins, and identify automation keys and their owners. The SSH Key Management Guide covers discovery and clean-up.
- Host certificates first: sign host keys and distribute the
@cert-authorityline to clients. This is low risk because plain known_hosts entries keep working alongside it. - Pilot user certificates: enable
TrustedUserCAKeyson a pilot ring of non-production hosts while authorized_keys still works, and confirm sshd logs show certificate logins with key ID and serial. - Move automation: convert CI and configuration management jobs to job-length certificates, one pipeline at a time, and delete the static key each replaces.
- Enforce: set
AuthorizedKeysFile none(or a root-owned path for named exceptions) ring by ring, starting with production bastions, then the wider estate. - Harden and measure: introduce
AuthorizedPrincipalsCommand, KRL distribution, FIDO-backed keys for privileged users, and the metrics below.
Detection and measurement
At the default LogLevel INFO, OpenSSH sshd login records for certificate authentication include the certificate key ID, serial and the signing CA key fingerprint; avoid DEBUG levels in production, which the manual page warns violate user privacy. Forward these logs to the SIEM and alert on:
- Logins using plain public keys on hosts where certificates are enforced (a sign of a missed exception or a planted key, ATT&CK T1098.004 SSH Authorized Keys).
- Certificates signed by a CA fingerprint that is not in your registry, or key IDs that do not match the issuance service's audit log (possible forged certificate, compare T1649 Steal or Forge Authentication Certificates).
- Issuance spikes for one identity, privileged principals outside change windows, or certificates with validity longer than policy.
- Agent forwarding use where certificates do not permit it, and lateral movement patterns (T1021.004 SSH, T1563.001 SSH Hijacking).
| Metric | Target direction |
|---|---|
| Share of SSH logins authenticated by certificate | Towards 100% outside documented break-glass |
| Remaining authorized_keys entries in production | Falling to zero or a named exception list |
| Median user certificate lifetime | Hours, not days |
| Hosts with current host certificate | All, with renewal well before expiry |
| Time to publish a KRL update to all hosts | Minutes |
| Static automation keys replaced by job certificates | Rising each quarter |
Common failure patterns
- CA key on a disk: a CA private key stored as a file, backed up and copied, is worse than the static keys it replaced. High-value SSH keys do leak: the GitHub RSA SSH host key exposure came from a private host key briefly published in a public repository.
- Long-lived user certificates: a one-year user certificate is a static key with extra steps. Validity must be short or the model gives little benefit.
- authorized_keys left open: certificates are added but
AuthorizedKeysFilestays at the default, so attackers and users can still plant keys, includingcert-authoritylines. - Over-broad principals: a principal such as
rootor a shared admin role in every certificate defeats central authorisation. - Default permissions kept: certificates issued without
clearallow port and agent forwarding to every role, which helps lateral movement. - Brittle KRL distribution: an unreadable
RevokedKeysfile blocks all public key logins on that host, and an unreadableRevokedHostKeysfile blocks all host authentication, so a failed push becomes an outage.
Lessons from incidents
When GitHub replaced its RSA host key on 24 March 2023, every client that had pinned the old key saw a host identification warning, and automation that pinned it broke. Clients that trust a host CA only need the CA line, so the host key behind it can be replaced and re-signed without touching them. The same incident shows the custody point: the key that leaked was a single private key file. A CA key with the same exposure would let an attacker sign trusted host or user certificates, which is why custody in an HSM or KMS is not optional.
Framework mapping
| Framework | Reference | How SSH certificates help |
|---|---|---|
| NIST SP 800-53 Rev. 5 | IA-5(2) Public Key-based Authentication; IA-5 Authenticator Management | Certificate validity, binding to identity, revocation through KRLs |
| NIST SP 800-53 Rev. 5 | AC-17 Remote Access; AC-2 Account Management | Principals and short validity enforce authorised, time-bound remote access |
| NIST SP 800-53 Rev. 5 | SC-12 Cryptographic Key Establishment and Management; IA-3 Device Identification and Authentication | CA key custody; host certificates authenticate servers |
| NIST SP 800-53 Rev. 5 | AU-2 Event Logging | Key ID and serial in sshd logs |
| NIST CSF 2.0 | PR.AA-01, PR.AA-03, PR.DS-02 | Managed credentials, authenticated users and hosts, protected data in transit |
| MITRE ATT&CK v20 | T1098.004, T1552.004, T1021.004, T1563.001, T1557 | Removes plantable and stealable static keys; host certificates resist adversary-in-the-middle |
Practitioner checklist
- Create separate user and host CAs per environment, with keys generated and held in an HSM, KMS or PKCS#11 token.
- Put all signing behind an issuance service that checks SSO and MFA or workload identity and logs every certificate.
- Issue user certificates for hours, with unique serials, meaningful key IDs and
clearplus only the extensions each role needs. - Sign every host key and distribute
@cert-authoritylines so automation can run withStrictHostKeyChecking yes. - Replace static automation keys with job-length certificates constrained by
force-commandandsource-address. - Set
AuthorizedKeysFile noneor a root-owned path once certificates are proven, with a named break-glass exception. - Publish KRLs through configuration management, test them with
ssh-keygen -Q, and monitor that every host can read them. - Pre-publish a standby CA key and rehearse CA rotation and compromise response.
- Alert on plain key logins, unknown CA fingerprints and certificates that do not match the issuance log.
Standards and references
- IETF: RFC 4251 The Secure Shell (SSH) Protocol Architecture
- OpenBSD: ssh-keygen(1) manual page, Certificates and Key Revocation Lists
- OpenBSD: sshd_config(5) manual page
- OpenBSD: ssh_config(5) manual page
- OpenBSD: sshd(8) manual page, authorized_keys and known_hosts formats
- OpenSSH: Release notes
- NIST: IR 7966 Security of Interactive and Automated Access Management Using Secure Shell (SSH)
- NIST: SP 800-53 Rev. 5
Related NHI Mgmt Group resources: SSH Key Management Guide · OpenSSH Server Hardening Guide · Privileged Access Management Guide · Machine Identity, PKI and Certificate Lifecycle Guide