An SSH private key stays on the client and must remain secret, while the public key is placed on the server in authorized_keys. The public key identifies who may connect, but only the matching private key can prove possession during login. If the private key is exposed, the account associated with that key can be impersonated.
Why SSH keys work as a paired trust model
An SSH login keypair is a cryptographic trust relationship, not two interchangeable copies of the same secret. The server keeps the public key because it is safe to distribute and can be used to verify a signature or challenge response. The client keeps the private key because possession of that key is what proves authority to authenticate.
This split matters because SSH is designed so the server can recognise a legitimate key without ever learning the secret needed to use it. The public key in authorized_keys is effectively an allow-list entry, while the private key remains the proof material. That is why the two keys have different handling requirements and different blast radii if exposed. For broader identity and credential governance, Ultimate Guide to NHIs — What are Non-Human Identities is the best NHIMG reference point for how keys, tokens and certificates fit into lifecycle control.
In practice, the server is not “storing your key” in the sense of keeping your private secret. It is storing the matching public key so it can compare or validate the cryptographic proof presented during connection setup. That is why public keys can be replicated across systems, while private keys must be protected as sensitive authentication material.
A useful way to think about it is: the public key says, “this account will trust a holder of the matching private key,” and the private key says, “I can prove I am that holder.” The server-side record is permission data; the client-side private key is authentication proof.
For examples of how exposed authentication material becomes an access problem, the State of Secrets Sprawl 2025 and CI/CD pipeline exploitation case study show how mismanaged secrets turn into server compromise. Public-key login is different from password login, but it still depends on protecting the proof material on the client side.
What changes if the private key is exposed
The security difference is asymmetrical. If the public key is copied, nothing is immediately lost because it is already meant to be shared. If the private key is copied, an attacker can authenticate as the holder of that key until the key is revoked or replaced. That makes private-key compromise an account compromise issue, not just a file exposure issue.
The practical consequence is that private-key protection is about more than file permissions. Teams need passphrases, hardware-backed storage where appropriate, rotation, revocation, and inventory of where keys are installed. The server copy should be treated as a trust record that may need removal when an employee leaves, a workload is retired, or a key is suspected of exposure.
Key management guidance from NIST SP 800-57 Key Management supports that lifecycle view, while NIST SP 800-63 Digital Identity Guidelines reinforces the distinction between an authenticator and the verifier that checks it. For certificate-based variants of the same trust model, the CA/Browser Forum provides the public trust and revocation context that helps explain why proof material and verifier material are handled separately.
One useful operational signal is whether the same private key is reused across many hosts, environments, or automation flows. Reuse increases blast radius, complicates offboarding, and makes revocation slower than it should be. The public key can be broadly distributed; the private key should never become broadly reachable.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
NIST SP 800-63, CIS Controls v8 and NIST CSF 2.0 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST SP 800-63 | Authenticator and Verifier Separation — Digital Identity Assurance and Authenticator Handling | SSH keys split proof material from the verifier-side record. |
| Phishing Resistance and Authenticator Strength — Phishing-Resistant Authentication | Key-based SSH login relies on possession of a private key for proof. | |
| Recommendation — Keep the private key as the authenticator and treat the server entry as the verifier-side trust record. Prefer phishing-resistant authenticators when you rely on key-based remote access. | ||
| CIS Controls v8 | 6 — Access Control Management | SSH public keys grant access and must be revoked when no longer needed. |
| 4 — Secure Configuration of Enterprise Assets and Software | authorized_keys and private-key storage depend on secure host and client configuration. | |
| Recommendation — Review and remove stale SSH access grants as part of access control management. Harden key storage paths and SSH configuration to reduce exposure of authentication material. | ||
| NIST CSF 2.0 | PR.AA — Identity Management, Authentication, and Access Control | SSH keys are an identity and access mechanism that must be governed. |
| PR.DS — Data Security | Private keys are sensitive authentication data that must be protected at rest. | |
| Recommendation — Control SSH key issuance, authentication and revocation under identity and access policy. Protect private keys with strong storage controls and limit where they can be copied. | ||
Practitioner Guidance
What to verify: Confirm that the private key is stored only on the intended client, protected by file permissions and, where appropriate, a passphrase or hardware-backed authenticator. On the server, verify that only the public key appears in authorized_keys and that stale entries are removed when access changes.
Decision rule: If you suspect the private key has been exposed, treat it like compromised login proof material, not like a harmless configuration file. Revoke the server-side public key entry, rotate the keypair, and check for other systems where the same key was authorised.
What practitioners underestimate: The server-side public key is not the secret, but it still represents an access grant. The real control problem is lifecycle management, because a valid public key entry remains effective until it is deliberately removed.
Practitioner takeaway: ssh key security is about preserving the secrecy of the private key and the governance of the public key, because either side fails differently, but only the private side can be used to impersonate the account.
Related resources from NHI Mgmt Group
- What is the difference between private key encryption and public key encryption for practitioners?
- What is the difference between SSH password authentication and public key authentication?
- What is the difference between SSH public key authentication and SSH certificate-based authentication?
- What is the difference between installing Python packages from a public PyPI server and using a private PyPI server?
