RSA vs ECC: choosing key types for certificates, SSH and HSMs

Every new certificate, SSH key, signing key and KMS key forces a choice between RSA and elliptic curve cryptography, and most organisations make it by default rather than by policy. The choice affects key size, handshake cost, compatibility with old clients and appliances, what your HSM or cloud KMS can hold, and how much work the post-quantum transition will be. This guide compares the two key types for the places practitioners actually meet them: public and private TLS certificates, SSH, code and token signing, and hardware-protected keys. It is for PKI owners, platform engineers and security architects writing a key type policy. For how elliptic curves work, see the Elliptic Curve Cryptography guide; for the choice between the two curve signature schemes, see ECDSA vs EdDSA.

Key takeaways

• ECC is the default for new keys. P-256 gives 128-bit security with a 256-bit key; RSA needs 3072 bits for the same strength under NIST SP 800-57 Part 1 Table 2, and the RSA 2048 most estates use sits at 112 bits.
• RSA is still the compatibility key. Keep RSA where old clients, appliances, smart cards or partner integrations cannot handle ECDSA, and serve both certificate types where your servers allow it.
• Public TLS limits you to three curves. The CA/Browser Forum Baseline Requirements permit RSA keys of at least 2048 bits and ECDSA on P-256, P-384 or P-521 only; Ed25519 is common for SSH but not permitted for publicly trusted server certificates.
• Both are on the same quantum clock. NIST IR 8547 (final, October 2026) deprecates quantum-vulnerable RSA, ECDSA, EdDSA and ECDH after 2030 and disallows them after 2035, so the choice between RSA and ECC is a choice about the next decade, not the next three.
• Policy beats preference. Write a key type matrix per use case, enforce it at issuance (CA templates, KMS key policies, SSH server settings) and measure the estate against it.

Short answer

For new deployments choose ECC: ECDSA P-256 for TLS certificates and most signing, Ed25519 for SSH user and host keys, and X25519 or P-256 ECDHE for key exchange (which TLS 1.3 and modern SSH already do regardless of certificate type). Choose RSA, 3072 bits or more for anything expected to live past 2030, where a relying party cannot verify ECDSA, where a hardware token or appliance only supports RSA, or where you need RSA encryption for an existing protocol. Run both side by side where clients are mixed, and treat either choice as temporary: plan the post-quantum move alongside it.

Comparison at a glance

Strength values are from NIST SP 800-57 Part 1 Rev. 5 Table 2, which also notes that 80-bit security (RSA 1024) is no longer considered sufficient.

RSA in depth: threat model, architecture and failure modes

Threat model. RSA's security rests on the difficulty of factoring the modulus. Classical attacks scale with modulus size, which is why the required size grows so quickly: doubling security from 128 to 256 bits takes the modulus from 3072 to 15360 bits. In practice RSA keys fail far more often through implementation than through factoring: weak random number generation at key creation, unpadded or PKCS#1 v1.5 encryption exposed to padding oracle attacks, and keys copied off servers. Theft of private key files is catalogued by MITRE ATT&CK as T1552.004 Private Keys, and deliberately weakening key length as T1600.001 Reduce Key Space.

Architecture in words. An RSA key pair is a modulus and a public exponent, with the private exponent kept secret. The same key can sign and decrypt, which is why older TLS used RSA certificates for key transport: the client encrypted a secret to the server's public key. TLS 1.3 (RFC 8446) removed static RSA and Diffie-Hellman cipher suites, so today an RSA certificate in TLS 1.3 only signs the handshake, with ephemeral ECDHE providing the session key. RFC 8446 also limits RSASSA-PKCS1-v1_5 values to signatures in certificates; handshake signatures use RSA-PSS (rsa_pss_rsae_sha256 and related schemes).

Failure modes.

• Undersized keys: RSA 1024 still appears on old appliances and internal CAs. The CA/Browser Forum requires at least 2048 bits, evenly divisible by 8, for publicly trusted certificates.
• Long-lived 2048-bit keys: 2048 bits is 112-bit security. A root or code signing key created today at 2048 bits will be relied on into the period NIST has already scheduled for deprecation.
• Dual use: one key used for both signing and decryption widens the blast radius of any oracle attack. Separate keys by usage; cloud KMS services make usage a fixed property of the key.
• SHA-1 signatures: OpenSSH 8.8 disabled RSA signatures using SHA-1 by default, and the CA/Browser Forum sunset remaining SHA-1 use in certificates and CRLs from 15 September 2026. Systems still negotiating the legacy ssh-rsa signature scheme break on upgrade.

Controls. Generate in an HSM or KMS where the key is valuable, set 3072-bit minimum for new long-lived keys, use RSA-PSS for new signing and RSA-OAEP for any remaining encryption, set RequiredRSASize on OpenSSH servers (added in 9.1), and keep RSA public exponents in the range the Baseline Requirements recommend (2¹⁶+1 or higher).

ECC in depth: threat model, architecture and failure modes

Threat model. ECC rests on the elliptic curve discrete logarithm problem, which has no known sub-exponential classical attack on well-chosen curves. That is why a 256-bit key gives roughly the strength of a 3072-bit RSA key. The practical risks are in implementation: ECDSA requires a fresh, secret, unbiased nonce for every signature, and reusing or leaking it exposes the private key. Accepting public keys without checking they lie on the curve enables invalid curve attacks. Deterministic nonce generation (as in EdDSA or deterministic ECDSA) removes the first risk; the ECDSA vs EdDSA guide covers that choice.

Architecture in words. An ECC private key is a scalar; the public key is a point on an agreed curve. ECC does not encrypt directly: it signs (ECDSA, EdDSA) or agrees a shared secret (ECDH, X25519) that a symmetric cipher then uses. In TLS 1.3 an ECDSA certificate signs the handshake (ecdsa_secp256r1_sha256 and related schemes), while the key exchange is usually X25519 or P-256 ECDHE whatever the certificate type. In SSH, ssh-keygen in OpenSSH 9.9 documents Ed25519 as the default key type.

Failure modes.

• Curve mismatch: picking Ed25519 for a public TLS certificate (not permitted under the Baseline Requirements), or secp256k1 for anything other than blockchain use.
• Weak randomness in signing: ECDSA on devices with poor entropy sources, or custom implementations that derive nonces badly.
• Old clients: ECDSA-only deployments that silently cut off legacy clients, payment terminals or partner systems, discovered only after cutover.
• FIPS scope surprises: curves accepted by a library but outside your validated module's approved set; check against SP 800-186 and the module's security policy (see the FIPS 140-3 and Cryptographic Module Validation Guide).

Controls. Use vetted libraries and platform crypto, never custom curve code; prefer deterministic signatures where the policy allows; restrict curves to P-256 and P-384 for X.509 and Ed25519 for SSH; validate peer public keys (the Baseline Requirements recommend CAs check ECDSA keys with the NIST SP 800-56A full or partial public key validation routines).

Compatibility and HSM support

Hardware and cloud key services decide what you can actually deploy. All three major cloud key services hold both RSA and ECC keys, including HSM-protected options. Platforms are listed alphabetically; the list describes each service's documented key specs as of November 2026 and is not a ranking.

Availability notes. AWS KMS: key spec and key usage are fixed when the key is created. Azure Key Vault: HSM-protected keys (RSA-HSM, EC-HSM) require a Premium vault or Managed HSM; Standard vaults hold software-protected keys. Google Cloud KMS: each algorithm is offered at SOFTWARE and/or HSM protection level as listed on its algorithms page, with P-256, P-384 and all RSA algorithms at both levels. See the AWS KMS Security Guide, the Azure Key Vault Security Guide and the Cloud KMS and Cloud HSM Guide.

For on-premises HSMs, smart cards and hardware tokens, check the specific model's supported mechanisms and the curves listed in its FIPS 140-3 security policy before setting policy: support for P-256 is common, while Ed25519 and RSA sizes above 4096 vary by model and firmware. See the HSM Deployment and Operations Guide.

Choosing per use case

• Small organisations: accept platform defaults that are already ECC (certificate automation clients, OpenSSH), set a 3072-bit floor for any new RSA, and spend effort on inventory rather than tuning.
• Large or regulated organisations: publish a key type matrix like the one above, tie it to FIPS-validated modules where required, and enforce it in CA templates, KMS policies and SSH configuration management.

Post-quantum outlook for each

A large enough quantum computer running Shor's algorithm would break both RSA and ECC, so neither is a quantum hedge. NIST IR 8547, finalised in October 2026, says NIST will deprecate all quantum-vulnerable digital signature algorithms after 2030 and disallow their use under NIST standards and guidelines after 2035, with the same dates for quantum-vulnerable key establishment including elliptic curve Diffie-Hellman and RSA key transport. It also records an earlier federal requirement for high value assets and high impact systems to adopt post-quantum key establishment by 31 December 2030.

• Key exchange moves first: recorded traffic can be decrypted later, so hybrid key exchange matters before signatures. OpenSSH 10.0 made mlkem768x25519-sha256 the default key agreement. See the Hybrid Post-Quantum TLS Key Exchange Guide and the PQC SSH Migration Guide.
• Signatures follow trust chains: certificate and code signing keys move as CAs, platforms and relying parties support ML-DSA and SLH-DSA. See the ML-DSA and SLH-DSA Post-Quantum Signatures Guide.
• What this means for RSA vs ECC today: choose the key type that is cheapest to replace. Short-lived ECC certificates issued by automation are easier to swap than long-lived RSA keys embedded in devices. Do not create new long-lived RSA 2048 keys: they combine the lowest current strength with the longest exposure.

Worked scenario: one estate, three key populations

A software company runs public websites, an internal service mesh and a fleet of network appliances.

1. Public web: it moves public certificates to ECDSA P-256 and keeps an RSA 2048 certificate alongside for six months, logging which clients negotiate RSA. Under 1 percent do, mostly an old partner integration that is asked to upgrade; the RSA certificate is then retired.
2. Service mesh: the internal CA issues ECDSA P-256 workload certificates with lifetimes of hours, so a later move to post-quantum signatures is a CA configuration change, not a fleet re-key.
3. Appliances and roots: several appliances only accept RSA for SSH and management certificates. They stay on RSA 3072, are listed as exceptions with an owner and an end date, and are fenced off behind a management network. The offline root is re-planned as part of the post-quantum hierarchy rather than re-keyed to another classical algorithm.

Where identity fits

Most RSA and ECC private keys in an estate belong to machines: TLS server keys, workload certificates, SSH host keys, CI signing keys and token signing keys. Key type policy is therefore a non-human identity control. Inventory keys by owner, algorithm, size and expiry, and treat an unknown RSA 1024 key the same way you would an unowned privileged account. See the Machine Identity, PKI and Certificate Lifecycle Guide and the SSH Key Management Guide.

Rolling out a key type policy

1. Inventory: scan TLS endpoints, certificate transparency logs for your domains, internal CA databases, SSH authorized_keys and host keys, and KMS key lists for algorithm and size. See the Cryptographic Inventory and CBOM Guide.
2. Set the matrix: agree defaults and exception rules per use case, with 3072-bit minimum for new RSA and P-256, P-384 or Ed25519 for ECC.
3. Enforce at issuance: update CA certificate templates and ACME client defaults, KMS key creation policies, and SSH server settings (PubkeyAcceptedAlgorithms, RequiredRSASize).
4. Pilot with dual certificates: serve ECDSA and RSA side by side and log negotiated signature schemes before removing RSA.
5. Cut over by ring: internal services first, then public endpoints, then partner integrations, each with a rollback path (re-enable the RSA certificate or accept the RSA key type) kept for one renewal cycle.
6. Retire and record: revoke replaced keys, remove old authorized_keys entries, and record remaining RSA exceptions with owners and end dates in the post-quantum plan.

Detection and measurement

• Key type mix: percentage of certificates and SSH keys by algorithm and size, trended monthly.
• Below floor: count of RSA keys under 2048 bits (target zero) and new RSA keys under 3072 bits created since the policy date.
• RSA negotiation share: percentage of TLS handshakes still using the RSA certificate on dual-certificate endpoints, and SSH sessions using RSA keys.
• Exceptions: number of documented RSA-only exceptions with an owner and end date, and the number past their end date.
• Legacy signature use: SSH logins still attempting ssh-rsa (SHA-1) signatures, visible in server logs after upgrades.
• Post-quantum readiness: percentage of key populations with a named migration path and target date.

Common failure patterns

• Defaulting to RSA 2048 for everything: including roots and code signing keys that will outlive 2030.
• ECDSA cutover without telemetry: removing RSA certificates before measuring which clients need them, then rolling back under pressure.
• Using SSH key types on X.509: requesting Ed25519 public certificates that no public CA can issue under the Baseline Requirements.
• Ignoring the HSM: setting a curve or key size policy the HSM, token or KMS tier in use cannot hold, which pushes keys into software.
• Treating ECC as quantum safe: smaller keys are not a post-quantum defence; ECC is on the same NIST timeline as RSA.
• Re-keying twice: spending a cycle moving long-lived roots from RSA to ECC when the next move to post-quantum is already scheduled.

Framework mapping

Practitioner checklist

• Inventory every certificate, SSH key and KMS key by algorithm, size, owner and expiry.
• Set ECDSA P-256 as the default for new TLS certificates and Ed25519 for new SSH keys.
• Set a 3072-bit minimum for new RSA keys and remove all keys below 2048 bits.
• Run dual RSA and ECDSA certificates on public endpoints until telemetry shows RSA is no longer needed.
• Check HSM, token and KMS tier support before writing curve or size policy.
• Configure OpenSSH servers to refuse SHA-1 RSA signatures and set RequiredRSASize.
• Record every RSA-only exception with an owner and end date.
• Map each key population to its post-quantum migration path against the 2030 and 2035 NIST dates.

Standards and references

• NIST: SP 800-57 Part 1 Rev. 5 Recommendation for Key Management
• NIST: IR 8547 Transition to Post-Quantum Cryptography Standards
• CA/Browser Forum: Baseline Requirements for TLS Server Certificates
• IETF: RFC 8446 The Transport Layer Security (TLS) Protocol Version 1.3
• OpenSSH: Release notes
• AWS: KMS key spec reference
• Microsoft: About Azure Key Vault keys
• Google Cloud: Cloud KMS key purposes and algorithms

Related NHI Mgmt Group resources: Elliptic Curve Cryptography · SSL/TLS Certificate Types · PQC SSH Migration Guide · TLS Server Configuration Baseline Guide · Post-Quantum Cryptography Migration Guide