IPsec Security Guide: IKEv2, Algorithms and Hardening

Most IPsec weaknesses are not in the protocol but in the proposal list, the authentication method and the gateway software that someone configured years ago and never revisited. IPsec with IKEv2 remains the standard choice for site-to-site tunnels between different vendors' firewalls and cloud gateways, and it still carries a large share of remote access traffic. This guide goes below the protocol comparison level: it sets out which algorithms the IETF now marks MUST and MUST NOT, how to choose between certificates, pre-shared keys and EAP, how to prepare tunnels for post-quantum key exchange, and how to harden, monitor and patch the IKE daemon itself. It is written for network and security engineers, firewall administrators and auditors who own or assess IPsec configurations. For choosing between IPsec, WireGuard and OpenVPN in the first place, see the VPN Protocols Compared: WireGuard, IPsec, OpenVPN Guide.

Key takeaways

• IKEv1 is finished. RFC 9395 (April 2023) deprecates IKEv1 and moves its defining RFCs to Historic status; every remaining IKEv1 tunnel is a migration item, and IKEv1 code paths keep appearing in CISA's Known Exploited Vulnerabilities catalog.
• The proposal set is the security boundary. RFC 8247 (IKE) and RFC 8221 (ESP) list what must and must not be negotiated; a tunnel that still accepts DES, MD5 or Diffie-Hellman group 1 can be downgraded to it.
• Authentication decides the blast radius. Certificates with revocation checking or EAP behind a server certificate scale and resist offline cracking; short, human-chosen pre-shared keys do neither.
• Post-quantum work is standardised, not yet default. RFC 9370 lets IKEv2 add up to seven extra key exchanges, and RFC 8784 mixes a post-quantum pre-shared key into the derivation, so long-lived tunnels can start hedging against harvest-now-decrypt-later today.
• The gateway is the target. Hardening the IKE service, restricting UDP 500 and 4500, patching quickly and logging negotiation outcomes matter as much as cryptography.

How IPsec and IKEv2 fit together

IPsec is two layers. IKEv2 (RFC 7296, Internet Standard STD 79) authenticates the peers and negotiates keys; the Encapsulating Security Payload (ESP) then protects the traffic. Knowing which layer a setting belongs to avoids the most common audit confusion, where a strong ESP cipher hides a weak IKE proposal.

• IKE_SA_INIT: the first exchange agrees the cryptographic suite, swaps nonces and runs a Diffie-Hellman exchange. It is unauthenticated, which is why the responder must parse attacker-controlled input and why RFC 7296 provides a COOKIE notification for responders facing many half-open IKE SAs.
• IKE_AUTH: the peers prove their identity (signature, pre-shared key or EAP), exchange identities inside the encrypted channel and create the first Child SA with its traffic selectors (TSi and TSr). The responder may narrow the selectors the initiator proposes.
• CREATE_CHILD_SA: creates further Child SAs and rekeys both IKE and Child SAs. An optional KE payload gives the Child SA a fresh Diffie-Hellman exchange, which is how perfect forward secrecy (PFS) is configured for ESP.
• Ports and protocols: IKE uses UDP 500. When a NAT is detected, IKE and ESP move to UDP 4500 using the encapsulation in RFC 3948, so both share one NAT mapping. Without NAT, ESP travels as IP protocol 50. AH (IP protocol 51) authenticates without encrypting and does not survive NAT well; RFC 8221 prefers ESP even for integrity-only use.
• Tunnel versus transport mode: gateways use tunnel mode, which wraps the whole original packet. Transport mode protects only the payload and suits host-to-host protection, for example server-to-server traffic inside a data centre.

NAT traversal brings its own policy edge cases. RFC 3948 warns that peers behind different NATs may present overlapping private addresses to a gateway, and that several clients behind one NAT look like a single address to a transport-mode server, which must then not allow a mix of protected and cleartext traffic for the same flow. The NAT and Its Security Implications Guide covers the NAT side of this in depth.

Algorithm baseline: what to allow and what to remove

RFC 8247 and RFC 8221 use requirement levels, including MUST- (still required for interoperability but expected to drop) and SHOULD+ (expected to become MUST). Read MUST as "your implementation will support this", not "you must enable it": policy should allow only the strongest option both ends support.

Two points are easy to miss. First, ESP with an AEAD cipher and integrity set to none is both the fastest and the cleanest option, whereas ESP combined with AH is NOT RECOMMENDED by RFC 8221. Second, the strongSwan project notes that its default proposals still include 3DES and SHA-1 because some Windows clients rely on them, so "defaults" in any implementation are an interoperability compromise, not a security baseline. Write explicit proposals on both peers and check the negotiated result, not just the configured list. Map any negotiated weakening to MITRE ATT&CK T1689 Downgrade Attack when you assess a gateway.

Regulated environments: NIST SP 800-77 Rev. 1 (June 2020) is the US federal reference for IPsec VPNs and expects FIPS-validated cryptographic modules, which usually means AES-GCM with NIST curves or MODP groups rather than ChaCha20-Poly1305. For national security systems, strongSwan documents CNSA-aligned proposals such as AES-256 with SHA-384 and ECP-384.

Authentication: certificates, pre-shared keys and EAP

The method you choose in IKE_AUTH determines how a stolen secret is used against you and how hard it is to revoke one peer.

• PSK strength: strongSwan states that PSK-based IKEv2 authentication with user-chosen passwords is open to offline dictionary attack by a man-in-the-middle, and recommends random keys of more than 256 bits. A 12-character "partner password" agreed by email does not meet that bar.
• IKEv1 Aggressive Mode: sends a hash derived from the PSK before the peer is authenticated, enabling offline cracking. If any IKEv1 remains during migration, disable Aggressive Mode first.
• Identity binding: configure each peer to accept only the expected remote identity (certificate subject or SAN, or IKE ID) rather than any certificate from the trusted CA. Without this, any device holding a certificate from your VPN CA can impersonate any gateway.
• Revocation: enable CRL or OCSP fetching on the gateway and decide in advance whether an unreachable responder fails open or closed. Fail-open is common and means a revoked certificate still connects during an outage.

Where identity fits

Every IPsec gateway, branch firewall and cloud VPN connection is a non-human identity: it holds a private key or a shared secret, it authenticates without a person present, and it often outlives the engineer who built it. Treat those credentials as you would any machine credential.

• Inventory: record every tunnel with its owner, peer identity, authentication method, certificate expiry or PSK last-rotation date and the business service it supports.
• Lifecycle: issue gateway certificates from a managed CA with defined validity, renew automatically and revoke when a device is decommissioned or a partner relationship ends. See the Machine Identity, PKI and Certificate Lifecycle Guide.
• Secrets storage: keep PSKs out of tickets, emails and configuration backups in shared drives. Configuration exports often contain PSKs in reversible form.
• Remote access users: the user step needs phishing-resistant MFA and should be fed by the identity provider. See the Remote Access Identity Guide.

Preparing tunnels for post-quantum key exchange

A quantum computer able to run Shor's algorithm would break the (EC)DH exchange in IKE_SA_INIT, exposing any recorded traffic. For site-to-site tunnels that carry data with a confidentiality life of ten years or more, this harvest-now-decrypt-later risk is already relevant. Two IETF standards give options.

Check vendor support before planning: some open-source implementations already document proposals that pair a classical group with ML-KEM, while many appliances do not yet. Start with an inventory of tunnels by data sensitivity and lifetime, pilot hybrid key exchange on a tunnel you control at both ends, and confirm fragmentation and MTU behaviour through any middleboxes. See the Post-Quantum Cryptography Migration Guide and, for the wider debate, the QKD vs Post-Quantum Cryptography Guide.

Hardening the gateway and the IKE service

The IKE daemon parses unauthenticated packets from the internet, so its implementation bugs are directly reachable. CISA's Known Exploited Vulnerabilities catalog lists exploited IKE flaws across vendors, including CVE-2026-50751 (Check Point Security Gateway, an IKEv1 authentication bypass allowing a remote access VPN connection without a valid password, added June 2026), CVE-2025-14733 (WatchGuard Fireware OS iked out-of-bounds write affecting IKEv2 mobile and branch office VPN with a dynamic gateway peer), CVE-2026-33824 (Microsoft IKE Service Extensions double free) and CVE-2016-6415 (Cisco IKEv1 memory disclosure).

• Reduce reachability: for site-to-site, allow UDP 500, UDP 4500 and ESP only from known peer addresses. Remote access gateways cannot do this, which makes their patch speed the main control. Disable IKEv1 listeners entirely where the platform allows it.
• Patch on KEV timelines: subscribe to vendor advisories for the VPN feature specifically and treat a KEV-listed IKE flaw as an emergency change. Map exposure to ATT&CK T1190 Exploit Public-Facing Application and T1133 External Remote Services.
• Prevent cleartext leakage: on route-based or policy-based tunnels, make sure traffic is dropped, not routed in the clear, while an SA is down. strongSwan's start_action = trap and explicit drop policies for private ranges are examples; on firewalls, add a deny rule for the protected subnets on the internet-facing interface.
• Enable PFS on Child SAs: include a key exchange method in ESP proposals so each Child SA gets independent keys, and set realistic lifetimes so rekeying happens.
• Narrow traffic selectors: protect only the subnets each peer needs. A 0.0.0.0/0 selector to a partner turns a tunnel into a full network bridge, the condition described by ATT&CK T1599 Network Boundary Bridging.
• Filter inside the tunnel: decrypted traffic is still untrusted. Apply firewall policy to the tunnel interface as you would to any partner network. See the Network Segmentation Guide.
• Protect the management plane: keep gateway administration off the internet-facing interface and behind MFA. See the VPN Security Guide.

Implementation: a phased hardening approach

1. Inventory (weeks 1 to 2): export every IKE gateway configuration. List tunnels with IKE version, proposals, authentication method, PSK age, certificate expiry, traffic selectors, owner and peer organisation.
2. Observe what is negotiated: collect logs of the proposal actually selected per tunnel for two weeks. Peers that negotiate 3DES, SHA-1 or group 2 are your remediation list; configured but unused weak proposals can be removed immediately.
3. Define two profiles: a target profile (for example IKE AES-256-GCM, PRF SHA-384, ECP-384, ESP AES-256-GCM with PFS ECP-384) and a time-limited legacy profile for named partners, each with an expiry date recorded as a risk acceptance.
4. Pilot: change one internal tunnel where you control both ends. Confirm rekey, DPD behaviour, MTU and fragmentation, and failover. Keep the previous configuration exported for rollback.
5. Migrate partners: agree a change window with each external peer, swap PSKs for certificates or random keys, and move IKEv1 peers to IKEv2. Roll back by re-enabling the previous proposal on that tunnel only, never globally.
6. Remove legacy: when the legacy profile's expiry passes, delete it and disable IKEv1. Re-scan from outside to confirm.
7. Sustain: add the configuration to automated drift checks and review proposals at least annually against RFC 8247 and RFC 8221 updates.

Decision guidance

• Policy-based versus route-based: route-based tunnels (virtual tunnel interfaces) make firewall policy, routing and monitoring clearer and suit most new builds. Policy-based tunnels remain common for interoperability with older peers but are harder to audit as selectors multiply.
• Certificates versus PSK: choose certificates once you have more than a handful of peers or any remote access users. PSKs are defensible only for a small number of fixed site-to-site peers, with unique random keys.
• IPsec versus alternatives: for user remote access, many organisations are moving to identity-aware access rather than network tunnels. See the Secure Remote Access Beyond VPN Guide. For automated machine overlays where FIPS is not required, see the WireGuard Security Guide.
• Small organisations: a cloud provider's managed VPN gateway or a single firewall vendor's defaults, tightened to the target profile, is reasonable. The priorities are patching, removing IKEv1 and replacing weak PSKs.
• Large organisations: centralise proposal templates, automate certificate issuance and renewal, and gate configuration changes through review so one team cannot reintroduce legacy suites.

Detection and measurement

Most gateways log IKE negotiation success and failure, the selected proposal, authentication results, rekeys and dead peer detection events. On Windows, IKE and AuthIP are provided by the IKEEXT service, and IPsec audit events are written to the Security log when the relevant audit subcategories are enabled. Forward these to your SIEM and alert on:

• Bursts of IKE_SA_INIT from many sources, or many half-open SAs (scanning or exhaustion attempts).
• Authentication failures against a remote access gateway from new geographies, especially followed by a success (T1078 Valid Accounts).
• A tunnel negotiating a weaker proposal than its baseline, or any IKEv1 negotiation after IKEv1 should be off.
• Changes to tunnel configuration, traffic selectors or NAT rules on the gateway outside a change window.
• Gateway process crashes or restarts of the IKE service, which may indicate exploitation attempts.

Framework mapping

Common failure patterns

• Legacy proposals kept "just in case": a single partner needed 3DES and group 2 in 2016, so the global proposal list still accepts them. Fix it per tunnel and record an expiry.
• One PSK for many sites: branch rollouts reuse a single pre-shared key, so one leaked configuration backup exposes every branch. Use certificates or unique keys per peer.
• Accounts and profiles that never die: the Colonial Pipeline ransomware attack began with a legacy VPN profile that was no longer used, had no MFA and whose password had leaked. The protocol was not the weakness; the lifecycle was.
• Slow patching of the IKE service: IKE flaws in firewalls and Windows have reached the KEV catalog repeatedly. A gateway that is three releases behind is an initial access path regardless of its cipher suite.
• Any-to-any traffic selectors with no inner filtering: a compromised partner network then has the same reach as your own LAN.
• Revocation not enforced: CRL fetching is disabled or fails open, so certificates from lost or decommissioned devices still authenticate.

Practitioner checklist

• Inventory every IPsec tunnel with owner, peer, IKE version, proposals, authentication method and credential expiry.
• Disable IKEv1 and Aggressive Mode; plan each remaining IKEv1 peer's migration with a date.
• Remove every algorithm marked MUST NOT or SHOULD NOT in RFC 8247 and RFC 8221, and verify what each tunnel actually negotiates.
• Prefer AEAD (AES-GCM) for IKE and ESP, with PFS on Child SAs.
• Replace human-chosen PSKs with certificates or unique random keys of at least 256 bits, stored in a secrets manager.
• Enforce revocation checking and per-peer identity constraints on certificate-authenticated tunnels.
• Restrict UDP 500, UDP 4500 and ESP to known peers for site-to-site, and patch remote access gateways within KEV timelines.
• Narrow traffic selectors and filter decrypted traffic on the tunnel interface.
• Log negotiation outcomes and configuration changes to the SIEM, and alert on downgrades and IKEv1 use.
• Pilot RFC 9370 hybrid key exchange or RFC 8784 PPKs on long-lived, high-sensitivity tunnels.

Standards and references

• IETF: RFC 7296, Internet Key Exchange Protocol Version 2 (IKEv2)
• IETF: RFC 8247, Algorithm Implementation Requirements and Usage Guidance for IKEv2
• IETF: RFC 8221, Cryptographic Algorithm Implementation Requirements and Usage Guidance for ESP and AH
• IETF: RFC 9395, Deprecation of the IKEv1 Protocol and Obsoleted Algorithms
• IETF: RFC 9370, Multiple Key Exchanges in IKEv2
• IETF: RFC 8784, Mixing Preshared Keys in IKEv2 for Post-quantum Security
• NIST: SP 800-77 Rev. 1, Guide to IPsec VPNs
• strongSwan: Security Recommendations

Related NHI Mgmt Group resources: VPN Protocols Compared: WireGuard, IPsec, OpenVPN Guide · WireGuard Security Guide · VPN Security Guide · NAT and Its Security Implications Guide · Post-Quantum Cryptography Migration Guide