Encapsulating Security Payload (ESP)

What ESP does inside IPsec tunnels

Encapsulating Security Payload sits at the transport layer of IPsec and is the part that protects data as it crosses untrusted networks. In practice, it is what gives VPN traffic confidentiality and, when configured to do so, integrity and replay protection.

ESP is often paired with tunnel mode for site-to-site connectivity and remote access VPNs, where it helps preserve the privacy of internal traffic while still allowing routing over shared infrastructure. Because ESP protects the packet payload, it is central to the security value of the tunnel rather than a cosmetic add-on.

How ESP protects traffic in transit

ESP can encrypt payloads, authenticate packets, or do both, depending on the negotiated transform. That flexibility is useful, but it also means the real security property depends on the peers agreeing to strong algorithms and on the tunnel being established with the intended endpoints.

When ESP is working correctly, an observer should not be able to read the protected data, and a tampered packet should fail validation. That is why ESP is commonly used where confidentiality matters more than simple reachability, such as branch connectivity, partner links, and administrative access over the public internet.

The Network Ports and Protocols Security Guide is a useful companion for understanding where ESP fits alongside other network protocols and tunnel types.

Why ESP is valuable in VPN and site-to-site designs

ESP matters because it protects traffic without requiring application changes. That makes it one of the most practical controls for securing links between networks that still need to exchange sensitive data, especially when both sides are outside a single trusted perimeter.

Its value is strongest when the tunnel endpoints are tightly controlled and the cryptographic profile is current. If peers are misconfigured, or if a tunnel is established with weak or unexpected parameters, the protection can degrade even though the tunnel still appears to function.

For broader control context, NIST SP 800-53 Rev 5 Security and Privacy Controls is relevant because IPsec deployments intersect with access control, cryptographic protection, auditability, and secure configuration.

What defenders should validate and monitor

ESP is not a “set and forget” control. Its security depends on endpoint validation, strong keying material, and ongoing attention to negotiation failures, because repeated failed handshakes can signal misconfiguration, interoperability problems, or active probing.

Defenders should also treat tunnel inventory as part of security hygiene. If a peer, route, or certificate changes without review, the tunnel may continue to pass traffic while no longer providing the protection the organisation expects.

The NIST Cybersecurity Framework 2.0 and the NIST Privacy Framework both support the operational discipline needed to inventory protected channels, monitor control health, and reduce exposure from poorly governed network links.

Risk and Threat Considerations

ESP is only as strong as the cryptography and peer trust behind it. Weak keys, stale algorithms, endpoint spoofing, or tunnel misconfiguration can turn a supposedly protected link into a false assurance problem, where traffic still flows but the security boundary has been weakened.

Failure mechanism: Attackers or misconfigured systems can exploit poor peer validation, outdated cryptographic settings, or tunnel negotiation failures to downgrade protection, intercept traffic, or create blind spots in monitoring.

Impact: The result can be exposure of sensitive in-transit data, unauthorized access through a trusted tunnel, or delayed detection of compromise because encrypted traffic obscures inspection and weak tunnels may still look operational.

Practitioner Guidance

Why practitioners should care: ESP is a control that protects both confidentiality and trust in network interconnects, so it should be managed as a security boundary rather than a routing detail. In VPN and site-to-site designs, the cryptographic posture and peer identity assumptions deserve the same attention as firewall rules.

What to watch for: Repeated negotiation failures, unexpected tunnel re-establishment, unusual peer changes, or sudden shifts in traffic volume through a tunnel can all indicate configuration drift or abuse of the protected channel.

Practitioner takeaway: Treat ESP as a living control, verify that the tunnel is established with the intended peers and algorithms, and review failures as a potential security signal, not just an availability issue.

Related resources from NHI Mgmt Group

• Static Application Security Testing
• What breaks when email security tools cannot see the full rendered payload?
• Why do traditional email security tools miss payload-less BEC attacks?
• How should security teams defend against payload splitting in AI systems?