How does Encapsulating Security Payload (ESP) work?

How ESP fits into IPsec packet protection

ESP is the IPsec protocol that protects the payload of an IP packet, and in common deployments it can also hide most of the original traffic from passive observers. The sender wraps the protected data in an ESP header and trailer, then applies encryption, integrity protection, or both, depending on the negotiated security association and transform set.

In transport mode, ESP protects the upper-layer payload while leaving the original IP header mostly visible for routing. In tunnel mode, ESP protects an entire inner packet and adds a new outer IP header, which is why it is commonly used for site-to-site VPNs and remote-access tunnels. That mode choice affects what metadata remains exposed on the wire.

ESP is negotiated through IKE, which establishes the keys, algorithms, and security parameters used by both sides. The practical security outcome depends on the negotiated cipher suite, the freshness of keys, and whether authentication is actually enabled, because ESP can be configured for confidentiality only, or for confidentiality plus integrity and origin authentication.

What ESP changes on the wire

ESP alters both the format and the security properties of traffic in transit. It protects the payload, but it does not make traffic disappear from the network, because observers can still often see endpoints, packet sizes, timing, and tunnel endpoints. Those remaining clues matter when you are assessing traffic analysis risk or trying to understand what a monitoring system can still inspect.

Most ESP deployments rely on anti-replay protection as well, so a receiver can reject duplicated packets that arrive out of sequence. That matters because encryption alone does not stop replay, and replay resistance is part of what makes the protected channel usable for stateful protocols and sensitive application flows. The result is a channel that is both harder to read and harder to tamper with in transit.

For network defenders, ESP security is only as strong as the weakest negotiated parameter. Weak transforms, long-lived keys, or permissive tunnel policy can reduce the practical value of the protection even when ESP is technically enabled. The same is true when encryption is configured but integrity is omitted, because confidentiality alone does not tell you whether the packet was altered in flight.

Where ESP is strong, and where it is easy to misunderstand

ESP is strong at protecting data in motion between trusted endpoints or between security gateways, but it is not a substitute for endpoint security, access control, or secure application design. If a host is already compromised, ESP will still faithfully protect malicious traffic leaving that host. It also does not remove the need to secure keys, tunnel policies, and the systems that terminate the tunnel.

ESP can be misunderstood as a blanket guarantee of trust, yet it only protects the traffic path that is actually negotiated. If a routing change, policy error, or misconfigured selector sends sensitive traffic outside the tunnel, ESP does nothing for that path. Likewise, if administrators choose a mode or transform set that does not match the use case, the result may be secure in theory but weak in practice.

In many environments, the real design question is not whether to use ESP, but what it should protect and how much metadata must remain visible for operations. That balance determines whether transport mode or tunnel mode is the better fit, and whether additional controls such as segmentation, host hardening, or application-layer protection are still needed.

Risk and Threat Considerations

ESP reduces exposure on untrusted networks, but it also concentrates trust in key management, tunnel endpoints, and policy correctness. If those elements are weak, attackers may still gain access to protected traffic, replay packets, or exploit tunnels that were meant to reduce exposure.

Failure mechanism: Weak cipher choices, stale keys, disabled integrity protection, or incorrect tunnel selectors can leave traffic readable, replayable, or routed outside the protected path.

Impact: Sensitive data may be exposed in transit, tunnel trust may be abused for lateral movement, and defenders may mistakenly believe a flow is protected when it is not.

Practitioner Guidance

What to verify: Confirm the negotiated ESP transform set, key lifetime, and mode of operation, then check that the expected flows actually traverse the tunnel. If authentication is optional in your deployment, verify where integrity is required and where confidentiality alone is insufficient.

What good looks like: Strong ESP deployments use modern, approved ciphers, rotate keys on a defined schedule, enforce anti-replay protection, and make tunnel boundaries explicit in network documentation. Packet captures and device logs should agree on which traffic is protected and which is intentionally left outside the tunnel.

Practitioner takeaway: Treat ESP as a transport protection mechanism, not as a complete trust model, because the security value comes from correct negotiation, sound key handling, and accurate tunnel policy as much as from the protocol itself.

Related resources from NHI Mgmt Group

• Guide: Network Ports and Protocols Security Guide
• Encapsulating Security Payload (ESP)
• How should security teams split identity governance from implementation work?
• When should organisations prioritise NHI security over other identity work?
• When should security teams prioritise post-quantum readiness work?