XFRM ESP-in-TCP

What XFRM ESP-in-TCP Does

XFRM ESP-in-TCP is a transport workaround for IPsec, not a different protection model. It keeps ESP’s encrypted and integrity-protected payload intact while changing the outer delivery so the traffic can cross networks that interfere with native IPsec.

That distinction matters because the security value comes from ESP itself, while the TCP encapsulation only helps the packets survive hostile or restrictive network paths. In practice, the outer transport is about reachability and traversal, not about relaxing the cryptographic protection of the inner tunnel.

Why It Exists in Real Networks

Many environments allow ordinary TCP more readily than IPsec-native flows, especially when middleboxes, firewalls, or NAT devices mishandle protocol numbers or non-TCP traffic. XFRM ESP-in-TCP exists to preserve secure site-to-site or endpoint-to-endpoint communication when those network conditions would otherwise break the tunnel.

It is therefore most useful where the security requirement is “keep IPsec, but make it pass through difficult paths.” That makes it a compatibility mechanism for transport, while the actual confidentiality and integrity guarantees still depend on the IPsec configuration and endpoint trust model.

For the underlying IPsec design and hardening choices that normally surround this transport mode, see the IPsec Security Guide.

How It Changes the Operational Picture

Encapsulating esp in tcp can improve connectivity, but it also changes how the traffic behaves under loss, congestion, and inspection. TCP adds its own reliability and session semantics, which can make the tunnel easier to pass through some networks but also more sensitive to performance quirks, timeout handling, and interference from intermediaries.

That means the operator is no longer just thinking about cryptographic correctness. They also need to consider whether the path accepts long-lived TCP sessions, whether NAT or proxies terminate or reshape flows, and whether the added transport layer affects throughput or failure recovery in ways that alter the user experience.

What to Verify Before You Rely on It

XFRM ESP-in-TCP should be treated as a path-compatibility feature, not a substitute for a clean IPsec design. The important checks are whether the transport is actually needed, whether the endpoints and gateways are configured consistently, and whether the surrounding network permits the encapsulated flow without silently degrading or rewriting it.

It is also important to confirm that the use of TCP does not create false confidence about visibility or enforcement. Security teams still need to validate the IPsec policy, the endpoint authentication model, and the operational monitoring around tunnel health, because the encapsulation layer does not change the need for strong tunnel governance.

Related resources from NHI Mgmt Group

• What breaks when Linux hosts allow ESP-in-TCP on untrusted workloads?
• ESP-in-TCP
• Why do TCP and buffering still fail to guarantee security telemetry delivery?
• What is the difference between UDP, TCP, and TLS transport for syslog?