How does IPsec tunnel mode work?

How IPsec tunnel mode carries traffic across untrusted networks

ipsec tunnel mode wraps the original packet inside a new IP packet, so the inner source and destination stay hidden while the outer packet is used for delivery between tunnel endpoints. That makes it the usual choice for gateway-to-gateway VPNs and other routed overlays where the network in between cannot be trusted.

The practical consequence is that routing, peer trust, and security policy all matter at the tunnel boundary. The tunnel is only as strong as the endpoint configuration, because the protected payload can still be exposed by weak peer authentication, broken routing, or traffic that is steered outside the tunnel.

What IPsec actually does to the packet

In tunnel mode, the device takes the entire original IP packet, including its internal addressing, and encapsulates it inside a new outer packet. The outer header gives the network enough information to forward the traffic between the IPsec peers, while the encapsulated inner packet is protected by ESP or AH depending on the design.

This is different from transport mode, where the original IP header remains in place and only the payload is protected. Tunnel mode therefore changes the packet boundary, not just the payload protection, which is why it is commonly used when two networks need a secure path over an untrusted intermediary.

Because the inner packet is hidden from the transit network, tunnel mode can preserve private addressing and let organisations connect separate sites without exposing internal topology. It also means that the tunnel endpoints become security-critical translation points for encryption, integrity, and policy enforcement.

Where tunnel mode fits in a real network design

Most deployments use tunnel mode for site-to-site VPNs, branch connectivity, or remote access designs that need an encrypted routed path between distinct security zones. The endpoints may be security gateways, routers, firewalls, or hosts, but the design goal is the same, create a protected overlay that carries ordinary IP traffic safely between trusted peers.

That design choice also affects operations. Once traffic is encapsulated, the underlay network only needs to move the outer packet, while the overlay policy decides what the protected networks can reach. This separation is useful, but it also means that mistakes in route selection, NAT handling, or allowed selectors can break connectivity in ways that are hard to diagnose.

For teams that want a deeper VPN hardening reference, NHIMG’s VPN Security Guide covers IPsec alongside appliance hardening, split tunnelling, and replacement options.

Risk and Threat Considerations

IPsec tunnel mode reduces exposure on the transit network, but it also concentrates trust at the tunnel endpoints. If peer identity is weak, security associations are mismanaged, or routing is wrong, traffic can bypass protection, land on the wrong network, or become reachable through a tunnel that was assumed to be private.

Failure mechanism: Misconfigured selectors, weak peer authentication, or incorrect route advertisement can cause traffic to leave the intended encrypted path or create a tunnel that accepts unintended networks.

Impact: Confidential traffic may be exposed, internal segments may become reachable from an untrusted peer, and operators may believe they have encrypted connectivity when they actually have a partial or broken control boundary.

Practitioner Guidance

What to verify: Confirm the peer identity model, the exact traffic selectors, and the routing path on both sides before relying on the tunnel for production traffic. If the tunnel is meant to protect internal subnets, verify that the inner networks are not accidentally leaked through a parallel route, NAT rule, or fallback path.

Decision rule: If the tunnel carries sensitive production traffic, treat any peer mismatch, selector ambiguity, or asymmetric routing as a security issue, not just a connectivity problem. If availability is restored by widening selectors or weakening policy, the design is probably masking a configuration defect rather than solving it.

Practitioner takeaway: Tunnel mode is a packet encapsulation method, but operationally it is a trust boundary, so the real security work is proving that the right peers, routes, and traffic sets are protected end to end.

Related resources from NHI Mgmt Group

• Why does browser fingerprinting still work when users change user agent, incognito mode, or network connection?
• What breaks when remote users still have to tunnel back to the on-prem domain for everyday work?
• What is the biggest failure mode when employees use ChatGPT for work?
• What is the main failure mode when AppSec roles describe remediation as coordination work?