MACsec and IPsec both encrypt traffic in transit, but one protects a single Ethernet link and the other protects an IP path, so picking the wrong one leaves either a gap in coverage or a throughput and operations problem you did not plan for. MACsec (IEEE 802.1AE) encrypts frames between two directly connected Ethernet devices and decrypts them at the next one. IPsec ESP (RFC 4303) protects IP packets between two endpoints or gateways across any routed network, including the internet. This guide compares them on threat model, architecture, performance, key management and failure modes, then works through data centre interconnect, carrier Ethernet WAN, internet WAN and campus scenarios. It is for network architects, security engineers and anyone writing a transit encryption standard. Configuration depth for each protocol is in the MACsec guide and the IPsec Security Guide.
Key takeaways
- Layer decides scope. MACsec protects one Ethernet hop and every frame on it; IPsec protects IP traffic end to end between two peers, whatever lies in between.
- Every MACsec hop is a clear text point. Frames are decrypted inside each MACsec device, so a path through a switch or router you do not trust needs IPsec or application encryption on top.
- IPsec reaches where Ethernet cannot. Over the internet, an MPLS IP VPN or any routed network, MACsec cannot form a session; IPsec, or an overlay that uses it, is the option.
- Performance profiles differ. MACsec runs in port hardware at line rate on capable PHYs; IPsec throughput depends on crypto engines and CPU, and RFC 9611 records 10 Gbps links often cut to 2 to 5 Gbps per Child SA with AES-GCM.
- Both are machine identity systems. MACsec CAKs and IPsec PSKs or certificates are non-human credentials that need inventory, rotation and revocation.
The short answer
Use MACsec when you own or lease a Layer 2 Ethernet link end to end, need full line rate, and want everything on that link protected, including ARP, LACP, LLDP and routing protocol traffic: dark fibre and wavelength data centre interconnect, carrier Ethernet services that pass MKA frames, and switch uplinks in shared space. Use IPsec when traffic crosses a routed network you do not control, when you need a single encrypted path across several hops, or when the peers are hosts and virtual appliances rather than switch ports: internet site-to-site VPNs, cloud connectivity, and host-to-host transport mode. Use both when a Layer 2 link carries traffic that must also stay encrypted beyond the far switch, or when policy demands two independent layers.
How the two compare
| Dimension | MACsec (IEEE 802.1AE with MKA) | IPsec (ESP with IKEv2) |
|---|---|---|
| Layer | Layer 2, Ethernet frames | Layer 3, IP packets |
| Scope | One point-to-point Ethernet link; configured per link | Between two IPsec peers across any number of routed hops |
| What is protected | Almost all frames on the link, including LLDP, LACP, DHCP and ARP | IP traffic matching the traffic selectors |
| What stays visible | MAC addresses; VLAN tags when configured in the clear; optional confidentiality offset | Outer IP header (tunnel mode) or original IP header (transport mode) |
| Security services | Confidentiality, integrity, data origin authenticity, replay protection, privacy protection (802.1AEdk-2023) | Confidentiality, data origin authentication, integrity, anti-replay, limited traffic flow confidentiality (RFC 4303) |
| Key management | MKA (IEEE 802.1X) from a pre-shared CAK or 802.1X EAP-TLS | IKEv2 (RFC 7296) with certificates, PSK or EAP |
| Cipher suites | GCM-AES-128, GCM-AES-256, XPN variants for high speed links | AES-GCM-16 (MUST), ChaCha20-Poly1305 (SHOULD), AES-CBC with HMAC (RFC 8221) |
| Per-packet overhead | 24 bytes (8-byte SecTAG, 16-byte ICV) per Juniper documentation; more with optional fields | ESP header, IV, padding, trailer and ICV, plus a new outer IP header in tunnel mode and UDP 4500 encapsulation behind NAT |
| Throughput model | Port or PHY hardware, line rate on capable ports | Crypto engines or CPUs; per-SA limits unless per-resource SAs are used |
| Typical failure | Session down on must-secure link drops traffic; should-secure silently falls back to clear | Tunnel down routes traffic in clear unless dropped; weak proposals negotiated |
Where this guide cites vendor documentation for MACsec examples, platforms are listed alphabetically (Cisco, Juniper); order is not a ranking, and both are used only to illustrate standard behaviour.
MACsec: threat model and architecture
Threats addressed: passive taps on fibre, patch panels and carrier circuits (ATT&CK T1040 Network Sniffing), inline devices that alter or inject frames (T1557 Adversary-in-the-Middle), and unauthorised devices plugged into a protected port, which cannot join a connectivity association without the CAK or a valid certificate.
Architecture in words: the two ends share a Connectivity Association Key, either configured as a pre-shared key or derived per session from 802.1X EAP-TLS. MKA authenticates the peers, elects a key server and distributes a randomly generated Secure Association Key; the port hardware then encrypts each frame with the SAK and checks integrity on receipt. The IEEE describes MACsec as transparent to MAC clients, so routing protocols and applications need no change.
Limits:
- Clear text inside every device: each MACsec switch or router sees plain frames. A compromised or untrusted intermediate device sees everything.
- Layer 2 reach only: MKA frames must reach the peer. Carrier Ethernet services may not forward them by default; Cisco's WAN MACsec configuration example changes the EAPoL destination to broadcast and the EtherType from 0x888E to 0x876F so provider switches forward them.
- Header exposure: VLAN tags in the clear (needed for many carrier services) and MAC addresses reveal topology.
- Hardware dependent: it needs MACsec capable ports, and high speed links need XPN cipher suites.
IPsec: threat model and architecture
Threats addressed: interception and tampering anywhere along a routed path, including carrier IP VPNs and the internet, and limited traffic analysis in tunnel mode because the inner addresses are hidden.
Architecture in words: IKEv2 peers authenticate each other and negotiate Child SAs with traffic selectors; ESP then protects matching packets, in tunnel mode between gateways (the whole inner packet wrapped in a new IP header) or in transport mode between hosts. ESP uses IP protocol 50, or UDP 4500 when NAT is detected, with IKE on UDP 500. Each SA has a 32-bit SPI and a sequence number for replay protection; RFC 4303 sets a minimum replay window of 32 packets and a recommended default of 64.
Limits:
- IP only: Layer 2 control traffic on the underlying links (ARP, LACP, LLDP) is not protected.
- Exposed gateway: the IKE service parses unauthenticated input from the network and is a frequent target; patching and peer restrictions matter as much as cryptography.
- Throughput ceilings: RFC 9611 (Standards Track, July 2024) explains that most implementations tie one Child SA to one CPU or queue, and reports aggregate throughput rising from about 5 Gbps on one CPU to 40 to 60 Gbps across 25 to 30 CPUs with per-resource Child SAs.
- MTU and fragmentation: tunnel mode overhead reduces effective MTU and causes fragmentation problems when path MTU discovery is blocked.
Performance and operations
| Concern | MACsec | IPsec |
|---|---|---|
| Throughput planning | Check per-port MACsec support and line rate on the exact PHY or line card | Size crypto throughput per tunnel and per SA; check per-resource Child SA support for 10 Gbps and above |
| MTU | Raise interface MTU for overhead; Cisco's WAN example notes enabling MACsec reduces interface MTU by 32 bytes | Set tunnel MTU and TCP MSS clamping; test with the largest real packets |
| Rekeying | SAK rekey by key server; XPN reduces rekey frequency on fast links | Child SA and IKE SA lifetimes; PFS via CREATE_CHILD_SA key exchange |
| Key rollover | Key chains with overlapping CAK lifetimes; NTP so both ends agree | Certificate renewal or PSK rotation with overlapping validity |
| Redundancy | Per link; protect each member of a bundle | Tunnel pairs with routing failover and dead peer detection |
| Visibility for tools | Traffic is clear inside the switch, so on-box telemetry still works | Inspection only at or behind the decrypting gateway |
Integrity only modes: both protocols can authenticate without encrypting. Juniper documents MACsec integrity checks as always on with encryption optional; RFC 8221 keeps ESP with ENCR_NULL for authentication-only and prefers it to AH because of NAT traversal. Treat either as a documented exception.
Choosing by scenario
| Scenario | Recommended layer | Why |
|---|---|---|
| Data centre interconnect over dark fibre or wavelengths | MACsec | Point-to-point Ethernet you control end to end; line rate at 100 Gbps and above with XPN |
| Carrier Ethernet (E-Line) between sites | MACsec if the carrier passes MKA and tags; IPsec otherwise | Confirm with the carrier and in a pilot before committing |
| MPLS IP VPN or internet between sites | IPsec, directly or inside an SD-WAN overlay | Routed path; MACsec cannot reach the peer |
| Cloud connectivity | IPsec over the internet or private connection; MACsec only where the provider offers it on dedicated ports | Depends on provider service; check its documentation |
| Campus uplinks in shared risers | MACsec | Protects everything on the link without touching routing |
| Host-to-host inside a data centre | IPsec transport mode, or TLS at the application | End to end across switches you may not fully trust |
| Regulated data that must stay encrypted across every device | IPsec or TLS, optionally with MACsec underneath | MACsec alone decrypts at each hop |
Small organisations: usually only have IP WAN and internet links, so IPsec (often through a firewall or SD-WAN appliance) is the practical answer; MACsec appears only if they lease dark fibre. Large organisations: commonly run MACsec on DCI and campus cores, IPsec for branches and cloud, and TLS in the application, with one transit encryption standard that states which layer applies to which link class. See the SD-WAN Security Guide and the MPLS and Carrier WAN Security Guide.
Worked scenario: two data centres and forty branches
A financial services firm has two data centres 30 km apart on leased dark fibre, forty branches on an MPLS IP VPN with internet backup, and a public cloud region. Its policy requires encryption of all data leaving controlled premises.
- DCI: MACsec with GCM-AES-XPN-256 on the four 100 Gbps fibre pairs, must-secure, pre-shared CAKs unique per link and held in a vault, key chains with overlapping lifetimes. IPsec at this rate would need large crypto capacity per tunnel and add latency for storage replication.
- Branches: IPsec IKEv2 with certificates from the internal CA over both MPLS and internet, because neither path is Layer 2. The carrier offers no Ethernet service at the branches.
- Cloud: IPsec tunnels to the cloud VPN gateways, redundant pairs, with drop-on-tunnel-down policy so nothing routes in the clear.
- Payment application: TLS between application tiers, because MACsec on the DCI decrypts inside each core switch and the firm wants the card data encrypted across every device.
The design review records one residual gap: the core switches see clear text, so their administrative access moves behind PAM with session recording.
Implementation approach
- Classify links: list every link and path leaving controlled space, its type (dark fibre, carrier Ethernet, IP VPN, internet), line rate, and the data classes it carries.
- Assign a layer per link class: using the scenario table, and record exceptions with an owner.
- Check capability: MACsec port support, XPN and licence requirements for MACsec; crypto throughput, per-resource Child SA support and RFC 8221 algorithms for IPsec.
- Design keys: unique CAK per MACsec link or 802.1X EAP-TLS; certificates per IPsec gateway with identity binding; all in a vault or managed CA.
- Pilot: MACsec in should-secure on one link until sessions reach Secured on both ends, then must-secure; IPsec on one tunnel with full-size packet tests, rekey and failover tests.
- Roll out in waves with saved prior configuration for rollback per link or tunnel, never globally.
- Monitor and rotate: session state, counters and key events from day one; test a CAK and certificate rollover before the first real one.
Detection and measurement
- MACsec: MKA session state per link, integrity failure and replay drop counters, key server changes and SAK rekeys, links found in should-secure.
- IPsec: tunnel up and down events, negotiated proposals, rekey failures, replay and integrity failures (RFC 4303 lists these as auditable events), traffic sent while an SA is down.
Metrics:
- Percentage of in-scope links and paths encrypted at the layer the standard assigns (target 100 percent).
- MACsec links in must-secure and Secured state (target 100 percent).
- IPsec tunnels negotiating only target profile algorithms.
- CAKs, PSKs and certificates past rotation date (target zero).
- Unplanned session or tunnel drops per month by link class.
Common failure patterns
- Assuming MACsec is end to end: data is clear inside every switch on the path, including carrier or colocation equipment if it terminates MACsec.
- Should-secure left in production: a mismatched peer drops the link to clear text without an alert.
- Carrier circuit that swallows MKA frames: the link never secures, and someone switches to should-secure to get it up.
- IPsec sized on headline throughput: one large flow hits the single-SA ceiling and replication falls behind.
- Clear text on tunnel failure: routing falls back to an unencrypted path when the IPsec SA drops.
- Shared keys everywhere: one CAK or PSK reused across many links means one leaked configuration backup exposes all of them.
Where identity fits
Each MACsec port pair and each IPsec gateway authenticates with a credential and no person present. Pre-shared CAKs and PSKs often sit in configuration backups and automation repositories; certificates need a CA, renewal and revocation. Inventory them with owners, keep them in a secrets manager, restrict which automation accounts can read them, and revoke on decommissioning. See the Machine Identity, PKI and Certificate Lifecycle Guide and the Secrets Management Guide.
Framework mapping
| Framework | Reference | MACsec and IPsec contribution |
|---|---|---|
| NIST SP 800-53 Rev. 5 | SC-8 Transmission Confidentiality and Integrity; SC-8(1) Cryptographic Protection | Both protect data in transit at their layer |
| NIST SP 800-53 Rev. 5 | SC-12 Cryptographic Key Establishment and Management; SC-13 Cryptographic Protection | MKA and IKEv2 key management; approved cipher suites |
| NIST SP 800-53 Rev. 5 | IA-3 Device Identification and Authentication | CAKs, certificates and PSKs authenticate devices |
| NIST CSF 2.0 | PR.DS-02 | Data in transit protected |
| NIST SP 800-77 Rev. 1 | Guide to IPsec VPNs (final, June 2020) | US federal IPsec deployment guidance |
| ISO/IEC 27001:2022 | A.8.20 Networks security; A.8.24 Use of cryptography | Network protection and cryptographic policy |
| MITRE ATT&CK | T1040 Network Sniffing; T1557 Adversary-in-the-Middle | Defeats interception on protected links and paths |
Practitioner checklist
- Write a transit encryption standard that assigns a layer to each link class.
- Use MACsec in must-secure on owned or leased Layer 2 links, with XPN suites on 40 Gbps and faster links.
- Confirm with each carrier that MKA frames and VLAN tags pass before ordering MACsec circuits.
- Use IPsec IKEv2 with RFC 8221 AEAD ciphers for any routed path, and drop traffic when an SA is down.
- Size IPsec throughput per SA and check per-resource Child SA support for high bandwidth tunnels.
- Add TLS or IPsec on top of MACsec where data must stay encrypted across intermediate devices.
- Give every link and tunnel a unique credential, held in a vault or issued by a managed CA, with tested rollover.
- Monitor session state, integrity failures and negotiated algorithms, and alert on any fallback to clear text.
Standards and references
- IEEE 802.1: 802.1AE MAC Security (MACsec)
- IETF: RFC 4303 IP Encapsulating Security Payload (ESP)
- IETF: RFC 8221 Cryptographic Algorithm Implementation Requirements and Usage Guidance for ESP and AH
- IETF: RFC 9611 IKEv2 Support for Per-Resource Child Security Associations
- NIST: SP 800-77 Rev. 1 Guide to IPsec VPNs
- Cisco: Configure WAN MACsec on Catalyst 8500
- Juniper: Understanding Media Access Control Security (MACsec)
Related NHI Mgmt Group resources: MACsec · IPsec Security Guide · Data Centre Network Fabric Security · VPN Protocols Compared: WireGuard, IPsec, OpenVPN Guide