The three-way handshake is the TCP connection setup process between two endpoints. One side sends a SYN, the other replies with SYN-ACK, and the first side finishes with ACK, which confirms both directions can communicate and establishes initial sequence numbers for reliable transport.
How the Three-Way Handshake Works
The three-way handshake is the TCP setup sequence that turns a raw packet path into a stateful connection. The SYN begins the conversation, the SYN-ACK confirms the listener can reply, and the final ACK completes the exchange while both sides establish initial sequence numbers for reliable transport.
That exchange matters because TCP is not just “connected” by reaching a host. It is connected only after both endpoints prove bidirectional reachability and agree on the starting state that later supports ordered delivery, retransmission, and flow control.
Why It Matters for Network Reliability
The handshake is one of the simplest but most important reliability checks in networking. It distinguishes a live, reachable service from a port that merely receives traffic, and it gives operators a consistent point at which connection state begins.
For defenders, that means connection establishment is often the first observable moment in a communication path, which is why TCP state is so useful in logs, packet captures, and firewall telemetry. Networking Fundamentals for Defenders Guide is a useful companion when you want the handshake explained in the broader context of TCP, routing, and packet visibility.
What the Handshake Reveals to Defenders
Because the handshake happens before application data flows, it is often the first place a defender can infer whether a connection attempt is legitimate, repeated, or abnormal. Repeated SYNs without a completed ACK, for example, can indicate scanning, path failure, or resource exhaustion rather than a healthy session start.
The sequence also helps analysts separate network reachability from application success. A completed handshake means the endpoints can exchange TCP segments, but it does not prove the application behind the port is healthy, trusted, or allowed to serve the request.
When teams interpret TCP activity, they should treat handshake state as evidence of transport-level progress, not as proof of business-level trust. That distinction becomes important in troubleshooting, monitoring, and intrusion detection.
Common Failure Modes and Security Implications
Handshake failures can reflect simple network problems such as packet loss, asymmetric routing, stateful filtering, or overloaded services. They can also reflect deliberate abuse, especially when attackers use connection attempts to probe exposed services or consume resources with incomplete sessions.
At scale, repeated half-open or unsuccessful connection attempts can become both an availability issue and a visibility problem. The network may look “active” while the real issue is that connections never mature into usable sessions.
For that reason, the handshake is often one of the earliest places to look when a service appears intermittently unreachable or when transport-level traffic does not line up with expected application behaviour.
Risk and Threat Considerations
The three-way handshake is a natural control point for network abuse because it is the moment TCP allocates connection state and confirms bidirectional communication. Attackers can exploit that statefulness with floods, spoofed attempts, or large volumes of incomplete connections to consume resources and reduce service availability.
Failure mechanism: If an endpoint accepts or tracks too many incomplete handshakes, its connection tables, CPU, or memory can be exhausted before legitimate traffic is served.
Impact: The result can be degraded performance, dropped connections, or a service that becomes unreachable even though the host still appears online.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
MITRE ATT&CK addresses the attack and risk surface, while NIST SP 800-53 Rev 5 and NIST CSF 2.0 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| MITRE ATT&CK | T1498 — Network Denial of Service | Three-way handshake abuse commonly appears in connection-flood denial patterns. |
| Recommendation — Map handshake-flood traffic to T1498 and tune detection for excessive incomplete connections. | ||
| NIST SP 800-53 Rev 5 | SC-5 — Denial of Service Protection | TCP handshake exhaustion is a classic availability control concern for exposed services. |
| Recommendation — Apply SC-5 controls to limit connection exhaustion and preserve service availability. | ||
| NIST CSF 2.0 | DE.CM-01 — Networks and network services are monitored to find potentially adverse events | Handshake anomalies are a network monitoring signal used to detect transport-layer abuse. |
| Recommendation — Monitor handshake anomalies as adverse network events in your detection program. | ||
Practitioner Guidance
What to watch for: Monitor handshake completion rates, unexpected spikes in SYN traffic, and repeated retries from the same source or subnet. Those patterns are often more informative than application logs when the problem sits at the transport layer.
Practitioner note: Treat handshake state as an operational signal, not a trust decision. A completed TCP handshake only proves the transport path is viable, so access control and application authentication still need to happen above it.