The TCP three-way handshake establishes a connection by exchanging SYN, SYN-ACK and ACK messages between client and server. It confirms both sides can send and receive before data flows. Defenders should watch for incomplete handshakes, connection floods and unexpected inbound SYN traffic, which can indicate scanning or denial-of-service activity.
How the TCP three-way handshake establishes a connection
TCP uses a three-step exchange to agree on a session before payload data starts moving. The client sends a SYN to request a connection, the server replies with SYN-ACK to acknowledge and synchronise, and the client finishes with ACK. That final ACK is what turns a tentative exchange into an established TCP connection.
The handshake does more than “say hello.” It confirms that both endpoints are reachable, that each side can transmit and receive, and that both parties have agreed to the initial sequence numbers used to track ordered delivery. In practice, this is why TCP is connection-oriented while UDP is not.
The mechanism also creates a clear boundary between setup and traffic flow. Before the final ACK, the connection is incomplete and the server may be holding state for a half-open session. Once the handshake completes, the endpoints can exchange application data with TCP’s reliability features, including retransmission and in-order delivery.
What the handshake tells defenders about traffic and abuse
The handshake is also a useful visibility point for defenders. Abnormal patterns, such as many SYNs without completed ACKs, can indicate scanning, service pressure, or a denial-of-service attempt. Because the setup phase is lightweight and repeatable, attackers often abuse it to probe live hosts or to exhaust connection state.
Unexpected inbound SYN traffic is not always malicious, but it is a useful signal when it appears in bursts, from unusual sources, or against services that should not be exposed. In those cases, the issue is not the handshake itself, but the scale and context of the traffic around it.
Connection floods exploit the same setup behaviour by forcing a server to allocate resources for sessions that never fully establish. Defenders should treat repeated incomplete handshakes as a network telemetry problem first, then decide whether the pattern is benign load, misconfigured clients, or hostile activity.
Why TCP setup details still matter in modern networks
Even though the handshake is a basic protocol function, it still shapes how services are exposed, monitored, and protected. Firewalls, load balancers, intrusion detection systems, and reverse proxies all observe or influence handshake behaviour, so small changes in handshake volume or timing can have operational significance.
Because the handshake happens before application logic runs, it is often the earliest point at which defenders can distinguish “someone tried to connect” from “someone successfully used the service.” That makes it valuable for inventorying exposed services, validating reachability, and catching noisy network activity before it becomes an application-level incident.
For practitioners, the main takeaway is that TCP setup is not just protocol theory. It is also an observable trust boundary: once you understand how SYN, SYN-ACK, and ACK fit together, you can interpret failed or excessive handshakes as signals about availability, exposure, and attack pressure.
Risk and Threat Considerations
TCP handshake behaviour becomes risky when a service is exposed to untrusted networks and the endpoint is forced to hold state for many incomplete connection attempts. That can consume resources, distort monitoring, and make legitimate traffic harder to serve during an attack or outage.
Failure mechanism: An attacker or noisy client sends repeated SYNs, then withholds the final ACK or varies source characteristics to keep sessions half-open and create connection-state pressure.
Impact: The server may slow down, refuse new connections, or show misleading telemetry, which can look like a capacity problem when it is actually a network-level abuse pattern.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
MITRE ATT&CK addresses the attack and risk surface, while NIST CSF 2.0, NIST SP 800-53 Rev 5 and CIS Controls v8 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| MITRE ATT&CK | T1595 — Active Scanning | TCP handshake traffic often reveals scanning and exposure probing. |
| Recommendation — Correlate repeated SYN activity with scanning techniques and hunt for exposed services. | ||
| NIST CSF 2.0 | DE.CM-01 — Continuous Monitoring and Adverse Event Detection | Handshake anomalies are network events that should be continuously monitored. |
| Recommendation — Monitor connection setup anomalies as indicators of exposure or abuse. | ||
| NIST SP 800-53 Rev 5 | SC-7 — Boundary Protection | Handshake traffic crosses network boundaries and is shaped by perimeter controls. |
| Recommendation — Apply boundary protection to limit unsolicited inbound connection attempts. | ||
| CIS Controls v8 | CIS-13 — Network Monitoring and Defense | Handshake failures and floods are network telemetry that belongs in monitoring controls. |
| Recommendation — Log and alert on abnormal SYN and incomplete handshake patterns. | ||
Practitioner Guidance
What to verify: Confirm whether incomplete handshakes cluster around a specific service, source range, or time window. If the same pattern appears across many ports or destinations, treat it as broader reconnaissance or infrastructure noise rather than a single-service issue.
What to measure: Track the ratio of SYNs to completed handshakes, the lifetime of half-open sessions, and whether the pattern correlates with latency or connection failures. Those signals tell you whether the traffic is merely bursty or beginning to affect service quality.
Practitioner takeaway: The handshake is both a connection primitive and an early warning surface, so the best operational response is to monitor completion patterns, not just packet volume.