Detect it by spotting crafted TCP handshakes that vary options such as ordering, presence, or negotiation behaviour across repeated attempts. Such probes are used for fingerprinting, so defenders should compare packet captures with known client baselines and alert on bursts of SYN traffic that exercise unusual TCP option combinations.
What a TCP options probe is really testing
A tcp options probe is not trying to complete a normal connection. It sends crafted SYN packets to see how a stack handles option ordering, presence, and negotiation, which gives attackers or scanners a useful fingerprint of the target’s TCP implementation. Detection works best when you compare observed handshakes with what your environment normally emits, not just with generic port-scan patterns.
Security teams should treat this as a protocol fingerprinting problem. The interesting signal is often in the edge cases: uncommon option sets, repeated probes with subtle variations, or SYN bursts that do not lead to ordinary session establishment.
How to spot it in packet captures and telemetry
Start with packet captures that preserve TCP flags and options, then compare the captured SYNs against a baseline of known-good clients and services. A probe often stands out because the source repeats similar handshakes while changing option combinations such as MSS, window scale, SACK permitted, timestamps, or the order in which options appear.
In network telemetry, look for bursts of SYN traffic that are concentrated on one host or small set of hosts, especially when the traffic does not progress into completed three-way handshakes. Network Ports and Protocols Security Guide is useful background when you are separating routine TCP behaviour from suspicious probing across common protocol surfaces.
Also check whether the source rotates probes across ports, changes option layouts between attempts, or uses source addresses and timing that suggest automated enumeration rather than a real client. A good detector often correlates packet-level details with asset context, so a probe against a server port is judged differently from the same pattern on an internal client segment.
What to do when the probe is suspicious
If the probe is isolated and low volume, you usually do not need to block it immediately. The first decision is whether the traffic is consistent with a sanctioned scanner, a load balancer, a test harness, or another approved tool that intentionally varies TCP behaviour.
When it is not clearly authorised, focus on three checks: whether the source is new or unusual, whether the option pattern is being repeated across many destinations, and whether the probe is part of a broader scan sequence. For validation and escalation, compare the event with broader detection guidance in NIST Cybersecurity Framework 2.0 and your internal network security playbooks.
If the same source also shows other reconnaissance indicators, such as service discovery, port sweep behaviour, or targeted follow-up traffic, treat the TCP options probe as one stage in a larger enumeration campaign rather than as a standalone curiosity.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
MITRE ATT&CK addresses the attack and risk surface, while NIST CSF 2.0, CIS Controls v8 and NIST SP 800-53 Rev 5 set the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| NIST CSF 2.0 | DE.CM-01 — Networks and Network Services Monitored | TCP options probes are detected through network monitoring and packet analysis. |
| DE.AE-02 — Potentially Adverse Events are Analyzed | Suspicious probe patterns must be analysed to distinguish reconnaissance from normal traffic. | |
| Recommendation — Monitor network flows and packet metadata for repeated handshake variation and scan-like bursts. Analyze repeated SYN variation against baseline client behaviour to confirm suspicious reconnaissance. | ||
| CIS Controls v8 | CIS-13 — Network Monitoring and Defense | This control family supports detection of protocol-level reconnaissance and scan activity. |
| Recommendation — Inspect packet-level indicators and alert on protocol-fingerprinting bursts that deviate from baselines. | ||
| MITRE ATT&CK | T1046 — Network Service Scanning | TCP options probing is a reconnaissance technique used during network service discovery. |
| Recommendation — Map unusual TCP option probing to reconnaissance and hunt for adjacent scanning activity. | ||
| NIST SP 800-53 Rev 5 | AU-12 — Audit Record Generation | Packet-capture and telemetry generation are needed to preserve the TCP option details required for detection. |
| Recommendation — Generate detailed network audit records that retain TCP flags, options, and handshake outcomes. | ||
Practitioner Guidance
What to verify: Confirm that your detection logic preserves TCP options, because some tools and sensors normalise or truncate them. If the options are lost before analysis, you will miss the fingerprinting signal entirely.
What to prioritise: Build the baseline around your own client populations and operating systems, not around a generic “normal TCP” model. Small differences in stacks, middleboxes, and proxies can otherwise produce false positives or hide the probe you care about.
Common mistake: Treating every odd SYN as a scan. The better test is whether the source is repeatedly exercising unusual option combinations across attempts in a way that is inconsistent with a real session setup.
Practitioner takeaway: The most reliable detector is a comparison engine, not a signature alone, so anchor alerting to repeated option variance plus handshake failure, then validate whether the pattern fits an approved scanner before escalating.
Related resources from NHI Mgmt Group
Deepen Your Knowledge
Free weekly newsletter
Subscribe to the NHI & AI Identity Journal
The latest on NHI and Agentic AI security – articles, research, breaches, news and events every week.
Bonus 33% off our NHI Course when you subscribe.
Reviewed and updated by the NHIMG editorial team on October 20, 2026.
NHI Mgmt Group — the #1 independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org