A handshake is the initial exchange that establishes trust, shared keys, or session parameters between malware and its server. Here it also delivered configuration values used in later requests. In malware analysis, the handshake often reveals how subsequent communications are structured and what metadata the actor controls.
What a handshake does in malware communications
A handshake is the first meaningful exchange between the malware and its infrastructure. It usually confirms the actor controls the server, negotiates session settings, and establishes the parameters the implant will use before any operational traffic begins.
In practice, this step is more than a connection test. It can embed identifiers, configuration choices, or gating logic that shape what later requests look like, which makes the handshake a useful point for understanding protocol structure and campaign behaviour.
Why analysts care about the handshake phase
The handshake often reveals the communication model that follows. Analysts may infer encryption setup, message framing, client validation, timing expectations, and whether the malware expects a fixed command schema or server-driven configuration.
That makes the handshake a high-value artifact in reverse engineering and network analysis. If the exchange is decoded, it can expose how the malware authenticates the peer, what state is established up front, and which fields are likely controlled by the operator rather than the infected host.
Handshake structure and what it can expose
Common handshake elements include version markers, bot or host identifiers, shared secrets, session nonces, encryption metadata, and configuration blobs. Some families use the handshake to return tasking parameters immediately, while others use it only to agree on keys and transport rules.
The exact structure matters because it often distinguishes a simple beacon from a more capable command-and-control design. A handshake that is static, reusable, or weakly validated can also make the protocol easier to emulate, fingerprint, or disrupt during analysis.
How handshakes relate to later command-and-control traffic
Once established, the handshake usually sets the rules for all subsequent communications. That can include request formatting, message ordering, keepalive behaviour, and whether later responses are encrypted, compressed, or signed in a way the client will accept.
Because the handshake can deliver initial configuration values, it may indirectly define the malware’s next actions without requiring a separate configuration fetch. For defenders, that means the earliest exchange may already contain the best clues about infrastructure role, campaign state, and protocol intent.
Risk and Threat Considerations
Handshake design affects how easily malware can be observed, fingerprinted, or impersonated. Weak validation can allow defenders to emulate the server, harvest identifiers, or interrupt tasking, while overly rigid trust assumptions can make the malware vulnerable to protocol analysis and sinkholing.
Failure mechanism: If the handshake uses predictable values, weak peer validation, or reusable session material, analysts or defenders may be able to spoof the server, recover protocol structure, or force the malware into a detectable failure mode.
Impact: Exposed handshake logic can reveal campaign infrastructure, operator-controlled settings, and downstream message formats, which improves detection, reverse engineering, and potential disruption of command-and-control traffic.
Standards & Framework Alignment
This section maps relevant standards and security frameworks to the operational risks and controls described in this guidance.
MITRE ATT&CK and OWASP Non-Human Identity Top 10 address the attack and risk surface, while NIST SP 800-53 Rev 5 sets the governance and control requirements practitioners need to meet.
| Framework | Control / Reference | Relevance |
|---|---|---|
| MITRE ATT&CK | T1095 — Non-Application Layer Protocol | Malware handshakes often define a custom C2 protocol exchanged over non-app-layer channels. |
| T1071 — Application Layer Protocol | Handshake exchanges commonly establish malicious traffic over standard application protocols. | |
| T1105 — Ingress Tool Transfer | Handshake-negotiated sessions can precede payload delivery and tasking from attacker infrastructure. | |
| Recommendation — Map the handshake to custom C2 protocol behavior and hunt for unusual session negotiation patterns. Inspect early session negotiation for abuse of common application protocols carrying C2 traffic. Correlate handshake setup with subsequent payload retrieval and staging activity. | ||
| NIST SP 800-53 Rev 5 | SC-8 — Transmission Confidentiality and Integrity | Handshake exchanges often establish session protection and integrity expectations for later traffic. |
| AU-6 — Audit Record Review, Analysis, and Reporting | Handshake artifacts are high-value telemetry for understanding malicious session setup. | |
| SI-4 — System Monitoring | Handshake behavior is a detectable indicator of command-and-control initialization. | |
| Recommendation — Protect session establishment so attackers cannot tamper with negotiated malware traffic. Review first-packet telemetry to identify protocol negotiation and operator-controlled fields. Monitor unusual initial exchanges to detect malware beaconing and protocol setup. | ||
| OWASP Non-Human Identity Top 10 | NHI-04 — Insecure Authentication | Handshake logic may rely on weak trust checks and session authentication between malware and server. |
| NHI-07 — Long-Lived Secrets | Handshake exchanges may carry or derive session material that persists across malware communications. | |
| Recommendation — Verify that any trust establishment cannot be trivially spoofed or replayed. Reduce exposure from reusable session material that can be captured during negotiation. | ||
Practitioner Guidance
What to watch for: Treat the handshake as a primary artifact, not just a prelude to “real” traffic. If you can capture the first exchange, you often gain the best visibility into protocol state, trust assumptions, and configuration values that will govern the rest of the session.
Practitioner note: Preserve the raw handshake alongside decrypted or decoded follow-on traffic, because later messages often make more sense when read against the initial negotiation that established them.
Related resources from NHI Mgmt Group
- What should organisations do if PQC adoption could increase handshake latency?
- What breaks when MCP servers rely on session-bound state or the initialize handshake?
- Why does the TLS handshake use both public key and symmetric key cryptography?
- Why does TLS fingerprinting become less reliable when attackers randomize handshake settings?
Deepen Your Knowledge
Reviewed and updated by the NHIMG editorial team on September 25, 2026.
NHI Mgmt Group — the #1 independent authority on Non-Human Identity, IAM, and Agentic AI security. nhimg.org
