How do attackers use a custom command and control protocol?

How custom C2 protocols help attackers hide in plain sight

Attackers design custom command and control traffic to look less like a known malware family and more like ordinary application traffic. That reduces signature match rates and gives them more freedom to shape packet timing, message structure, encoding, and encryption around the environment they are already inside.

What matters technically is not just that the traffic is “different”, but that it still needs to move commands, status, and exfiltration without drawing attention. A custom protocol often borrows familiar transport properties, such as HTTP-like request cadence, DNS-like naming, or periodic polling, while keeping the payload format proprietary or lightly obfuscated.

That flexibility is why defenders should think in terms of behaviour, not just bytes. A protocol can be custom even when it rides on standard ports, uses TLS, or mimics a legitimate application flow. The detection problem is usually the mismatch between claimed service behaviour and what the session actually does over time. For protocol and port context, IANA is the authoritative registry, and attackers often abuse expectations tied to common identifiers.

What custom C2 looks like on the wire

Custom C2 is usually built to survive basic inspection by changing one or more of these elements: message framing, field order, padding, timing, domain usage, encryption wrapper, or retry logic. The result may look like a normal request stream until you compare it against baseline application behaviour, expected destinations, and the relationship between request and response sizes.

Attackers also use protocol imitation to make malicious channels blend into known services. Some mimic web APIs, some emulate chatty enterprise software, and others tunnel through DNS, ICMP, or other less-scrutinised flows. The more the protocol can borrow the shape of normal traffic, the more it can evade tools that rely on fixed signatures or simplistic allowlists. NHIMG’s Command and Control (C2) Guide covers the main techniques defenders use to spot beaconing, tunnelling, and egress abuse.

Encryption does not make traffic benign, but it does remove content visibility and pushes defenders toward metadata. If the destination is unknown, the timing is rigid, the session is long-lived, and the application profile does not match the host role, custom C2 becomes much easier to spot through correlation than through payload inspection alone.

Why protocol abuse matters to defenders

The key defensive issue is that custom C2 changes the inspection model. Instead of looking for a known malware signature, analysts need to identify protocol anomalies, unexplained beacons, malformed messages, unusual session duration, and outbound connections that do not match approved services. That is especially important when adversaries deliberately choose uncommon encodings or low-and-slow cadence to avoid threshold-based detections.

Known-bad intelligence still helps, but custom C2 is designed to outlive fixed indicators. That means defenders need egress control, protocol normalisation where possible, and a clear view of which hosts are actually expected to talk to which external services. If a workstation or server is generating traffic that resembles an internal business application but points to an unfamiliar external host, that is often more interesting than whether the payload is readable.

Attackers also benefit from persistence. A custom protocol can be reused across campaigns, tuned for a specific target, and updated without changing the visible infrastructure footprint. That makes hunting for transport behaviour, rather than just infrastructure names, a more durable detection strategy. NHIMG’s Red Teaming Guide is useful here because it frames C2 as part of the broader adversary emulation chain, not just a standalone network artefact.

Risk and Threat Considerations

Custom C2 is attractive to attackers because it lowers the chance of detection while preserving reliable operator control. The main risk is not only stealth, but also the defender’s delayed understanding of what the channel is doing, which can extend dwell time and make containment harder once the intrusion is underway.

Failure mechanism: Security controls that depend on known protocol signatures, fixed ports, or readable payloads fail when the adversary changes framing, cadence, or encryption while keeping the channel functionally intact.

Impact: The environment may miss early command execution, treat malicious traffic as ordinary service communication, and allow theft, lateral movement, or exfiltration to continue longer than it should.

Practitioner Guidance

What to verify: Check whether the talking host should be making outbound connections at all, then confirm whether the destination, cadence, and message sizes match the application’s normal role. Traffic that is stable, repetitive, and externally pointed from an unexpected asset deserves more attention than a one-off noisy anomaly.

• Compare the session against approved service inventory, not just against a port-based allowlist.
• Look for repeated beacon intervals, identical request shapes, and responses that are too regular for the claimed application.
• Inspect the egress path for unknown hosts, unusual domains, or protocol mismatches between host role and network behaviour.

Common mistake: Treating encrypted traffic as inherently trustworthy or assuming that “unknown protocol” means harmless custom software. The practical question is whether the communication pattern is explainable for that system and that business function.

Practitioner takeaway: Custom C2 is best detected as a behavioural mismatch, so prioritise outbound baselining and destination control over trying to enumerate every possible malicious protocol variant.

Related resources from NHI Mgmt Group

• What breaks when attackers use legitimate AI APIs as command and control channels?
• How should security teams stop command and control traffic before attackers can use it for remote control and malware spread?
• What happens when attackers use Redis replication to hide command-and-control traffic?
• What happens when attackers use a web-based command and control platform to deploy payloads and collect data?