How can security teams tell whether a custom protocol is leaking sensitive data?

How do you confirm a custom protocol is exposing data on the wire?

Start by capturing traffic at a point where you can see the full session, then check whether the payload still reveals business records, usernames, session material, configuration values, or other stable identifiers. A custom protocol may look binary or proprietary, but if the same fields appear in clear form across flows, the protocol is still leaking information.

It also helps to compare what the application thinks it is sending with what actually crosses the network. If packet boundaries, message length, or repeated tokens make sensitive fields easy to reconstruct, you do not need the protocol to be “plaintext” in the usual sense for it to be exposed.

A useful test is whether someone who only sees traffic can infer the meaning of the conversation without access to the application. If they can read or reliably reconstruct sensitive content from captures, the protocol is disclosing more than it should.

What leakage patterns are most important to look for?

Readable fields are the obvious case, but many leaks show up indirectly. Repeated identifiers, timestamps, hostnames, tenant names, error strings, and configuration values can still be sensitive because they help an observer map users, systems, and workflows. Even when the body is encoded or compressed, metadata can reveal enough to support profiling or correlation.

Pay attention to values that should not be stable across sessions. If the same token, account reference, or environment marker appears repeatedly, it can support replay, tracking, or privilege discovery even if the protocol was designed to hide the main payload.

Also watch for “almost protected” traffic, where some fields are encrypted but adjacent data remains exposed. That creates a false sense of safety because the most sensitive object may be protected while the surrounding context still reveals operational detail.

What should teams do once they prove the protocol leaks sensitive content?

Use the finding to decide whether the problem is confidentiality, authentication, or both. If the wire data includes secrets, reusable identifiers, or configuration values, the protocol needs stronger transport protection and a tighter message design, not just a claim that it is proprietary. If the traffic also supports session or identity replay, the protocol must be treated as an access-control issue as well as a data exposure issue.

That is where Encryption Guide is useful as a reference point, because the real fix is often to move from obscured payloads to authenticated encryption and better key handling. For protocol design and deployment, Network Ports and Protocols Security Guide helps teams think about exposure at the transport layer, while Data Security Guide reinforces the need to discover where sensitive data is actually moving, not where teams assume it is moving.

If the protocol is used in production, you should also test whether observability, logging, or debugging features are leaking the same information in other paths. A protocol can be fixed on the wire and still remain exposed through logs, traces, or support tooling.

Risk and Threat Considerations

Custom protocols often fail because teams assume obscurity is equivalent to protection. That breaks down quickly when a capture tool, proxy, compromised host, or insider can observe the stream and reconstruct sensitive fields. The same weakness can also assist replay, correlation, and lateral movement when identifiers or session material are exposed alongside the payload.

Failure mechanism: Sensitive data is transmitted in a form that is recoverable from packet captures, message structure, or repeated values, allowing an observer to read, correlate, or replay information that should have been protected in transit.

Impact: Confidential data exposure can lead to credential theft, session abuse, workflow mapping, and broader trust in the protocol collapsing because observers can infer meaningful business or identity data from the wire.

Practitioner Guidance

What to verify: Confirm exposure from an actual capture point, not from application assumptions. If a tester can recover business meaning, identity references, or reusable tokens from the stream, treat the protocol as sensitive-data-bearing and not merely “non-HTTP.”

Decision rule: If the protocol carries anything that would be damaging to disclose in a log file, assume it also needs protection on the wire, because “custom” does not reduce the confidentiality requirement. If you cannot remove the exposed content, redesign the message so the sensitive field is protected independently of the surrounding transport.

Practitioner takeaway: The key judgement is whether an observer can understand or reuse the traffic without application access, because that is the point where protocol design, encryption, and authentication all become mandatory.

Related resources from NHI Mgmt Group

• How do security teams know whether a RAG pipeline is leaking sensitive data?
• How can security teams tell whether an AI coding assistant is ingesting too much sensitive data?
• How can teams tell whether cloud data security controls are actually reducing risk?
• How can security teams tell whether dark data governance is working?