TLS Inspection vs Encrypted Traffic Analysis: When to Decrypt and What Metadata Still Catches

Security teams face a choice they used to avoid: decrypt traffic to see inside it, or leave it encrypted and detect threats from what the connection reveals on the outside. TLS inspection terminates and re-encrypts sessions so content engines can read them. Encrypted traffic analysis keeps the session intact and works from handshake fields, fingerprints such as JA4, certificates where visible, DNS, and flow timing and volume. TLS 1.3, Encrypted Client Hello (ECH), QUIC and certificate pinning have moved the boundary between the two, so the right answer is now a mix decided per traffic class rather than a single network-wide setting. This guide compares the two by threat coverage, privacy, breakage and the risk of the interception CA, and gives decision criteria, failure modes and a worked scenario. It is written for network security architects, SOC leads and privacy officers. For the mechanics of running a decrypting proxy, see the TLS Inspection Guide; this page does not repeat them.

Key takeaways

• Decrypting sees content; not decrypting sees behaviour. Inspection is the only way to scan files, apply full URL and DLP policy and read payloads. Encrypted traffic analysis cannot see content, but it scales to every flow, including the ones you are not allowed or able to decrypt.
• The visible surface is shrinking, not gone. TLS 1.3 hides server certificates and ECH hides the server name, but the 5-tuple, packet sizes, timing, the client fingerprint and the outer ClientHello remain observable to a passive sensor.
• Inspection creates its own high-value target. An interception CA trusted by every managed device, and a proxy that re-validates every upstream certificate, are both points where a defect weakens security for the whole organisation.
• Pinning, mutual TLS and QUIC decide much of the answer for you. Traffic that cannot be intercepted without breaking it must be covered by metadata and endpoint telemetry, so build that coverage before you need it.
• The workable design is layered. Decrypt a defined set of high-risk categories, analyse the metadata of everything, and correlate both with endpoint process data, measured against coverage metrics rather than assumed.

The short answer

Use TLS inspection where you need to act on content: blocking malicious downloads, enforcing data loss prevention on uploads, and applying URL path policy for uncategorised and newly registered destinations. Use encrypted traffic analysis for everything else, and as the only network control for flows you exempt from decryption because of privacy, pinning, mutual TLS or protocol. Neither replaces endpoint detection: the endpoint sees plaintext before encryption and knows which process opened the connection, which neither network approach can see on its own.

What each option can detect

RFC 8404, the IETF's informational review of how encryption affects operators, notes that a significant share of malware hides inside TLS, and also that connections to command and control servers are typically encrypted already, so detection methods built for botnets are expected to change little. That tension is the core of this comparison: some threats are only visible in content, others are visible in behaviour whether or not you decrypt.

See the Beaconing and C2 Detection Guide for the timing analysis behind the C2 rows.

How encrypted traffic analysis works without decryption

Research by Anderson, Paul and McGrew in 2016, covering millions of TLS flows and 18 malware families, found that the observable features of TLS can detect malware communication while preserving the privacy of benign traffic, and could attribute families from a single flow. They also found that families which change their cryptographic behaviour are harder to classify, which is the main limit of the approach. The signals fall into five groups.

• Client fingerprint: JA4 summarises the ClientHello as a readable prefix (protocol t, q or d for TCP, QUIC or DTLS; TLS version; whether SNI is present; cipher and extension counts; first ALPN value) plus hashes of the sorted cipher list and the sorted extension list with signature algorithms. Because the lists are sorted, the hash does not change when a client reorders extensions. See the JA3 and JA4 TLS Fingerprinting Guide.
• Server response and certificate: JA4S fingerprints the server's reply and JA4X the certificate's structure. In TLS 1.2 the certificate chain is in clear text; in TLS 1.3 it is encrypted. Zeek's documentation shows that for TLS 1.3 its ssl.log has no certificate details, while server_name, version, cipher and curve are still logged.
• Flow metadata: RFC 8404 observes that the content length of encrypted traffic is effectively the same as the cleartext and that bitrate and pacing are application specific. Beacon intervals, upload to download ratios and session duration survive encryption. See the NetFlow and IPFIX Flow Telemetry Guide.
• DNS context: the query before the connection often names the destination when SNI does not. RFC 8404 notes that content filtering relying on DNS fails when DNS is encrypted, so this signal depends on forcing managed devices to your resolvers.
• Latency and distance: JA4L estimates client to server distance from handshake timing, which helps spot relays and proxies in the path.

Licensing note: the FoxIO repository states that JA4 itself is BSD 3-Clause licensed, while the other JA4+ methods (JA4S, JA4H, JA4X, JA4SSH, JA4T and others) are under the FoxIO License 1.1, which allows internal business use but requires an OEM licence for products that sell the capability. The repository lists Zeek, Wireshark and Arkime among JA4+ implementations, with Suricata support marked as under development.

TLS 1.3, ECH, QUIC and pinning: what changes for each option

Two practical points follow. First, when you block QUIC to keep inspection working, RFC 9312 advises blocking whole flows rather than dropping a share of packets, because partial loss can let a handshake complete and then prevent fallback to TCP. Second, Zeek's documentation shows an ECH connection with DNS over HTTPS leaving no identifying information in its ssl.log, which is what encrypted traffic analysis looks like at its weakest. See the Encrypted Client Hello Guide, the HTTP/2, HTTP/3 and QUIC Security Guide and the Certificate Pinning guide.

Threat model of each option

TLS inspection

Architecture in words: the client connects to a proxy that presents a certificate for the requested site, signed by an interception CA in the client's trust store. The proxy opens a second TLS session to the real server, validates its chain, decrypts both directions and passes content to scanning engines.

• Interception CA compromise: anyone holding the CA key can impersonate any website to every managed device. Treat it as a tier zero machine identity: hardware-protected key, a dedicated subordinate CA rather than your enterprise root, name constraints where your products support them, and monitoring of what it issues. See Certificate trust stores and root CA management.
• Weaker upstream validation: Durumeric and colleagues, at NDSS 2017, found that nearly all of the interception middleboxes and client security products they studied reduced connection security and many introduced severe vulnerabilities. The user can no longer see a bad upstream certificate, so the proxy's validation is the only check.
• Concentrated plaintext: the proxy, its logs and any cloud provider operating it hold decrypted data from across the organisation, which raises insider and provider risk.
• Exemption abuse: anything on the bypass list is uninspected, so an attacker who chooses a bypassed category or a pinned channel walks past content controls.

Encrypted traffic analysis

Architecture in words: a passive sensor or gateway receives a copy of traffic (span, tap or cloud packet mirroring) or flow records, extracts handshake fields and fingerprints, joins DNS and flow data, and sends results to analytics and the SIEM. Nothing is changed in the path.

• Mimicry: malware that uses the operating system TLS library or a common browser stack produces the same JA4 as legitimate software, so fingerprints alone rarely convict.
• Shared infrastructure: C2 hosted behind a large CDN or cloud platform shares IP addresses, certificates and, with ECH, public names with legitimate services.
• Blind spots by placement: sensors see only the segments they are fed; remote users on direct internet access and east-west traffic in cloud are often missed.
• Metadata is still personal data: destination and timing records reveal behaviour, so retention and access still need a lawful basis and limits.

Decision criteria

Small organisations: a cloud gateway that decrypts a short list of high-risk categories, plus EDR and enforced DNS resolvers, usually gives more value than an on-premises decrypting proxy. Encrypted traffic analysis at small scale often means using the handshake logging your gateway already produces rather than buying a separate sensor.

Large organisations: decrypt once, at the outermost point, and run a separate passive analysis layer on internal segments and cloud traffic mirrors where decryption is impractical. See NDR vs EDR for how a network layer and an endpoint layer divide the work.

Where identity fits

Two identity questions sit inside this decision. The interception CA is a machine identity with more reach than almost any other key you own, and should be inventoried, owned and rotated like one. Second, workload and service traffic (API clients, update agents, CI runners) is where encrypted traffic analysis performs best: a service account's client fingerprint, destinations and volumes are stable, so a new JA4 or a new destination from a build runner is a strong signal of a stolen token being used from tooling. Decrypting that traffic is rarely practical because workloads carry their own trust stores, and developers who hit interception errors tend to disable certificate verification. See the mTLS Guide for service traffic that must stay end-to-end.

Implementing a layered design

1. Inventory traffic classes. Group egress by user browsing, SaaS, pinned and mutual TLS applications, workloads, QUIC and unmanaged devices, and record which can be decrypted at all.
2. Turn on metadata everywhere first. Log TLS version, SNI, JA4 and JA4S, certificate fields where visible, DNS and flow records from gateways and sensors into the SIEM. This costs no breakage and gives a baseline.
3. Set your ECH and QUIC position for managed devices. Either disable them by browser and operating system policy so inspection and SNI logging keep working, or accept them and rely on DNS and endpoint telemetry. Write the decision down; do not let it happen by default.
4. Decrypt a narrow first scope in audit mode. Start with uncategorised, newly registered and high-risk categories plus file downloads for a pilot group; log what would be blocked and which applications break.
5. Build the exemption list from evidence. Add pinned, mutual TLS and privacy categories with an owner and review date, and give each exempt class a compensating metadata detection.
6. Expand by ring and test rollback. Widen to further groups only when the breakage rate is stable; keep a policy switch that returns a group to analysis-only within minutes if a business application fails.
7. Correlate with the endpoint. Join network detections to process data so an analyst can see which binary produced an odd fingerprint, which removes most fingerprint false positives.

Worked scenario

A 4,000 person firm runs a cloud gateway that decrypts all web traffic. A trading application pins its certificate and fails, a partner bank requires mutual TLS, and the privacy officer raises staff health and banking traffic. Chrome on managed laptops has started using HTTP/3. The team moves to a layered model: decryption continues for uncategorised, newly registered and file-sharing categories; health, banking, the trading application and the partner bank are exempt; QUIC is blocked at the gateway so browsers fall back to TCP; ECH is disabled by managed browser policy. All traffic, decrypted or not, now feeds JA4, SNI, DNS and flow records into the SIEM. Two months later an alert fires on an exempt banking category: a laptop shows a JA4 never seen in the estate, regular 60 second connections and steady small uploads to a host behind a CDN. EDR shows the process is an unsigned binary in a user profile, not a browser. The content gateway never saw it, because it sat in a bypassed category; the metadata layer did.

Detection and measurement

• Decryption coverage: percentage of managed-device web bytes that pass through content inspection, tracked against the share you intended to inspect.
• Metadata coverage: percentage of egress flows with a JA4 and either SNI or a correlated DNS answer. A falling figure means ECH, encrypted DNS or QUIC is spreading unmanaged.
• Exemption hygiene: number of exemptions without an owner or past their review date, target zero.
• Breakage rate: helpdesk tickets attributed to interception per thousand users per month.
• Fingerprint novelty: count of first-seen JA4 values per week by host group, with the share resolved to known software through endpoint correlation.
• Interception CA issuance: alerts on certificates issued by the interception CA outside the proxy, and on the CA appearing on unmanaged devices.

Common failure patterns

• Decrypting everything and analysing nothing: the exemption list grows, the bypassed traffic has no detection at all, and attackers who use pinned or exempt channels are invisible.
• Treating fingerprints as verdicts: blocking on a single JA4 match produces false positives against shared libraries and misses malware that uses the operating system stack.
• Ignoring QUIC and ECH drift: browsers update, HTTP/3 and ECH arrive silently, and both SNI logging and the proxy lose coverage without anyone noticing.
• Using the enterprise root as the interception CA: a compromise of the proxy then exposes the trust anchor for everything else the root signs.
• Breaking workload trust: intercepting server and CI traffic leads developers to disable certificate validation, which removes protection the proxy never restores.
• No privacy assessment: decrypting personal content without a documented assessment creates legal exposure; see the DPIA Guide.

Framework mapping

See M1020 SSL/TLS Inspection for the ATT&CK mitigation in detail.

Practitioner checklist

• Classify egress traffic and record which classes can and cannot be decrypted, and why.
• Log JA4, JA4S, SNI, TLS version, certificate fields, DNS and flow records for all egress, decrypted or not.
• Decide and document your ECH, QUIC and encrypted DNS position for managed devices.
• Limit decryption to categories where a content verdict changes the outcome, starting in audit mode.
• Give every exemption an owner, a review date and a compensating metadata detection.
• Run the interception CA as a dedicated, hardware-protected subordinate with issuance monitoring.
• Correlate network detections with endpoint process data before blocking on fingerprints.
• Complete a privacy assessment for any decryption of staff traffic and review it annually.
• Track decryption coverage, metadata coverage and breakage monthly.

Standards and references

• IETF: RFC 9849 TLS Encrypted Client Hello
• IETF: RFC 9312 Manageability of the QUIC Transport Protocol
• IETF: RFC 8404 Effects of Pervasive Encryption on Operators
• NIST: SP 800-53 Rev. 5
• MITRE ATT&CK: SSL/TLS Inspection (M1020)
• FoxIO: JA4+ Network Fingerprinting
• Zeek: ssl.log documentation
• NDSS: The Security Impact of HTTPS Interception

Related NHI Mgmt Group resources: TLS Inspection Guide · JA3 and JA4 TLS Fingerprinting Guide · NDR vs EDR · NDR Guide · Encrypted DNS Enterprise Guide