Subscribe to the Non-Human & AI Identity Journal

Notifications
Clear all

Happy eyeballs and DNS timeouts: what broke in this CI path?


(@nhi-mgmt-group)
Member Moderator
Joined: 1 year ago
Posts: 15051
Topic starter  

TL;DR: A timeout in DNS-over-TLS connection handling surfaced during an OCaml migration, exposing how sequential connection attempts, unclear timeout propagation, and incomplete fallback logic can break modern client networking in CI environments, according to Semgrep. The case shows that resilience in networked tooling depends on concurrency, not just longer timeouts.

NHIMG editorial — based on content published by Semgrep: a guest post on debugging an OCaml happy-eyeballs timeout in CI

Questions worth separating out

Q: How should teams design retry logic for remote security tooling?

A: Design retry logic so one slow or silent failure cannot block the whole request path.

Q: Why do DNS failures often look like application timeouts?

A: Because DNS is usually the first network dependency in the request chain, and a stalled resolver prevents the application from ever reaching HTTP.

Q: What breaks when connection attempts happen sequentially in distributed systems?

A: Sequential attempts assume the first path will fail fast, but silent packet loss can leave the client waiting until the timeout expires.

Practitioner guidance

  • Map every remote dependency in build and security pipelines Identify where scanners, policy engines, and CI jobs depend on DNS, HTTP, TLS, package registries, or external config services.
  • Test fallback paths under silent-drop conditions Validate behaviour when packets are dropped rather than rejected, because many real failures do not produce explicit negative responses.
  • Instrument timeout propagation across layers Make sure resolver, transport, and application timeouts are visible in logs and metrics so one stalled dependency cannot masquerade as another.

What's in the full article

Semgrep's full post covers the debugging detail this analysis intentionally leaves out for brevity:

  • The exact log traces that exposed the DNS-over-TLS timeout path in CI.
  • The OCaml happy-eyeballs interface changes, including concurrent connection attempts and cancellation handling.
  • The resolver behaviour differences between UDP, TCP, and DNS-over-TLS in the test setup.
  • The implementation details behind the final fix across happy-eyeballs, dns-client, and related libraries.

👉 Read Semgrep's analysis of the OCaml happy-eyeballs timeout bug →

Happy eyeballs and DNS timeouts: what broke in this CI path?

Explore further

View Full Forum →  |  NHI Foundation Course →



   
Quote
(@mr-nhi)
Member Moderator
Joined: 3 months ago
Posts: 14635
 

Connection resilience is now a governance issue, not only an implementation detail. Modern build and security tooling depends on layered network paths that can fail in different ways, and those failure modes often surface as opaque application errors. When CI, scanners, or policy engines rely on remote config retrieval, a timeout in DNS can look like an HTTP failure and consume operational time. The practical conclusion is that resilience controls must be designed into the client path, not added after outages appear.

A question worth separating out:

Q: How can security teams know whether network fallback is actually working?

A: Measure how often alternate paths are attempted, how quickly they are launched, and whether failed attempts are cancelled cleanly after success. If logs only show a final timeout, the fallback mechanism is not observable enough to trust. Effective resilience requires both behaviour and evidence.

👉 Read our full editorial: Happy eyeballs failures show why DNS and HTTP retries need redesign



   
ReplyQuote
Share: