GitHub traffic failover shows why routing resilience needs identity controls

At a glance

What this is: This is a resilience and infrastructure design article showing how Blacksmith bypassed degraded ISP routing by proxying GitHub traffic through an authenticated tailnet path.

Why it matters: It matters because identity and access controls increasingly govern which services can reach critical build systems, and secure failover now depends on both network design and service authentication.

By the numbers:

• GitHub traffic from Blacksmith's datacenter saw five to 20-second HTTP stalls on roughly 7-10% of requests during the outage.
• The problem took 16 hours to isolate to the problematic ISP.

👉 Read Tailscale's analysis of GitHub traffic failover with Tailscale Services

Context

GitHub traffic resilience is not just a network reliability issue. In build and delivery environments, routing failures can stop authentication-dependent workflows, interrupt package retrieval, and break access to critical developer services even when applications themselves are unchanged. For identity and access teams, the important question is how to preserve trusted service access when the path to those services degrades.

This article is especially relevant where network controls, service authentication, and infrastructure identity meet. The proxy model uses a stable, authenticated service path inside a tailnet, which means reachability is governed by device trust and service membership rather than ad hoc network exposure. That is a typical pattern for modern platform teams facing external dependency failures, not an edge case.

Key questions

Q: How should teams design failover for critical developer services?

A: Design failover around the smallest reachable scope that restores service, not around full-network redundancy. Route only the critical dependency through the alternate path, keep the rest of the environment on the normal route, and require authenticated membership for the fallback service. That preserves control, makes troubleshooting easier, and avoids turning resilience into a broad exception.

Q: Why does authenticated service reachability matter in resilience design?

A: Because an alternate path that is open to everyone is just another exposed control surface. When a service is reachable only through managed identity, device trust, or a tailnet, the organisation can restore access without exposing the proxy itself. That gives operations a safer way to recover from routing failure while keeping access governed.

Q: What breaks when failover paths are not scoped to specific services?

A: Broad failover paths create unnecessary exposure, more complex troubleshooting, and harder logging. They also make it difficult to prove which traffic used the backup route and why. Service-scoped routing keeps the exception bounded, preserves normal paths for everything else, and makes later review much more reliable.

Q: Which frameworks help teams govern alternate network paths and service access?

A: NIST SP 800-207 fits the zero trust side because it emphasizes continuous verification and controlled access to resources. NIST SP 800-53 Rev 5 also applies where teams need access control, system integrity, and configuration management around proxying, routing changes, and service reachability.

Technical breakdown

How transparent proxying preserves GitHub access during routing failure

Transparent proxying intercepts traffic without changing the client application. In this case, packets destined for GitHub were rewritten at the kernel layer, forwarded to a local proxy manager, and then carried through an HTTP CONNECT tunnel to a proxy pool with direct GitHub peering. The important design choice is selectivity: only GitHub-bound traffic is redirected, so the rest of the environment keeps using the default network path. That limits blast radius while keeping recovery fast.

Practical implication: isolate failover only to the critical destinations that need resilience, not the whole network.

Why ipset and atomic CIDR swaps reduce operational risk

ipset is a Linux kernel structure that matches packets against large IP sets efficiently. Instead of maintaining dozens of sequential iptables rules for each GitHub CIDR, the environment loads the ranges into one set and swaps them atomically when the ranges change. That avoids connection drops and race conditions because in-flight traffic continues matching the old set until the swap completes. This is a control-plane choice as much as a networking one, because it keeps policy updates predictable under load.

Practical implication: use atomic policy updates for IP-based routing rules where live traffic cannot tolerate partial state.

Tailscale Services as authenticated service reachability

Tailscale Services provided load distribution, health checking, and encrypted connectivity without a separate public load balancer. Each Squid instance joined the service over the tailnet and became reachable through a stable service name rather than exposed IP addresses. That creates a trust boundary based on tailnet membership and device identity, which is operationally different from opening proxy ports to the internet. The security value lies in reducing exposed surface while keeping service discovery stable for clients.

Practical implication: prefer authenticated service membership over public exposure when failover endpoints must stay reachable and controlled.

NHI Mgmt Group analysis

Routing resilience is becoming an access control problem, not just an availability problem. When build systems depend on third-party services such as GitHub, the question is not only whether the network is up. It is whether critical paths remain reachable through controlled, authenticated channels when upstream routing degrades. That matters to IAM and NHI programmes because service reachability, service authentication, and workload identity are increasingly coupled. Practitioners should treat resilience paths as governed access paths, not emergency exceptions.

Selective failover is a stronger governance pattern than blanket network redundancy. The article shows a narrow proxy path for GitHub traffic only, which is a cleaner control model than rerouting everything through a secondary network. That aligns with least privilege thinking: the smaller the failover surface, the easier it is to reason about trust, logging, and blast radius. For identity architects, the lesson is to scope alternate paths by service and by trust boundary, not by convenience.

Authenticated service membership is the real control plane for modern infrastructure dependencies. The proxy pool was reachable only inside the tailnet, which turns availability tooling into an access-governed system. That is a useful named pattern: controlled failover identity, where fallback connectivity is only usable by approved devices and services. The implication for practitioners is that resilience design now belongs in identity governance conversations, especially where CI, package delivery, and workload automation depend on external platforms.

Network fallback should be designed with auditability and lifecycle control from day one. If a disaster recovery path cannot be enumerated, logged, and revoked cleanly, it becomes another standing exception. The article's approach avoids public exposure, but the same principle applies to any alternate route, proxy, or service tunnel used by automation. Teams should assume that recovery paths will persist longer than intended unless they are explicitly governed.

What this signals

Controlled failover identity will become a more common operating pattern as platform teams route automation through authenticated fallback services instead of exposing emergency endpoints. That shift matters because the backup path is still an access path, and access paths need policy, logging, and revocation. Teams that already use zero trust principles can apply the same discipline to recovery routes by aligning them with NIST SP 800-207 Zero Trust Architecture.

The practical signal for security programmes is that resilience and identity governance are converging around service membership, not just user login. If CI runners, bots, and deployment systems can reach critical external services only through managed authentication, then those service identities need lifecycle controls, visibility, and offboarding just like human accounts. That is where machine identity governance stops being a niche concern and becomes an operational dependency.

As platform teams add alternate paths for third-party dependencies, the hidden risk is that emergency routing becomes a permanent exception. The control question is whether every fallback path can be enumerated, scoped, and revoked as cleanly as a privileged account. If not, the organisation has built availability at the cost of governance.

For practitioners

• Define service-specific failover paths Document which upstream dependencies are allowed to bypass normal routing during degradation, and keep the exception list limited to critical destinations such as GitHub, package registries, or artifact stores.
• Gate fallback endpoints with identity controls Require tailnet membership, device trust, or equivalent service authentication before any proxy or alternate route can be used by automation.
• Use atomic updates for destination policy Load destination IP ranges into a single policy set and swap the set atomically so active connections are not dropped during list changes.
• Test failover against partial routing degradation Rehearse scenarios where only one ISP, one edge path, or one region is impaired so teams can verify that routing recovery works without broad network disruption.

Key takeaways

• Routing failures in CI are not only an uptime problem. They are a control problem because service reachability determines whether critical automation can still operate safely.
• Scoped fallback routing is preferable to broad network redundancy because it keeps the exception narrow, auditable, and easier to revoke.
• Authenticated service membership turns resilience into governed access, which is the right model for automation that depends on external platforms.

Key terms

• Transparent proxy: A transparent proxy intercepts traffic without requiring the client to change how it connects. In this design, the network rewrites traffic destined for a chosen service and forwards it through an alternate path while preserving the original destination and user experience.
• Tailnet: A tailnet is a private network formed by devices authenticated into the same Tailscale environment. It acts as an access boundary where connectivity is based on identity and policy rather than public IP exposure, making device membership a governance issue, not just a networking one.
• ipset: ipset is a Linux kernel data structure for storing large groups of IP addresses or ranges efficiently. It lets packet filtering rules check membership in constant time, which is useful when many destination ranges must be updated without disrupting live connections.
• Controlled failover identity: Controlled failover identity is a governance pattern in which alternate operational paths are available only to approved services, devices, or workloads. It treats resilience routes as access-controlled assets, so emergency connectivity remains visible, scoped, and revocable.

What's in the full article

Tailscale's full article covers the implementation detail this post intentionally leaves for the source:

• Kernel-level packet rewriting logic using iptables and SO_ORIGINAL_DST to preserve original destinations
• Squid proxy configuration and tailnet service registration commands for the GitHub failover path
• How GitHub CIDRs are loaded into ipset and atomically swapped during IP range changes
• Operational notes on why the proxy design avoids public IP exposure and separate load balancers

👉 Tailscale's full post covers the proxy design, ipset handling, and tailnet service model.

Deepen your knowledge

NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, machine identity security, and secrets management. It helps security and platform teams apply identity controls to automation, service access, and operational resilience.