Cloud Region Outage Resilience: Design for a Provider Region Going Down

A workload deployed across three availability zones in one cloud region is still a single-region workload, and when that region or a global service anchored in it degrades, multi-zone design does not help. Large regional events at every major provider have shown the same pattern: applications that were "highly available" stopped because their failover depended on the region that failed, on a global control plane, or on DNS and identity changes that could not be made during the event. This guide is the technical design view: multi-region patterns, control plane and data plane dependencies, DNS and traffic steering, identity during an outage, data replication and region evacuation testing. It covers Amazon Web Services (AWS), Google Cloud and Microsoft Azure; providers are listed alphabetically, and the order is not a ranking. The strategic questions of provider concentration and exit plans are covered in the Cloud Concentration Risk and Exit Strategy Guide.

Key takeaways

• Zones protect against zone failure, not region failure. Region-level resilience needs a second region with its own copy of data, capacity and configuration, and a tested way to move traffic to it.
• Fail over using data plane operations only. All three providers' guidance points the same way: recovery steps that need create, update or delete calls against a degraded control plane can stall exactly when you need them.
• Know which global services are anchored where. AWS documents that IAM, Route 53, CloudFront and Organizations have control planes in us-east-1 (Route 53 now also offers a standby control plane endpoint in us-west-2 for public record changes); Azure and Google Cloud describe global identity and DNS services built for redundancy. Your design should assume you cannot change any of them mid-incident.
• Pick a pattern per workload, not per company. Backup and restore, pilot light, warm standby and active-active differ by an order of magnitude in cost and recovery time; tier workloads and match them.
• Only an evacuation drill proves the design. Measure time to serve from the secondary region at full load, including DNS propagation, quotas and identity, and repeat after architecture changes.

How region outages actually break applications

Region events are rarely "every server switched off". More often a foundational service in one region degrades (storage, DNS, a database control plane, networking) and the failure spreads through dependencies. The October 2025 us-east-1 event, analysed in the Major IT Outage Lessons Guide, started with a DNS automation defect in one regional database service and spread to services and customers that depended on it, including some outside the region. Four dependency types decide whether you ride through such an event:

• Regional data dependencies: the primary database, object store or queue lives only in the affected region, so a second region has compute but nothing to serve.
• Control plane dependencies: failover requires launching instances, scaling groups, creating records or changing policies, all of which are control plane calls that may be throttled or failing.
• Global service anchors: a service marketed as global may have its management plane in one region, so changes (not reads) depend on that region.
• Hidden third-party dependencies: SaaS, identity providers, CI/CD, secrets managers and monitoring tools that themselves run in the affected region.

Control planes, data planes and static stability

The control plane handles management operations (create, update, delete, list); the data plane does the runtime work (serving DNS answers, authenticating requests, moving packets, reading objects). Data planes are generally built to higher availability than control planes, so a design is statically stable when it keeps working, and can fail over, without making control plane changes during the event. Each provider documents this in its own terms.

Amazon Web Services

The AWS Fault Isolation Boundaries whitepaper lists global services whose control planes are in us-east-1 with data planes in each region or edge location: IAM, AWS Account Management, AWS Organizations, Amazon Route 53 (public and private hosted zones and health checks), Amazon CloudFront, AWS WAF and AWS Certificate Manager for CloudFront, and AWS Shield Advanced. Application Recovery Controller, Global Accelerator and Network Manager have control planes in us-west-2. The whitepaper also notes that S3 CreateBucket, DeleteBucket and bucket configuration calls depend on us-east-1 even when made against another region. Its recommendations: keep global control plane operations and consoles out of the recovery path, use regional STS endpoints rather than the global endpoint, avoid depending on specific new bucket names during recovery, and use the Route 53 standby control plane endpoint in us-west-2 (announced December 2025) if public DNS records must change during a us-east-1 impairment.

Google Cloud

Google Cloud's geography and regions documentation separates zonal, regional, multi-regional and global resources. Multi-regional resources are designed to keep functioning after the loss of a single region. Global resources such as IAM policies, global firewall rules, global load balancer configurations and Pub/Sub topics are stored in replicated databases, and Google states that it deploys updates to internal multi-region services one region at a time. It also notes that access to Google Cloud APIs depends on DNS, global load balancers and points of presence, and recommends replicating data to another region or snapshotting it to a multi-regional resource, then testing recovery with fault injection and simulated failures.

Microsoft Azure

The Azure Well-Architected Framework names Microsoft Entra ID, Azure DNS and Azure Key Vault as global services with built-in redundancy that need no redundancy configuration, and states that Azure DNS public and private zones are zone- and geo-redundant. For hybrid name resolution it recommends two or more Azure DNS Private Resolvers in different regions with primary and secondary conditional forwarding. It also recommends CanNotDelete locks on data stores, DNS zones and traffic routing layers so accidental deletion does not remove redundancy. Azure's region pairs guidance is explicit that deploying to a paired region does not by itself provide failover, and that Microsoft-managed failover of geo-redundant storage happens only in catastrophic situations, so it should not be the primary recovery approach.

Choosing a multi-region pattern

The four patterns below use AWS's names from its disaster recovery whitepaper; Azure's Well-Architected guidance describes the same range as cold spare, warm spare and active-active (at capacity or overprovisioned), and Google Cloud's planning guide describes cold, warm and hot patterns.

• Scale-up is a control plane dependency: AWS notes that auto scaling is a control plane activity; Azure notes that "at capacity" active-active can degrade during scale-up. If the secondary must absorb full load immediately, provision it at full capacity (overprovisioned), which is the statically stable choice.
• Replication is not backup: all three providers' guidance separates replication from protection against corruption or malicious deletion. Continuous replication copies the bad write too, so keep point-in-time backups alongside it. See the Backup Security and Immutable Backup Guide.
• Consistency costs availability: Azure recommends eventual consistency where requirements allow and partitioning so one failure affects a subset of transactions; Google Cloud's multi-regional archetype uses synchronous replication only where the RPO or cross-region consistency need demands it.
• Small organisations: a cold or pilot light design in a second region, with infrastructure as code and tested restores, is achievable and far better than relying on zones alone. Active-active usually only pays for itself in a few services.

Traffic steering and DNS

Moving users to the healthy region is where many failovers stall. Design the switch so it uses data plane mechanisms prepared in advance.

• Pre-create everything: records, health checks, certificates and load balancers in the secondary region should exist before the event. Creating them during the event is a control plane dependency.
• Keep TTLs honest: a record with a long TTL keeps clients on the failed region however fast you switch. Set TTLs on failover records to match the RTO.
• Prefer manual decision, automated execution: AWS warns that false alarms in automatic failover cause the same losses as real failovers, and Azure notes automation can produce false positives. A common answer is a human decision that triggers a fully scripted, pre-tested switch.
• Protect DNS itself: registrar and DNS administration accounts are recovery-critical; see the DNS Security Guide.

Where identity fits: access during a region outage

Identity is the dependency teams most often forget to test. If engineers cannot sign in, or workloads cannot obtain tokens, the secondary region is unreachable however well it is built.

• AWS: IAM's control plane is in us-east-1 while authentication is served regionally, so plan to make no IAM changes during a us-east-1 event. Pre-create the roles and policies the recovery needs, and configure SDKs and tools to use regional STS endpoints; AWS notes that credentials from a regional STS endpoint are valid in any region.
• Google Cloud: IAM policies are global resources. Google's DR planning guide recommends replicating IAM policy with Terraform, managing VM administrative access with OS Login, syncing accounts with Cloud Identity and verifying that permissions in the DR environment match production.
• Microsoft Azure: Microsoft Entra ID is a global service with built-in redundancy. Azure's DR guidance asks you to duplicate accounts, RBAC permissions and policies in the secondary region at the same security baseline, and to store DR credentials and certificates in highly available locations replicated across regions, with offline copies for the worst case.

Across all three, keep tested break-glass accounts that do not depend on a federated identity provider hosted in the affected region, and make sure workload identities (service accounts, managed identities, roles) are bound in the secondary region before you need them. See the Break-Glass Account Guide and Emergency Identity Operations When the Primary IdP Is Unavailable.

Implementation: a phased approach

1. Tier workloads: agree RTO and RPO per workload with the business, as Google Cloud's planning guide and Azure's criticality tiers both recommend, and assign a pattern from the table above.
2. Map dependencies: for each tier 1 workload list every regional and global dependency, including identity, DNS, secrets, CI/CD, observability and SaaS. Mark each call made during failover as control plane or data plane.
3. Remove control plane steps from the failover path: pre-provision capacity or accept the dependency explicitly, pre-create DNS, certificates and IAM, switch to regional endpoints.
4. Replicate data and keep backups: cross-region replication for low RPO, plus point-in-time backups in a separate account for corruption and ransomware.
5. Codify the secondary region: infrastructure as code deploys both regions; check service quotas and reserved capacity in the secondary region.
6. Write the runbook: who declares, activation criteria, the scripted switch, post-failover tasks (DNS, connection strings) and a separate failback plan, as Azure's guidance requires.
7. Test in rings: tabletop, then non-production dry run, then a production evacuation of one service, then full workload evacuation. Roll back by reversing the traffic switch, which should itself be tested.

Testing region evacuation

• Test the real switch: shift traffic away from the primary region using the same mechanism you would use in an incident, and keep it there long enough to see full-load behaviour.
• Simulate control plane loss: run the failover with permissions that deny control plane calls you claim not to need. If the drill fails, the design was not statically stable.
• Test restores, not only replication: AWS notes restores are control plane operations and recommends regular scheduled restores; Azure includes restoring from cold storage in component tests.
• Test access paths: Google Cloud recommends testing the case where your usual access path to the cloud fails and keeping an alternative.
• Use provider tooling where it helps: AWS Resilience Hub, Application Recovery Controller and Fault Injection Service on AWS; fault injection and simulated failure scenarios on Google Cloud; Azure Chaos Studio and Azure Site Recovery drills on Azure. See Security chaos engineering.
• Review twice a year: Azure suggests reviewing the DR plan about every six months and after architecture changes; use measured drill times as the planning minimum.

Detection and measurement

Watch the provider health dashboards (AWS Health, Google Cloud Service Health, Azure Service Health) through alerts that do not themselves depend on the affected region, and monitor synthetic transactions from outside the provider so you see user impact before the status page does.

Framework mapping

Common failure patterns

• Calling multi-zone "disaster recovery": every copy of the data is in one region, so a regional storage or control plane event removes all of them at once.
• A failover runbook full of control plane calls: launching instances, scaling groups, editing records or changing IAM during the event, when those APIs are the ones degraded.
• Tooling in the failed region: CI/CD, secrets manager, bastion hosts, monitoring or the runbook wiki live only in the primary region, so the team cannot act.
• Relying on provider-managed geo-failover: treating paired-region or provider-initiated storage failover as the plan, when Azure, for example, says it happens only in catastrophic situations.
• Unplanned failback: traffic moves successfully, then data reconciliation and failback cause a second outage because no one designed the return.
• No quota in the secondary region: the secondary cannot scale to production size because limits were never raised there.

Practitioner checklist

• Every tier 1 workload has an agreed RTO, RPO and multi-region pattern, documented and owned.
• A dependency map marks each failover step as control plane or data plane, and global service anchors are listed.
• Failover uses pre-created DNS records, health checks, certificates and identities; no IAM or DNS changes are required mid-event.
• Data is replicated cross-region for RPO and also backed up point-in-time in a separate account.
• Regional STS endpoints, regional service endpoints and break-glass access are configured and tested.
• CI/CD, secrets, monitoring and runbooks are reachable when the primary region is down.
• Secondary region quotas and capacity support full production load.
• A production region evacuation drill runs at least yearly per tier 1 workload, with measured times feeding the plan, and a tested failback.

Standards and references

• AWS: Fault Isolation Boundaries, Global services
• AWS: Disaster Recovery of Workloads on AWS, Disaster recovery options in the cloud
• AWS: Application Recovery Controller Region switch
• Google Cloud: Geography and regions
• Google Cloud: Disaster recovery planning guide
• Google Cloud: Multi-regional deployment archetype
• Microsoft: Azure Well-Architected Framework, redundancy and multi-region design
• Microsoft: Azure Well-Architected Framework, disaster recovery strategy
• Microsoft: Azure region pairs and nonpaired regions

Related NHI Mgmt Group resources: Cloud Concentration Risk and Exit Strategy Guide · Major IT Outage Lessons Guide · Business Continuity and Disaster Recovery Guide · Cyber Resilience Guide