Secrets Manager Resilience Guide: keeping secrets available through outages and regional failures

When the secrets manager is unavailable, every workload that fetches a credential at start-up, renews a lease or rotates a key stops with it, so the secrets manager is a tier 0 dependency for every non-human identity you run. Teams spend years moving credentials out of code and into a central vault, then discover during the first serious outage that nothing can authenticate, that the failover region holds stale values, or that the break-glass path itself depends on the vault. This guide covers availability architecture for self-hosted and cloud secrets managers, how clients should behave when the service is degraded, regional failover for secrets, outage operations and testing. It is for platform, SRE and identity teams who run or depend on a secrets manager. Recovering a destroyed vault, seal keys and backups is covered in the Recovery of Cryptographic Keys and Secrets Guide, and securing the vault against attackers in Securing the secrets manager itself.

This is an independent guide from NHI Mgmt Group, based on each vendor's public documentation as of November 2026. Products are named only as examples, listed alphabetically; naming is not a ranking. It is not sponsored or reviewed by any vendor named. Product features change, so confirm settings against current vendor documentation.

Key takeaways

• Availability and recovery are different problems. High availability keeps the service answering through a node or zone failure; disaster recovery brings it back after a cluster or region is lost. You need both, tested separately.
• Client behaviour decides the blast radius. A workload that caches its credential and retries with backoff survives a short outage; one that fetches on every request fails immediately and adds load during recovery.
• Replicas differ in what they carry. Some replicas serve reads and keep their own tokens, some mirror tokens but serve nothing until promoted, and some regional failovers are read-only. Know which one you have before an outage.
• Rotation and failover interact. Rotating during a degraded period can leave regions, caches and consumers holding different values. Pause scheduled rotation during failover and resume it deliberately.
• Break-glass must not depend on the thing that failed. Keep a sealed, separately stored path to the credentials needed to restore the vault and its dependencies.

Why the secrets manager is a tier 0 dependency

A secrets manager sits on the start-up path of every workload that pulls a database password, API key, certificate or cloud credential from it. With dynamic secrets and short leases the dependency is continuous: workloads renew or reissue credentials many times a day (see the Dynamic Secrets and Ephemeral Credentials Guide). The vault in turn depends on its storage, its unseal mechanism (a KMS or HSM for auto-unseal), DNS, the network path, the identity provider used for operator login, and the platform identity used by workloads to authenticate (Kubernetes service account tokens, cloud instance identity). An outage anywhere in that chain looks like a secrets outage.

Start with a dependency map, recorded in your business continuity plan (see the Business Continuity and Disaster Recovery Guide):

• Upstream: storage backend, auto-unseal key service, certificates for the vault listener, DNS and load balancer, operator identity provider.
• Downstream: every workload, pipeline and agent that reads from the vault, with how it authenticates, whether it caches, and what happens when a read fails.
• Circular dependencies: a vault whose storage credentials, TLS keys or monitoring tokens are stored in the vault itself cannot be restarted cleanly. Find and remove these first.

Set recovery objectives per tier of consumer. Payment or authentication services may need a recovery time objective of minutes; batch jobs may tolerate hours. The vault's objective is the strictest of its consumers.

Availability models by platform

See the platform guides for configuration detail: HashiCorp Vault Guide, AWS Secrets Manager, Azure Key Vault Security Guide and the Cloud Secrets Managers Guide.

Self-hosted cluster design

• Quorum sizing: HashiCorp states a majority of voting nodes must be alive for Integrated Storage to operate. Three voters tolerate one failure and five tolerate two; spread voters across at least three failure domains so one zone loss does not take quorum.
• HA is not scale: HashiCorp notes HA improves availability, not throughput, because standbys forward requests to the active node. Size the active node for peak load including the thundering herd after an outage.
• Load balancing: health checks should route only to unsealed, healthy nodes; Vault's api_addr must be set correctly so 307 redirects (used when forwarding fails) point somewhere reachable.
• Quorum loss: the documented last resort is a raft/peers.json recovery on the surviving nodes, which HashiCorp warns can lose data and implicitly commits outstanding log entries. Rehearse it in a lab, not first in production.
• Seal dependency: with auto-unseal, every node restart needs the KMS or HSM. Put the seal key in a region and account that survives the same failures as the vault, and see the recovery guide for what happens if it is lost.

Choosing a resilience pattern

Decision guidance: a small team on one cloud usually gets the best return from the provider's zone redundancy plus client caching, with a tested restore. Large or regulated estates need a cross-region pattern with a written promotion decision and owner. Remember that replication protects against infrastructure loss, not against a destructive change or a compromised administrator: deletion and policy changes replicate too, so keep point-in-time backups and deletion protection as well (covered in the recovery guide).

Designing clients to survive an outage

Most of the resilience you can buy cheaply is in the consumer, not the vault. Microsoft's Key Vault guidance tells customers to retry transient failures with exponential backoff and to cache secrets in memory where possible.

• Cache with a bounded lifetime: AWS's Python caching client (SecretCache, configured with SecretCacheConfig) refreshes secrets every hour by default and does not invalidate on rotation, so a cached value can be stale for up to the refresh interval. Choose the interval against your rotation overlap, and note AWS states the cache is not security hardened.
• Agents and sidecars: Vault Agent caches tokens and leased secrets it manages and renews them; HashiCorp documents that shutting the agent down stops renewals but does not revoke the secrets. Vault Proxy is the documented option for static secret caching.
• Kubernetes sync: the External Secrets Operator re-reads the provider every refreshInterval and marks the ExternalSecret not Ready when a refresh fails; with the default deletionPolicy: Retain the synced Kubernetes Secret is kept if data disappears from the provider. Synced secrets keep pods starting during a short provider outage, at the cost of a second copy to protect (see the External Secrets Operator security guide).
• Lease and TTL headroom: set dynamic credential TTLs long enough to outlast your expected recovery time, or workloads will lose database access mid-outage when leases cannot be renewed.
• Fail closed, visibly: on a hard failure, a workload should stop and alert, never fall back to a hard-coded or environment default credential.
• Jittered retries: exponential backoff with jitter prevents thousands of pods hammering a recovering active node.

Rotation, consistency and failover

• Rotation runs in one place: AWS rotates a multi-Region secret in the primary Region and propagates the new value. If the primary Region is impaired, rotation cannot run until a replica is promoted, and after promotion the replica is a standalone secret whose rotation must be configured again.
• Read-only failover blocks rotation: an Azure Key Vault in failover can serve secrets but cannot accept writes, so scheduled rotation jobs fail. Expect that, suppress the alerts, and rotate once the primary returns.
• Dual-credential overlap: rotation schemes that keep the previous and current credential valid (two database users, two API keys) tolerate stale caches and lagging replicas. Single-credential rotation does not.
• After a performance-secondary promotion: HashiCorp documents that applications must reauthenticate and obtain new leases from the new primary. Make sure every client's auth method works against the secondary cluster before you need it.
• Compromise during an outage: if you must rotate because of a suspected leak while degraded, treat it as an incident and follow the If your secrets manager is compromised playbook.

Outage operations and break-glass

1. Detect and declare: alert on vault health endpoints, seal status, Raft quorum, replication lag and client error rates, not only on the cloud provider's status page. Microsoft notes it does not automatically notify you of zone or region failures; subscribe to Resource Health or the equivalent.
2. Freeze changes: pause scheduled rotation, policy deployments and secret writes that would diverge between regions.
3. Decide on failover: a named owner decides whether to promote a replica or wait, using the expected recovery time against the data loss from replication lag. Record the decision time.
4. Promote and redirect: promote the DR or performance secondary, or switch application configuration to the standby region; confirm clients can authenticate, not only reach the endpoint.
5. Use break-glass only if needed: sealed credentials for the vault's own dependencies (storage, KMS, cloud root or emergency accounts) held outside the vault, with dual control. See the Break Glass Access Procedure Template.
6. Fail back and reconcile: compare secret versions between regions, resume rotation, rotate any break-glass credential used, and review audit logs for the outage window.

Testing and rollback

• Game days in rings: start with a non-production cluster, then a single production node, then a zone, then a full regional promotion in a maintenance window.
• Test what clients see: measure how long each tier of workload keeps running with the vault unreachable; this is your real recovery objective.
• Rehearse promotion and demotion: know how to return to the original primary without a split brain, and document the replication re-establishment steps.
• Restore test: restore a snapshot to an isolated environment at least quarterly (an NHI Mgmt Group recommendation) and verify it unseals and serves secrets.

Detection and measurement

Framework mapping

Common failure patterns

• The vault depends on itself. Storage credentials, TLS keys or the monitoring token live in the vault, so a cold start is impossible until someone finds a copy.
• Replicas treated as backups. A deletion or bad policy replicates to every region within seconds; without point-in-time backups and deletion protection there is nothing to fail over to.
• Backups and replicas less protected than the primary. Every snapshot holds every secret. In the LastPass breach, stolen DevOps secrets gave access to cloud backups of customer vault data, which shows that backup copies need the same protection as the live store.
• Fetch-per-request clients. Applications without caching fail in the first second of an outage and then overload the vault on recovery.
• Untested promotion. The DR secondary exists but no one has promoted it, workload auth methods were never configured against it, and the runbook references retired hosts.
• Break-glass stored in the vault. Emergency credentials for the cloud account or KMS are themselves secrets in the unavailable vault.

Practitioner checklist

• Map the vault's upstream dependencies and every downstream consumer, and remove circular dependencies.
• Set recovery time and recovery point objectives from the strictest consumer.
• Run HA across at least three failure domains, with quorum sized for one domain loss.
• Choose and document a cross-region pattern, including who decides to promote.
• Make tier 1 clients cache with bounded lifetime, retry with jittered backoff and fail closed.
• Align dynamic secret TTLs and rotation overlap with expected recovery time.
• Pause rotation and policy changes during failover; reconcile versions after fail-back.
• Store break-glass credentials for the vault's own dependencies outside the vault, with dual control.
• Exercise node, zone and region failover and a snapshot restore on a fixed cycle, and record measured recovery time.

Standards and references

• HashiCorp: Vault High Availability
• HashiCorp: Vault Enterprise Replication
• HashiCorp: Integrated Storage
• AWS: Replicate Secrets Manager secrets across Regions
• Microsoft: Azure Key Vault availability and redundancy
• Google Cloud: Choose a replication policy for Secret Manager
• External Secrets Operator: ExternalSecret
• NIST: SP 800-53 Rev. 5

Related NHI Mgmt Group resources: Secrets Management Guide · Recovery of Cryptographic Keys and Secrets Guide · Securing the secrets manager itself · Business Continuity and Disaster Recovery Guide · Break-Glass Account Guide