Credential Rotation at Scale: Zero-Downtime Patterns

Rotating one credential is a change ticket; rotating thousands of machine credentials across services, pipelines, partners and regions is a distributed systems problem, and most rotation outages come from a consumer nobody knew about or a cut-over that left no valid credential in place. Teams that cannot rotate safely stop rotating, and long-lived secrets accumulate until an incident forces an emergency rotation under the worst possible conditions. This guide is about the engineering of rotation at fleet scale: mapping dependencies, choosing an overlap pattern for each credential type, sequencing and staging the roll, rolling back, and measuring whether the programme is working. It is for platform, SRE and secrets management teams. It complements the step-by-step Secrets Rotation Runbook Template and the post-incident lessons in Incomplete rotation breaches.

Key takeaways

Zero downtime needs overlap. Every safe rotation pattern keeps two valid credentials for a window: alternating users, dual keys, versioned secrets or overlapping signing keys. Single-credential swaps always have a gap.
The dependency map is the real work. You cannot rotate safely what you cannot trace to every consumer, including caches, pipelines, partner systems and disaster recovery copies.
Disable before you delete. Disabling the old credential is a reversible test that the new one is in use everywhere; deletion is the irreversible final step.
Consumers must reload without redeploying. Rotation at scale fails when every rotation needs a restart; build applications that fetch on start, refresh on schedule and re-fetch on authentication failure.
Routine rotation is emergency rehearsal. An organisation that rotates on a schedule can rotate everything quickly after a compromise; one that never rotates will discover its dependencies during the incident.

Why rotation breaks at scale

Most rotation outages share a short list of causes. A consumer holds a cached copy and keeps using the old value after it is revoked. A second application, a cron job or a partner integration was quietly reusing the same credential. The rotation updated the target system and the secret store at slightly different moments, and requests in between failed. A downstream artefact derived from the credential, such as a signed token, was invalidated along with it. Or the rotation job itself failed halfway, leaving the secret store and the target out of step.

Each of these is predictable, which means each can be designed out. The AWS Secrets Manager documentation is candid about the single-user case: while rotation is happening there is a short period between when the password changes in the database and when the secret is updated, and it recommends an appropriate retry strategy for that window. The patterns below remove the window rather than retrying through it.

Map dependencies before you rotate

A rotation plan starts with an inventory of each credential and every consumer of it. The OWASP Secrets Management Cheat Sheet's audit fields are a useful baseline for what to record: who requested the secret, when and by what it was used, expiry, and failed or expired-secret use. For each credential, capture:

Field	Why it matters for rotation	Where to get it
Owner and system of record	Someone must approve and verify; the secret store must be the single source	Secrets inventory, CMDB, NHI inventory
Target system and credential type	Determines which overlap pattern is possible (two users, two keys, versions, none)	Target documentation
All consumers	Every service, job, pipeline and partner that must pick up the new value	Secret store access logs, target authentication logs, code and config search
How each consumer loads it	Hot reload, restart, manual paste or partner change request	Application owners, deployment manifests
Derived artefacts	Tokens, SAS URLs or sessions minted from the credential that rotation will invalidate	Target documentation; for example, Azure notes that rotating a storage account key revokes account-level SAS tokens generated from it
Copies outside the store	DR sites, backups, CI variables, developer machines, vendor-held copies	Secret scanning, CI configuration, vendor contracts
Last used and by whom	Unused credentials can be disabled instead of rotated; unknown users surface hidden consumers	Provider last-used data and authentication logs

The consumer list is where hidden risk lives. Compare the consumers your secret store knows about with the source addresses and workloads in the target's authentication logs; anything in the second list but not the first is an undocumented consumer that will break on rotation. Google Cloud's guidance for service account keys suggests Cloud Asset Inventory to find keys created before a cut-off date, which is a good model for building the population of credentials due for rotation. See NHI Inventory Reconciliation for reconciling inventory sources.

Choose an overlap pattern per credential type

Pick the pattern from what the target system supports, not from what is convenient for the secret store.

Pattern	How it works	Typical targets	Trade-offs
Alternating users	Two accounts with identical rights; each rotation changes the inactive one and switches the current pointer	Databases, directory service accounts	No gap; the two accounts' permissions must be kept in step, and a privileged credential is needed to change them
Dual keys	The target issues two keys; regenerate the one not in use, publish it, then retire the other on the next cycle	Storage account keys, some SaaS API keys	No gap; consumers get a full cycle to pick up the new key
Versioned secret with staging labels	New value is created as pending, set on the target, tested, then promoted to current; the old version is kept as previous	Any target behind a managed secret store with rotation hooks	Clean rollback to the previous version; still a brief gap if the target holds only one credential
Overlapping signing keys	Publish the new verification key, wait for caches to refresh, start signing with the new key, retire the old key after tokens signed by it expire	JWT and OIDC signing keys, SAML certificates	Depends on verifier cache lifetimes; see the JSON Web Key Sets (JWKS) and Signing Key Rotation Guide
Create, migrate, disable, delete	Issue a new key alongside the old, move every consumer, disable the old, monitor, delete	Cloud service account keys, access keys, API tokens	Simple and reversible; relies on a complete consumer list
Replace with short-lived credentials	Remove the static secret; workloads obtain credentials per session or lease	Cloud workloads, CI/CD, databases with dynamic credentials	Removes scheduled rotation for that secret entirely; needs workload identity

Alternating users: AWS Secrets Manager's alternating users strategy clones the original user on the first rotation and alternates between the two afterwards, and the documentation notes that both credentials are valid after rotation, that it needs a separate superuser credential, and that permission changes to the original user must also be made to the clone. AWS suggests it for high-availability applications.

Dual keys: the Azure Key Vault dual-credential tutorial stores each key as a version of the same secret, alternating between primary and secondary. A SecretNearExpiry event, published 30 days before expiry, triggers a function that regenerates the alternate key and adds it as the new version, so applications get a full rotation cycle to pick it up. Tags such as CredentialId and ValidityPeriodDays record which key is current and set the next expiry.

Versioned secrets: AWS rotation functions run four steps. createSecret stores the new value under the AWSPENDING label, setSecret changes the credential on the target, testSecret uses the pending version to access the target, and finishSecret moves AWSCURRENT to the new version, with AWSPREVIOUS applied to the old one so the last known good value is retained. The documentation recommends that setSecret first checks that the current and pending versions refer to the same user and host, which protects against a rotation job being pointed at the wrong target.

The strongest pattern is not rotating at all: where a workload can authenticate with federated workload identity or obtain dynamic credentials, the scheduled rotation problem disappears for that secret. Google recommends using a more secure alternative to service account keys wherever possible, and the OWASP cheat sheet recommends dynamic secrets where possible. See the Dynamic Secrets and Ephemeral Credentials Guide.

Make consumers rotation-ready

A rotation pattern only works if consumers pick up the new value in time. Set a standard for every application that uses a managed secret:

Fetch, do not bake: read the secret from the store or a mounted file at start; never build it into images or configuration committed to source control.
Refresh on a schedule shorter than the overlap: if the old credential stays valid for 24 hours after promotion, every consumer must refresh well within that. Agents such as HashiCorp Vault Agent illustrate the mechanics: static secrets are re-fetched every five minutes by default, and leased dynamic credentials are re-fetched at 90 per cent of their TTL.
Re-fetch on authentication failure: on a 401 or login failure, fetch the current value once and retry before raising an error. This closes the remaining gap for single-credential targets.
Drain connection pools: existing database connections usually survive a password change, as AWS notes, so pools must recycle connections to start using the new credential and to reveal failures before the old one is disabled.
Expose the version in use: emit the secret version or key identifier (never the value) in health checks or metrics so you can see which consumers have moved.

Sequencing and staged rollout

Classify and batch. Group credentials by pattern, owner and blast radius. Start with credentials that have one known consumer and a dual-credential or alternating pattern.
Pre-flight checks. For each credential: confirm the current value works, the consumer list is reconciled against authentication logs, the rotation identity has the rights it needs, and the change window and approver are recorded.
Create and publish the new credential. Issue it on the target, store it as a new version, and test it directly against the target before any consumer depends on it.
Move consumers in rings. Canary consumers first, then the rest by region or service tier. Watch authentication failure rates per ring; halt automatically if they rise above baseline.
Promote. Make the new version current once every consumer reports it, or once the refresh interval has safely passed.
Disable the old credential and observe. Disable rather than delete, and watch the target's logs for any use of the old credential. Google's service account key procedure follows this order: create new keys, replace them in all applications, disable the replaced keys and monitor, then delete.
Delete and record. After an observation period with no use, delete the old credential, update the inventory with the rotation date and next due date, and close the change.

Rate and concurrency: rotating thousands of credentials at once can hit target-side API rate limits and overwhelm on-call teams. Throttle rotation jobs, cap concurrent rotations per target system, and add jitter so scheduled rotations do not all fire at the same minute. Be careful with stale in-progress markers: AWS notes that if a pending version is left attached to a different version than current, later rotations assume one is still in progress and return an error, so clean-up after failures must be part of the job.

Rollback design

Stage reached	Rollback action	Condition
New credential created, not promoted	Discard the pending version; nothing else changes	Always safe
Promoted, old credential still valid	Move the current pointer back to the previous version	Safe while the old credential is enabled on the target
Old credential disabled	Re-enable it on the target and point consumers back	Safe only for routine rotation; never during incident response
Old credential deleted	Roll forward: fix the failing consumer with the new value	No rollback exists, which is why deletion comes last

Decide in advance which rotations are allowed to roll back. In a routine rotation, re-enabling the old credential for an hour while a missed consumer is fixed is reasonable. After a suspected compromise, it is not: the old credential must stay revoked and broken consumers are fixed forward. The Incomplete rotation breaches guide covers that incident sequence.

Emergency and vendor-driven rotation

A provider breach can require you to rotate everything that provider holds, at short notice. The Sumo Logic Breach is a measured example: after a compromised AWS credential was used to access one of its accounts in November 2023, the company rotated its own potentially exposed credentials and advised customers to rotate API access keys, collector credentials, credentials stored for third-party data collection and user passwords. Customers with a current inventory of what they had given the provider, and an exercised rotation process, could do that quickly; others had to find their dependencies first. Keep a per-provider list of credentials you have issued to each vendor or integration, and run at least one rotation exercise per year against it. See the Secrets Rotation Runbook Template for the emergency procedure.

Detection and measurement

Monitor the rotation system itself and the targets around it. Alert on rotation job failures and partial completions, any use of a credential after it was disabled (an undocumented consumer or an attacker), authentication failure spikes during a rotation ring, and credentials past their next-due date. The OWASP cheat sheet recommends detection rules for attempts to use revoked secrets, which is the single most useful rotation signal.

Metric	Target direction
Credentials rotated within policy, by tier	Rising towards 100%
Rotations causing a customer-visible incident	Zero, trending down per quarter
Uses of disabled credentials detected during observation	Falling as consumer maps improve
Time to rotate all credentials held by one provider in an exercise	Hours, falling
Share of secrets replaced by workload identity or dynamic credentials	Rising
Credentials with a complete, reconciled consumer list	Rising towards 100%

Common failure patterns

Rotating the store but not the target, or the reverse: the rotation job fails between steps and leaves the two out of step. Use idempotent steps with a pending version and test before promotion.
Single-credential swaps on busy systems: a password change with no overlap causes a burst of failures every cycle, so teams lengthen the interval until rotation effectively stops.
Deleting first: the old credential is deleted on promotion, and the first missed consumer turns into an outage with no rollback.
Forgotten derived artefacts: SAS tokens, signed URLs or sessions minted from the old key fail after rotation and nobody linked them to it.
Clone drift: in alternating-user schemes, permissions are changed on one account but not its twin, so the next rotation breaks features.
Credentials that are never rotated because they are "too critical": these are the ones an incident will force you to rotate under pressure. Give them an overlap pattern and an exercised plan.

Framework mapping

Framework	Reference	Relevance
NIST SP 800-53 Rev. 5	IA-5 Authenticator Management; SC-12 Cryptographic Key Establishment and Management; CM-3 Configuration Change Control; CM-8 System Component Inventory; SI-4 System Monitoring	Authenticator changes, key lifecycle, controlled change, consumer inventory, monitoring for use of old credentials
NIST SP 800-57 Part 1 Rev. 5	Key management, originator-usage and recipient-usage periods	The basis for overlap windows between issuing and accepting a key
NIST CSF 2.0	PR.AA-01; PR.PS-01; RS.MI-02	Credentials managed; configuration management applied; incidents eradicated, including credential revocation
MITRE ATT&CK	T1078.004 Cloud Accounts; T1552 Unsecured Credentials	Rotation limits the useful life of stolen or exposed credentials

Practitioner checklist

Every managed credential has an owner, a target type, a reconciled consumer list and a next-due date.
Each credential type has an assigned overlap pattern; single-credential swaps are limited to low-traffic targets with retry logic.
Applications fetch secrets at start, refresh within the overlap window, re-fetch once on authentication failure and report the version in use.
Rotation jobs use pending, current and previous versions, test before promotion and clean up after failure.
Rollouts move consumers in rings with automatic halt on authentication failure spikes.
Old credentials are disabled and observed before deletion; use after disablement raises an alert.
Rotation concurrency is throttled per target, and schedules are jittered.
A per-provider list of issued credentials exists and an emergency rotation exercise runs at least yearly.
Credentials that can move to workload identity or dynamic secrets are on a dated migration plan.

Credential Rotation at Scale: Rotating Thousands of Secrets Without Outages