Secret Rotation Outage Troubleshooting Guide

When a secret rotation takes production down, the fastest fix is usually to put the old credential back, and that is exactly the habit that leaves leaked secrets valid for months. Rotation outages fall into a small number of patterns: the rotation job failed half way, the new value never reached the consumers, the old value was revoked before everyone moved, or the new credential does not carry the same permissions. This guide is a diagnostic reference for those patterns across AWS Secrets Manager, Kubernetes and External Secrets Operator, HashiCorp Vault, Azure Key Vault and Microsoft Entra application secrets. For each symptom it lists likely causes, where to look, the fix and how to prevent a repeat. Error strings are quoted from vendor documentation checked on 11 November 2026. It is for platform, SRE and secrets management engineers. Planning rotation programmes is covered in Credential Rotation at Scale, which this page does not repeat.

Key takeaways

Classify the outage before rolling anything back. A failed rotation, a stale consumer, an early revocation and a permission gap look identical from the application (authentication failed) but have different fixes.
Stale consumers are the most common cause. Kubernetes environment variables and subPath mounts never see an updated Secret, and connection pools and in-process caches hold the old value until restart.
Overlap is the cure, not rollback. Alternating users, dual keys and multiple client secrets keep two valid credentials during the change, so consumers can move at their own pace.
If the rotation was triggered by a leak, the old secret must not come back. Restore service by pushing the new value to consumers, not by revalidating the exposed one.
Every rotation needs a test step and an owner on call. AWS's rotation functions have a dedicated testSecret step; your own tooling should verify that the new credential works before the old one is retired.

Triage: which kind of rotation outage is it?

Observation	Most likely class	First check
Secret store still holds the old value, or a version is stuck in a pending state	Rotation job failed part way	Rotation job logs and the secret's version history
Store holds the new value; some pods or hosts fail, others work	Stale consumers	When did each failing instance start and how does it read the secret?
All consumers fail at the same moment, shortly after rotation	Old value revoked with no overlap, or new value wrong at the target	Can you authenticate manually with the new value?
Authentication succeeds but operations fail with access denied	New credential lacks permissions	Compare grants of old and new identity
Many unrelated services fail after one rotation	Shared or root credential rotated	Which config objects reference the rotated credential?
Failures begin days later, not at rotation time	Lease or expiry, not rotation	Lease TTL, max TTL, secret expiry date

Stabilise first, without undoing the security fix

Establish why the rotation happened. Scheduled rotation: rolling back to the previous value is acceptable as a short-term measure. Rotation after exposure: the previous value is compromised; do not revalidate it.
Check whether the previous version is still valid. In AWS Secrets Manager the prior version carries the AWSPREVIOUS label; with alternating users both users remain valid after rotation, so pointing a consumer at AWSPREVIOUS buys time without touching the database.
Push the current value to the failing consumers (restart pods, force a sync, recycle connection pools) before considering any rollback.
Freeze further rotation on the affected secret until the cause is understood, so a scheduled run does not compound the problem. Record the freeze and its end date.
Keep evidence: rotation logs, secret version IDs and labels, audit events and the first failing request time.

Symptom 1: the rotation job failed (AWS Secrets Manager)

Lambda-based rotation runs four steps: createSecret, setSecret, testSecret and finishSecret. AWS's first instruction is to check CloudWatch logs and confirm every step was attempted. The documented errors and their causes:

Log line or error	Cause (AWS documentation)	Fix
No activity after `Found credentials in environment variables`	Function cannot reach the Secrets Manager endpoint and times out	Secrets Manager VPC interface endpoint, or a NAT route if policy allows
No activity after `createSecret`	Network ACLs blocking HTTPS, timeout too short, VPC endpoint security group or endpoint policy not allowing the function; for alternating users with an RDS-managed superuser secret, no route to the RDS API	Open HTTPS, raise the timeout, fix endpoint policy, add an RDS interface endpoint
`Access to KMS is not allowed`	Function lacks permission to decrypt with the customer managed key	Add the key policy statement for the rotation role
`Key is missing from secret JSON`	Secret JSON does not match the structure the function expects	Match the documented JSON structure for the secret type
`setSecret: Unable to log into database`	Function cannot reach the database (long duration), wrong credentials (short duration), `scram-sha-256` with an old `libpq`, or TLS required but not used	Same VPC and security group rules; test `AWSCURRENT` and `AWSPREVIOUS` manually; recreate older rotation functions
`Pending secret version ... was not created by Lambda ... Remove the AWSPENDING staging label and restart rotation`	Assumed-role or cross-account rotation without a `RotationToken`	Pass `RotationToken` to `put_secret_value`
Intermittent failures looping between createSecret and setSecret	Reserved concurrency too low (for example 1)	Reserved concurrency of at least 10, or unreserved

Stuck pending version: if rotation keeps failing, AWS's advice is to clear the AWSPENDING staging label and retry, after fixing the cause.
Races: avoid calling PutSecretValue or other mutating APIs on a secret while RotateSecret is running.
Old runtimes: Unable to import module 'lambda_function' after an automatic Python upgrade means the function needs recreating on a current runtime.

Service-specific settings are covered in the AWS Secrets Manager guide.

Symptom 2: the store has the new value, consumers still use the old one

Likely causes, most to least common

Kubernetes Secret consumed as an environment variable: the value is read when the container starts; pods keep the old value until restarted.
Kubernetes Secret mounted with subPath: the documentation states that a container using a Secret as a subPath volume mount does not receive automated Secret updates.
Normal volume mount, but propagation is not instant: Kubernetes updates mounted Secret data with an eventually consistent approach; the delay can be as long as the kubelet sync period plus the cache propagation delay. The application must also re-read the file.
External Secrets Operator has not synced: refreshInterval sets how often values are read from the provider; zero means fetch once. With refreshPolicy: CreatedOnce the Secret is synced once and not refreshed on a schedule; OnChange ignores refreshInterval and only updates when the ExternalSecret changes.
Application cache or connection pool: the app read the secret at start-up and keeps it in memory, or pooled database connections authenticated with the old password are reused until they fail.

Diagnose

Compare versions: secret version or hash in the store, in the Kubernetes Secret (resourceVersion), in the mounted file and in the process.
ExternalSecret status: the Ready condition is False if the last refresh was not successful; check its reason and the controller logs, then the provider credentials the operator itself uses.
Pod start times: failing pods that started before the rotation and working pods that started after point straight at environment variables or start-up caching.

Fix

Immediate: rolling restart of affected workloads; for ESO, force a sync by updating an annotation on the ExternalSecret (documented example: force-sync=$(date +%s)).
Durable: mount Secrets as volumes without subPath and have the application re-read on authentication failure, or use a reloader that restarts workloads when a referenced Secret changes. Fetch secrets at runtime where possible. See External Secrets Operator security guide and the Kubernetes Secrets Security Guide.

Symptom 3: everything fails right after rotation

Single-user rotation window: AWS states that with single-user rotation there is a short window between the password change and the secret update in which the database may deny calls using the rotated credentials, and recommends a retry strategy. Clients without retries fail.
Old value revoked immediately: custom scripts that set the new password and invalidate the old one in one step give consumers no time to move. Use an overlap pattern (next section).
New value wrong at the target: generated password breaks a character rule, length limit or shell quoting at the consumer. Test login manually with the stored value.
Entra application secrets: AADSTS7000215 (invalid client secret is provided) usually means the app is sending a removed or mistyped secret, often the secret ID instead of the value; AADSTS7000222 (InvalidClientSecretExpiredKeysProvided) means the secret expired. Entra applications can hold more than one client secret, so add the new one, deploy it, then remove the old one. See How to find expiring app secrets and certificates in Entra ID.
Shared key regenerated: Microsoft notes that rotating an Azure Storage shared key revokes any account-level SAS generated from it. Every SAS consumer fails at once unless SAS tokens are regenerated.

Symptom 4: new credential authenticates but is denied

Alternating users drift: AWS creates the clone with the same permissions as the original, but if you later change the original's permissions you must change the clone's too. Grants added to one user only break every other rotation.
Object ownership: where one role owns tables and a second accesses them, objects created by the clone may not be accessible to the original. AWS recommends alternating users for that owner-and-accessor model and single user where clones would not have the same permissions.
New identity, missing grants: a key or service principal recreated rather than rotated loses role assignments made to the old object ID.

Symptom 5: Vault rotation and lease failures

Root credential in a static role: HashiCorp warns not to use static roles for root database credentials, because Vault does not distinguish root from standard credentials when rotating. After rotation the config/ password is no longer valid and every dynamic and static user under that connection fails.
Root rotated, humans locked out: after database/rotate-root the root password is known only to Vault. That is intended; break-glass access needs a separate account.
Static role rotation: rotation runs on rotation_period (default 24 hours) or a cron-style rotation_schedule, and on onboarding unless import rotation is skipped. A consumer holding the password longer than the period fails at the next rotation.
Dynamic credentials expiring: defaults are a 1-hour TTL and 24-hour max TTL. An application that renews its lease still fails at max TTL and must request new credentials. Failures exactly 24 hours after deployment are the signature.

See the HashiCorp Vault Guide and Dynamic Secrets and Ephemeral Credentials Guide.

Choosing a rotation pattern that does not cause outages

Pattern	How overlap works	Outage risk	Choose when
Single user	None: one password changes	Short deny window; needs client retries	Low-traffic systems, interactive or one-off users (AWS recommendation)
Alternating users	Two users; both valid after each rotation	Low; propagation across server farms can still deny briefly	High availability databases where clone permissions stay identical
Dual key (primary and secondary)	Regenerate the key not in use; consumers get a full cycle to move	Low, if dependent SAS tokens are handled	Storage keys and other two-key services; Key Vault's near-expiry event fires 30 days before expiry
Multiple concurrent secrets	Add new, deploy, remove old	Low, if removal waits for zero use of the old secret	OAuth client secrets, API keys with multi-key support
Dynamic or federated credentials	No shared long-lived secret to rotate	Shifts to lease and token expiry handling	New builds; preferred where the platform supports it

Microsoft's own advice for Azure Storage is to prefer Entra authorisation with a managed identity and, where allowed, disable Shared Key so there is nothing to rotate. See How to replace Azure app secrets with federated credentials.

A rotation runbook that can be rolled back

Inventory consumers of the secret and how each reads it (env var, file, SDK call, cache).
Create the new credential alongside the old using the overlap pattern for that type.
Test the new credential against the target system from the consumer's network path.
Distribute to consumers in rings (one instance, one service, all), watching authentication error rates.
Confirm zero use of the old credential from logs before revoking it.
Revoke the old credential and record the time.
Rollback point: until step 6, rollback is pointing consumers back at the old value. After step 6, roll forward only.

Detection and measurement

Signals: CloudTrail RotationFailed and rotation function errors; ExternalSecret Ready=False; Vault lease revocation and renewal errors; Entra sign-in logs for service principals with AADSTS7000215 or AADSTS7000222; database authentication failures by user.
Use of the old credential after rotation: the clearest sign a consumer was missed, and, for a leaked secret, a possible attacker.

Metric	Target
Rotation success rate per run	Above 99% on scheduled runs
Rotations causing a customer-visible incident	Zero; trend down per quarter
Time from rotation to last use of the old credential	Within the overlap window
Secrets with rotation frozen or disabled	Each with an owner and an end date
Consumers reading secrets via environment variables	Down, replaced by runtime retrieval or reload

Common failure patterns

Rolling back to a leaked secret: outage pressure revalidates the exposed credential. The Cloudflare Thanksgiving breach followed credentials left unrotated after the Okta incident.
Abandoned rotation: one consumer fails, rotation is disabled, and the secret is never rotated again. The Internet Archive breach involved Zendesk keys left unrotated. See Incomplete rotation breaches.
Root credential under automated rotation with no separate break-glass account.
Permissions granted to one half of an alternating pair.
Rotation scheduled during peak load with no on-call owner.
Removing the old secret on a timer rather than on evidence of zero use.

Framework mapping

Framework	Reference	How it applies
NIST SP 800-53 Rev 5	IA-5 Authenticator Management; SC-12; CM-3 Configuration Change Control; CP-10	Changing authenticators; key management; controlled change; recovery from a failed change
NIST CSF 2.0	PR.AA-01; ID.AM-08	Credentials for services managed; lifecycle management of systems and data
ISO/IEC 27001:2022 Annex A	A.5.17 Authentication information; A.8.24 Use of cryptography; A.8.32 Change management	Rotation as managed change of authentication information
MITRE ATT&CK v20	T1552.001 Credentials In Files; T1078.004 Cloud Accounts	Why exposed credentials must not be revalidated during an outage

Practitioner checklist

Every rotated secret has a consumer inventory and an owner on call during rotation.
Overlap pattern chosen per credential type; no single-step revoke-and-replace.
New credential tested from the consumer path before distribution.
No Kubernetes subPath or environment variable consumption for secrets that rotate, or a reloader in place.
ESO refreshInterval and refreshPolicy reviewed for rotated secrets.
Old credential revoked only after logs show zero use.
Rotation freezes recorded with an end date and reviewed weekly.
Leak-driven rotations never rolled back to the exposed value.