Secrets Rotation Runbook Template: Planned and Emergency Key Rotation

Most rotation outages and most incomplete incident clean-ups have the same cause: nobody wrote down which systems use the secret, how the new value reaches them, or how to get back to the old one when the cutover fails. This template is a ready-to-adopt secrets rotation runbook for API keys, passwords, cloud and service account keys, client secrets, private keys and encryption keys. It covers the secret record, triggers, single or dual key rotation, dependency checks, the planned procedure, verification, rollback, emergency key rotation and manual key rotation, with AWS Secrets Manager, Azure Key Vault and Google Cloud Secret Manager examples. It is for platform, DevOps and security engineers who own production secrets. For the wider programme see the Secrets Management Guide.

How to use this template

• Replace text in square brackets, such as [Organisation], [Secret owner] and [24 hours], with your own teams, tools and targets.
• Complete sections 3 and 6 once per secret (or per class of identical secrets). Sections 7 to 11 are the procedures engineers follow; keep them as a key rotation checklist in your ticketing or change tool.
• "Must" is mandatory; "should" may be skipped with a recorded reason.
• Why rotation is hard for machine identities is covered in the Guide to NHI Rotation Challenges; this runbook covers how to do it safely.

1. Purpose and scope

This runbook defines how [Organisation] rotates secrets and cryptographic keys on a schedule and in response to suspected compromise, so that rotation does not cause outages, every consumer is moved to the new value, the old value is revoked, and each step is recorded. It implements [Secrets Management Standard] clause [X].

• In scope: API keys and tokens; database and service account passwords; cloud and service account keys; OAuth client secrets; SSH, TLS, signing and encryption keys; webhook secrets and connection strings used by applications, pipelines, AI agents and integrations in [environments].
• Out of scope: human user passwords, which OWASP says "should only be rotated if there is suspicion or evidence that they have been compromised"; these follow [Password Policy].

2. Roles and responsibilities

3. Secret record

Every secret in scope must have a record before it can be rotated safely. OWASP lists the metadata a secrets solution should hold, including when a secret was "created/consumed/archived/rotated/deleted", "Who to contact" and "When you need to rotate it, if done manually".

4. Rotation triggers and intervals

A cryptoperiod is "the time span during which a specific key is authorized for use" (NIST SP 800-57 Part 1 Rev. 5), and NIST lists the cost of key revocation and replacement among the factors that shape it. Set intervals by tier and by how the secret is used, not one number for everything. OWASP's reason to rotate is "so that any stolen credentials will only work for a short time."

5. Choose the rotation pattern

The pattern decides whether rotation can be done without downtime and how rollback works. Record it in section 3.

Safe key rotation for anything with more than one consumer should use the dual credential or versioned pattern. Alternating users on AWS needs a separate superuser secret to clone the user, because "most users don't have permission to clone themselves"; protect that secret as Tier 1. Before investing in rotation, ask whether the secret can be removed: Microsoft's own tutorial advises preferring managed identity and, where compliance allows, disabling Shared Key authorisation to "remove the need for key rotation entirely".

6. Pre-rotation dependency checks

Complete this secret rotation checklist before any planned rotation. Do not start if any item fails.

1. Confirm the consumer list in section 3 against evidence: secrets manager access logs for the last [30 days], issuer authentication logs, and a code and configuration search for the secret ID.
2. Confirm how each consumer loads the value: on every request, on a cache timeout, on start-up or only on redeploy. Consumers that load only at start-up need a restart or redeploy step.
3. Check that the operator or rotation function can create and revoke credentials at the issuer, and that the issuer allows a second credential if the pattern is dual.
4. Check the platform state: on AWS, no version stuck with AWSPENDING from a failed rotation, since a later rotation then "assumes that a previous rotation request is still in progress and returns an error".
5. Identify dependent artefacts that break silently: rotating an Azure storage account key "revokes any account-level shared access signature (SAS) generated from that key"; signed URLs, cached sessions and derived tokens may do the same.
6. Confirm the rollback method in section 9 works for this secret, and that the previous value or version will be retained.
7. Schedule a window outside [peak periods], with consumer owners on call, and raise the change record.

7. Planned rotation procedure

This key rotation procedure follows the create, set, test, finish sequence used by AWS rotation functions and recommended by OWASP: "Test the pending credential before promoting its version to AWSCURRENT."

1. Create: generate the new value with the issuer or the secrets manager's generator, never by hand. Store it as a new pending version (AWS AWSPENDING; a new secret version in Azure Key Vault or Google Cloud Secret Manager). Do not overwrite the current version.
2. Set: register the new value at the issuer. For dual credential, regenerate the inactive credential only; the active one keeps serving traffic.
3. Test: authenticate with the pending value from a test client with the same network path and permissions as a consumer, and perform a read-only operation.
4. Promote: make the new version current (AWS moves AWSCURRENT and labels the old version AWSPREVIOUS, "so that you retain the last known good version"). On Google Cloud, where workloads should bind to a specific version rather than latest, update the pinned version in deployment configuration.
5. Propagate: roll the new value to consumers in stages: [one canary instance], then [10%], then all. Google warns that resolving latest continuously "risks immediate, service-wide outage"; staged adoption limits the blast radius of a bad value.
6. Verify: run section 8 before touching the old credential.
7. Retire the old credential: after [one full cache or session lifetime, minimum 24 hours], disable the old version or credential, watch for errors for [24 hours], then revoke it at the issuer and destroy the old version. Google's sequence is to disable, "wait a reasonable amount of time to allow a human to intervene", and only then destroy.
8. Record: update the section 3 record and close the change with evidence.

For encryption keys, retirement is different: old versions must stay enabled for decryption. Microsoft states that "Both old and new key versions must remain enabled until re-wrapping is complete", and that Azure services can take "one hour to 24 hours or more" to adopt a new customer-managed key version.

8. Verification

• Consumers: every consumer in section 3 shows successful authentications using the new credential or version (issuer logs, application health checks, error rates against the baseline).
• Old value unused: zero successful authentications with the old credential for [24 hours] before revocation. Any remaining use is either a missed consumer or misuse; investigate before revoking.
• After revocation: a test call with the old value must fail. A "rotation" that leaves the old credential valid is not complete.

9. Rotation rollback plan

Rollback is not available once the old credential has been revoked at the issuer and destroyed, so those steps come last. In an emergency rotation (section 10), never roll back to a compromised value: fix forward.

10. Emergency key rotation

Use this procedure when a secret is exposed or suspected compromised. OWASP: "Keys that were exposed should undergo immediate revocation", and "Restarting the application does not itself revoke stolen credentials." The order matters: if the attacker still has the access path that exposed the secret, they will collect the new one.

1. Declare and assign: open an incident under [Incident Response Plan], assign an owner, and start a UTC log.
2. Close the exposure path: remove the secret from the repository, image, log or ticket, and remove the attacker's access where the exposure came from a compromised system. OWASP requires that rotated secrets "be removed from the exposed system immediately".
3. Revoke or rotate: if the service can tolerate an outage, revoke first and then issue the new value. If not, run section 7 at speed with the canary step shortened, and revoke the old value as soon as consumers have moved, within [the deadline in section 4].
4. Scope the blast radius: list every secret the exposed one could read or that sat in the same location, and rotate those too. Rotating one credential and leaving its neighbours is the most common gap.
5. Investigate use: answer OWASP's questions from logs: "Who had access?", "When did they use it?", "When was it previously rotated?" Search issuer logs for use of the old value from unfamiliar addresses during the exposure window.
6. Confirm and close: old value fails; no further use; record lessons and change the rotation pattern if the incident exposed a weakness.

The Cloudflare breach of 2023 shows why step 4 matters: four credentials left unrotated after the Okta support breach let a nation-state attacker in. The Sumo Logic breach shows how one compromised cloud credential can lead to a customer-wide rotation. For first-hour actions after a leak, use the Leaked Credential Response Playbook.

11. Manual key rotation

Where the issuer has no API or automation is not yet built, manual key rotation must follow the same section 7 sequence, plus these controls. OWASP warns that rotation "implemented manually ... can lead to mistakes".

• Two people: one performs, one checks each step against this runbook. Neither pastes the value into chat, tickets or email.
• The new value is generated by the secrets manager or issuer and entered directly into it; nobody types or records it elsewhere.
• Use a near-expiry alert as the trigger: Azure Key Vault can publish a near-expiry event to Event Grid "as a reminder for manual rotation", and Google Cloud Secret Manager sends a SECRET_ROTATE Pub/Sub message at next_rotation_time (the rotation period cannot be less than one hour).
• Every manual rotation creates a ticket to automate the secret, with a target date.

12. Automation standards

• Schedules: AWS rotation uses rate() or cron() expressions in UTC, "as often as every four hours within a rotation window as small as one hour". Azure key rotation policies have a minimum rotation time of seven days. Google rotation schedules only notify; [Organisation] must run a subscriber that performs the rotation.
• Reentrancy: rotation code must survive being stopped part way and restarted. Google advises that the process "must be able to restart"; AWS relies on AWSPENDING for idempotency.
• Alerting and privilege: failed or skipped rotations page the owner (Google skips a scheduled rotation "if there is an in-flight rotation"), and the rotation function's identity can change only the credentials it rotates.

13. Records, metrics and review

• Keep, per rotation: secret ID, trigger, operator, start and end times, versions, verification evidence, rollback taken, and revocation time.
• Report monthly: percentage of secrets rotated within interval; percentage rotated automatically; failed rotations and rollbacks; median time from exposure to revocation; secrets with an incomplete section 3 record.
• Review this runbook after any failed rotation, any emergency rotation and at least [annually]. Rehearse section 10 on one Tier 1 secret in [each half year], alongside the break-glass credentials that OWASP says should be "tested routinely".

Implementation guidance

1. Inventory first: fill section 3 for Tier 1 secrets before scheduling any rotation. A runbook without a consumer list is the root cause of most rotation outages.
2. Pilot: run section 7 end to end in non-production, then on one Tier 3 production secret, timing each step.
3. Move multi-consumer secrets to dual credential and automate by risk: Tier 1 database and cloud credentials first, using native rotation where it exists.
4. Remove where you can: each secret replaced by workload identity no longer needs this runbook. See the Cloud Workload Identity Guide.

Small and large organisations: a small team can run this runbook with one owner per secret and native cloud rotation, keeping section 3 in the secrets manager's tags (Azure's dual rotation tutorial stores CredentialId, ProviderAddress and ValidityPeriodDays as tags). A large organisation should hold section 3 in an NHI inventory, enforce section 6 checks in the pipeline and report section 13 metrics per business unit.

Common failure patterns: rotating at the store but not revoking at the issuer; missing a consumer that only reads the secret at start-up; deleting the old version before the canary has finished, which removes rollback; restarting services and assuming that revoked a leaked credential; and rotating the exposed secret while leaving others from the same location, as in the Internet Archive breach, where unrotated keys allowed a second breach within a month.

Standards and references

• NIST: SP 800-57 Part 1 Rev. 5 Recommendation for Key Management: Part 1, General
• OWASP: Secrets Management Cheat Sheet
• AWS: Rotation strategies in AWS Secrets Manager
• AWS: Lambda rotation functions
• Microsoft Learn: Automate rotation for resources with two sets of credentials
• Microsoft Learn: Configure cryptographic key auto-rotation in Azure Key Vault
• Google Cloud: Secret Manager rotation schedules
• Google Cloud: Secret Manager rotation recommendations

This template is provided for guidance and must be adapted to your organisation. Related NHI Mgmt Group resources: Secrets Management Guide · Guide to NHI Rotation Challenges · Leaked Credential Response Playbook · Cryptographic Key Management Guide · API Key Management Guide