A mass credential reset after a serious compromise fails in one of two ways: it misses a credential the attacker still holds, or it breaks production because nobody knew what depended on the credential it changed. This runbook template gives [Organisation] a sequenced, enterprise-wide reset of every credential class an attacker may have taken: user and administrator passwords, the krbtgt account, Active Directory trusts, service accounts, cloud access keys and role sessions, API keys, OAuth tokens, CI/CD secrets, SSH keys, certificates and signing keys. It is built around a dependency map so each reset is ordered, tested and reversible where possible. Most of the work, and most of the risk, sits with non-human identities. It is written for incident leads, identity and platform engineers, application owners and change managers. Active Directory domain compromise detail lives in the Active Directory Domain Compromise Playbook; this page coordinates the reset across the whole estate. Replace every [Organisation] placeholder before use.
How to use this template
- Invoke it from a parent playbook (domain compromise, identity provider tenant compromise, ransomware, secrets platform breach) once the incident lead decides a broad reset is needed. It is not a substitute for scoping.
- Build the dependency map before you need it. Section 3 cannot be done well during an incident without an inventory. See the Service Account Inventory Template and NHI Inventory Reconciliation.
- Work in waves (section 4). Each wave has an owner, a start condition, a verification test and a rollback note.
- Track every credential in one reset register: ID, class, system, owner, dependants, wave, reset time, verified by, status. The register is the evidence that nothing was skipped.
- Do not reset into a compromised control plane. If the attacker still controls domain controllers, the identity provider or the vault, new credentials go straight to them. Containment of those systems comes first.
1. Scope and triggers
1.1 Use this runbook when any of these is confirmed or cannot be ruled out:
- Theft of the Active Directory database or replication of password data (T1003.006 DCSync), or a forged Kerberos ticket (T1558.001 Golden Ticket).
- Administrative compromise of the workforce identity provider, privileged access management platform or secrets vault.
- Bulk theft of credentials from files, repositories, wikis or CI/CD variables (T1552.001 Credentials In Files), or of application tokens (T1528 Steal Application Access Token).
- Ransomware or destructive attack where the attacker held domain or cloud administrator rights.
1.2 Credential classes in scope and the action that actually invalidates each:
| Class | Invalidating action | Common trap |
|---|---|---|
| User and admin passwords (AD and hybrid) | Reset; Microsoft's emergency revocation guidance for hybrid users says reset the password twice in AD, then revoke cloud refresh tokens | Reset without session revocation leaves the attacker signed in |
| Cloud identity sessions (Entra ID) | Revoke sessions (Revoke-MgUserSignInSession); access tokens last about one hour by default unless continuous access evaluation applies | Applications issuing their own session tokens are not revoked by Entra ID |
| krbtgt | Reset twice, at least 10 hours apart (default maximum ticket lifetime), per Microsoft's forest recovery guidance; separate krbtgt_<number> accounts for read-only domain controllers | A single reset leaves the previous key valid, because krbtgt keeps a password history of two |
| Domain and forest trusts | Reset the trust password on the trusting side and set the same value on the trusted side (Microsoft's netdom procedure for breach-related recovery) | Forgetting external and child-parent trusts |
| AD service accounts and gMSAs | Reset passwords or rotate gMSA keys; update every host and application using them | Hard-coded passwords in scheduled tasks and scripts break silently |
| Cloud access keys | Create new key, deploy, deactivate old, then delete once verified | Long-lived keys on old build servers nobody inventoried |
| Cloud role sessions (AWS) | Revoke active sessions: IAM attaches the AWSRevokeOlderSessions deny policy based on aws:TokenIssueTime | Does not affect IAM user access keys, service-linked roles, or new sessions if the attacker can still assume the role |
| OAuth grants, refresh tokens, PATs | Revoke the grant or token at the issuer; rotate client secrets | Rotating the client secret does not revoke refresh tokens already issued |
| API keys and vault secrets | Rotate at the provider and in the vault; revoke dynamic secret leases | The OWASP Secrets Management Cheat Sheet notes that restarting an application does not revoke stolen dynamic credentials |
| SSH keys, certificates and signing keys | Remove from authorized_keys and trust stores; revoke and reissue certificates; rotate signing keys with overlap | Forgotten host keys and private CAs |
2. Roles and contacts
| Role | Named lead | Responsibilities |
|---|---|---|
| Reset coordinator (reports to incident lead) | [Name, phone] | Owns the reset register, wave start and stop decisions, status reporting |
| Directory engineer | [Name] | AD passwords, krbtgt, trusts, gMSAs, replication health |
| Identity provider administrator | [Name] | Session and token revocation, app registrations, federation certificates |
| Cloud platform owners | [Per provider] | Access keys, role sessions, service account keys, workload identity |
| Secrets and PAM owner | [Name] | Vault rotation, privileged account check-in, break-glass accounts |
| Application and service owners | [From inventory] | Update consumers, verify service health, approve outage windows |
| DevOps lead | [Name] | CI/CD secrets, deploy keys, registry and package publishing tokens |
| Change manager | [Name] | Emergency change records, business communication of outage windows |
| Service desk lead | [Name] | User password reset support, identity verification, call surge |
3. Detection and triage
3.1 Fix the compromise window. Take the earliest evidence of attacker access from the parent incident. Every credential that existed, was readable by a compromised identity, or was used from a compromised host since then is in scope.
3.2 Build the in-scope list from the inventory, then add what the inventory misses: search code, CI/CD variables, wikis and ticket attachments for secrets; list cloud keys by last-used date; list OAuth grants and personal access tokens by issuer.
3.3 Map dependencies. For each credential record which systems present it and which systems accept it. A database password has a consumer (the application) and a verifier (the database); both sides must change together.
3.4 Prioritise by attacker value:
| Priority | Credentials | Why |
|---|---|---|
| P0 | krbtgt, domain and enterprise admins, identity provider global admins, federation and token signing keys, vault root and unseal material, cloud organisation root and break-glass accounts | Can mint or reset every other credential |
| P1 | Service accounts with admin rights, cloud keys with IAM write, CI/CD and deploy credentials, backup system accounts | Can create persistence or reach production at scale |
| P2 | Application service accounts, API keys to third parties, OAuth integrations, database credentials | Data access and lateral movement |
| P3 | All remaining user passwords and low-privilege machine credentials | Broad but lower individual impact |
3.5 Decision (incident lead): targeted or global reset. Choose global when password hashes for the whole directory were taken, the compromise window is unknown, or logging gaps prevent scoping. Choose targeted when the evidence reliably limits which credentials were exposed. Record the evidence for the choice.
4. Containment
4.1 Run the waves in order. A wave starts only when the previous wave's verification has passed.
- Wave 0, secure the control planes: confirm attacker access to domain controllers, the identity provider, PAM and the vault is removed. Create new break-glass accounts and store their credentials offline. Confirm AD replication is healthy.
- Wave 1, P0 credentials: reset tier 0 administrator accounts, revoke their sessions, then the first krbtgt reset. Rotate identity provider federation and token signing certificates. Rotate vault root material. Reset cloud root and organisation admin credentials.
- Wave 2, P1 machine credentials: rotate privileged service accounts, cloud keys with IAM write, CI/CD secrets and deploy keys, and revoke cloud role sessions. Update consumers in the same change.
- Wave 3, second krbtgt reset after the 10 hour wait (or longer if ticket lifetime policies differ), then trust passwords.
- Wave 4, P2: application credentials, third-party API keys (at the provider), OAuth client secrets and grants, database passwords, SSH keys and certificates.
- Wave 5, P3: force a password change for all users at next sign-in, staged by business unit, with session revocation.
4.2 For each machine credential: create the new credential, deploy it to consumers, verify, then disable the old one, observe for [24] hours, then delete. Where a provider allows two active keys (AWS allows a maximum of two access keys per user), use the overlap for zero-downtime rotation. Where the attacker is active, disable first and accept the outage.
4.3 Decision (reset coordinator with service owner): disable now or rotate with overlap. Disable now for P0 and P1, or any credential seen in use by the attacker. Rotate with overlap for P2 and P3 where there is no evidence of use.
5. Eradication
5.1 Confirm every row in the register shows the old credential disabled or deleted, not just a new one issued.
5.2 Hunt for use of old credentials: authentication failures with the old key or password are expected from forgotten consumers, but successful use after the reset time means a missed copy or a second credential. Alert on both for [30] days.
5.3 Remove attacker-created credentials found during the reset: new access keys, app registration secrets, OAuth consents, SSH keys in authorized_keys, and personal access tokens.
5.4 Retire credentials found to be unused instead of rotating them, but only after verifying non-use from logs. The Cloudflare breach began with one service token and three service accounts left unrotated after an earlier incident because they were mistakenly believed unused. See Incomplete rotation breaches.
6. Recovery
6.1 Verify service health after each wave against a checklist of critical business transactions, not just process status.
6.2 Clear cached credentials where tools keep them, for example the AWS CLI credential cache after role session revocation, and restart services that cache Kerberos tickets or tokens.
6.3 Replace static credentials with federation, workload identity or managed identities where the reset showed it is feasible, so the next reset is smaller. See the Secrets Rotation Runbook Template.
6.4 Return break-glass and emergency access to its normal sealed state and record who accessed it during the incident.
7. Communication and notification
| Audience | Owner | When and what |
|---|---|---|
| All staff | Incident lead with communications | Before Wave 5: why passwords are changing, how to reset, how the service desk will verify them; warn of phishing that imitates the reset |
| Application owners | Reset coordinator | Before each wave: credentials affected, window, rollback contact |
| Third parties holding [Organisation] credentials | Vendor management | When keys or tokens they use will change, and to request rotation of credentials they issued to [Organisation] |
| Regulators and customers | Privacy and legal | Per the parent incident's notification decision; the reset itself is evidence of containment for those notices |
8. Post-incident review
8.1 Record: number of credentials in scope, number rotated, retired and missed; time per wave; outages caused and their cause.
8.2 Metrics to track:
- Coverage: share of in-scope credentials with a verified reset (target 100%).
- Inventory gap: credentials found during the reset that were not in the inventory.
- Time to reset P0 and P1 from decision.
- Reset-caused outages and minutes of downtime.
- Old credential use after reset (successful attempts should be zero).
- Static credential share: machine credentials still static after the incident.
Implementation guidance
- Rehearse a P0 and P1 reset in a test forest and a test cloud account once a year, and time it.
- Keep owners current. Every machine credential needs a named owner and a documented consumer list. See the Service Account Security Guide.
- Automate rotation for vault-managed secrets so an emergency reset is a trigger, not a project. See the Secrets Management Guide.
- Prepare revocation commands per platform (Entra ID, AWS, Google Cloud, GitHub, SaaS) and store them with this runbook.
| Runbook element | NIST CSF 2.0 | NIST SP 800-53 Rev. 5 |
|---|---|---|
| Inventory and dependency map | PR.AA-01, ID.AM-08 | IA-5 Authenticator Management |
| Waves and verification | RS.MI-01, RS.MI-02 | IR-4 Incident Handling |
| Recovery and service health | RC.RP-01 | IR-4 Incident Handling |
| Stakeholder communication | RS.CO-02 | IR-6 Incident Reporting |
Standards and references
- Microsoft Learn: AD Forest Recovery, Reset the krbtgt password
- Microsoft Learn: AD Forest Recovery, Reset a trust password on one side of the trust
- Microsoft Learn: Revoke user access in Microsoft Entra ID
- AWS: Revoke IAM role temporary security credentials
- OWASP: Secrets Management Cheat Sheet
- NIST: SP 800-61 Rev. 3 Incident Response Recommendations and Considerations for Cybersecurity Risk Management
Related NHI Mgmt Group resources: Active Directory Domain Compromise Playbook · Identity Provider Tenant Compromise Playbook · Leaked Credential Response Playbook · Golden Ticket, Silver Ticket and DCSync Guide