Organisations that announce "all credentials have been rotated" keep getting breached a second time through the one credential that was not. This guide looks at three incidents in the NHI Mgmt Group breach database where a rotation was declared and the attacker came back anyway: Trivy in 2026, Cloudflare in 2023 and the Internet Archive in 2024. It sets out why post-incident rotation fails, how to scope it from the attacker's reach rather than from your own list, how to sequence it so the attacker cannot watch the new secrets arrive, and how to prove that every old credential is dead. It is written for incident responders, platform and secrets owners and the security leads who sign off "contained".
Key takeaways
- Rotation is scoped by attacker reach, not by your inventory. Start from what the compromised identity, host or repository could read, then find every credential in that blast radius, including ones you believe are unused.
- "Unused" is a hypothesis. Cloudflare skipped four credentials it believed were unused after the Okta support breach, and those four were how the attacker got in and stayed.
- Non-atomic rotation leaks the new secrets. In the Trivy case the rotation "wasn't atomic", so an attacker with residual access could see refreshed tokens and publish a poisoned release weeks later.
- Revoke, do not just reissue. Issuing a new key while the old one stays active is not rotation. Deactivate, test, then delete, and confirm the old value fails.
- Proof is a deliverable. Close the incident only with evidence: a scoped credential list, a revocation record per item, failed-use tests and log searches showing no later use of old values.
What the breach data shows
NHI Mgmt Group, The State of NHI & AI Agent Breach Report 2026 (October 2026), analysed more than 200 breaches involving non-human identities and AI agents. Leaked secrets were the starting point for 48 of them, and "secret or password" was the most-exploited identity type, in 84 breaches. The report also found that 56% of the breaches involved more than one kind of non-human identity, and it notes that "one leaked key rarely stays one key: attackers use it to find the next credential, then the next."
That chaining is exactly why incomplete rotation is so common. The first credential is usually found and revoked. The second, third and fourth, found by the attacker inside code, wikis, CI secrets or support tickets, are the ones that survive. The report's lesson for the Trivy case is direct: "A rotation that misses one credential is no rotation: revoke everything, then prove it."
Three cases, one failure
| Case | What was rotated | What survived | How it was used |
|---|---|---|---|
| Cloudflare Thanksgiving Breach 2023 | Thousands of credentials after the October 2023 Okta support system compromise | One Moveworks service token and three service accounts (Smartsheet with Jira admin rights, Bitbucket, an AWS environment), skipped because they were mistakenly believed unused | Reconnaissance from 14 November, persistence on the Atlassian server via ScriptRunner and the Sliver framework, access to 120 of almost 12,000 repositories (76 treated as exfiltrated) |
| Trivy Supply Chain Attack 2026 | Credentials rotated on 1 March 2026 after a stolen GitHub PAT | Still-valid credentials, including access to the aqua-bot service account; Aqua confirmed on 1 April that the rotation was incomplete | Malicious Trivy v0.69.4 on 19 March, trivy-action and setup-trivy tags force-pushed, malicious Docker Hub images on 22 March, downstream compromise of LiteLLM and Checkmarx KICS |
| Internet Archive Breach 2024 | Reported partial rotation after the first breach of 31 million records | Zendesk API tokens found in the stolen source code; the GitLab token behind the first breach had reportedly been exposed since at least December 2022 | On 20 October 2024 the attacker replied to support tickets from the Archive's genuine Zendesk account; the emails passed DKIM, DMARC and SPF |
The three cases fail at different points. Cloudflare scoped correctly but filtered on a belief about usage. Trivy scoped broadly but executed in a way that left a window. The Internet Archive did not treat every secret in stolen code as compromised. The controls below address each failure point in turn. For the wider Trivy chain and its downstream victims, see the TeamPCP Threat Actor Profile Guide.
Why rotation fails: five root causes
- Scope from the wrong list: Responders rotate what the secrets manager or CMDB knows about. Attackers use what they found in source code, Confluence, HAR files, CI variables and tickets. Credentials outside the vault are exactly the ones not on the list.
- Usage assumptions: A credential with no known consumer is skipped to avoid breaking something. Cloudflare later called its "unused" judgement incorrect. Without last-used data, nobody can tell an idle credential from one the attacker is using quietly.
- Non-atomic sequencing: If the attacker still holds any foothold (a bot account, a workflow, a runner, a session) while new secrets are issued, the new secrets can be read as they land. This is the Trivy pattern.
- Reissue without revoke: Many platforms allow two live keys (AWS allows two access keys per IAM user). Creating a new key without deactivating the old one leaves both valid.
- Machine identities left out: Bot accounts, service accounts, publishing tokens held by former staff, GPG and signing keys, OAuth refresh tokens and SaaS integration tokens are often owned by nobody. In the Trivy case the Open VSX publishing token belonged to a former employee.
Scope the rotation from attacker reach
Build the rotation list from the attacker's point of view. For each compromised foothold, ask what it could read, not what it was supposed to read.
| Compromised foothold | Where to look for reachable credentials | Typical survivors |
|---|---|---|
| Source code repository or GitLab/GitHub token | Full git history (not just HEAD), config files, CI variables, deploy keys, linked webhooks | SaaS API tokens in code, database credentials, deploy keys |
| CI/CD runner or workflow | Repository and organisation secrets, environment secrets, OIDC trust policies, cached artefacts, runner memory | Publishing tokens, cloud keys, bot PATs, signing keys |
| Identity provider or SSO session | Service tokens, API tokens issued by the IdP, apps the session could consent to, service accounts with SSO | Integration service tokens, service accounts with admin roles |
| Wiki, ticketing or support system | Pages, attachments and HAR files containing tokens; integration tokens of connected apps | Session tokens, API tokens pasted for troubleshooting |
| Cloud account or role | Secrets stores, instance metadata, parameter stores, cross-account roles, additional keys created by the attacker | Second access keys, new users or app credentials (ATT&CK T1098.001 Additional Cloud Credentials) |
Two rules make the list complete. First, treat every secret found in stolen data as compromised, whatever its believed status. Second, hunt for credentials the attacker created, not just the ones they stole: new access keys, new OAuth apps, new deploy keys, new self-hosted runners and new service account keys. For triaging secrets found in code, see the Developer Secret Scanning Triage.
Sequence the rotation so it holds
Rotation after an incident is an eradication step, not a maintenance task. The order matters more than the speed.
- Evict before you reissue. Remove the attacker's footholds first: disable compromised bot and service accounts, revoke sessions, remove unknown runners, workflows, OAuth apps and deploy keys, and isolate affected hosts. Issuing new secrets into an environment the attacker still controls repeats the Trivy failure.
- Freeze the release path. Where the incident touches build or publishing, pause releases and require manual approval on release jobs until rotation is complete, so a surviving credential cannot ship a poisoned version.
- Rotate in one window, highest privilege first. Group the credentials by blast radius (publishing and signing keys, cloud admin, identity provider tokens, then the rest) and rotate each group in a single planned window rather than piecemeal over days.
- Deactivate, then delete. For each credential, issue the replacement, update the consumers, deactivate the old value, test, and only then delete. AWS documents exactly this overlap for IAM access keys and provides
aws iam get-access-key-last-usedto confirm the old key is no longer in use. - Replace, do not rotate, where you can. A long-lived key that has just been stolen is a good candidate to remove entirely. Move CI to workload identity federation, publishing to trusted publishing and service accounts to short-lived tokens. See the GitHub Actions OIDC Federation Hardening Guide.
- Distrust delivery channels. Do not deliver new secrets through the channel that was compromised (the same repository, wiki or chat). Inject them through the vault or the platform's secret store only.
The step-by-step mechanics of a planned and emergency rotation are in the Secrets Rotation Runbook Template. This guide is about making sure that runbook is applied to the right list, in the right order.
Prove revocation before you close
"Rotated" should mean the old value has been shown not to work. Keep evidence for each credential on the scoped list.
| Evidence | How to produce it | What it rules out |
|---|---|---|
| Revocation record | Ticket or vault log entry with the credential ID, owner, revocation time and who did it | Items silently skipped |
| Failed-use test | Call the API with the old value from a controlled host and record the 401/403 | Reissue without revoke, second key still active |
| Last-used check | Provider last-used fields (for example the AWS access key last-used timestamp) before and after | "Unused" credentials that were in fact in use |
| Log search for old values | Search cloud, SaaS and IdP audit logs for the old key IDs and token fingerprints after the revocation time | Residual access through a missed path |
| Created-credential sweep | List keys, apps, runners, deploy keys and OAuth grants created during the incident window | Attacker persistence through new credentials |
| Independent re-scan | Second team or tool rescans repositories, wikis and tickets for live secrets | Secrets the first pass missed |
A useful closing test is to ask the people who did the scoping to name, for each credential they excluded, the evidence that it could not have been reached. If the answer is a belief rather than a log, it goes back on the list.
Decision guidance
- Rotate everything versus rotate the scoped list: Cloudflare's eventual response was to rotate every production credential, more than 5,000, and triage 4,893 systems. That is right when the attacker reached an identity provider, source code at scale or a CI platform, because the reachable set is effectively everything. Where the foothold is narrow and well logged (one workload with one role), a scoped list with proof is enough.
- Rotate now or evict first: If you cannot yet show the attacker is out, rotate only the credentials that give the widest access (identity provider, cloud admin, publishing) and accept that you may rotate them again after eviction. Do not run the full rotation into a live intrusion.
- Small organisations: Fewer credentials makes "rotate everything reachable" practical. Use the provider revocation features (GitHub's credential revocation API, revoking all sessions and tokens from account settings) and keep the evidence table even if it is a spreadsheet.
- Large organisations: Split scoping and execution. A small incident team owns the reach analysis and the evidence; platform teams execute rotation by group. Without that split, each team rotates what it owns and the unowned credentials are left.
Detection and measurement
- Alert on old values: After revocation, keep detections for the old key IDs and token fingerprints for at least the length of the investigation. A failed attempt with an old value is evidence the attacker still has data.
- Watch survivor types: Service accounts doing new things (plugin installs, bulk repository reads, new releases) were the signal in both Cloudflare and Trivy.
- Watch the release path: Alert on tag force-pushes, releases outside the normal workflow and releases by bot accounts at unusual times.
Metrics that show the process works:
- Percentage of credentials on the scoped list with a recorded failed-use test (target 100% before closure).
- Percentage of NHIs with last-used data available (the input that makes "unused" decisions safe).
- Time from incident declaration to revocation of the highest-privilege group.
- Number of credentials found after closure in the blast radius (target zero; any non-zero value is a process failure to review).
- Percentage of rotated long-lived credentials replaced with short-lived or federated credentials.
Framework mapping
| Framework | Reference | How it applies |
|---|---|---|
| NIST CSF 2.0 | RS.AN-03, RS.MI-01, RS.MI-02 | Establish what took place, contain, and eradicate, which includes revoking every reachable credential |
| NIST CSF 2.0 | PR.AA-01 | Identities and credentials for services are managed, so an owner and inventory exist before the incident |
| NIST SP 800-53 Rev 5 | IA-5, IR-4, SC-12 | Authenticator management, incident handling, key management |
| NIST SP 800-53 Rev 5 | CM-8, AU-6, SI-4 | Inventory, log review and monitoring that produce last-used data and detect use of old values |
| MITRE ATT&CK v20 | T1552.001, T1078, T1098.001, T1550.001 | Credentials in files, valid accounts, additional cloud credentials and application access tokens: the techniques that let an attacker survive a partial rotation |
Common failure patterns
- Excluding "unused" credentials without evidence: the Cloudflare Thanksgiving Breach 2023 started from exactly four of these.
- Rotating while the attacker still has a foothold: the Trivy Supply Chain Attack 2026 shows a rotation that was not atomic and a bot account that stayed usable.
- Not treating secrets in stolen code as compromised: in the Internet Archive Breach 2024 Zendesk tokens from the stolen code were used weeks after the first breach.
- Forgetting tokens held by former staff: marketplace and registry publishing tokens are rarely in the offboarding checklist.
- Closing on a statement, not evidence: "all credentials rotated" in a status update, with no per-credential record, cannot be checked and is often wrong.
- Leaving secrets in wikis and HAR files: Cloudflare deleted HAR files from its wiki during remediation; the Okta Support System Breach 2023 shows why.
Practitioner checklist
- Map each compromised foothold to everything it could read, including git history, CI secrets, wikis and tickets.
- Put every secret found in stolen data on the list, regardless of believed status.
- Sweep for credentials, apps, runners and keys created during the incident window.
- Evict the attacker and freeze release paths before issuing new secrets.
- Rotate in planned groups by blast radius, highest privilege first, in one window.
- Deactivate, test, then delete; never leave two live values.
- Replace stolen long-lived keys with federated or short-lived credentials where possible.
- Record a failed-use test and a log search for each revoked credential.
- Keep detections on old values after closure.
- Assign an owner and last-used tracking to every NHI so the next "unused" decision is based on data.
Standards and references
- NHI Mgmt Group: The State of NHI & AI Agent Breach Report 2026 (October 2026)
- NIST: Cybersecurity Framework 2.0
- NIST: SP 800-53 Rev 5 Security and Privacy Controls
- AWS: Update IAM user access keys
- GitHub Docs: Token expiration and revocation
- GitHub Docs: Secure use reference for GitHub Actions
- MITRE ATT&CK: T1098.001 Additional Cloud Credentials
Related NHI Mgmt Group resources: Secrets Rotation Runbook Template, Developer Secret Scanning Triage, NHI lifecycle automation, Non-Human Identity (NHI) Guide.