If your secrets manager is compromised: contain it, rotate every secret and rebuild trust

When the system that holds every machine credential is compromised, you are no longer responding to one leaked secret: you are responding to all of them at once, and the tool you would normally use to rotate them may be the thing you cannot trust. A secrets manager sits underneath CI/CD, Kubernetes, databases, cloud accounts and SaaS integrations, so an attacker with administrative access, its root encryption key or a broad read policy can collect credentials for most of your estate in minutes. This guide is a response playbook for that scenario, written for a vendor-neutral product class: self-hosted vaults, cloud secrets services and CI platforms that store secrets centrally. It covers how the compromise shows up, triage questions, containment order, how to decide which secrets are exposed, rotation at scale, rebuild versus restore, evidence and notification. It is for incident responders, platform engineers and NHI owners, and assumes the prevention work in Securing the secrets manager itself.

Key takeaways

• Scope by capability, not by evidence of reads. If the attacker held a policy, token or key that could read a path, treat every secret on that path as exposed, as CircleCI told customers to do for every secret stored during its exposure window.
• Secure your own path first. Set up an out-of-band channel and a trusted administrative route into the secrets manager before you revoke anything, or you can lock out responders while the attacker keeps a token.
• Revoke before rotate, rotate before delete. The OWASP Secrets Management Cheat Sheet sets that order and warns that restarting an application does not revoke credentials it has already handed out.
• Rotate in tiers. Credentials that can mint other credentials (cloud admin keys, identity provider tokens, signing keys) come first, then data access, then the long tail; Cloudflare was breached through four credentials it skipped because it believed they were unused.
• Rebuild rather than restore when the root of trust was reached. If the store's encryption key, unseal material or backups were accessible, a restored copy carries the attacker's knowledge with it.

What a secrets manager compromise looks like

MITRE ATT&CK tracks reading secrets from these systems as T1555.006 Cloud Secrets Management Stores, noting that an adversary with high-privileged cloud accounts, or control of a service that is allowed to retrieve secrets, can simply request them through the normal API. That is the usual shape: the platform works as designed for the wrong caller. The scenarios differ in what the attacker can do next, and that determines how far you must go.

Triggers and first-hour triage

Invoke this playbook when any of the following is seen: an administrator or root-equivalent token used from an unknown network or outside a change; a new authentication method, role binding or policy granting broad read; audit logging disabled, redirected or showing a gap; a spike in secret reads by one identity, or reads of paths that identity never touches; recovery or unseal operations nobody scheduled; a vendor advisory for an exploited flaw in your secrets platform; or downstream evidence, such as a cloud or database credential used from attacker infrastructure, whose only copy lived in the store.

1. Preserve logs first: export the secrets manager's audit log, the logs of the KMS or HSM that protects its root key, the identity provider sign-in logs for its administrators, and infrastructure logs for its hosts. The OWASP cheat sheet stresses that audit records must be tamper resistant and correctly timestamped; copy them somewhere the attacker cannot reach before anyone changes configuration.
2. Identify the attacker's identity and capability: which token, account or trust was used, what policies were attached at the time (not now), and whether it could create tokens, change policies or read audit configuration.
3. Establish the window: earliest possible access to latest observed activity. Every secret that existed in readable paths at any point in that window is in scope.
4. Decide whether the root of trust was reached: any access to the root key, unseal or recovery shares, KMS key policy, host memory or backups moves you straight to the rebuild path.
5. Set severity and mode: targeted response when the identity and its scope are known and narrow; full containment when an admin path, an auth method or the root key is involved, or when scope is unknown.

Containment in the right order

The order matters more here than in most incidents, because the store is both the crime scene and the tool for recovery. Run these steps with the incident commander's approval and a timestamped decision log.

1. Move responders out of band: use a chat and bridge that do not authenticate through any credential held in the store, and assume the attacker can read tickets and chat that integrate with it.
2. Secure the administrative path: confirm break-glass access works (OWASP recommends break-glass credentials backed up in a secondary secrets system, and the NCSC says the break-glass process must be carefully protected), then restrict admin access to named responders from managed devices. See the Break-Glass Emergency Access Guide.
3. Remove attacker trust paths: disable authentication methods, role bindings, identity federation trusts and policies the attacker created or widened. Do this before revoking tokens, or the attacker can log in again.
4. Revoke tokens and leases: revoke the attacker's tokens and every token issued to the compromised admin or role, including child tokens. Revoke leases for dynamic credentials issued during the window, which also invalidates the database or cloud credentials behind them where the platform supports lease revocation.
5. Decide on read lockdown: if scope is unknown, restrict reads to a minimal allow-list of production paths, accepting that some deployments will fail. A full seal or shutdown stops the attacker but also stops every workload that fetches secrets at start-up; record the business decision.
6. Contain the hosts and agents: isolate compromised servers, sync operators and CI runners. Stopping a runner or restarting an app does not revoke what it already fetched.
7. Protect the root key: if the backing KMS or HSM key may be exposed, restrict its key policy to the rebuild team and disable any other principal's decrypt permission. See the Encryption Key Compromise Playbook.

Deciding which secrets are exposed

Audit logs tell you what was read, but only if the attacker did not change logging, did not use an export or snapshot function, and did not read storage directly. Treat logged reads as a minimum, not the answer.

Remember secret versions: many stores keep previous values for rollback, so a "rotated last month" secret can still be exposed if an older value remains valid at the source. Also list secrets the store generates or brokers rather than stores, such as dynamic database users and signed certificates, because their issuing credentials (the store's own connection to the database or CA) are themselves secrets in scope.

Rotating at scale

Rotation must happen at the source system that honours the credential, not just in the store. Changing a value in the store without revoking the old one at the database, cloud account or SaaS provider leaves the attacker's copy working. Prioritise by what the credential can do.

• Use a tracker, not memory: export the in-scope list to a sheet with owner, source system, revocation method, dependent services, rotated time and verified time. Cloudflare rotated more than 5,000 production credentials after its 2023 incident; the four it missed were the way back in. See the Cloudflare Thanksgiving Breach.
• Do not skip "unused" secrets: revoke them at the source instead of rotating. An unused credential that still works is an attacker's best option.
• Replace rather than rotate where you can: swap static keys for workload identity federation or short-lived dynamic credentials during the rebuild. See the Dynamic Secrets and Ephemeral Credentials Guide.
• Verify revocation: attempt authentication with the old value in a controlled way, or confirm in the source system that it is disabled. Log attempts to use revoked secrets; NHI Mgmt Group recommends alerting on them for at least 90 days.

Eradication: rebuild or restore

When restoring configuration from backup, verify the backup's integrity and date first: CSF 2.0 outcome RC.RP-03 asks that backups and restoration assets are checked before use. Never import secret values from a backup taken during or after the window into a rebuilt store; issue new ones at the source. Before cutover, close the entry point: the stolen SSO session, the over-broad admin role, the exposed recovery share or the unpatched flaw.

Evidence to preserve

• Audit logs: the store's audit stream and any secondary copy, with hashes, covering at least the full window.
• Configuration snapshot: policies, auth methods, roles, token settings, audit devices and plugins as found, before you revert anything.
• Token and lease inventory: accessor IDs, creation times, parents and policies of all live tokens and leases at the time of discovery.
• Root key evidence: KMS or HSM key usage logs and key policy history.
• Host evidence: memory and disk images of store servers and sync agents where code execution is suspected. See the Memory Forensics Guide.
• Downstream evidence: cloud, database and SaaS logs showing use of in-scope credentials during the window, which drive the scope of the wider investigation.

Notification triggers

• Personal data: if any in-scope credential gave access to personal data and you cannot show it was not used, legal must assess breach notification duties. See the Cyber Incident Reporting Requirements Map.
• Customers: if the store held customer credentials (as CI and integration platforms do), customers must be told which secrets to rotate and from which date to investigate, as CircleCI did. See the CircleCI Breach.
• Providers and partners: notify SaaS and cloud providers whose keys were exposed so they can watch for abuse.
• Platform vendor: open a security case if the cause may be a product flaw.

Detection and measurement

The detections that would have caught the attack become the heightened monitoring for the next 90 days: admin token use outside change windows, new auth methods or policies with wildcard paths, audit configuration changes, read volume per identity against its baseline, reads of paths outside an identity's normal set, recovery or unseal operations, and any attempt to use a revoked secret at a source system.

• Time to export audit logs for a 90-day window (tested in exercises).
• Percentage of in-scope secrets with a documented revocation method and owner, which predicts rotation speed.
• Tier 0 rotation time from declaration to verified revocation.
• Share of workloads using short-lived or federated credentials, which shrinks the next incident.
• Revoked-secret use attempts after the incident, which should fall to zero.

Common failure patterns

• Rotating in the store but not at the source: the new value is stored, the old value still works at the database or cloud provider.
• Skipping credentials believed to be unused: the pattern behind the Cloudflare intrusion after the Okta support breach.
• Trusting encryption at rest: CircleCI's data was encrypted, but the attacker extracted keys from a running process.
• Restoring the old store from backup: brings back every value and policy the attacker knew, and sometimes their auth method too.
• Forgetting cached copies: secrets synchronised into Kubernetes Secrets, CI variables or config files stay valid after the store is cleaned. See the Kubernetes Cluster Compromise Playbook and the CI/CD Pipeline Compromise Playbook.
• A central store of everyone's tokens: when one location holds credentials for many customers or systems, one access opens all of them, as in the Sisense Breach.

Framework mapping

Practitioner checklist

• Keep a tested out-of-band channel and break-glass path that do not depend on the store.
• Stream the store's audit log and its KMS or HSM key logs to a separate system with at least 12 months of retention.
• Maintain an inventory of every secret with owner, source system, revocation method and dependent services.
• Write tier 0 rotation runbooks and time them in a tabletop.
• Keep store configuration as code so eradication can diff against known good.
• Know where secrets are cached outside the store and how to purge each location.
• Pre-decide who can approve a read lockdown or full shutdown, and its business impact.
• Replace static secrets with federated or dynamic credentials during recovery, not "later".

Standards and references

• NIST: SP 800-61 Rev. 3, Incident Response Recommendations and Considerations for Cybersecurity Risk Management
• NIST: SP 800-53 Rev. 5, Security and Privacy Controls
• NIST: Cybersecurity Framework 2.0
• OWASP: Secrets Management Cheat Sheet
• OWASP: Non-Human Identities Top 10
• MITRE ATT&CK: Cloud Secrets Management Stores
• NCSC: Secure system administration, use privileged access management
• CircleCI: CircleCI incident report for January 4, 2023 security incident

Related NHI Mgmt Group resources: Secrets Management Guide · Securing the secrets manager itself · Leaked Credential Response Playbook · If your privileged access management (PAM) vault is compromised · Cloud Credential Compromise Playbook