A compromised artefact or container registry lets an attacker swap what your clusters, servers and customers install, while every deployment still points at a name you trust. Registries hold the images and packages that production pulls, the push credentials your pipelines use, the pull secrets spread across clusters and the caches that proxy public sources. The urgent questions are which artefacts can still be trusted, which credentials must be rotated, and how to rebuild without redeploying the attacker's content. This playbook is for platform, DevOps and incident response teams running a self-hosted or cloud registry for container images, language packages or build outputs. It is vendor-neutral and assumes OCI-style registries where content is addressed by digest. Preventive design is in the Container Registry Security Guide and Securing the artefact and container registry itself.
Key takeaways
- Freeze writes before you investigate. Stop pushes, tag moves and deletions so evidence stays intact and the attacker cannot keep replacing content.
- Trust digests and signatures, not tags. A tag is a movable pointer; a digest is a hash of content. Compare what each tag points at now with your build records and signatures.
- Verification by the consumer is what saves you. SLSA's distribution channel threat assumes the registry itself can be modified, and the mitigation is verifying provenance or signatures at deploy time.
- The registry is full of machine credentials. Robot accounts, pipeline push tokens, cluster pull secrets, replication and webhook credentials and storage keys all need rotation.
- Rebuild from source, not from the registry. Re-push artefacts from a verified build, not by copying what is currently stored.
Scope: what a registry compromise covers
| Entry point | Typical cause | What the attacker can do | ATT&CK |
|---|---|---|---|
| Registry administrator account | Phished admin, reused password, no MFA | Change permissions, create robot accounts, disable scanning or signing policy, delete evidence | T1078.004 Cloud Accounts, T1685 Disable or Modify Tools |
| Push token or robot account | Token leaked from CI, a workstation or a public image | Push or overwrite images and packages under trusted names | T1525 Implant Internal Image, T1195.002 Compromise Software Supply Chain |
| Pull secret or read token | Copied from a cluster or developer machine | Download private images and the secrets baked into them | T1552.007 Container API (where reached through it) |
| Registry host or storage backend | Unpatched server, exposed storage bucket key | Modify stored blobs and metadata directly, outside the registry's audit log | T1525 |
| Pull-through cache or remote proxy | Upstream poisoned or cache configuration changed | Serve a malicious upstream artefact to internal consumers | T1195.002 |
| Replication, webhook or scanner integration | Over-privileged integration credentials | Spread content to other regions or registries, trigger deployments | T1098.001 Additional Cloud Credentials |
If the incident started in the build system, run the CI/CD Pipeline Compromise Playbook alongside this one. A pipeline that pushed poisoned artefacts with valid credentials looks like legitimate publication to the registry.
How the compromise shows up
- Tag moved without a build: a release tag now points at a digest no pipeline run produced.
- Pushes from unexpected identities or networks: robot accounts used from outside CI runner addresses, or human accounts pushing to release repositories.
- New robot accounts, access tokens or permission changes in the registry audit log.
- Signature or admission failures: admission controllers rejecting images that fail signature or provenance checks, or a sudden switch of policy from enforce to audit.
- Scanning or retention policy changes: vulnerability scanning switched off, immutability removed, or deletion enabled on release repositories.
- External signals: a vendor advisory, a public report of a poisoned image, or downstream customers reporting unexpected behaviour. The Checkmarx KICS supply chain attack reached users through poisoned images published under a trusted name.
Triage: the first hour
- Freeze writes: set affected repositories read-only or revoke push permissions for all non-responder identities. Disable tag deletion and manifest deletion if the registry allows it; under the OCI Distribution Specification a registry with deletion disabled answers 400 or 405 to delete requests.
- Snapshot the state: export the audit log, the list of repositories, every tag with its current digest and timestamps, robot accounts, tokens, permissions and integration settings. Hash the exports.
- Preserve storage: snapshot the storage backend and database, so blobs changed outside the API can be found later.
- Pause automated promotion: stop pipelines and GitOps controllers that deploy on tag change, so a moved tag cannot reach production during triage.
- Decide fail-closed for deploys: if you run admission control that verifies signatures, keep it in enforce mode; if you do not, freeze deployments of affected repositories until integrity is proved.
Proving which artefacts you can still trust
Integrity checking relies on the fact that a digest is a hash of the content. The OCI Distribution Specification requires clients to verify each layer's digest against the manifest, so if you know the right manifest digest, you know the right content.
| Evidence you hold | How to use it | Confidence |
|---|---|---|
| Signatures from a key or identity the attacker did not control | Verify every release digest against the expected signer identity and issuer | High, if signing keys or identities were outside the registry and pipeline |
| Build provenance attestations | Check the provenance subject hash matches the artefact and the builder and source are as expected | High (SLSA mitigation for tampering after build) |
| Digests recorded in deployment manifests in Git | Compare the digest each tag now points to with the digest deployed and committed | Medium to high, depends on Git integrity |
| CI build logs with output digests | Match pushed digests to build runs | Medium; logs may be in the attacker's reach |
| Tags and timestamps only | Treat as unverified | Low |
- SLSA framing: SLSA calls this the distribution channel threat: an adversary modifies the package on the registry through an administrative interface or infrastructure compromise. It is mitigated by the consumer verifying provenance or a verification summary from a trusted issuer, not by trusting the registry. See the SLSA and Build Provenance Guide.
- Signing keys: if signing keys were stored in the registry, in pipeline variables the attacker could read, or were used by a compromised robot account, signatures stop being evidence. Rotate the key and treat artefacts signed during the exposure window as unverified. See the Compromised Code Signing Certificate Playbook and the Sigstore Guide.
- Running workloads: list the image digests actually running in clusters and hosts and compare them with the verified set. A malicious image already pulled keeps running after the registry is clean (T1610 Deploy Container).
Containment by priority
- Remove attacker access: disable unknown admin and robot accounts, revoke all access tokens created or used in the suspect window, and reduce administrators to named responders.
- Quarantine suspect artefacts: move or block unverified digests rather than deleting them, so you keep evidence and stop pulls.
- Repoint tags to verified digests: only after verification, and with two-person approval.
- Close side doors: disable replication rules, webhooks and pull-through cache upstreams that were added or changed, and purge caches that may hold poisoned upstream content.
- Stop affected workloads: scale down or redeploy workloads running unverified digests in production, starting with internet-facing and privileged ones.
- Protect the logs: confirm audit logs stream to a store registry administrators cannot alter.
Credential rotation inventory
| Credential | Where it lives | Rotation action | Better pattern afterwards |
|---|---|---|---|
| Pipeline push tokens and robot accounts | CI secrets, runner hosts | Revoke all, issue new per pipeline with push scope to one repository | OIDC federation from CI so no long-lived push secret exists |
| Cluster and host pull secrets | Kubernetes secrets, node configuration, host credential helpers | Revoke and redeploy cluster by cluster | Workload identity or node identity with read-only scope |
| Developer access tokens | Workstations, credential helpers, shell history | Revoke all; reissue with expiry | SSO-backed short-lived tokens |
| Admin accounts and API keys | Registry, identity provider | Reset, re-enrol phishing-resistant MFA, revoke sessions | SSO only, break-glass account in a vault |
| Replication, webhook and scanner credentials | Registry integrations, other registries | Revoke and reissue with minimum scope | One credential per integration, owner and expiry |
| Storage backend keys and database credentials | Registry configuration | Rotate; check storage access logs | Registry-only identity, no human access to storage |
| Signing keys (if exposed) | KMS, HSM, CI | Rotate, re-sign verified artefacts, update verification policy | Keyless or KMS-held keys outside the registry |
| Secrets baked into images | Image layers | Rotate every secret found in images an attacker could pull | No secrets in images; inject at runtime |
Read access matters too: an attacker who could pull private images may hold any secret baked into them. The Docker Hub breach of 2019 exposed GitHub and Bitbucket tokens linked to around 190,000 accounts, and the tokens needed revoking on the code hosts as well. See How to find hard-coded secrets in container images and the Workload Identity Federation Guide.
Eradication and rebuild
- Rebuild or patch the registry service: for a self-hosted registry, rebuild hosts from a trusted image on a supported version and restore configuration from a reviewed export, not from the live system. For a cloud registry, confirm with the provider whether their side was involved.
- Rebuild release artefacts from source: run clean builds from verified source commits on a trusted builder, sign them with new or verified keys, generate provenance and push to fresh repositories or freshly reviewed ones.
- Re-baseline policy: restore immutability on release tags, disable deletion on release repositories, re-enable scanning and set admission control to enforce signature and provenance checks.
- Clean downstream: redeploy every workload from verified digests, purge proxy caches and mirrors, and notify internal teams that pulled during the window to clear local caches.
- Hunt for follow-on activity: look at what malicious images did when they ran: outbound connections, credential access in clusters and new cloud identities.
Recovery, testing and rollback
- Staged unfreeze: re-enable pushes for one pipeline at a time with its new federated identity, confirm signatures and provenance are produced and verified, then widen.
- Admission tests: try to deploy an unsigned test image and an image with a mismatched provenance; both must be rejected. See the Kubernetes Admission Control Guide.
- Rollback point: keep the list of verified digests per release as the known-good set, so a bad recovery step can be reverted by pinning to it.
- Residual monitoring: alert on any use of revoked tokens and on pushes from identities outside the new federated set.
Small and large organisations
| Decision | Small team, one registry | Large estate, many registries and clusters |
|---|---|---|
| Scope of rebuild | Rebuild every release artefact from source | Rebuild by risk tier: internet-facing and privileged first, verified-signature artefacts last |
| Pull secret rotation | Rotate all at once | Rotate per cluster with a tracking list; federate as you go |
| Customer impact | Notify customers who pulled public images in the window | Publish an advisory listing affected digests and verified replacements |
Evidence and notification
- Preserve: audit logs, tag-to-digest exports before and after containment, quarantined artefacts, storage snapshots, registry host images, CI logs for the window and identity provider sign-ins for admin accounts.
- Customers and users: if you publish images or packages externally, tell consumers which digests are affected and which verified digests replace them. Under NIST SP 800-61 Rev. 3, coordination with affected third parties is part of response (CSF 2.0 RS.MA-01). Legal duties vary; see the Cyber Incident Reporting Requirements Map.
Detection and measurement
- Share of production workloads running digests that verify against signature and provenance policy (target 100%).
- Release tags that are immutable (target 100%).
- Long-lived push tokens remaining after migration to federation (target zero).
- Time to answer "which digests did tag X point to, and when", measured in exercises.
- Pushes by identities outside the approved federated set (target zero; each one investigated).
- Revoked credentials seen in use after rotation (target zero).
Framework mapping
| Area | NIST SP 800-53 Rev. 5 | NIST CSF 2.0 | ATT&CK addressed |
|---|---|---|---|
| Artefact integrity and signatures | SI-7, SI-7(15), CM-14 | RS.AN-03 | T1525, T1195.002 |
| Provenance and component authenticity | SR-4, SR-11 | RC.RP-02 | T1195.002 |
| Registry credentials and robot accounts | IA-5, AC-6 | RS.MI-02 | T1078.004, T1098.001 |
| Audit log protection | AU-9 | RS.MI-01 | T1685 |
| Incident handling and recovery | IR-4, CP-10 | RS.MA-01 | T1610 |
NIST SP 800-204D sets out how to build these supply chain controls into CI/CD pipelines, including artefact attestations.
Common failure patterns
- Deleting suspect images first: evidence is lost and you cannot show customers what was served.
- Verifying tags instead of digests: the attacker moves the tag back after you check it.
- Signing keys in the same blast radius: keys held in pipeline variables or the registry make signatures worthless after compromise.
- Rotating push tokens but not pull secrets: read access to private images keeps leaking embedded secrets.
- Forgetting caches and mirrors: poisoned content keeps being served from a pull-through cache after the main repository is clean.
- Leaked storage credentials: blobs changed directly in storage never appear in the registry audit log; the Codecov breach began with a leaked cloud storage credential used to alter a distributed script.
Practitioner checklist
- Freeze pushes, tag changes and deletions on affected repositories and pause automated promotion.
- Export audit logs, tag-to-digest maps, accounts and tokens, and snapshot storage, before changing anything.
- Verify every release digest against signatures and provenance from identities outside the blast radius.
- Quarantine unverified digests and redeploy workloads from verified ones.
- Revoke and reissue push tokens, pull secrets, admin keys, integration and storage credentials.
- Rotate signing keys if exposed and any secrets baked into images an attacker could pull.
- Rebuild artefacts from source on a trusted builder and enforce admission checks.
- Purge proxy caches and mirrors and inform consumers of affected and replacement digests.
Standards and references
- Open Container Initiative: Distribution Specification
- SLSA: Threats and mitigations
- Sigstore: Verifying signatures with Cosign
- NIST: SP 800-204D Strategies for the Integration of Software Supply Chain Security in DevSecOps CI/CD Pipelines
- NIST: SP 800-61 Rev. 3 Incident Response Recommendations and Considerations for Cybersecurity Risk Management
- NIST: SP 800-53 Rev. 5 Security and Privacy Controls
- MITRE ATT&CK: T1525 Implant Internal Image
- MITRE ATT&CK: T1195.002 Compromise Software Supply Chain
Related NHI Mgmt Group resources: Container Registry Security Guide · CI/CD Pipeline Compromise Playbook · Kubernetes Cluster Compromise Playbook · Software Supply Chain Security Guide · If your secrets manager is compromised