CI/CD Pipeline Compromise Playbook: Responding to Build System Attacks

A compromised build pipeline hands an attacker two things at once: the secrets your pipelines hold and the ability to ship code that your customers and production systems trust. Pipeline incidents rarely look like a classic intrusion. They arrive as a repointed action tag, a planted workflow file, a stolen bot token, a poisoned dependency or a breach at your CI provider, and the evidence sits in workflow logs that may be public or short-lived. This template is a playbook for responding to a compromise of source control, CI/CD workflows, runners, build tooling or release credentials. It is written for security operations, platform and DevOps teams, with decision points for incident managers and executives. Adapt roles, timings and tool names to [Organisation].

How to use this template

• Replace text in square brackets, such as [Organisation], [Incident manager] and [Platform engineering], with your own names and contact routes.
• Steps are numbered and owned. Steps marked Decision: need a recorded decision, decision maker and time.
• Examples use GitHub Actions terms because they are widely used; map them to your platform (GitLab CI, Azure Pipelines, Jenkins, CircleCI and others have equivalents).
• Use it with your main incident response plan and the Supply Chain Compromise Response Playbook when the problem started upstream.

1. Scope and triggers

In scope: source code repositories and their settings, workflow and pipeline definitions, hosted and self-hosted runners, third-party actions and plugins, build dependencies, artefact and package registries, signing keys, and the credentials pipelines use (secrets, deploy keys, bot tokens, OIDC trust relationships). The NSA and CISA guidance Defending Continuous Integration/Continuous Delivery (CI/CD) Environments frames three threat scenarios, and each is a trigger here.

2. Roles and contacts

3. Detection and triage

1. [SOC analyst] records the trigger and identifies affected repositories, workflows, runners and the exposure window. For a third-party action, list every workflow that referenced it by tag or branch rather than commit SHA during the window.
2. [SOC analyst] preserves evidence before logs expire or are deleted (step 4.1).
3. [SOC analyst] lists every secret that affected jobs could read: repository, environment and organisation secrets, the job's GITHUB_TOKEN permissions, cloud roles reachable through OIDC, and credentials on self-hosted runners.
4. [SOC analyst] checks for output: secrets printed to logs (the tj-actions/changed-files compromise wrote runner secrets into workflow logs, readable by anyone on public repositories), outbound connections from runners, and new releases or packages.
5. Decision: [Incident manager] sets severity. Critical: release credentials, signing keys or production deployment roles exposed, or a tainted artefact published. High: secrets exposed but no evidence of release tampering. Medium: suspicious change blocked before execution.

4. Containment

1. Preserve evidence ([SOC analyst]). Export workflow run logs, job metadata and artefacts for the window; source host audit logs (for GitHub, including events such as org.update_actions_secret); repository history including force pushes and tag changes; runner images and disks for self-hosted runners; registry publish logs; cloud logs for OIDC role sessions. Hash and store them read-only.
2. Stop execution ([Platform engineering]). Disable affected workflows, or Actions for the repository or organisation; take self-hosted runners offline and keep their disks; block the malicious action or dependency through the allowed actions policy.
3. Halt releases ([Release manager]). Freeze deployments and publishing from affected pipelines. Decision: [Incident manager] whether to freeze all releases organisation-wide where the scope is unclear.
4. Revoke and rotate ([Secret owners], [Cloud team]). Treat every secret readable by an affected job as compromised. Revoke first, then reissue: PATs, bot and app tokens, deploy keys, registry publish tokens, cloud keys, database credentials. Tighten OIDC trust conditions on the cloud side (for example the sub, repository, ref and environment claims) and revoke active role sessions. After the CircleCI breach, the provider told customers to rotate every secret stored on the platform and assume it had been accessed.
5. Lock accounts ([IAM team]). Revoke sessions and tokens for compromised developer and maintainer accounts; enforce phishing-resistant MFA before access is restored.

Use this table to agree containment actions per credential type in advance, so the on-call team does not have to work them out during the incident.

Revoke before you rotate: replacing a stored value does not stop an attacker who already copied the old one. Scope the exposure window generously. If the first malicious run cannot be dated, start from the last change to the affected action, dependency or workflow that you can prove was clean.

5. Eradication

1. [SOC analyst] hunts for persistence: new or changed files under .github/workflows (or equivalent), new webhooks, deploy keys, apps, runners, branch protection or ruleset changes, and secrets created during the window. In the GhostAction campaign, attackers committed a workflow under the victim's own identity that sent the repository's secrets to an external server.
2. [Platform engineering] removes malicious commits, restores branch protection and re-pins every third-party action to a reviewed full-length commit SHA.
3. [Platform engineering] rebuilds self-hosted runners from clean images; prefer ephemeral just-in-time runners from here on.
4. [Cloud team] uses the cloud logs to check what any stolen credential did in the window, and removes created users, keys and resources.
5. [Release manager] identifies every artefact built during the window and compares it with a rebuild from verified source.

6. Recovery

1. [Release manager] rebuilds from known-good source on clean runners and publishes new versions with provenance and signatures. Deprecate or remove tainted versions in the registry and tell users which versions to avoid.
2. [Platform engineering] re-enables pipelines in stages, starting with low-risk repositories, with default GITHUB_TOKEN permissions set to read-only and secrets restricted to protected environments.
3. Decision: [Incident manager] and [Release manager] lift the release freeze once rotation is confirmed complete and no persistence remains.
4. [SOC analyst] applies heightened monitoring for [30] days: workflow file changes, runner egress to new hosts, use of any revoked credential.

7. Communication and notification

• Internal: engineering-wide notice of the release freeze and what developers must do (rotate personal tokens, re-authenticate).
• Customers and downstream users: if a tainted artefact was published, publish affected versions, hashes and the safe version, and coordinate a security advisory.
• Providers: report to the source host, CI provider, package registries and upstream maintainers so they can remove malicious versions.
• Regulators: [Legal and privacy] assess personal data breach notification (UK and EU GDPR Article 33: 72 hours) and sector duties such as NIS2 and DORA.

8. Post-incident review

1. [Incident manager] records the timeline: initial access, first malicious run, detection, workflows disabled, last secret rotated, clean release.
2. Answer: which long-lived secret made the impact possible, and could OIDC have replaced it? Were actions pinned? Could anyone push a workflow without review? Were logs retained long enough?
3. Map gaps to the NSA and CISA mitigations and track actions with owners and dates. Update this playbook and the SOC Runbook Template.

Implementation guidance

Prepare before you need it

1. Secrets inventory: every pipeline secret with owner, scope and rotation procedure, so step 4.4 takes hours, not weeks.
2. Federation: replace stored cloud keys with OIDC, which issues a token per job that the cloud provider exchanges for short-lived access. See the Workload Identity Federation Guide.
3. Hardening: pin actions to commit SHAs, set the default GITHUB_TOKEN to read-only, avoid pull_request_target with untrusted code, add .github/workflows to CODEOWNERS, and do not use self-hosted runners on public repositories, all per GitHub's security hardening guidance. See the GitHub and GitHub Actions Security Guide.
4. Logging: stream source host audit logs and runner telemetry to the SIEM with retention beyond the platform default.
5. Exercise: a tabletop for a compromised third-party action, then a timed rotation drill of the top [20] pipeline secrets.

Decision guidance

• Rotate everything or targeted: rotate everything readable when secrets reached logs, a provider was breached or the window is unknown. Targeted rotation is reasonable only when logs prove which secrets the malicious step accessed.
• Small organisations: on a hosted platform, focus on account security, pinned actions, OIDC and a written rotation list.
• Large organisations: enforce allowed actions and required workflows centrally, and use organisation rulesets so one change contains many repositories.

Metrics

• Time from detection to affected workflows disabled, and to last exposed secret rotated.
• Percentage of cloud deployments using OIDC rather than stored keys.
• Percentage of third-party action references pinned to a full commit SHA.
• Percentage of repositories requiring code owner review for workflow changes.
• Percentage of released artefacts with verifiable provenance.

Common failure patterns

• Rotating secrets but leaving the compromised bot token or developer session active.
• Deleting a malicious commit and assuming the secrets it read are safe.
• Build tokens able to push to release branches: in the Amazon Q Developer extension compromise, AWS said an inappropriately scoped GitHub token in its CodeBuild configuration let a malicious commit reach a release.
• Reusing persistent self-hosted runners after the incident.
• No list of which secrets each pipeline holds, so rotation stalls.

Framework mapping

Standards and references

• NSA and CISA: Defending Continuous Integration/Continuous Delivery (CI/CD) Environments
• GitHub Docs: Security hardening for GitHub Actions
• GitHub Docs: OpenID Connect
• NIST: SP 800-61 Rev. 3, Incident Response Recommendations and Considerations for Cybersecurity Risk Management
• MITRE ATT&CK: Poisoned Pipeline Execution
• MITRE ATT&CK: Compromise Software Supply Chain
• NIST: SP 800-53 Rev. 5

This template is provided for guidance and must be adapted to your organisation. Related NHI Mgmt Group resources: CI/CD Security Guide · CI/CD Pipeline Identity Security Guide · Software Supply Chain Security Guide · Leaked Credential Response Playbook