On 4 January 2023, CircleCI told every customer to rotate every secret stored on its CI/CD platform. Its investigation found that malware on a CircleCI engineer's laptop, undetected by antivirus, had stolen a valid SSO session backed by two-factor authentication on 16 December 2022. Using that session, an attacker impersonated the engineer, who had privileges to generate production access tokens, and escalated into production. On 22 December the attacker exfiltrated customer environment variables, tokens and keys. The data was encrypted at rest, but the attacker extracted the encryption keys from a running process. CircleCI rotated GitHub OAuth tokens, revoked all project and personal API tokens, worked with Atlassian and AWS on further rotation and notification, and later reported that fewer than five customers had told it of unauthorised access to third-party systems.
Key takeaways
- Infostealer malware on an engineer's laptop stole a 2FA-backed SSO session cookie on 16 December 2022, so MFA was already satisfied when the attacker used it.
- The engineer could generate production access tokens, which let the attacker reach databases holding customer environment variables, tokens and keys.
- Exfiltration happened on 22 December 2022; the attacker also took encryption keys from a running process, so encryption at rest did not protect the data.
- CircleCI told customers to assume every secret stored on the platform in that period was accessed.
- The identity lesson: a CI/CD platform is a vault of other companies' machine credentials, and one stolen session can put all of them at risk.
At a glance
| Organisation | CircleCI and its customers |
|---|---|
| When | Laptop compromised 16 December 2022; reconnaissance 19 December; exfiltration 22 December; customer alert 29 December 2022; disclosure 4 January 2023; incident report 12 January 2023 |
| Attacker | Unattributed |
| Entry point | Malware on a CircleCI engineer's laptop that stole a valid 2FA-backed SSO session cookie |
| Identities abused | The engineer's SSO session; production access tokens the engineer could generate; customer environment variables, API tokens, OAuth tokens, SSH keys and cloud keys stored in CircleCI; platform encryption keys |
| Impact | Customer secrets exfiltrated and potentially decryptable; fewer than five customers reported unauthorised access to third-party systems; mass rotation across GitHub, Bitbucket and AWS |
| Category | NHI. Incident class: confirmed NHI breach (customer CI/CD secrets and tokens exfiltrated) |
What happened
CircleCI's investigation began on 29 December 2022, when a customer reported suspicious GitHub OAuth activity. By 30 December CircleCI knew that customer's GitHub OAuth token had been compromised, and on 31 December it began rotating all GitHub OAuth tokens on customers' behalf. By 4 January 2023 it had found the entry point: "an unauthorized third party leveraged malware deployed to a CircleCI engineer's laptop in order to steal a valid, 2FA-backed SSO session. This machine was compromised on December 16, 2022. The malware was not detected by our antivirus software."
"Because the targeted employee had privileges to generate production access tokens as part of the employee's regular duties, the unauthorized third party was able to access and exfiltrate data from a subset of databases and stores, including customer environment variables, tokens, and keys," CircleCI wrote. It saw reconnaissance on 19 December and exfiltration on 22 December, its last record of unauthorised production activity. "Though all the data exfiltrated was encrypted at rest, the third party extracted encryption keys from a running process, enabling them to potentially access the encrypted data."
On 4 January CircleCI shut down the compromised employee's access, cut production access for nearly all employees and replaced all potentially exposed production hosts. It disclosed the breach that evening and told customers to rotate their secrets, including OAuth tokens, project API tokens and SSH keys. On 5 and 6 January it revoked all project API tokens and older personal API tokens, and worked with Atlassian to rotate Bitbucket tokens. It finished rotating GitHub OAuth tokens on 7 January and worked with AWS to notify customers of potentially affected AWS tokens, which finished on 12 January.
CircleCI said it could not know whether stolen secrets were used against customers' own systems. "If you stored secrets on our platform during this time period, assume they have been accessed and take the recommended mitigation steps," it wrote. At the time of its report, "fewer than 5 customers have informed us of unauthorized access to third-party systems as a result of this incident."
Timeline
| Date | Event |
|---|---|
| 16 December 2022 | Malware on a CircleCI engineer's laptop steals a 2FA-backed SSO session. |
| 19 December 2022 | The attacker carries out reconnaissance in CircleCI systems. |
| 22 December 2022 | Customer environment variables, tokens and keys are exfiltrated. |
| 29 December 2022 | A customer alerts CircleCI to suspicious GitHub OAuth activity. |
| 31 December 2022 | CircleCI begins rotating all customers' GitHub OAuth tokens. |
| 4 January 2023 | CircleCI identifies the entry path, removes access, replaces production hosts and tells customers to rotate all secrets. |
| 7 January 2023 | GitHub OAuth token rotation completes; AWS begins notifying affected customers. |
| 12 January 2023 | CircleCI publishes its incident report. |
How it happened: the identity attack path
- Endpoint compromise. Malware on an engineer's laptop, missed by antivirus, stole a session cookie.
- Session replay past MFA. The stolen cookie represented a completed 2FA-backed SSO login, so the attacker impersonated the engineer from a remote location.
- Token minting. The engineer's role allowed generating production access tokens, which the attacker used to reach production.
- Bulk secret theft. Customer environment variables, tokens and keys were exfiltrated from databases and stores.
- Encryption bypassed. Encryption keys taken from a running process made the stolen data potentially readable.
- Downstream use. At least one customer's GitHub OAuth token was misused, and fewer than five customers reported unauthorised access.
Impact
- Secrets: customer environment variables, API tokens, OAuth tokens, SSH keys and cloud keys stored in CircleCI were exfiltrated and potentially decryptable.
- Customers: every customer was told to rotate all secrets; fewer than five reported unauthorised access to third-party systems by the time of the report.
- Ecosystem: CircleCI, GitHub, Atlassian and AWS coordinated token rotation and notifications.
What this means for NHI governance
CI/CD platforms hold other organisations' machine credentials by design: cloud keys to deploy, tokens to pull code, API keys to call services. That makes them one of the most concentrated stores of non-human identities anywhere. Here, one engineer's session was enough to reach them, and encryption at rest did not help because the keys were available in memory.
For customers, the lesson is to keep as few long-lived secrets in CI/CD as possible. OIDC federation lets pipelines get short-lived cloud credentials on demand, so there is nothing durable to steal. For platforms, the lesson is that access to production and token minting needs step-up authentication and device trust, which CircleCI added afterwards. See our CI/CD Pipeline Identity Security Guide and Token and Session Security Guide.
Recommendations
- Replace stored cloud keys in pipelines with OIDC federation. Short-lived, per-job credentials leave nothing long-lived to exfiltrate. See the CI/CD Pipeline Identity Security Guide.
- Know every secret you store with a CI/CD provider. Keep an inventory so you can rotate everything within hours if the provider is breached. See the Leaked Credential Response Playbook.
- Bind sessions to devices. Use device-bound sessions, token binding or continuous evaluation so stolen cookies fail elsewhere. See the Token and Session Security Guide.
- Require step-up authentication for production and token minting. Privileged actions should not ride on an everyday SSO session.
- Scope secrets to least privilege. A key stored for deployment should only be able to deploy, so theft yields as little as possible.
Frequently asked questions
How was CircleCI breached?
Malware on a CircleCI engineer's laptop stole a 2FA-backed SSO session cookie on 16 December 2022. The attacker used it to impersonate the engineer, generate production access tokens and exfiltrate customer secrets on 22 December.
What data was stolen from CircleCI?
Customer environment variables, tokens and keys stored on the platform. The data was encrypted at rest, but the attacker also took encryption keys from a running process.
What should CircleCI customers have done?
Rotate every secret stored in CircleCI, including OAuth tokens, API tokens, SSH keys and cloud keys, and review their own systems for suspicious activity from 16 December 2022 until rotation was complete.
Related NHI Mgmt Group resources
Slack GitHub Breach 2022 · Codecov Breach · CI/CD Pipeline Identity Security Guide · Token and Session Security Guide · Leaked Credential Response Playbook
How NHI Mgmt Group can help
CI/CD pipelines often hold the most powerful credentials an organisation has. We help teams inventory pipeline secrets, move to federated short-lived credentials and rehearse provider breach rotation. See our NHI and AI agent security training.
References
- CircleCI: CircleCI Jan 4, 2023 security incident report (12 January 2023)
- Help Net Security: CircleCI breach post-mortem, attackers got in by stealing engineer's session cookie (16 January 2023)
- SecurityWeek: CircleCI Hacked via Malware on Employee Laptop (16 January 2023)