On 22 April 2021, HashiCorp disclosed that the private GPG key it used to sign the checksums of its product downloads had been exposed in the Codecov supply chain attack. Between 31 January and 1 April 2021, an attacker had quietly altered Codecov's Bash Uploader, a script that many of its customers ran inside their continuous integration (CI) pipelines, so that it sent each pipeline's environment variables to a server the attacker controlled. A subset of HashiCorp's CI pipelines used that script, and one of the secrets in their environment was the GPG private key that signs the SHA256SUMS files customers use to verify downloads of Terraform, Vault, Consul and other HashiCorp tools. HashiCorp said it found no evidence the key had been used, but it revoked it, published a new key, re-signed existing releases and shipped Terraform patches so the tool would trust the new key. Its investigation closed on 4 May 2021 with no evidence of tampered code or binaries.
Key takeaways
- HashiCorp's GPG private key for signing product download checksums was exposed when a subset of its CI pipelines ran Codecov's maliciously altered Bash Uploader.
- The altered script sent CI environment variables, including secrets, to an attacker-controlled server between 31 January and 1 April 2021, according to Codecov.
- HashiCorp revoked the key, published a new one, re-signed existing releases and released five Terraform patch versions so provider verification used the new key.
- HashiCorp said its investigation had not revealed evidence of unauthorised use of the key, and later found no evidence of malicious changes to its code or binaries.
- The identity lesson: a signing key placed in a CI environment variable is exposed to every tool that pipeline runs, including third-party scripts fetched at build time.
At a glance
| Organisation | HashiCorp, maker of Terraform, Vault, Consul and other infrastructure tools; a customer of Codecov |
|---|---|
| When | Codecov's Bash Uploader altered from 31 January 2021, discovered 1 April 2021; Codecov disclosed 15 April 2021; HashiCorp disclosed 22 April 2021 |
| Attacker | Unknown. Codecov said it had not been able to determine conclusively who carried out the attack |
| Entry point | A third-party CI tool: Codecov's Bash Uploader, altered after the attacker extracted a credential through an error in Codecov's Docker image creation process |
| Identities abused | The GPG private key HashiCorp used to sign SHA256SUMS files for product downloads, held in CI environment variables |
| Impact | Signing key exposed and replaced; releases re-signed; no evidence of unauthorised use of the key or of modified HashiCorp code or binaries |
| Category | NHI. Incident class: confirmed NHI breach (signing key harvested in a confirmed supply chain attack; no evidence of use) |
What happened
Codecov is a code coverage service. Many projects ran its Bash Uploader script in their CI pipelines by downloading it fresh on every build. On 15 April 2021, Codecov disclosed that a third party had made periodic, unauthorised changes to the script since 31 January. The attacker had gained access because of "an error in Codecov's Docker image creation process that allowed the actor to extract the credential" needed to modify the script, a Google Cloud Storage key according to Codecov. The added line sent the output of the pipeline's environment, and its Git remote information, to a server the attacker controlled. A customer spotted a checksum mismatch on 1 April, and Codecov secured the script that day. Our Codecov breach page covers that part of the story.
HashiCorp began its response on 15 April. Its security bulletin, HCSEC-2021-12, published on 22 April, said a subset of its CI pipelines used the affected Codecov component, and that the GPG private key used for signing hashes that validate HashiCorp product downloads was exposed. "While investigation has not revealed evidence of unauthorized usage of the exposed GPG key, it has been rotated in order to maintain a trusted signing mechanism," HashiCorp wrote. Only the SHA256SUMS signing mechanism was affected; HashiCorp said macOS code signing and notarisation, Windows Authenticode signing and Linux package signing were not.
The key mattered because of what it vouched for. Customers and automated tools verify a HashiCorp download by checking its hash against a SHA256SUMS file, then checking the GPG signature on that file. Anyone holding the private key could in principle sign a checksum file for a tampered binary that would pass that check. Terraform also uses the key to verify providers it downloads, so HashiCorp released Terraform 0.11.15, 0.12.31, 0.13.7, 0.14.11 and 0.15.1 on 26 April to trust the new key.
HashiCorp validated existing releases, compared logs, signatures and storage metadata with known-good copies, and re-signed releases with the new key. Its FAQ of 27 April said: "There is no evidence of malicious modification to HashiCorp code or binaries at this point in time." On 4 May it reported that its initial response and investigation were complete. SecurityWeek described HashiCorp as the first company to publicly acknowledge exposure in the Codecov incident; Twilio was among those that followed.
Timeline
| Date | Event |
|---|---|
| 31 January 2021 | Unauthorised changes to Codecov's Bash Uploader begin, according to Codecov. |
| 1 April 2021 | A customer reports a checksum mismatch; Codecov secures the script. |
| 15 April 2021 | Codecov discloses the incident and notifies affected users; HashiCorp begins its response. |
| 22 April 2021 | HashiCorp publishes HCSEC-2021-12, disclosing the GPG key exposure and its rotation. |
| 26 April 2021 | HashiCorp releases Terraform patch versions that use the new key for provider verification. |
| 27 April 2021 | HashiCorp publishes an FAQ: no evidence of customer data disclosure or modified code or binaries. |
| 4 May 2021 | HashiCorp reports its initial response and investigation complete. |
How it happened: the identity attack path
- Upstream credential extracted. The attacker extracted a cloud storage credential from Codecov's Docker image and used it to alter the Bash Uploader.
- Trusted script, fetched at build time. HashiCorp pipelines downloaded and ran the uploader as part of normal builds, giving it the same access to environment variables as the build itself.
- Signing key in the environment. The GPG private key for signing download checksums was available to those pipelines as an environment variable.
- Exfiltration. The altered script was built to send the whole environment, which included the key, to the attacker's server; HashiCorp treated the key as exposed.
- Revocation and re-signing. HashiCorp revoked the key, issued a new one, re-signed releases and updated Terraform to trust the new key.
Impact
- Confirmed: HashiCorp's private GPG key for signing product download checksums was exposed to the attacker's collection mechanism and had to be replaced.
- Not found: HashiCorp said its investigation did not reveal unauthorised use of the key, customer data disclosure or malicious modification of its code or binaries.
- Customer effort: anyone who verified HashiCorp downloads with the old key, and users of older Terraform versions, had to update to the new key or patched releases.
- Potential: a valid signing key could have been used to sign checksums for tampered downloads that would pass standard verification.
What this means for NHI governance
A release signing key is one of the most sensitive non-human identities a software company owns. It does not log in anywhere; it vouches. Every customer who checks a signature is trusting that only the vendor's release process can produce one. In this case the key sat in a CI environment, which meant it was available not only to HashiCorp's own build steps but to every script those steps ran, including a third-party uploader that was downloaded fresh on each build. When that script was poisoned upstream, the key left with everything else in the environment.
HashiCorp's response was quick and transparent, and its rotation and re-signing were the right moves. The broader lesson is about placement. Signing keys belong in a hardware security module or managed signing service, called only by a dedicated release job, never in the general environment of a build that also runs third-party tools. The same theme runs through the GitHub code signing certificate theft and the NVIDIA code-signing certificate theft. Our CI/CD Pipeline Identity Security Guide and Cryptographic Key Management Guide cover these controls.
Recommendations
- Move signing keys out of CI environment variables. Use an HSM or managed signing service that signs on request without releasing the key. See our Cryptographic Key Management Guide.
- Isolate the signing step. Run signing in a dedicated job with no third-party tools and no other secrets, so a compromised build step cannot reach the key. See our CI/CD Pipeline Identity Security Guide.
- Pin and verify third-party build scripts. Do not pipe scripts from the internet into a shell; pin versions and check hashes, which is how a Codecov customer caught the change.
- Give each pipeline only the secrets it needs. Scope secrets per job and use short-lived, federated credentials where possible. See our Secrets Management Guide.
- Plan for key rotation before you need it. Publish how customers obtain and trust a new key, and build tools that can update trusted keys, as HashiCorp had to do for Terraform. See the Leaked Credential Response Playbook.
- Treat supplier breaches as your incident. When a CI tool you use is compromised, assume every secret in affected pipelines is exposed and rotate them all.
Frequently asked questions
Was HashiCorp affected by the Codecov breach?
Yes. HashiCorp said a subset of its CI pipelines ran Codecov's altered Bash Uploader, which exposed the GPG private key used to sign the checksums of HashiCorp product downloads. HashiCorp disclosed this on 22 April 2021, revoked the key and re-signed its releases.
Were HashiCorp downloads tampered with?
HashiCorp said its investigation did not reveal unauthorised use of the exposed key and found no evidence of malicious modification to HashiCorp code or binaries. It validated existing releases against known-good copies before re-signing them with a new key.
Why was the HashiCorp GPG key important?
The key signed the SHA256SUMS files that customers and Terraform use to verify that downloads are genuine. Anyone holding it could have signed checksums for altered binaries that would pass standard verification, so it had to be replaced even without evidence of misuse.
Related NHI Mgmt Group resources
Codecov Breach 2021 · GitHub Code Signing Certificate Theft 2022 · NVIDIA Code-Signing Certificates Stolen 2022 · CI/CD Pipeline Identity Security Guide · Cryptographic Key Management Guide
How NHI Mgmt Group can help
Build pipelines hold some of the most powerful machine identities a software company has, from publishing tokens to release signing keys. We help teams find which secrets each pipeline can reach, move signing into protected services and plan rotations before they are needed. See our NHI and AI agent security training.
References
- Codecov: Bash Uploader Security Update (15 April 2021)
- HashiCorp: HCSEC-2021-12, Codecov Security Event and HashiCorp GPG Key Exposure (22 April 2021)
- The Register: HashiCorp reveals exposure of private code-signing key after Codecov compromise (26 April 2021)
- SecurityWeek: Twilio, HashiCorp Among Codecov Supply Chain Hack Victims (10 May 2021)