The Codecov Bash Uploader Breach and the Problem With Curl to Bash in CI

There’s a line in a lot of CI configs that nobody reads anymore. It fetches a script from a vendor’s URL and runs it straight away, usually right after the tests pass, and it’s been working fine for years. Last week Codecov told its users that for roughly two months, the script behind one of those lines wasn’t the script they thought it was.

For the incident facts I’m sticking to Codecov’s own security notice and its documentation. I’m not going to guess at who did it, because Codecov says it hasn’t been able to determine that, and it has reported the matter to law enforcement.

What Codecov said happened

Codecov’s Bash Uploader security update, dated April 15, says that on April 1 the company learned someone had gained unauthorized access to its Bash Uploader script and modified it. According to the notice, the way in was an error in Codecov’s Docker image creation process, which let the actor extract a credential that could modify the script. The FAQ fills in a bit more: the credential was a Google Cloud Storage key, and Codecov’s forensic work so far shows periodic unauthorized access to it beginning January 31, 2021.

What the altered script did is the part that matters for everyone downstream. Codecov says the change could take information from users’ CI environments and send it to a third-party server outside Codecov’s infrastructure. In the company’s words, that potentially covers any credentials, tokens, or keys passing through the CI runner that were reachable when the uploader ran, anything those credentials could reach (services, datastores, application code), and the git remote information for the repository, meaning the URL of the origin.

It wasn’t only people calling the script directly. Codecov says three of its integrations use the Bash Uploader under the hood, so the Codecov GitHub Action, the Codecov CircleCI Orb, and the Codecov Bitrise Step were affected too. Codecov’s other uploaders, like its Ruby gem and its Node package, use separate implementations that don’t rely on the Bash script. Self-hosted Codecov installs are very unlikely to be affected, according to the notice, unless the CI pipeline was fetching the script from codecov.io rather than from the self-hosted server.

The detail I keep coming back to is how it was caught. Codecov says a customer reported it on the morning of April 1 because they were checking the shasum of the downloaded script against the one Codecov publishes on GitHub, and the two didn’t match. One team doing the boring verification step found a problem that had been running since the end of January.

What Codecov told users to do

The advice is blunt. If you used any of the Bash Uploaders between January 31 and April 1, 2021, and you didn’t do a checksum validation, Codecov recommends that you re-roll every credential, token, or key in the environment variables of those CI processes. Codecov emailed the users it identified as affected and put a banner in the app, but its FAQ still suggests rotating if you used the uploaders in that window without checking the checksum, whether or not an email arrived.

To figure out what was exposed, Codecov suggests running the env command in your CI pipeline and treating anything private or sensitive in the output as something to invalidate and regenerate. It also says to audit how those tokens have been used, and if you keep a locally stored copy of the uploader, to check it for the altered code Codecov describes and replace it with the current version.

That env suggestion is simple and correct, but I’d go a bit wider when working out what to rotate. The environment at the moment the uploader ran is what counts, and it tends to hold more than the secrets you remember adding. Think about cloud provider access keys for deploys, package registry publish tokens for npm, PyPI, or RubyGems, Docker registry credentials, database URLs with passwords embedded in them, API keys for third-party services, signing keys, and any tokens that let CI push to your own repositories. Also look at your checkout step. If your CI clones with a token embedded in the remote URL, the git remote information Codecov mentions could include that token.

Rotating is the easy half. The notice also asks you to audit usage, and that’s the half people skip. A revoked key tells you nothing about what it was used for before you revoked it, so pull the access logs for your cloud accounts, registries, and source host for the window between late January and the day you rotated.

Why curl piped to bash is the real lesson

The README in Codecov’s own codecov-bash repository shows the standard usage as running bash on the output of a curl request to codecov.io. That pattern is everywhere, not just with Codecov, and it has one property that’s easy to forget: you’re executing whatever that URL returns at the moment your build runs. There’s no version, no review, and no record of what you actually ran. If the file changes on the server, every pipeline picks up the change on its next build.

That’s what made this incident so effective. The script is designed to run in CI, right next to your secrets, and it’s expected to make network calls. Nothing about it looks unusual in a build log.

A checksum check doesn’t make a vendor script trustworthy, but it does turn a silent swap into a failed build. Codecov publishes SHA1, SHA256, and SHA512 checksum files in that GitHub repository, and its uploader documentation walks through validating the script against them. The useful property is that the checksums live on GitHub while the script is served from Codecov’s side, so an attacker who can change one doesn’t automatically get to change the other. Here’s roughly what that looks like as a CI step:

curl -fsSL -o codecov https://codecov.io/bash
VERSION=$(grep -o 'VERSION="[0-9.]*"' codecov | cut -d'"' -f2)
curl -fsSL -o SHA256SUM "https://raw.githubusercontent.com/codecov/codecov-bash/${VERSION}/SHA256SUM"
grep ' codecov$' SHA256SUM > codecov.sha256
shasum -a 256 -c codecov.sha256 && bash codecov

If the hash doesn’t match, shasum exits non-zero and the script never runs. The grep is there because the checksum file lists more than one file, and older versions of shasum don’t have an option to ignore the entries you didn’t download.

My own preference goes a step further. If a script is going to run next to production credentials, I’d rather commit a known-good copy (or its hash) into my own repository and update it deliberately, the same way I’d bump a dependency in a lockfile. Then a change on the vendor’s side is a pull request I get to look at, not something that shows up in a build at 3am.

The same idea applies to GitHub Actions, including Codecov’s own action. GitHub’s security hardening guide for Actions warns that a compromised action can access every secret configured on your repository, and it recommends pinning third-party actions to a full length commit SHA because that’s the only way to treat an action as an immutable release. The guide also points out that a tag can be moved or deleted if a bad actor gets access to the repository behind the action.

Least privilege makes the next one smaller

Verification narrows the window. Least privilege shrinks what’s inside it. The coverage upload step doesn’t need your AWS deploy keys, but if they’re exported as environment variables for the whole job, every step can read them, including the one that talks to a third party.

A few habits help a lot here. Scope secrets to the specific job or step that needs them rather than the whole pipeline. Split jobs so that anything running third-party scripts, like coverage uploads, runs in a job without deploy credentials. Use tokens with narrow permissions and short lifetimes where your provider supports them. GitHub’s hardening guide makes the same points about credentials having the least privileges required, and about auditing and periodically rotating secrets so a leaked one isn’t useful forever.

GitHub also shipped something timely this week. On April 20 it announced that you can now control the permissions of the GITHUB_TOKEN with a new permissions key at the workflow or job level, and that any permission you leave out is set to none. The default is still read and write for all scopes, but you can change the repository or organization default to read-only contents, which the changelog says is enough for workflows that only need to clone and build. If a script in your workflow ever does go bad, a read-only token is a lot less interesting to steal.

The takeaway

If you used any of Codecov’s Bash-based uploaders between January 31 and April 1, rotate what was in that environment and check the logs for what those credentials did in the meantime. Then look at every other line in your CI config that downloads something and runs it. Verify it against a hash you didn’t get from the same place, keep the secrets away from steps that don’t need them, and give your tokens as little power as they can get away with. The customer who caught this wasn’t doing anything clever. They were just checking.