The Docker Hub Breach Is a Good Reason to Audit Your Build Credentials

Automated builds are the kind of thing you set up once and forget. You connect Docker Hub to GitHub or Bitbucket, a push kicks off an image build, and the credential that makes all of that work sits on someone else’s servers for years. Last week Docker told users that one of the databases holding those credentials had been accessed by someone who shouldn’t have been there. Even if you don’t use autobuilds, it’s a good excuse to look at every token and key your build pipeline holds.

I’m sticking to Docker’s own notice, its FAQ, and its status page for the incident facts, and to Docker, GitHub, and Atlassian documentation for the cleanup. I’m not going to guess at who did it or how, because Docker hasn’t said.

What Docker said happened

Docker’s support article, titled “Unauthorized access to Docker Hub database,” says that on Thursday, April 25, Docker discovered unauthorized access to a single Hub database storing a subset of non-financial user data. According to the notice, sensitive data from approximately 190,000 accounts may have been exposed during a brief period, which Docker puts at less than 5% of Hub users. The data it lists is usernames and hashed passwords for a small percentage of those users, along with GitHub and Bitbucket tokens for Docker autobuilds.

As for what Docker did, the notice says it revoked GitHub tokens and access keys for users with autobuilds that may have been impacted, and it warns that this means those autobuilds will fail until you reconnect your repositories. It asks affected users to check the security logs on their GitHub or Bitbucket accounts for unexpected actions, and it says Docker is reviewing its policies and has put additional monitoring tools in place. Docker’s status page posted an incident early on Saturday, April 27, saying GitHub and Bitbucket access tokens had been reset and that emails had gone out to everyone affected.

The first version of the notice asked users to change their Docker Hub password, plus the password on any other account that shared it. By Sunday the article had grown a FAQ that softens this a bit. It says that if your password hash was potentially exposed, Docker has already invalidated your password and emailed you a reset link, and that there’s otherwise no action required beyond relinking autobuilds. If you got an email and never used autobuilds, the FAQ says you may have linked GitHub or Bitbucket to Hub at some point, and Docker has unlinked it.

Two more FAQ answers are worth knowing. Docker says no Official Images were compromised, and points to GPG signatures on git commits and Notary signing as the extra protection it uses for them. And to people who asked why their tokens vanished before any email arrived, Docker’s answer is that revoking them first was done to protect users while it worked on securing the site.

Why the autobuild tokens are the part to care about

A leaked password hash has a familiar fix. The autobuild tokens are more interesting, because they were keys to other people’s source code hosts. When you link Docker Hub to GitHub, Docker’s linking docs have you authorize an OAuth application called Docker Hub Builder, and organization owners can grant that same application access to their organization’s repositories.

GitHub’s page on authorizing OAuth apps has a note that’s easy to skim past: you can’t scope source code access to read-only. So when you let an OAuth app at your repository code, you’re trusting whoever holds the token with more than the ability to read it. Docker revoked the tokens, which is the right move, but revoking a token tells you nothing about what it was used for before that. That’s the point of Docker’s advice to check your security logs.

Relink, but clean up the old link first

The fix for broken autobuilds is to unlink and relink, and the Docker docs have a detail that matters here. To fully revoke Docker Hub’s access to your GitHub account, you have to unlink it in two places: in Docker Hub’s Linked Accounts settings, and in your GitHub settings, where you revoke the Docker Hub Builder application. The same goes for Bitbucket, where the docs send you to OAuth under Bitbucket settings to revoke the Docker Hub entry. Clicking the plug icon in Hub alone only disables the link. The docs also point out that each repository set up as an autobuild source has a webhook pointing at Docker Hub, and revoking access doesn’t remove it.

My advice is to do the full revoke on both sides, then relink only the repositories you still build from. Docker’s FAQ says relinking creates a new read-only deploy key on your GitHub or Bitbucket source repository, so after you relink, go look at what’s actually attached to each repo. GitHub’s guide to reviewing deploy keys is short: in the repository’s settings, open Deploy keys and delete anything you don’t recognize or that’s out of date. While you’re in there, clear out webhooks for repos you no longer build.

One more place secrets hide. Docker’s autobuild docs let you set build environment variables, including an SSH_PRIVATE variable that holds a private key so builds can clone private submodules. Docker’s notice doesn’t list build environment variables among the exposed data, but if you ever pasted a key or token into one, replacing it costs you very little.

Review every OAuth app and key, not just Docker’s

While you’re in the settings pages, don’t stop at Docker. GitHub’s page on reviewing your authorized OAuth apps walks you to the Authorized OAuth Apps list, and GitHub’s own advice there is to check that nothing with expansive permissions, like access to private repositories, has been authorized without you knowing. Revoke anything you don’t recognize or don’t use anymore, because every entry is a token stored by some other company.

Then read your GitHub security log. As of this writing it covers the last 90 days, and it records activity around OAuth app authorizations and the SSH keys on your account, so an app authorization or key you didn’t create should show up there. Bitbucket has an audit log for account changes too, and Docker’s notice points Bitbucket users to it.

Rotate what your CI holds

Autobuilds are only one pipeline. Most of us also have a CI service with a pile of secrets in its settings: a Docker Hub username and password for pushing images, a GitHub personal access token for tagging releases, SSH keys for checkouts, cloud credentials for deploys. Docker’s notice doesn’t mention any of those, but if you’re already rotating, this is the cheapest time to rotate everything in your build and release path.

If CI pushes to Docker Hub with a username and password, anyone who gets that password can push to your repositories. Use a dedicated account for CI rather than your personal one (Docker’s docs already recommend a service account for team autobuilds), and change its password now. Pass the password on stdin rather than as a flag, which Docker’s login docs describe as a way to keep it out of shell history and logs:

echo "$DOCKER_PASSWORD" | docker login --username "$DOCKER_USERNAME" --password-stdin

For GitHub, when you generate a personal access token for CI you pick its scopes, so give it only what the job needs. For read-only checkouts a deploy key is often better, because GitHub’s deploy key guide notes that deploy keys are limited to a single repository and are read-only unless you give them write access when adding them. Bitbucket’s equivalent is a repository access key, which Atlassian documents as granting read-only access to a repository, and for API calls Bitbucket has app passwords, which let you choose permission scopes per password. My rule of thumb is one credential per job, scoped to one repository where possible, and replaced on a schedule rather than only after a breach.

Check what actually got built and pushed

Docker hasn’t said anything was pushed with those tokens, and I’m not suggesting it was, but checking is cheap. For each source repository that was linked to Hub, look at recent commits across every branch and tag, not just master:

git fetch --all --tags
git log --all --since="2019-04-01" --format="%h %an %ad %s"

On Docker Hub, go through the Builds tab and build reports for each repository, and compare them with builds you remember triggering. Anything you can’t explain deserves a closer look before anyone pulls it.

Downstream, remember that a tag can be moved to a different image, but a digest can’t. Docker’s pull docs describe pulling by digest as a way to pin an image to one exact version, so once you’ve confirmed an image is the one you built, reference it that way in deploys:

docker pull myorg/myapp@sha256:<digest-you-verified>

The takeaway

Docker revoked the tokens first and explained afterward, which is the order you’d want. What’s left is on us. Autobuilds and CI quietly collect long-lived, broadly scoped credentials for our most sensitive systems, and we rarely look at them after the first green build. Use this week to prune them, and swap broad tokens for narrow, per-repository ones so the next breach somewhere else has less to hand over.