SolarWinds Orion and the Build Pipeline You Probably Aren't Guarding

Most of us spend our security effort on the software we ship. The machinery that turns source into a signed installer gets far less attention, and it’s the part our customers trust without looking. About ten days ago SolarWinds said that trust was abused in its Orion Platform, and plenty of teams who have never touched Orion should be looking hard at their own build pipelines this week.

For the incident itself I’m sticking to SolarWinds’ advisory, CISA’s directive and alert, and Microsoft’s published defensive guidance. I won’t get into who did it, how the inserted code behaves, or indicators to hunt for, because none of that changes what you should do with your own pipeline.

What SolarWinds and CISA have said

The SolarWinds security advisory says a cyberattack on the company’s systems inserted a vulnerability into Orion Platform builds for versions 2019.4 HF 5, 2020.2 with no hotfix installed, and 2020.2 HF 1. It calls this a supply chain attack and says the affected builds have been removed from its download sites. The first version of the advisory named a wider range, 2019.4 through 2020.2.1, and SolarWinds narrowed it over the following days.

As of its December 20 update, the advisory asks customers on 2020.2 or 2020.2 HF 1 to upgrade to Orion Platform 2020.2.1 HF 2 as soon as possible, and customers on 2019.4 HF 5 to update to 2019.4 HF 6. It says it doesn’t believe systems on 2019.4 HF 4 or earlier were compromised by this vulnerability. For anyone who can’t upgrade right away, its primary mitigations are keeping Orion behind firewalls, disabling its internet access, and limiting ports and connections to what the platform requires.

CISA moved fast. Emergency Directive 21-01, dated December 13, ordered federal civilian agencies to immediately disconnect or power down Orion versions 2019.4 through 2020.2.1 HF1, block traffic to and from outside hosts for any machine where any version of Orion was installed, and report to CISA by noon Eastern on December 14. After cleanup, agencies must treat hosts monitored by Orion as compromised, rebuild them from trusted sources, and reset every credential used by or stored in the software. Supplemental guidance on December 18 split the version range into affected and unaffected builds and told agencies to label and isolate backups of affected versions so the bad code can’t sneak back in.

The detail that matters most for the rest of us is in CISA’s alert AA20-352A from December 17. It says a malicious binary was added into the SolarWinds software lifecycle and then signed by the legitimate SolarWinds code signing certificate. Customers got an update from their vendor, the signature checked out, and it didn’t help.

A signature tells you who, not what

Code signing proves a binary came out of a process holding a particular key. It says nothing about whether everything that went into that process was supposed to be there. If someone can change source, swap a build step, or tamper with output before signing, your customers get a perfectly valid signature on code you never meant to ship.

So the question for your pipeline isn’t “do we sign releases?” It’s “who and what can influence the bytes between a reviewed commit and a signed artifact, and would we notice?” NIST’s Secure Software Development Framework white paper, published in April, is a good map. Its Protect the Software practices cover protecting all forms of code from tampering, giving consumers a way to verify release integrity, and archiving each release, and it’s something concrete to point at when someone asks why build infrastructure deserves budget.

Guard build servers like production

Build agents often get treated as utility boxes with broad network access, stale patches, and lots of people who can log in to debug a job. Yet a build server that ships to customers is arguably more sensitive than any single production host, because whatever runs there ends up running everywhere your software is installed.

My advice is to give build infrastructure production controls: patched images, no interactive logins by default, strong authentication for the few who need access, and logs shipped somewhere the build system can’t edit. Better still, make agents ephemeral so each job starts from a known image and any tampering dies with the job. The SSDF’s toolchain practice calls for assessing the security of each tool and monitoring tool logs for security issues, and its build practice includes validating the authenticity and integrity of compilers and build tools.

Separate building from signing, and protect the key

If the job that compiles code also holds the signing key, anyone who can influence that job can get anything signed. Split them. Let the build produce unsigned artifacts plus hashes, and have a separate, tightly controlled signing service that only signs artifacts from approved builds, keeps its own record of what it signed, and can’t be reached from ordinary CI jobs.

Keep the private key where it can’t be copied off a disk. Microsoft’s customer guidance is about identity federation rather than code signing, but it recommends considering hardware security for token signing certificates, and the same thinking fits a code signing key. With a hardware security module or managed key service, an attacker has to use the key through your signing service, where you can log and alert, instead of walking off with it. The SSDF also suggests periodically reviewing code signing processes, including certificate renewal and protection. Decide now how you’d revoke and replace a signing certificate, because working that out mid incident is miserable.

Scope CI credentials tightly

CI systems quietly collect registry tokens, cloud deploy keys, and source host tokens, and each one is a way out of a compromised pipeline. Microsoft’s guidance says service accounts and service principals with admin rights should use high entropy secrets stored securely, that you should monitor for changes to those secrets and for anomalous use of service accounts, and that you should reduce permissions on the applications you keep. That describes CI credentials well too. In practice, give each job its own credential with the narrowest scope that works, keep secrets away from pull requests built from forks, and alert when a pipeline credential shows up anywhere other than your build agents.

Aim for builds you can verify

Reproducible builds, where the same source and toolchain produce bit for bit identical output, are hard to get fully right because timestamps, file ordering, and embedded paths get in the way. It’s still worth aiming for, because it lets you build the same tagged commit on two independently managed builders and compare the results before anything is signed.

diff <(cd builder-a/dist && sha256sum *) <(cd builder-b/dist && sha256sum *)

Even without identical output, you can record hashes for every artifact at build time, have the signing service refuse anything that doesn’t match the build record, and publish hashes with releases, which the SSDF lists as one way to let consumers verify integrity. Its archiving practice recommends keeping a copy of each release and all its components, which is exactly what you’ll want when someone asks which builds are affected.

Review what your build pulls in

Every dependency, base image, and build plugin is code running inside your pipeline. Pin versions and verify hashes so a changed upstream package fails the build instead of slipping in. Pip’s hash checking mode requires every requirement and dependency to be pinned and hashed, and npm ci installs from your lockfile, which records an integrity value for registry packages, and exits with an error if the lockfile and package.json disagree.

python -m pip install --require-hashes -r requirements.txt
npm ci

Make dependency changes visible in code review like any other change. The SSDF also suggests an organization wide repository of vetted open source components and a software bill of materials for each package you create, so “do we ship this?” takes minutes instead of days.

Watch what build and update infrastructure talks to

SolarWinds told its customers to firewall Orion and cut its internet access. Apply the same idea to your own build and update systems. A build agent needs your source host, your package mirror, and your artifact store, and almost never needs arbitrary hosts on the internet. Put build agents and update servers behind an egress allowlist, route dependency downloads through a proxy you control, and alert on any outbound connection that isn’t on the list. An unexpected DNS lookup from a build agent is cheap to log and very expensive to miss.

Have a plan to tell customers

Look at what Orion customers needed in the first days: which versions were affected, which fixed versions to install, and what to do if they couldn’t upgrade yet. The SSDF’s vulnerability response practice suggests a playbook that covers a major ongoing incident involving multiple parties, not just a single reported bug. Write it now. Decide who can pull a release from your download site, how you’d publish an affected versions list, how you’d reach customers who don’t read your blog, and how you’d ship a rebuilt release from infrastructure you trust.

CISA’s alert adds a wrinkle: it says findings should be protected by operational security measures, including out of band communications for staff and leadership. If your build systems were compromised, assume your email and chat might be too, and settle how your response team will talk before you need to.

The takeaway

Orion customers installed signed updates from their vendor, which is exactly what everyone tells them to do. That puts the burden on those of us who produce updates. Guard build servers like production, keep signing separate with the key in hardware, scope CI credentials tightly, verify what goes in and comes out, watch outbound traffic, and write the customer notification plan before you need it.