After left-pad: Keeping Your Builds Alive When an npm Package Disappears

Last Tuesday afternoon a lot of JavaScript builds started failing at the npm install step. The package they couldn’t find was left-pad, a tiny module whose registry description is just “String left pad.” Many of the projects that broke never asked for it directly. They got it through something that depended on something that depended on it.

For what actually happened I’m sticking to what npm, Inc. has said in its own posts and docs. The rest of this is advice for the rest of us on making builds survive the next time a package vanishes.

What npm says happened

npm published its account of the incident on March 23. By npm’s telling, the author of a package named kik and the company Kik disagreed over the name, Kik asked npm for help, and npm applied its package name dispute resolution policy and concluded the name should be maintained by Kik. Under that policy an existing package with a disputed name normally stays on the registry and the new owner publishes with a breaking version number, so anyone already using the old package would keep finding it. Instead, npm says, the author unpublished kik and 272 other packages without warning to developers of dependent projects. One of those was left-pad.

Shortly after 2:30 PM Pacific on Tuesday, March 22, npm says it began seeing hundreds of failures per minute as dependent projects, and their dependents, failed when requesting the now unpublished package. Within ten minutes another developer published a functionally identical left-pad as version 1.0.0. That was allowed because left-pad is open source and npm lets anyone use an abandoned package name as long as they don’t reuse version numbers. The errors kept coming, though. Dependency chains including babel and atom pulled left-pad in through a package called line-numbers, which explicitly requested 0.0.3, and a brand new 1.0.0 doesn’t satisfy that. So npm took what it called an unprecedented step and republished the original 0.0.3 from a backup, since republishing isn’t otherwise possible. It announced the plan at 4:05 PM, finished by 4:55 PM, and puts the whole disruption at 2.5 hours.

The same post says plainly that unrestricted unpublishing caused a lot of pain. npm promised to make it harder to unpublish a version when doing so would break other packages, and to replace a fully unpublished package that has known dependents with a placeholder so the name can’t be grabbed right away by someone with bad intentions.

The new unpublish policy

Yesterday npm followed up with changes to its unpublish policy. You can still unpublish a version that’s less than 24 hours old, and it’s removed completely. Once a version is older than that, the unpublish fails with a message to contact npm support, and support checks whether removing it would break other installs. If it would, npm won’t remove it, and your options are transferring ownership or getting the dependents to change. If every version of a package does end up removed, a security placeholder takes the name so it can’t be squatted. npm calls this a first step and says it may later factor in things like download activity and dependency checking.

The CLI docs were already pointed this way. The npm unpublish docs for npm 3.8.5 warn that removing versions others depend on is generally considered bad behavior, suggest npm deprecate if you just want people to upgrade, and say a name and version combination can never be reused once it’s unpublished.

This is a real improvement, but I wouldn’t treat it as the fix for your own builds. Anything older than a day comes down to a support decision, and npm’s own examples note that private applications that aren’t published to the registry don’t count as packages that depend on something. Your company’s app is exactly that kind of invisible dependent. And a registry outage, or a flaky network between your CI server and the registry, fails a build just as hard as an unpublish does.

Pinning versions helps less than you’d think

The first instinct is to pin everything. By default npm install --save writes a caret range. The npm config docs list the save-prefix default as ^, so a package at 1.2.3 gets saved as ^1.2.3, and turning on save-exact (or passing --save-exact) saves the exact version instead. The semver docs explain what a caret allows, including the detail that ^0.0.3 allows no updates at all.

Exact pins give you predictability, and that’s worth having. But they only cover your direct dependencies. The shrinkwrap docs make this point with an example: when you and the author of one of your dependencies aren’t the same person, you have no way to stop that dependency from picking up newly published versions of its own dependencies. Pinning also does nothing for availability. line-numbers asked for one exact version, and that exact version was the one that disappeared. A pin to a missing version fails just like a range with nothing left to match.

Shrinkwrap locks versions, not bytes

The lockfile npm gives you today is npm shrinkwrap. Run it in a project whose node_modules is installed and working, and it writes npm-shrinkwrap.json describing the whole tree, recursively, with each package’s version and a resolved location. Per the shrinkwrap docs, an install with a shrinkwrap present reproduces the tree in that file, fetching the files listed in resolved when they’re available and falling back to normal resolution by version when they aren’t, and then installs anything missing. It leaves out devDependencies unless you pass --dev. To add or update something later, you npm install --save that package by name, which updates both package.json and the shrinkwrap.

Commit a shrinkwrap for anything you deploy, so the build you tested and the build you ship install the same tree. Just be clear about what it protects. In the docs’ example, resolved points to a tarball URL on registry.npmjs.org, and the unpublish docs say unpublishing removes the tarball. A shrinkwrap tells npm precisely what to fetch, but if that file is gone from the registry, the fetch still fails. The shrinkwrap docs point the same way in their caveats: if you need full confidence that you can reproduce a build, check your dependencies into source control or use something that verifies contents rather than versions.

Keep your own copy

For apps you deploy, as opposed to libraries you publish, I think keeping your own copy of your dependencies is the most direct defense. The blunt version is committing node_modules. The repo gets bigger and the diffs get noisier, but a deploy only needs what’s already in the repository. Watch out for compiled addons: if you commit on a Mac and deploy to Linux, those binaries won’t match, and the npm rebuild docs describe npm rebuild as the way to recompile C++ addons, so run it on the target machine.

A tidier variant is bundledDependencies. The package.json docs describe it as a list of package names that get bundled into the tarball npm pack produces, which you can then install elsewhere by pointing npm install at that file.

Or put a registry in front of the registry

If you have lots of machines and developers, a private registry or caching proxy is the other route. npm’s registry setting defaults to https://registry.npmjs.org/, and you can point it somewhere else. Sinopia describes itself as a private and caching npm repository server. It keeps its own small database, asks the public registry for packages it doesn’t have, and keeps only the ones you use. Its README is modest about the upside, describing the cache as limited failover when the public registry is down, so don’t treat it as a guarantee. npm, Inc. also offers npm On-Site, a private npm registry and website you run on your own server, and its On-Site docs include a section on mirroring the public registry.

A cache can only hand back what somebody already installed through it, so route CI and developer machines through it from the start, and back up its storage. A cache that gets wiped is just an empty registry.

Weigh the tiny dependencies

Every dependency is more than its code. It’s also a name on a registry, a maintainer you don’t know, and a network fetch in every clean build. For a web framework that trade is obviously worth it. For a few lines of string handling, writing the helper yourself can be the cheaper option. Something like this runs on any Node you’re likely to have, since it’s plain ES5:

// Pad the start of a string to a target length (ES5, no dependencies)
function padLeft(value, length, fill) {
  var str = String(value);
  var ch = fill === undefined ? ' ' : String(fill);
  var needed = length - str.length;
  var padding = '';
  if (!(needed > 0) || ch.length === 0) {
    return str;
  }
  while (padding.length < needed) {
    padding += ch;
  }
  return padding.slice(0, needed) + str;
}

module.exports = padLeft;

padLeft(7, 3, '0') gives you '007'. I’m not arguing against small modules in general, since they come with tests and fixes somebody else maintains. But when a function is short enough to check by reading it, owning it removes one more thing that can vanish between your commit and your deploy.

The takeaway

By npm’s own account, one unpublish disrupted many thousands of projects for about two and a half hours, and npm has since changed its policy so that established versions can’t simply be removed. That’s good, but your build is still only as reliable as its weakest fetch. Pin where predictability matters, commit a shrinkwrap so every environment installs the same tree, and for anything you deploy, keep the actual bytes somewhere you control, whether that’s a checked in copy, a bundled tarball, or a registry cache of your own.