Yarn and yarn.lock: JavaScript Installs You Can Repeat

Last Tuesday Facebook open sourced Yarn, a new package manager for JavaScript that installs from the same npm registry you already use. The announcement describes it as a collaboration with Exponent, Google, and Tilde. A second client for a registry that already has one sounds like churn, but the problem it goes after is one most JavaScript teams have hit: run npm install on two machines a few days apart and you can end up with two different sets of dependencies.

Everything here describes Yarn 0.15.1, which is what npm install -g yarn gives you right now, based on its docs and its source at that tag. It’s a young project, so expect details to move.

Why Facebook built it

The Facebook engineering post is candid about the workarounds that came first. As its codebase and engineering team grew, Facebook says, the npm client gave it trouble with consistency, security, and performance. Checking in only package.json broke down in continuous integration, which Facebook keeps sandboxed and cut off from the internet. Checking in all of node_modules worked, but a minor babel update produced an 800,000 line commit, and React Native’s 68 listed dependencies became 121,358 files once installed. Zipping node_modules onto an internal CDN meant engineers needed internet access to build.

Shrinkwrap had rough edges too. The post says shrinkwrap files aren’t generated by default, fall out of sync when people forget to regenerate them, and are huge JSON blobs with unsorted keys, so Facebook wrote a tool to verify them and a script to sort them. On security, it mentions concerns with the way the npm client runs code from some dependencies automatically.

What they built instead is a client with a deterministic install, backed by a lockfile, a global cache, and checksums.

Why two installs drift apart

A package.json usually lists ranges, not versions. A caret range like ^1.2.0 accepts any later 1.x release, so what you actually get depends on what’s been published by the time you install. Facebook’s post points out that semver relies on package authors not making mistakes, so a bug or an accidental breaking change can ride in on a range you never touched. It also says npm lays out node_modules non-deterministically, so the order in which dependencies get installed can change the structure of the folder from one person to the next. Either way you get “works on my machine” bugs that take ages to track down.

What yarn.lock records

The yarn.lock docs put it simply: to get consistent installs across machines, Yarn needs to store exactly which version of each dependency was installed, and it keeps that in a yarn.lock file at the root of your project. Here’s a real entry from Yarn’s own lockfile at the 0.15.1 tag:

acorn@^2.1.0, acorn@^2.4.0:
  version "2.7.0"
  resolved "https://registry.npmjs.org/acorn/-/acorn-2.7.0.tgz#ab6e7d9d886aaca8b085bc3312b79a198433f0e7"

The key is the exact name and range as some package.json asked for it, and two ranges that settle on the same version share one entry. Under it sits the version that was picked, the tarball URL it came from, and, after the #, the SHA-1 checksum the registry lists for that tarball. Entries whose packages have dependencies of their own list those too, as the ranges they request. The file starts with a comment saying it’s autogenerated and shouldn’t be edited directly, and Facebook says the format uses ordered keys so changes stay small and easy to review.

The docs are clear about how to treat it. Yarn manages the file and updates it as you add, upgrade, and remove dependencies, you shouldn’t edit it by hand, and every yarn.lock should be checked into source control so your coworker’s laptop and the CI server get the same dependency tree.

What yarn install does with it

Facebook breaks an install into resolution, fetching, and linking, and the 0.15.1 CLI shows a fourth step that builds packages by running their install scripts. Reading the source for the install command and its helpers, here’s what happens to a package that already has a lockfile entry.

During resolution, the npm resolver checks the lockfile for that exact pattern first. On a hit it takes the locked version, URL, and checksum and doesn’t ask the registry what the newest match is. During fetching, Yarn looks in its global cache, which the source puts at ~/.yarn-cache on macOS and Linux and under %LOCALAPPDATA% on Windows. If the package is there it’s reused. If not, the tarball is downloaded and hashed as it streams in. Linking copies the files from the cache into your project’s node_modules, and the build step runs lifecycle scripts, which you can skip with --ignore-scripts.

At the end of an install Yarn writes yarn.lock from what it just resolved, so a range you added to package.json by hand gets an entry. The install docs list two flags that change this: --pure-lockfile means don’t generate a lockfile, and --no-lockfile means don’t read or generate one. None of the 0.15.1 install flags makes an install fail when package.json and yarn.lock disagree. For that, yarn check reports an error for any package.json pattern the lockfile doesn’t contain.

Checksums, and what they actually prove

Yarn’s README says it uses checksums to verify the integrity of every installed package before its code runs. In the 0.15.1 source, the tarball fetcher computes a SHA-1 of each tarball as it arrives and compares it with the hash from the lockfile. If they differ, it rejects the package with a “Bad hash” security error, and for a regular dependency the install stops before linking or running any scripts. Facebook’s post says checksums are stored in the lockfile so you get the same package on every install.

That’s a real improvement, but I’d be precise about it. The checksum gets recorded the first time a version is resolved, so it proves later installs received the same bytes as that first one, not that the first one was trustworthy. The check runs when a tarball is downloaded or read from an offline mirror, while a package already in your global cache is reused as it is. So pin and checksum your installs, and then treat lockfile changes like code. A pull request that changes the resolved URL or hash for a version that didn’t change deserves a question.

The cache and working offline

Because every download lands in the global cache, the README promises that a package you’ve installed before can be installed again without an internet connection. The CLI has yarn cache ls to print what’s cached and yarn cache clean to empty it, and when Yarn thinks you’re offline it suggests the --offline flag, which resolves packages from the cache without making network requests.

For machines that never had a warm cache, like Facebook’s sandboxed CI, the announcement says dependencies can also be kept in source control as tarballs for fully offline installs. The docs site doesn’t explain that yet, but the 0.15.1 source reads a yarn-offline-mirror setting from .npmrc, copies downloaded tarballs into that folder, and records the mirrored file in the lockfile along with its hash. Tarballs read back from the mirror go through the same checksum comparison.

How this compares with npm shrinkwrap

The npm shrinkwrap docs for npm 3.10.9, the current release, describe a different model. You run npm shrinkwrap yourself, and it writes npm-shrinkwrap.json from whatever is installed in node_modules right now. It fails if required dependencies are missing or extraneous packages are present, and it leaves out devDependencies unless you pass --dev. The example in the docs records a version, a from field, and a resolved URL for each package, with no checksum field. The caveats section even says that if you want to lock down the specific bytes in a package, you should check dependencies into source control or use something that verifies contents rather than versions. Yarn is built around that second option.

Yarn’s docs call yarn.lock similar to shrinkwrap but not lossy, and the migration guide says plainly that Yarn doesn’t support shrinkwrap files because they don’t hold enough information for its algorithm, so switching may give you a different set of dependencies. In 0.15.1, running an install next to an npm-shrinkwrap.json prints a message that the file won’t be updated or respected. If your project relies on a shrinkwrap, the guide suggests moving everyone to Yarn at once, deleting the shrinkwrap, and checking in the new yarn.lock.

Getting started

The day to day commands map closely onto npm’s.

npm install -g yarn
yarn init
yarn add react
yarn add <package> --dev
yarn install --pure-lockfile
yarn upgrade

yarn init interactively creates a package.json. yarn add installs a package and updates both package.json and yarn.lock, with --dev saving it to devDependencies. Plain yarn or yarn install installs everything in package.json, using the lockfile when there is one, and yarn install <name> stops with a message telling you to use yarn add instead. yarn upgrade updates every dependency to the latest version its range allows and recreates yarn.lock. In 0.15.1 it takes no package names, so it’s a whole tree refresh, not a single package bump.

My advice for an app: commit yarn.lock with package.json, use --pure-lockfile in CI so builds never write it, change dependencies through yarn add, yarn remove, and yarn upgrade rather than hand edits, and read the lockfile diff in review.

Facebook reports installs on some projects dropping from several minutes to seconds, but speed isn’t the part I’d get excited about. The real win is that “the dependencies we tested” becomes a file you can commit, diff, and verify, down to a checksum for every tarball.