Shellshock: What the Bash Bug Is and What to Patch

Last Wednesday, September 24, the bash maintainers put out a patch for a bug that turned out to be reachable from a lot more places than anyone would like. It got the CVE ID CVE-2014-6271 and the nickname Shellshock, and within a week it had picked up a handful of sibling CVEs and several rounds of package updates. If you run Linux servers, a Mac, or anything with a web server that still has a cgi-bin folder, this is worth half an hour of your time. Here’s what the bug actually is, why it’s remotely reachable, how to check a machine, and what to update.

What bash was doing with environment variables

Bash lets you export a shell function to child bash processes, the same way you export a variable. There’s no special channel for that, so bash stuffs the function into an ordinary environment variable. The CERT/CC vulnerability note describes the format: a variable named after the function, with a value that starts with () {. When a new bash process starts, it walks through its environment, and any value that begins with those characters gets treated as a function definition to import.

The problem was how it imported them. Bash glued the variable name onto the value and passed the whole string to its normal command parser and executor. A function definition is just one command, but nothing stopped the string from containing more commands after the closing brace, and bash ran those too. The NVD entry for CVE-2014-6271 puts it as GNU Bash through 4.3 processing trailing strings after function definitions in the values of environment variables. Stephane Chazelas is credited with reporting it.

Notice that the check was on the value, not the name. Any variable at all could carry a “function”, and that is the part that matters for remote attacks. CERT also says the bug is being actively exploited.

Why a web request can reach it

Environment variables feel local, but plenty of software fills them with data that came from somewhere else and then starts a shell. CGI is the classic case. RFC 3875 section 4.1.18 says a CGI server puts request header values into meta-variables whose names begin with HTTP_, so a User-Agent header shows up in the script’s environment as HTTP_USER_AGENT, holding whatever text the client sent. If the CGI script is a bash script, or it runs anything through a shell that happens to be bash, that bash process reads the header value on startup.

The US-CERT alert for Shellshock (TA14-268A) calls out Apache mod_cgi and mod_cgid scripts written in bash, scripts that spawn bash subshells, and any system where /bin/sh is bash. That last one is the sneaky one, because a Perl or Python CGI script that calls out to a shell command can still end up running bash. Red Hat’s original write up of the issue notes that mod_php, mod_perl and mod_python don’t pass data to scripts through environment variables, and they believe those aren’t affected. I wouldn’t lean on that too hard if your PHP app shells out, since the shell it starts inherits whatever environment the app has.

It’s not only web servers. OpenSSH’s ForceCommand option runs the forced command through the user’s login shell with -c and puts the command the client originally asked for in SSH_ORIGINAL_COMMAND. People use this to lock an account down to something like rsync or git, and Red Hat points out that with a vulnerable bash as the login shell, that restriction can be sidestepped to run any command. The DHCP client is the other big one. Red Hat notes that dhclient uses environment variables and runs bash to configure the network interface, so connecting to a malicious DHCP server could let it run code on the client. Red Hat also says CUPS filters are believed to be affected. The common thread, in NVD’s words, is any place where setting the environment crosses a privilege boundary before bash runs.

Checking a machine

The standard local test, which appears in both the CERT note and Red Hat’s article, is harmless. All it does is print some text:

env x='() { :;}; echo vulnerable' bash -c "echo this is a test"

That sets a variable called x whose value looks like an empty function followed by echo vulnerable, then starts a new bash that just prints a test message. If the output includes the word vulnerable, the child bash executed the trailing command while importing the environment, and that bash needs updating. A bash with the first fix prints warnings about ignoring a function definition attempt instead. The exact warnings differ between the upstream patches and each distro’s build, so the thing to look for is simply whether vulnerable shows up.

Run it on every bash you have, not only /bin/bash. A second copy built from source in /usr/local/bin or pulled in by a package manager won’t be fixed by your distro’s update. Also don’t judge a distro’s bash by its version number. Red Hat, for example, backports fixes into packages like bash-4.1.2-15.el6_5.2 on RHEL 6, so compare the installed package against your vendor’s advisory rather than against upstream patch levels.

Why the first patch wasn’t the end of it

Upstream’s first fix, bash43-025, along with matching patches for the 3.0 through 4.2 release lines, changed the import so that bash only accepts a function definition and stops after a single command. That blocks the trailing command trick, but it still hands attacker controlled text to the bash parser, and in my opinion that was always going to be the weak spot, because a shell parser is a lot of old, complicated C to expose to text from strangers.

Tavis Ormandy showed almost immediately that you could still get the parser into a bad state, and NVD published CVE-2014-7169 the same day, noting that it exists because of an incomplete fix for CVE-2014-6271 and that it lets attackers write to files or possibly worse. Upstream shipped bash43-026 on September 26 to fix it, and Red Hat shipped RHSA-2014:1306 the same day. If you patched on Wednesday, you needed to patch again.

Then Florian Weimer and Todd Sabin found memory handling bugs in the parser, CVE-2014-7186 (the redir_stack issue with here documents) and CVE-2014-7187 (an off by one error with deeply nested loops). Ubuntu’s USN-2364-1 fixed both on September 27. By September 30 two more IDs were in play, CVE-2014-6277 and CVE-2014-6278. Red Hat says the details of those two aren’t public yet, and that its current bash packages already mitigate them.

The change I think matters most came in bash43-027 on September 27, credited to Florian Weimer. Instead of looking at every variable’s value, bash now only imports functions from variables whose names start with BASH_FUNC_ and end with %%. An HTTP_USER_AGENT can contain whatever it likes and bash never feeds it to the parser, so parser bugs like the ones above can’t be reached through arbitrary variables anymore. Ubuntu’s September 27 update describes the same hardening as adding prefixes and suffixes around environment variable names that contain shell functions, and Apple’s update adds its own namespace for exported functions to stop HTTP headers from passing through to bash as functions.

What to update and what to restart

On Red Hat Enterprise Linux it’s yum update bash, and on Debian and Ubuntu it’s a normal apt-get update followed by upgrading the bash package. Then run the check again. That’s not paranoia: Ubuntu had to issue USN-2363-2 on September 26 because a build issue meant the CVE-2014-7169 fix didn’t get applied to the 14.04 LTS package the first time. Given how fast things have been moving, I’d keep an eye on your vendor’s advisories for the next week or so as well.

On a Mac, Apple released OS X bash Update 1.0 on September 29 for Lion 10.7.5, Mountain Lion 10.8.5 and Mavericks 10.9.5, covering CVE-2014-6271 and CVE-2014-7169 along with that function namespace change.

Restarts are less dramatic than you might fear. Bash reads its environment when a process starts, so the next bash launched after the update is the fixed one. The catch is the new function naming. On September 30 Red Hat updated RHSA-2014:1306 to say that installing the packages without restarting services does address the vulnerability, but some functionality may break until affected services are restarted. Certain services, screen sessions and tmux sessions may need a restart, and affected interactive users may need to log in again, because the update changes how the names of exported functions are handled. If nothing on your system passes exported bash functions around, you’re unlikely to notice. If something does, restart it.

Beyond the package update, this is a good moment to go looking for CGI scripts you forgot about, and for any service that starts a shell with data it got from the network. Anything you don’t need in cgi-bin can go.

The takeaway

Shellshock isn’t really a story about one bad line of code. It’s about an old convenience feature that trusted the contents of every environment variable, sitting underneath years of software that treats the environment as a handy place to put untrusted input. The quick fix blocked the trick that was public, and the durable fix was to stop handing arbitrary variables to the parser at all. Update bash, check it again, and treat the environment your programs hand to a shell as input from whoever controls it.