HTTPoxy: Strip the Proxy Header Before It Becomes HTTP_PROXY

On Monday, CERT/CC published a vulnerability note for a bug with a catchy name, HTTPoxy, and the strange thing about it is that no single piece of software is really broken. Two old conventions collide. One says a web server running a CGI program turns each request header into an environment variable. The other says HTTP client libraries can read an environment variable called HTTP_PROXY to find out which proxy to send outbound requests through. Put those side by side and you can probably see where this goes: a request carrying a header named Proxy gets turned into exactly the variable those libraries trust.

Here’s the mechanism, what’s been assigned and fixed as of this week, and the mitigations the projects themselves recommend. The good news is that the main fix is one line of web server config.

What CERT/CC says

VU#797896 went public on July 18, 2016, crediting Dominic Scheirlinck and Scott Geary of Vend. Its overview says web servers running in a CGI or CGI-like context may assign the value of a client’s Proxy request header to an internal HTTP_PROXY environment variable, and that this can be used for man-in-the-middle attacks on the server’s internal subrequests or to make the server open connections to arbitrary hosts.

The note quotes the researchers’ two conditions, and you need both. First, something in the stack (the web server, the language runtime, the framework, or occasionally the app) sets HTTP_PROXY from the Proxy header. Second, the application then uses HTTP_PROXY unsafely, so an outbound request goes through a proxy the client chose. If your app never makes outbound HTTP calls, or its HTTP client ignores that variable, the header ends up as harmless data.

How a header turns into proxy settings

The first convention is written down in RFC 3875 section 4.1.18, the CGI 1.1 spec. Meta-variables that start with HTTP_ hold values from the client’s request headers. The server takes the header name, uppercases it, swaps each hyphen for an underscore, and puts HTTP_ on the front. So User-Agent becomes HTTP_USER_AGENT, and a header named Proxy becomes HTTP_PROXY. The client picks the header names, so the client effectively picks some of the names in your script’s environment too.

The second convention is just a habit many tools share. The nginx team’s post on the issue, Mitigating the HTTPoxy Vulnerability with NGINX, gives PHP’s Guzzle library as an example: it’s configured in part by HTTP_PROXY, and when that variable is set, the requests it makes get relayed to that proxy address. The same post says Go’s net/http package and Python’s Requests module trust HTTP_PROXY the same way. Under CGI, though, part of the environment comes from whoever sent the request.

The nginx post also points out that Proxy isn’t a standard HTTP header, which is why requests that carry it can be treated as suspicious, and that only requests with the bogus header are directly affected. That’s why removing it should be painless for real clients.

The CVEs and who’s affected

CERT/CC’s note lists six CVE IDs. CVE-2016-5385 is PHP, CVE-2016-5386 is Go, CVE-2016-5387 is Apache HTTP Server, CVE-2016-5388 is Apache Tomcat, CVE-2016-1000109 is HHVM, and CVE-2016-1000110 is Python. The NVD entries for the first four are up, and the one for CVE-2016-5385 is typical: PHP through 7.0.8 doesn’t address the RFC 3875 section 4.1.18 namespace conflict, so an application that calls getenv(‘HTTP_PROXY’), or PHP running in a CGI configuration, can have its outbound HTTP traffic redirected by a crafted Proxy header.

The Apache entries come with a twist. The Apache Software Foundation’s statement in the CERT note says it found no vulnerability as such in its software, which follows the CGI spec, and its advisory describes CVE-2016-5387 as the identifier assigned to a mitigation, with CVE-2016-5388 tracking a planned one for Tomcat. Whatever you call it, the practical outcome is the same, since the ASF says future httpd releases will stop passing the header through.

Beyond the CVEs, the vendor list in the note also marks HAProxy, lighttpd, nginx and Microsoft as affected, and the workaround section gives header filters for several of them. The nginx post, for its part, says the vulnerability doesn’t directly affect nginx but that nginx can stop attacks based on it. Microsoft’s statement is aimed at people running PHP or another third party framework on top of IIS. Ruby is listed with an unknown status.

Remove the header at the edge

This is the fix I’d do first, because it protects every script and language behind the server at once.

For nginx in front of PHP-FPM or another FastCGI app, the nginx post says to set the HTTP_PROXY FastCGI parameter to an empty string. When nginx proxies to an upstream over HTTP instead, set the Proxy header to an empty string. The proxy_set_header documentation says a header with an empty value isn’t passed to the proxied server at all.

fastcgi_param HTTP_PROXY "";
proxy_set_header Proxy "";

One gotcha from the nginx docs: fastcgi_param and proxy_set_header are only inherited from the enclosing level when the current level defines none of its own. If a location block already has fastcgi_param lines, which it does if it includes the stock fastcgi_params file, put the new line in that same block. If you drop it into a location that had none, it can quietly replace the parameters that used to be inherited. The nginx post also shows logging any request that includes a Proxy header to its own access log, which is a cheap way to see whether anyone is poking at you.

For Apache httpd, the ASF’s advisory for VU#797896 says the most direct fix is to drop any Proxy header coming from the client or an upstream proxy, and it gives two lines for httpd.conf. Skip the LoadModule line if mod_headers is already loaded or compiled in, and adjust the path if it isn’t.

LoadModule headers_module modules/mod_headers.so
RequestHeader unset Proxy early

The early keyword matters here. The mod_headers docs explain that early directives run right at the start of request processing, before the configuration for the request path is applied, and that also means they only work in the main server or a virtual host, not inside Directory or Location. The docs normally pitch early mode as a testing aid, but the advisory’s goal is to remove the header from every incoming request before further processing. The same advisory includes a source patch for server/util_script.c that stops httpd’s shared CGI environment code from turning Proxy into HTTP_PROXY. That covers mod_cgi, mod_cgid, mod_isapi, mod_proxy_fcgi and mod_fcgid, and the ASF says the change will ship in httpd 2.4.24 and 2.2.32. For Tomcat, whose CGI Servlet is off by default, the advisory offers a filter or valve that rejects requests carrying a Proxy header, or a servlet patch.

On IIS, Microsoft’s statement in the CERT note gives a URL Rewrite rule that sets the HTTP_PROXY server variable to an empty value. The note also includes filters for HAProxy and lighttpd if either of those sits at your edge.

Update the runtimes too

Header stripping is the fast fix, but the runtimes are moving as well. Go’s release history lists go1.6.3, released July 17, with security fixes to net/http/cgi and to net/http when it runs in a CGI environment. Reading the source at that tag, the CGI host code now skips the Proxy header, and the default proxy lookup refuses to use HTTP_PROXY when REQUEST_METHOD is set, which is a reasonable sign that the program is running under CGI.

PHP shipped 5.5.38, 5.6.24 and 7.0.9 yesterday, and the PHP 7 changelog lists the fix for bug #72573, HTTP_PROXY being improperly trusted by some PHP libraries and applications, under CVE-2016-5385. The same fix appears in the 5.5 and 5.6 changelogs. From my reading of the source, the change is to how getenv() looks up HTTP_PROXY, so I’d still strip the header at the server rather than count on the upgrade alone.

If you install httpd from your distribution, check its advisory. Ubuntu published USN-3038-1 for apache2 on July 18, and Red Hat’s RHSA-2016:1421 for httpd on RHEL 5 and 6 came out the same day, noting that after the update httpd no longer passes the Proxy request header value to scripts through HTTP_PROXY.

Don’t trust HTTP_PROXY in application code

The deeper lesson for app code is that under CGI, anything starting with HTTP_ in your environment might have come from the client. If your app needs an outbound proxy, set it explicitly in your own configuration and pass it to the HTTP client, instead of letting the library hunt through the environment. If you maintain a library that honors HTTP_PROXY, the Go fix is a nice pattern: refuse to use that variable when the process looks like it’s handling a CGI request. It isn’t even a new idea. The ASF advisory notes that Perl’s LWP has long avoided honoring HTTP_PROXY when serving CGI requests.

It’s also worth searching your code and dependencies for anything that reads HTTP_PROXY, and noting which apps run as CGI or behind FastCGI. That’s where both of CERT’s conditions can line up.

Why this feels familiar

If this reminds you of earlier trouble, that’s because request headers turning into environment variables has caused problems before. The CGI convention is more than twenty years old, and every so often something downstream trusts one of those variables more than it should.

My takeaway is to treat the client’s headers as data all the way down, even after they’ve been renamed into something that looks like configuration. Strip Proxy at the web server today, pick up the Go, PHP and distribution updates as you can, and make your outbound proxy settings something your app sets on purpose.