IMDSv2 and SSRF: Putting a Session in Front of EC2 Metadata

This summer’s Capital One incident put a spotlight on a question every team running on EC2 should be able to answer: if one of my servers can be tricked into fetching a URL, what can it reach? On Tuesday AWS shipped a second version of the EC2 Instance Metadata Service that puts a session token in front of the credentials your instances carry. I went through the announcement and the updated docs, and I think it’s worth turning on soon.

What Capital One and the government said

I want to be careful here, because plenty of confident explanations have gone around since July. Capital One announced the incident on July 29. It said that on July 19 it determined an outside individual had gained unauthorized access and obtained personal information about credit card customers and people who had applied for its credit cards, and that the access happened on March 22 and 23. The company said about 100 million people in the United States and about 6 million in Canada were affected. The largest category was application data from 2005 through early 2019, things like names, addresses, dates of birth, and self-reported income. It said no credit card account numbers or log-in credentials were compromised, while about 140,000 Social Security numbers, about 80,000 linked bank account numbers, and about 1 million Canadian Social Insurance Numbers were.

On the cause, Capital One described “a specific configuration vulnerability” in its infrastructure that it fixed, and it said this type of vulnerability is not specific to the cloud. The Department of Justice’s press release from the same day announced an arrest and said that, according to the criminal complaint, the intrusion occurred through a misconfigured web application firewall. Neither statement uses the words SSRF or metadata, and I’m not going to guess at anyone’s configuration. The general class of problem is worth understanding on its own, and it’s what AWS is now adding defenses for.

SSRF in plain terms

Server side request forgery is what happens when your application fetches a URL and an attacker gets to choose, or bend, which URL that is. Think of a feature that imports an image from a link, a webhook tester, a PDF renderer that pulls in remote assets, or a proxy that forwards requests. OWASP’s SSRF page describes it as an attacker supplying a different URL, or manipulating how one is built, so the server reads data from services that aren’t directly exposed to the internet. The request comes from your server, inside your network, so firewalls that keep outsiders away from internal services don’t help much.

Why the metadata endpoint is the prize

Every EC2 instance can reach the Instance Metadata Service at the link-local address 169.254.169.254. AWS’s announcement explains that the link-local address means only software running on the instance can access it. That sounds safe until you remember that an SSRF bug is, by definition, software running on the instance doing what an attacker asks.

The metadata service holds more than the instance’s IP address and AMI ID. If an IAM role is attached to the instance, the EC2 docs on retrieving security credentials from instance metadata explain that applications get that role’s credentials from the metadata path iam/security-credentials/ followed by the role name. Those credentials are temporary and rotated automatically, which is a big improvement over keys baked into an AMI, but while they’re valid they carry whatever permissions the role has. The same docs page carries a warning I suspect a lot of people skimmed past: if you run services that make HTTP calls on your behalf, like HTTP proxies, HTML and CSS validators, or XML processors that support XML inclusion, make sure they don’t expose your credentials.

With the original service, now called IMDSv1, getting those credentials is a single unauthenticated GET request. If an attacker can make your server fetch a URL and show them the response, they can walk the metadata tree, find the role name, and read the keys.

How IMDSv2 changes the conversation

AWS announced IMDSv2 on November 19 in a Security Blog post and a What’s New entry. The core idea is that every request now belongs to a session. Software on the instance starts a session with an HTTP PUT to a token endpoint, including a header that says how long the token should live, anywhere from one second to six hours. The service hands back a secret token, and every subsequent metadata request has to carry it in another header. Here’s the sequence from the EC2 instance metadata documentation, which requests a six hour token and then uses it:

TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"` \
&& curl -H "X-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/latest/meta-data/

curl -H "X-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/latest/meta-data/ami-id

The docs add a few details that matter for security. The token only works on the instance that created it. When tokens are required, a request without a valid token gets a 401. PUT requests that include an X-Forwarded-For header are rejected. And by default the response to the PUT has a hop limit, the IP time to live, of 1, so the packet carrying the token can’t make it off the instance.

AWS’s post walks through why each piece is there, and it maps onto four kinds of mistakes. A misconfigured open WAF usually passes requests through unchanged, but AWS says its analysis found the vast majority of those don’t permit PUT requests. Open reverse proxies also rarely allow PUT, and they normally add X-Forwarded-For, which IMDSv2 refuses. For SSRF, the post makes a point I found clarifying: requiring a static header only helps when the attacker controls just the URL, and AWS found many SSRF bugs where the attacker can set arbitrary headers too. Needing a PUT first, then a secret token, is what AWS calls strictly more effective than a static header, and it says this protects against the vast majority of SSRF vulnerabilities it looked at. Finally, if an instance has been misconfigured as an open router, NAT, or VPN, the TTL of 1 means the token response dies on the way out.

None of that makes an SSRF bug harmless, but it raises the bar a lot when an attacker can only influence a GET request’s URL.

Requiring IMDSv2

Both versions are enabled by default, and AWS says IMDSv1 will stay supported. AWS also recommends adopting v2 and restricting instances to it. The catch is that anything on the instance still speaking v1 breaks the moment you flip the switch, so the docs lay out a transition. First update the AWS SDKs and CLIs on your instances, since the latest versions support IMDSv2, and update any scripts that call the metadata service directly. Then watch the new CloudWatch metric MetadataNoToken, which counts calls that aren’t using tokens. Once it sits at zero, require tokens. At launch time, that’s the metadata-options parameter on run-instances or the RunInstances API. For instances that are already running, the modify-instance-metadata-options command changes the setting in place, and the EC2 user guide says you don’t need to restart:

aws ec2 modify-instance-metadata-options --instance-id i-1234567898abcdef0 --http-tokens required

The same command can raise the PUT response hop limit (the docs give container services on the instance as a reason you might need to) or turn the metadata endpoint off entirely with --http-endpoint disabled for instances that don’t use it. As of this week these options are available through the API and CLI only, not the console.

To keep things from drifting back, the docs describe an IAM condition key, ec2:MetadataHttpTokens, that lets a policy allow launching instances only when they require tokens. There’s also ec2:RoleDelivery, which lets a policy or an Organizations service control policy reject API calls made with role credentials that came from IMDSv1. That’s a nice backstop once everything has moved over.

The rest of the defense

IMDSv2 is one layer among several. Start with the instance role. Ask what an attacker could do with its credentials, and cut its permissions down to the specific actions and resources the application actually uses. A role scoped to one table or one prefix is a much smaller problem than a role with broad access across the account.

Then look at every place your code fetches a URL based on user input. My advice is to allow only the schemes and hosts the feature really needs rather than trying to block bad ones. Resolve the hostname and refuse private, loopback, and link-local addresses like 169.254.169.254, and make sure a redirect can’t send the request somewhere you wouldn’t have allowed in the first place. Don’t hand raw responses back to the user if the feature doesn’t need it. If a service only fetches from a known partner, an egress rule that says so beats any amount of string matching. AWS’s post also mentions local firewall rules restricting access to the metadata service as an existing mitigation that works alongside IMDSv2.

A web application firewall with rules that look for metadata addresses and internal hostnames in parameters is reasonable defense in depth too, as long as nobody mistakes it for the fix. And audit the WAF and proxy configuration itself, because a device meant to protect you that forwards arbitrary requests inward is its own version of this problem.

Turn it on

The metadata service solved a real problem by getting long lived keys off instances. What IMDSv2 fixes is how easy its credentials were to reach for anything that could make the instance send a GET. Update your SDKs, watch MetadataNoToken fall to zero, require tokens, and then go find the URL fetchers in your code. The session token helps a lot, but the bug that let someone aim your server at an internal address is still yours to fix.