After the S3 Outage: Designing for a Region That Goes Away

Last Tuesday, February 28, Amazon S3 in the US-EAST-1 region in Northern Virginia stopped serving requests for several hours, and other AWS services in the region that rely on S3 were affected along with it. On Thursday AWS published its post-event summary, and it’s short enough that you should read the whole thing. I’ll recap it briefly, because the more useful conversation is about our own systems and what happens to them the next time a region, or one service in a region, disappears for an afternoon.

What AWS says happened

The S3 team was debugging a billing system that was running slower than expected. At 9:37 AM Pacific, an authorized S3 team member following an established playbook ran a command meant to take a small number of servers out of a subsystem that billing uses. One of the inputs was entered incorrectly, and a much larger set of servers was removed. Those servers supported two other subsystems. The index subsystem tracks the metadata and location of every object in the region and is needed for GET, LIST, PUT and DELETE requests. The placement subsystem allocates storage for new objects during PUTs and depends on the index. Losing that much capacity forced a full restart of both, and S3 couldn’t serve requests while they came back.

AWS says it hadn’t fully restarted those subsystems in its larger regions for many years, and with S3’s growth the restart and its metadata integrity checks took longer than expected. The index could handle GET, LIST and DELETE again by 12:26 PM and was fully recovered at 1:18 PM. PUTs needed placement too, which finished recovering at 1:54 PM. The summary names the S3 console, new EC2 instance launches, EBS volumes that needed data from an S3 snapshot, and AWS Lambda among the affected services, and some of them had backlogs to work through after S3 recovered.

There’s one detail I keep coming back to. Until 11:37 AM, AWS couldn’t update individual services on its own Service Health Dashboard, because the dashboard’s administration console depended on S3. They posted updates through the @AWSCloud Twitter account and a banner on the dashboard instead.

What AWS says it’s changing

The tool that removed capacity now removes it more slowly, and it refuses to take any subsystem below its minimum required capacity. AWS is also auditing its other operational tools for similar safety checks. On recovery time, the summary describes how AWS breaks services into small partitions it calls cells to limit blast radius and make recovery testable, and says the further partitioning of the S3 index that was planned for later this year is being moved up to start immediately. The dashboard’s administration console now runs across multiple AWS regions.

The dependencies you didn’t draw

That dashboard story is the lesson for the rest of us. Most architecture diagrams show the obvious dependency, like the bucket your images live in. They rarely show the status page hosted in the same region, the deploy pipeline that pulls build artifacts from a bucket there, or the recovery plan that amounts to “launch more instances,” when new instance launches in the region were one of the things affected.

A useful exercise this week is to walk through an incident in which one region’s S3 is gone, and ask how you would tell your users, how you would ship a fix, and where that fix would run. If any answer points back at the broken region, that’s where to start.

Keep a copy somewhere else with cross-region replication

S3 has had a feature for this since the cross-region replication launch in March 2015: once it’s set up on a bucket, new objects are automatically copied to a bucket in a different region. The developer guide describes that copying as asynchronous, so the replica can lag behind the source. The requirements matter. Both buckets need versioning enabled. They have to be in different regions, and the developer guide says a source bucket can replicate to only one destination bucket. S3 also needs an IAM role it can assume, with permission to read object versions and their ACLs from the source and to replicate objects and delete markers into the destination.

Replication only covers objects created after you turn it on, so existing objects need a one-time copy of your own. According to the developer guide, objects encrypted with customer-provided keys or with KMS-managed keys aren’t replicated at all, while objects using S3-managed keys are. Deleting a specific object version isn’t replicated either, which the docs frame as protection against malicious deletes. Replicas keep the same key names, metadata, version IDs and ACLs. You pay for storage in the second bucket plus inter-region data transfer. Create the destination bucket well ahead of time, too, since the S3 docs warn that bucket creation works against a centralized, global resource space and doesn’t belong on your high-availability code path.

With the AWS CLI, a rule that replicates everything looks like this, saved as replication.json:

{
  "Role": "arn:aws:iam::123456789012:role/s3-crr-assets",
  "Rules": [
    {
      "ID": "assets-to-us-west-2",
      "Prefix": "",
      "Status": "Enabled",
      "Destination": {
        "Bucket": "arn:aws:s3:::example-assets-usw2",
        "StorageClass": "STANDARD"
      }
    }
  ]
}

Apply it to the source bucket with aws s3api put-bucket-replication --bucket example-assets-use1 --replication-configuration file://replication.json. An empty prefix means every object.

Serving from the second region

A copy of your data doesn’t help until traffic can reach it. Route 53 DNS failover is the usual tool: you create a primary and a secondary record with the same name and type, attach a health check to the primary, and Route 53 answers with the secondary when the primary is unhealthy. Health checkers in locations around the world send requests every 10 or 30 seconds, and the status flips after the number of consecutive failures you set as the failure threshold. An HTTP check has four seconds to connect and two more to get a 2xx or 3xx response. Resolvers cache answers, so the Route 53 docs recommend a TTL of 60 seconds or less on records tied to a health check.

Static sites hit a naming snag here. To point a Route 53 alias at an S3 website endpoint, the bucket name has to match the hostname. Bucket names are global, and the Route 53 docs on failover alias records spell out the consequence: only one record in a failover group can route to an S3 bucket. You can’t have a www.example.com bucket in both regions. S3 website endpoints also don’t support HTTPS. So at least one side has to be something else, like a CloudFront distribution with your hostname as an alternate domain name and a regional bucket as its origin, or your own servers, and if the site needs HTTPS the website endpoint is out on both sides. Route 53 doesn’t let you turn on Evaluate Target Health for an alias that targets a CloudFront distribution, so give that record a real health check. Point the check at something specific to that region rather than the shared hostname, since the docs warn that a check using the record’s own name gives unpredictable results. This is the fiddly part of the whole design, so test the full path and fail over on purpose before an outage does it for you.

Let CloudFront absorb some of it

Even with a single origin, CloudFront gives you some cushion. When an object in the edge cache has expired and the origin answers with a 5xx error, CloudFront keeps serving the expired copy for the error caching minimum TTL, then tries the origin again. That’s where the help ends. A 4xx goes straight to the viewer, an object that isn’t cached gets the error, and CloudFront can evict objects that aren’t requested often. Objects expire after 24 hours by default, so assets with versioned filenames can carry a much longer Cache-Control max-age and stay at the edge.

The summary doesn’t say which status codes S3 returned during the event, so I wouldn’t count on this alone. Treat it as one layer.

Degrade instead of falling over

The recovery timeline has a design hint in it: GETs and LISTs came back about an hour and a half before PUTs. An app that can live without uploads for a while could have been serving pages again around 12:30, while one that needs a PUT for every request stayed broken until almost two. Put short timeouts on S3 calls, keep reads and writes on separate paths, queue uploads for retry, and keep the replica bucket’s name in config so reads can switch regions without a deploy. For communication, do what AWS ended up doing and keep a channel, whether a status page or a social account, that doesn’t share the failure you’re reporting.

None of this is free, and a small site might reasonably decide an occasional afternoon of downtime is acceptable. Just make that decision on purpose, with every dependency on the diagram, rather than finding out which ones you forgot while the region is down.