After the Dyn Attack: Keeping Your Cameras, DVRs and Routers Out of a Botnet

Last Friday morning a lot of people on the US East Coast couldn’t reach sites they use every day. The attack wasn’t aimed at those sites. It was aimed at Dyn, the company that answers DNS questions for many of them, and Dyn now says a significant volume of the attack traffic came from Mirai-based botnets. According to US-CERT, the devices in recent Mirai incidents were mostly home routers, network cameras and DVRs, which is to say the kind of thing a lot of us have plugged in at home.

For the incident itself I’m sticking to what Dyn has said in its own statements and what US-CERT published about Mirai the week before. The rest of the post covers the part you can actually do something about: making sure the devices on your own network aren’t helping.

What Dyn says happened

Dyn’s chief strategy officer, Kyle York, posted a statement on the Dyn blog on Saturday. He said the attack on Dyn’s Managed DNS infrastructure began at approximately 7:00 am Eastern on Friday, October 21, and was mitigated with service restored about two hours later. During that window, people directed to Dyn’s East Coast servers couldn’t reach some of Dyn’s customers’ sites. A second, more global wave arrived just before noon, and service was restored at about 1:00 pm Eastern. A third attempt was mitigated without customer impact, and York wrote that Dyn never had a system-wide outage.

On Wednesday Dyn published a more detailed analysis from Scott Hilton, its EVP of Product. It puts the two main attacks at roughly 11:10 to 13:20 UTC and 15:50 to 17:00 UTC, and describes high-volume floods of TCP and UDP packets sent to destination port 53 from a large number of source IP addresses. It confirms the Mirai botnet as the primary source of the malicious traffic, and says Dyn won’t speculate about who did it or why because it’s collaborating in a criminal investigation.

One number changed between the two posts. Saturday’s statement mentioned tens of millions of IP addresses associated with Mirai. Wednesday’s analysis explains that the attack set off a storm of legitimate retries as recursive DNS servers tried to refresh their caches, 10 to 20 times normal traffic volume, which made the set of attacking endpoints look far bigger than it was. The estimate in that report is up to 100,000 malicious endpoints.

Why one DNS provider matters to so many sites

Your browser can’t connect to a name. It needs an IP address first. RFC 1034 lays out the pieces: resolvers fetch answers on behalf of programs, and name servers hold authoritative data for zones. Your computer usually asks a recursive resolver, often one run by your internet provider, which either answers from its cache or asks the authoritative name servers for the domain. Every record carries a TTL, which RFC 1034 describes as how long the record can be cached before it should be discarded. RFC 1035 says name servers are reached on port 53 over both UDP and TCP, which matches the traffic Dyn described.

Lots of unrelated companies pay the same provider to run their authoritative name servers. When those servers can’t answer, a resolver whose cached answer has expired has nowhere to get a fresh one. The sites’ own servers can be running perfectly well, and nobody can look up where they are. Meanwhile resolvers keep retrying, which is the storm Dyn described piling onto the attack.

The specs saw part of this coming. RFC 1034 requires every zone to be available on at least two servers, and RFC 2182 says secondary servers should be topologically and geographically dispersed, so that a failure in one segment of the network, such as a service provider, doesn’t make all of them unreachable. My read is that two name servers at one provider satisfy the first rule while missing the spirit of the second. For the rest of us, the useful question is what we have plugged in at home.

What US-CERT warned about the week before

On October 14, US-CERT published alert TA16-288A, Heightened DDoS Threat Posed by Mirai and Other Botnets. It says the IoT devices in the latest Mirai incidents were primarily home routers, network-enabled cameras and digital video recorders, and that Mirai finds them by scanning the internet and trying a short list of 62 common default usernames and passwords. Because so many devices are unsecured or weakly secured, that short list gets it into hundreds of thousands of them. The alert names a second family, Bashlite, that also gets in through default credentials.

Nothing there requires a clever exploit. That’s the uncomfortable part, and also the good news, because the fixes are mostly settings you control.

Start with an inventory

The alert says consumers should be aware of the capabilities of the devices in their homes, and that starts with a list. Many home routers have a page showing attached devices or DHCP clients. Match every entry to something physical, then walk around for what you forgot: the DVR under the TV, the camera by the back door, the printer, the smart TV itself. For each one, note whether it has a web login, whether its app works from outside the house, and who made it.

Change every default password

The alert tells you to change all default passwords to strong ones, noting that defaults for most devices are easy to find online. Router makers aren’t immune. In February the FTC announced a settlement with ASUS, and its complaint alleged the company set, and let consumers keep, the same admin and admin login on every router. If a device won’t let you change its password, I’d treat it as untrustworthy, make sure it isn’t reachable from the internet at all, and think about replacing it.

If you suspect something is already infected, the alert gives a specific order. Disconnect the device from the network, reboot it while it’s disconnected (the alert says Mirai lives in dynamic memory, so a reboot clears it), change the password from the default to a strong one, and only then reconnect. Reconnect before changing the password, the alert warns, and the device could be reinfected quickly.

Update firmware from the manufacturer

Next, the alert says to apply security patches as soon as they’re available. For most gadgets that means firmware, and I’d check the manufacturer’s support page for your exact model rather than trusting what the device says. The FTC complaint alleged that ASUS’s update tool often told people they were current when newer firmware with critical security fixes existed. Under the settlement ASUS has to notify consumers about updates, including through an option to register for direct security notices, so if a manufacturer offers that kind of registration, sign up. The alert also suggests buying from companies with a reputation for providing secure devices.

Close the doors you didn’t mean to open

The alert says to disable Universal Plug and Play on routers unless it’s absolutely necessary. One thing UPnP does on a home router is let devices inside your network create port mappings by themselves; RFC 6970 describes those mappings being constructed by control points on the local side. That’s handy for a camera’s remote viewing feature, and it’s also the problem, since the device decides on its own to be reachable from the internet. Turn UPnP off, then check the port forwarding page and delete any rule you don’t recognize.

While you’re there, find the setting called remote management or remote administration and turn it off unless you really use it. According to the FTC, hackers could exploit bugs in the ASUS routers’ web control panel to change security settings without the owner knowing. Admin pages have bugs like anything else, and the fewer places yours answers to, the better.

The alert also says to monitor TCP ports 23 and 2323 for Telnet attempts to take control of IoT devices. It doesn’t mention SSH, but I’d treat any remote login service the same way: if a device has a Telnet or SSH setting you don’t use, turn it off, and never forward those ports to anything on your network.

Give the gadgets their own network

The FBI’s 2015 public service announcement on IoT risks, which the US-CERT alert cites, recommends that people isolate IoT devices on their own protected networks. Many consumer routers have a guest network that gets internet access without seeing your other devices, which makes a decent IoT network if that isolation is actually enabled, so check the setting rather than assuming. If your equipment supports VLANs, a dedicated VLAN with firewall rules is the tidier version. Printers and streaming boxes you control from your phone can get awkward on a separate network, so expect some trial and error.

Be clear about what that buys you. A separate network protects your laptops and phones from a compromised camera. It doesn’t stop the camera from being recruited into a botnet. The passwords, updates and closed ports do that.

The takeaway

Dyn’s analysis says the attack highlighted vulnerabilities in the security of IoT devices that need to be addressed, and a lot of that work is ours. For most homes it’s an evening: make the list, change the passwords, update from the manufacturer, turn off UPnP and remote management, and move the gadgets off your main network. The next botnet will be built from whatever is still easy to get into, and there’s no reason for the camera by your back door to be part of it.