Dyn (DynDNS) DDoS Attack

By Red Button

October 21, 2016

On Friday, October 21st, millions of users in North America and across the world experienced connectivity issues with many prominent sites, like Twitter, PayPal, Spotify, AWS and more. This was due to a very large attack that common DNS provider Dyn was experiencing, and it is already being considered one of the most prominent attacks. According to Dyn, the attack started 11:00 and 17.

Dyn, also known as DynDNS, is a very large DNS provider that caters to multiple prominent customers (PayPal, Netflix, SalesForce, Deutsche Telekom, TripAdvisor, LinkedIn and more). The first alert that Dyn provided on its status page was at 11:10 UTC. The Dyn DDoS attack impacted both its DNS service and its advanced service monitoring. The attack continued until at least 19:00 UTC, spanning a total of eight hours and comprising three major strikes.

Many websites that rely on Dyn, including Amazon, PayPal, Twitter and GitHub, were reported as being affected. See the table below for more details. An organization affected by this can see short- and long-term recommendations specified below.

According to some websites [1], it has been confirmed that the botnet behind the attack is the Mirai botnet, which became famous recently as it was also used to attack the KerbsonSecurity blog. The Mirai botnet is known for its usage of IoT devices (at least partially), which are the reason for its enormous power.

Customers that were impacted

Here is a list of customers that were impacted by the Dyn DDoS attack.

The following sites were reported to be affected by the site:


Customer	Notification	Source
AWS	“AWS reported small-scale connectivity issues to small number of end-points. The cause was “an availability event that occurred with one of our third party DNS service providers. We have now applied mitigations to all regions that prevent impact from third party DNS availability events.”	AWS Status Page
PayPal	Alert – Major Impact to PayPal APIs and PayPal Website.	PayPal Merchant Notification Page
Twitter	Issues reported on 21st for all dev services/API.	Twitter Dev API Status Page
GitHub	19:36 BST: We are investigating DNS resolution issues for GitHub.com. 21:35 BST We have migrated to an unaffected DNS provider. Some users may experience problems with cached results as the change propagates.	GitHub Staus Page


Customer	Notification	Source
AWS	“AWS reported small-scale connectivity issues to small number of end-points. The cause was “an availability event that occurred with one of our third party DNS service providers. We have now applied mitigations to all regions that prevent impact from third party DNS availability events.”	AWS Status Page
PayPal	Alert – Major Impact to PayPal APIs and PayPal Website.	PayPal Merchant Notification Page
Twitter	Issues reported on 21st for all dev services/API.	Twitter Dev API Status Page
GitHub	19:36 BST: We are investigating DNS resolution issues for GitHub.com. 21:35 BST We have migrated to an unaffected DNS provider. Some users may experience problems with cached results as the change propagates.	GitHub Staus Page

Attack Analysis

The main source of information about the Dyn DDoS attack is Flashpoint, which has investigated the attack for Dyn [1]. Here are the highlights.

“While the attacks were still ongoing, Flashpoint was able to confirm that at least one portion of the attack was initiated by a Mirai Command and Control server.”
“Mirai botnets were previously used in the DDoS attacks earlier this month against the ‘Krebs on Security’ blog and the French internet service and hosting provider OVH.”
“Despite public speculation, Flashpoint assesses with a moderate degree of confidence that the perpetrators behind this attack are most likely not politically motivated, and most likely not nation-state actors.”

Red Button recommends taking this data with some caution, and especially to consider that whatever is published must protect a site’s customer reputation.

Open Questions

What were the Dyn DDoS attack vectors and volume?
Was Dyn targeted directly or was it one of its customers?
What does Dyn use for mitigation? is it using its sheer size or also a DDoS mitigation technology? If so, which?
Why did the Dyn DDoS mitigation fail? Was it more because of the attack size or because of a lack of effective mitigation? In other words, did Dyn “screw up” or was it really a groundbreaking attack?

Recommendations

Red Button’s recommendation is to not rely on a single DNS provider. De Facto, our customers that used a second provider in addition to Dyn have weathered the attack without impact.

Following is the recommended setup, provided by order of DDoS resiliency. We understand that for some organizations these recommendations are easier said than done, so we have also included possible caveats and workarounds under the ‘Considerations‘ section below.

Two active DNS servers
Use two DNS servers that both publish in “active-active” mode.
Use two DNS servers, one as primary and another as backup
The secondary server will be used only if the primary fails.
Game plan
If you cannot set a secondary DNS server, develop and define on your own what your actions would be if your DNS server failed and you hadn’t yet acquired a secondary one. For example, you can use your own DNS server or quickly acquire a different DNS service.

Considerations (of using two DNS servers)

Two Active DNS Servers

Organizations with very critical and sensitive services who periodically need to investigate a DNS issues with their providers may find that using two may complicate things too much. The investigation process may be two complicated when you in addition have to investigate which provider is in charge of the issue.

Active-Passive DNS servers

The main problem with this approach is that changing your primary DNS service to a secondary is an action that can take time with your registrar.

For more information, recommendations of DNS vendors and emergency support, contact us here.

History of DNS Attacks & Technical Details

DNS DDoS attacks are extremely common and have become the “weapon of choice” of hackers for several reasons:

DNS service is a point of failure for Internet services. When you take down a DNS server, you take down all the services which are dependent upon it.
DNS is UDP based. It allows spoofing, has modest resources to generate attacks due to connection less protocol, and allows for an attack amplification technique – 1Mbps of attack traffic can end up becoming 100Mbps reflected on the victim.
DNS DDoS mitigation technologies are not as mature and proofed as HTTP DDoS mitigation; in other words, it is harder to stop DNS attacks.

In the past, organizations have maintained their own authoritative DNS servers, but over the years many have chosen to migrate to external DNS service providers like Dyn. DDoS was one of the reasons for this migration – they were simply unable to mitigate these attacks and “outsourced” this problem to someone else.

As a result of this transition in the last few years, DNS providers are handling extremely large and complicated attacks. In some cases, they accept a new customer that is under an ongoing attack, although some providers have also rejected such customers.

There is no question that DNS providers are handling attacks much better than the end customer is. However, this also comes with a risk – if they are unable to do so, their entire customer base goes down.