DNS Made Easy's response to the DDOS attack

Dear DNS Made Easy Client,

On August 07, 2010 DNS Made Easy was the target of a large multi Gb/s attack against all of our name servers. The attack started at 8:00 UTC and was fully mitigated by 14:00 UTC. During this time period there were regional outages from some or all of our name servers. Regional outages means that certain regions of the world were not able to resolve your DNS and other regions of the world were resolving normally. When all name servers were not reachable a DNS query would have been lost, when some name servers were not reachable then DNS performance would have been slower than normal but still operational.

The regional downtime was in very small periods but it still did affect the overall resolution for all of our client's DNS. It is for this reason that we are explaining the situation in full to all of our clients now.

1) How long were the DNS outages? In some regions there were no issues, in other regions outages lasted a few minutes, while in other regions there were sporadic (up and down) outages for a couple of hours. In Europe for instance there was never any downtime. In Asia downtime continued longer than other regions. In United States the west coast was hit much harder and experienced issues longer than the central and east coast.

2) Many clients have asked us if in fact there was downtime since they did not notice issues. Many clients did not notice any DNS downtime. In fact many clients would not have noticed this issue if we had not sent this email. But we feel disclosure of this issue is something that we owe our client base. If you want to see if there is a significant loss of DNS queries you can quickly compare your daily queries from this Saturday to last Saturday in the DNS Made Easy control panel. Overall query statistics comparing this Saturday's query load (minus attack traffic) to recent Saturdays' query loads shows that our servers properly responded to a query total this Saturday within a 2% difference from recent Saturdays.

3) Where did the attack come from? We believe that the DDoS came from a botnet attack originating from Asia. Most attack traffic originated in or transited through China. The source IPs appear to be mostly spoofed but the vast majority are assigned by APNIC to Chinese Networks and Chinese ISPs. Traffic levels reported to us by our bandwidth providers regarding their connections through which this traffic entered their networks also points to origins in Asia.

4) How large of an attack was this? This attack hit levels that were so high that our Tier1 upstreams were suffering latency and network issues for other clients at many of their locations due to this attack. This caused some of our Tier1 bandwidth providers to use their last resort response of null routing traffic to some of our IPs from some networks to prevent major service degradation to their core networks. Measuring the exact size of this attack is rather difficult. However, discussions with our Tier1 bandwidth providers during the attack led to an estimate of 50 Gb/s in size. This was based on reports of multiple 10Gb/s lines being saturated at multiple different providers in different geographic regions. During our after-action discussions internally and with our providers after the attack was mitigated we analyzed all information available to us through monitoring systems and traffic reports and we revised our estimate of the attack size to be fluctuating between 20Gb/s and 40Gb/s during the attack. We will never know the true size of this attack as we actively moved traffic around to different locations throughout the attack and IPs were temporarily null routed into and through various networks, and some traffic was blocked from provider to provider in response to the attack. We do know that due to the service implication to the Tier1 providers, networking teams from China Netcom, China Telecom, Level3, GlobalCrossing, Tiscali, and Arbinet were involved to stop the attacks. Level3 and Arbinet both played special heroic roles in facilitating that the co...