Anycast isn't difficult to implement and is widely used to create CDNs. It is simply costly. You need to buy a large block of IPs and have multiple data centers.
After you have that, it is trivial to block all DNS packets, especially due to the fact that Cloudflare doesn't need to receive DNS packets on their public endpoints because they only serve HTTP traffic.
You are being rude, but I care about learning more than rudeness, so if you'll kindly help me out:
But DNS servers can also be queried for the IP addresses of huge swaths of the Internet, putting the person listed as making the request on the receiving end of a massive response. In a blog post published Wednesday, CloudFlare CEO Matthew Prince said each DNS request sent by the Spamhaus attackers was likely only 36 bytes long, while each response was about 3,000 bytes. By spoofing the requests to make them appear as if they originated with Spamhaus, the attackers can turn the firepower of all those networks against their opponent, all but guaranteeing it won't be available to process legitimate traffic.
I'm still not really getting it. Which IP addresses are they referring to when they say "huge swaths of the internet"?
To get Spamhaus back online, CloudFlare relied on Anycast, a routing technique that distributes the same IP address across 23 data centers across the world. Internet traffic almost always chooses the shortest physical path. Anycast allows the geographically dispersed junk traffic to be absorbed by dozens of individual centers, where each packet is then inspected. When it bears signatures found in the attack traffic—for example, if it's a 3,000-byte response from an open DNS resolver—it is discarded in the CloudFlare data center. Only Legitimate Web requests are allowed to be forwarded to the Spamhaus data center.
How do they differentiate between a legit query and a DDoS query?
The "huge swaths of the internet" part is poorly phrased in the article. What they're basically saying is that there are a large number of DNS servers on the Internet that allow anyone to query them. Attackers query those servers and forge the source address that the query originates from such that it points at the IP address of their target. The attacker needs only send 36 bytes of data to the DNS server and the server responds with up to thousands of bytes of response -- directly at the target of the attack.
I don't know precisely how CloudFlare mitigated it but I can make some guesses:
- They may have just blocked UDP traffic on their edge. Since their DDoS mitigation service is specifically for HTTP and HTTPS, UDP is safe to simply drop.
- They may have determined that the attack responses had payloads that fell within a size range, and configured their mitigation hardware or routers to drop packets in that size envelope.
- They may have analyzed inbound UDP traffic to see what open resolvers were flooding them and surgically blocked UDP traffic from those IPs.
I'm sorry but it is clear to me that you don't understand a major portion of the article you are telling people here to read as though you can understand the technical details. This is both disturbing and amusing at the same time.
If I am coming off as rude and people are going to downvote me for pointing this out, then so be it.
What facts? You mean the fact that he is telling HN that it is a technical marvel to block such a large attack and then he doesn't even understand how the attack was done as was explained in the article he told people to read?
This is a clear example of how the signal to noise ratio on Hacker News is going way down.
I'm still not really getting it. Which IP addresses are they referring to when they say "huge swaths of the internet"?
Most IP addresses... most notably including the victim's IP address. Scenario: Attacker sends 36 byte DNS query with a falsified sender address (that of his victim) to, say, Google's DNS servers. If the origin isn't verified through something like IPsec, Google won't be able to tell that the sender was forged. Google then sends 3000 bytes to the victim, thinking they're simply replying to a request, whereas they're really (unknowingly) facilitating a DNS amplification attack (so called because the attacker turns e.g. 2Gb/s of botnet traffic to DNS servers into 200Gb/s of DNS responses to the victim.)
Your random residential ISP may limit their DNS service to customers only -- unless the victim also happens to be a customer of said ISP (the very rare case), the victim's IP address will be refused service, and not sent a response. While this blocks "legitimate" requests (without forged senders) as well unless you're actually on that ISPs network, in practice this is rarely a problem -- DHCP is probably advertising the DNS servers which will respond to you on whatever network you're on. Aside from preventing other ISPs from having their customers freeload off your ISP's DNS server, this also prevents the attack (at least with regards to DNS servers.)
How do they differentiate between a legit query and a DDoS query?
CloudFlare, being on the sending side of the equation, can track what DNS requests are going outbound even without IPsec, and drop inbound responses which can't be matched up against one of those requests. That sounds expensive however. I imagine they do something much cheaper, like simply drop all inbound DNS traffic except the approved list of DNS servers that are probably advertised over DHCP anyways. Again, this can block technically "legitimate" requests where someone really did want to intentionally use some other DNS server, but again, rarely a problem.
I wager it's pretty rare for the following conditions to exist simultaneously:
- Small enough that $2k a month is a serious spend.
- Big enough to draw the attention of someone who cares to DDoS you.
- Important enough to one's customers that a few hours of downtime
is a serious issue.
If you somehow cause all three to exist at once, either you're going out of your way to piss off the wrong people, or you're not charging enough.
(Also, there's always the option to roll your own DDoS protection. It's complicated, but not super complicated, at least for a 90% solution. If I'm not mistaken, it basically involves detecting anomalous traffic and telling upstream routers that you don't exist for the source IP generating that traffic. Someone who knows more about it might be able to fill in some details.)
$2k is just the beginning. If you looked at the forum I just posted, attacks can go up to 60 gbps and still only cost a couple of dollars.
This can easily cost as much as a full time employee, and many large sites can be run by 1-2 people. Having to spend 33% more just to stay online is not negligible.
You can't just roll your own DDoS protection. If you do this you are looking at rolling your own data centers. Once an attack gets bigger than the port at one of your standard hosts, your host is going to null route you and even kick you off if it happens too often.
You are going to be hard pressed to find a host that is willing to broadcast your /24, let alone getting a /24 with this IP shortage if you were going to do your own cloudflare.
"If I'm not mistaken, it basically involves detecting anomalous traffic and telling upstream routers that you don't exist for the source IP generating that traffic."
This is called null routing. It involves telling your upstream that the IP can't be routed to. This blocks ALL traffic to that IP so your port doesn't get maxed. Large transit providers are going to charge you an arm and a leg to give you an API to insert ACLs because routers have a limited number of rules.
> Large transit providers are going to charge you an arm and a leg to give you an API to insert ACLs because routers have a limited number of rules.
It's been my experience that pretty much every transit provider supports this (and at no extra cost). All of my transit providers do.
I offer RTBH'ing (by tagging /32s with a specific community) to my customers because I can propagate those to my upstreams in order stop them from sending the traffic to me. Those providers would rather drop 10 Gbps of DDoS traffic at the edge than to worry about an extra entry in routing table.
A lot of deceptively large websites are barely holding on due to the collapse of the ad industry in 2008... never mind them being able to employ people. An attack is the sort of milestone that would encourage them to give up.
> If I'm not mistaken, it basically involves detecting anomalous traffic and telling upstream routers that you don't exist for the source IP generating that traffic.
You're thinking of real-time blackholing (RTBH). "Source-based" blackholing is not an option during most DDoS attacks (spoofed IP addresses) so destination-based blackholing (what everyone refers to when they're talking about RTBH'ing) is your only option.
By employing RTBH'ing, you effectively DDoS yourself. Say, for example, that you have a 5 Gbps pipe and an attacker is sending 10 Gbps of traffic to your web server with IP address 203.0.113.42. That level of traffic will saturate your network connection completely, making it completely unusable. Your only choice is to blackhole your web servers' IP address upstream by having your ISP(s) drop traffic destined to 203.0.113.42. At that point, your web server is unreachable by anyone and you've had to DDoS yourself, in effect, in order to be able to use your network connection again.
And if they spread their DDoS across all of your IP address space? You'll be able to use your network connection again when the attackers decide to stop. That's hardly rolling your own "DDoS protection".
Here's what I don't understand about DDoS mitigation.
Say, they targeted x.x.x.x IP and it get RTBH'd, so the DDoS traffic is effectively stopped closer to its origins. Then, in order to bring my service back up, I move to another IP (or I adjust the load-balancing mechanism not to use the original IP). The part I don't understand is why wouldn't DDoS controller simply redirect its attack to another IP? This is surely not that hard to do, and so we are back to where we started and this cycle can in theory continue forever.
Also, it follows from the above that the only way to dismantle DDoS attacks is to do traffic fingerprinting, detect malicious requests and then, ideally, have respective ISPs kick originating IPs off the Internet. Does anyone know which providers/services do this and how much it costs?
Attackers will switch to attacking an IP that isn't null routed.
You could theoretically keep changing your DNS entries. But many ISPs cache entries for a day instead of whatever your TTL is, so you get customers locked out.
Source verification is called BCP38, and as for the cost, this standard was published 13 years ago, and the majority of networks haven't implemented it.
Even a really, really good SysAdmin can't work around what's currently possible with network technology. You really can't serve beyond your network saturation point no matter how much "tweaking" you do.
That's gonna cost infrastructure you may not have (and/or cost far more than you can afford).
You'd be lucky to get 1/50th of the advertised Gbps. Hackforums is where script kiddies go. It's just a bunch of unemployed kids trying to show off with software/code that they didn't make themselves, but they will sure as hell try to convince you they did.
"Spanish police said that upon his arrest, the suspect identified himself as a diplomat, saying he was the Minister of Telecommunications and Foreign Affairs for the Republic of Cyberbunker."
I think from monday, at work, I will identify myself as a diplomat, the "Minister of Telecommunications and Foreign Affairs of the Republic of the Local Domain". Sounds nice.
It depends on the type of DOS attack being used by the attackers. If they are using a SYN attack you can change the source ip as you don't care to hear back. But a SYN attack can be mitigated a bit easier once you know about it.
If you are doing a more application specific attack, you need to establish a connection (assuming it is TCP), so you can't hide you IP in that case.
35 comments
[ 2.4 ms ] story [ 85.6 ms ] threadhttp://arstechnica.com/security/2013/03/spamhaus-ddos-grows-...
http://arstechnica.com/security/2013/03/how-whitehats-stoppe...
After you have that, it is trivial to block all DNS packets, especially due to the fact that Cloudflare doesn't need to receive DNS packets on their public endpoints because they only serve HTTP traffic.
But DNS servers can also be queried for the IP addresses of huge swaths of the Internet, putting the person listed as making the request on the receiving end of a massive response. In a blog post published Wednesday, CloudFlare CEO Matthew Prince said each DNS request sent by the Spamhaus attackers was likely only 36 bytes long, while each response was about 3,000 bytes. By spoofing the requests to make them appear as if they originated with Spamhaus, the attackers can turn the firepower of all those networks against their opponent, all but guaranteeing it won't be available to process legitimate traffic.
I'm still not really getting it. Which IP addresses are they referring to when they say "huge swaths of the internet"?
To get Spamhaus back online, CloudFlare relied on Anycast, a routing technique that distributes the same IP address across 23 data centers across the world. Internet traffic almost always chooses the shortest physical path. Anycast allows the geographically dispersed junk traffic to be absorbed by dozens of individual centers, where each packet is then inspected. When it bears signatures found in the attack traffic—for example, if it's a 3,000-byte response from an open DNS resolver—it is discarded in the CloudFlare data center. Only Legitimate Web requests are allowed to be forwarded to the Spamhaus data center.
How do they differentiate between a legit query and a DDoS query?
I don't know precisely how CloudFlare mitigated it but I can make some guesses:
- They may have just blocked UDP traffic on their edge. Since their DDoS mitigation service is specifically for HTTP and HTTPS, UDP is safe to simply drop. - They may have determined that the attack responses had payloads that fell within a size range, and configured their mitigation hardware or routers to drop packets in that size envelope. - They may have analyzed inbound UDP traffic to see what open resolvers were flooding them and surgically blocked UDP traffic from those IPs.
If I am coming off as rude and people are going to downvote me for pointing this out, then so be it.
This is a clear example of how the signal to noise ratio on Hacker News is going way down.
Most IP addresses... most notably including the victim's IP address. Scenario: Attacker sends 36 byte DNS query with a falsified sender address (that of his victim) to, say, Google's DNS servers. If the origin isn't verified through something like IPsec, Google won't be able to tell that the sender was forged. Google then sends 3000 bytes to the victim, thinking they're simply replying to a request, whereas they're really (unknowingly) facilitating a DNS amplification attack (so called because the attacker turns e.g. 2Gb/s of botnet traffic to DNS servers into 200Gb/s of DNS responses to the victim.)
Your random residential ISP may limit their DNS service to customers only -- unless the victim also happens to be a customer of said ISP (the very rare case), the victim's IP address will be refused service, and not sent a response. While this blocks "legitimate" requests (without forged senders) as well unless you're actually on that ISPs network, in practice this is rarely a problem -- DHCP is probably advertising the DNS servers which will respond to you on whatever network you're on. Aside from preventing other ISPs from having their customers freeload off your ISP's DNS server, this also prevents the attack (at least with regards to DNS servers.)
How do they differentiate between a legit query and a DDoS query?
CloudFlare, being on the sending side of the equation, can track what DNS requests are going outbound even without IPsec, and drop inbound responses which can't be matched up against one of those requests. That sounds expensive however. I imagine they do something much cheaper, like simply drop all inbound DNS traffic except the approved list of DNS servers that are probably advertised over DHCP anyways. Again, this can block technically "legitimate" requests where someone really did want to intentionally use some other DNS server, but again, rarely a problem.
Thanks for the thorough and very clear explanation.
Previous submissions about this around that time:
https://news.ycombinator.com/item?id=5614179
https://news.ycombinator.com/item?id=5613891
Any competitor can buy a 30+ gbps attack for a few dollars.
http://hackforums.net/forumdisplay.php?fid=232
And it costs $2000/month just for 20 gbps of protection.
https://www.staminus.net/ddos-protection https://ordering.blacklotus.net/cart.php?pid=10
DDoS attacks are going to get larger and more frequent.
(Also, there's always the option to roll your own DDoS protection. It's complicated, but not super complicated, at least for a 90% solution. If I'm not mistaken, it basically involves detecting anomalous traffic and telling upstream routers that you don't exist for the source IP generating that traffic. Someone who knows more about it might be able to fill in some details.)
This can easily cost as much as a full time employee, and many large sites can be run by 1-2 people. Having to spend 33% more just to stay online is not negligible.
You can't just roll your own DDoS protection. If you do this you are looking at rolling your own data centers. Once an attack gets bigger than the port at one of your standard hosts, your host is going to null route you and even kick you off if it happens too often.
You are going to be hard pressed to find a host that is willing to broadcast your /24, let alone getting a /24 with this IP shortage if you were going to do your own cloudflare.
"If I'm not mistaken, it basically involves detecting anomalous traffic and telling upstream routers that you don't exist for the source IP generating that traffic."
This is called null routing. It involves telling your upstream that the IP can't be routed to. This blocks ALL traffic to that IP so your port doesn't get maxed. Large transit providers are going to charge you an arm and a leg to give you an API to insert ACLs because routers have a limited number of rules.
It's been my experience that pretty much every transit provider supports this (and at no extra cost). All of my transit providers do.
I offer RTBH'ing (by tagging /32s with a specific community) to my customers because I can propagate those to my upstreams in order stop them from sending the traffic to me. Those providers would rather drop 10 Gbps of DDoS traffic at the edge than to worry about an extra entry in routing table.
You're thinking of real-time blackholing (RTBH). "Source-based" blackholing is not an option during most DDoS attacks (spoofed IP addresses) so destination-based blackholing (what everyone refers to when they're talking about RTBH'ing) is your only option.
By employing RTBH'ing, you effectively DDoS yourself. Say, for example, that you have a 5 Gbps pipe and an attacker is sending 10 Gbps of traffic to your web server with IP address 203.0.113.42. That level of traffic will saturate your network connection completely, making it completely unusable. Your only choice is to blackhole your web servers' IP address upstream by having your ISP(s) drop traffic destined to 203.0.113.42. At that point, your web server is unreachable by anyone and you've had to DDoS yourself, in effect, in order to be able to use your network connection again.
And if they spread their DDoS across all of your IP address space? You'll be able to use your network connection again when the attackers decide to stop. That's hardly rolling your own "DDoS protection".
Say, they targeted x.x.x.x IP and it get RTBH'd, so the DDoS traffic is effectively stopped closer to its origins. Then, in order to bring my service back up, I move to another IP (or I adjust the load-balancing mechanism not to use the original IP). The part I don't understand is why wouldn't DDoS controller simply redirect its attack to another IP? This is surely not that hard to do, and so we are back to where we started and this cycle can in theory continue forever.
Also, it follows from the above that the only way to dismantle DDoS attacks is to do traffic fingerprinting, detect malicious requests and then, ideally, have respective ISPs kick originating IPs off the Internet. Does anyone know which providers/services do this and how much it costs?
You could theoretically keep changing your DNS entries. But many ISPs cache entries for a day instead of whatever your TTL is, so you get customers locked out.
Source verification is called BCP38, and as for the cost, this standard was published 13 years ago, and the majority of networks haven't implemented it.
That's gonna cost infrastructure you may not have (and/or cost far more than you can afford).
"Spanish police said that upon his arrest, the suspect identified himself as a diplomat, saying he was the Minister of Telecommunications and Foreign Affairs for the Republic of Cyberbunker."
I think from monday, at work, I will identify myself as a diplomat, the "Minister of Telecommunications and Foreign Affairs of the Republic of the Local Domain". Sounds nice.
If you are doing a more application specific attack, you need to establish a connection (assuming it is TCP), so you can't hide you IP in that case.