HN Explorer

Ask HN: How would you deal with DDoS?

129 points by samratjp ↗ HN
In my recent memory - Posterous dealt with it once, the 4Chan and Tumblr fiasco and in the last few days, the Visa, MasterCard and PayPal mess.

What's a good way to deal with DDoS? Bonus points if you can elaborate more than "buy a stinkin' Cisco firewall"

EDIT: Do tell us your war stories as well.

67 comments

[ 2.8 ms ] story [ 81.1 ms ] thread
brianwillis
DDoS attacks aren't aren't technical problems - they're political ones. Political problems need political solutions. Technology isn't the cause of a DDoS attack, just the medium.
jacquesm
DDoS attacks can be used for political leverage, but more likely they're just used for what they are best at: extortion.

And they're definitely technology, they're a tool used to attack the owner/normal beneficiary of the tool with that tool (your uplink).

petercooper
That seems like telling someone to eat more healthily when he's presenting with a heart attack. And like heart attacks, not all DDoS attacks have logical causes that can be mitigated against.
jeffiel
War is a political problem too, doesn't mean you shouldn't have an army ready.
csomar
A competitor can attack you to take your service down.
wlievens
Has this ever happened?
dfox
I assume that this is pretty common situation in some (even non-criminal) circles, I'm thinking of various small web hosts, content-free portal pages, direct internet resellers of something etc. Few times some of these people actually went to us seeking help with something almost clearly unethical and possibly illegal. This is in Czech republic, but from various stories on internet I extrapolate that this is probably global phenomena, maybe even more pronounced in other countries.
dedward
Yes it has. Unfortunately I can't really say much about the parties involved or what happened (I wasn't the bad guy though, promise), but it absolutely has, and if I've seen it, then either I'm a unique little snowflake (yay!), or it's happened a LOT more.
neworbit
Yeah, it used to be a fairly commonplace story from the guys running online gambling shops.
dedward
They're both. Even with willing ISPs and countries on all sides, the attacks continue. Machines are far to insecure, which makes it far too easy for an individual or small group to harvest the resources of millions of machines and then hide, while giving them orders.

It's when they try to collect money that law enforcement has the best opportunity to catch them. There is a monetary limit over which it becomes internationally interesting to the FBI,HTCU, etc, to get involved and start using international resources to track the money back and make arrests.

troels
ddos is a very broad category of attacks, so there isn't one single solution. What really matters, is that you have some sysadmin type on board who knows his network protocols.
lsc
why was this modded down? There are many kinds of DoS attacks, with many mitigation strategies. If it's a pipe-filling attack, there's not much you can do besides get a bigger pipe, or have your upstream do filtering for you (which is quite often more expensive than just getting the bigger pipe... see, /someone/ needs the capacity to soak up the attack, you or your upstream, and that capacity costs money.)

I personally got hit with a flood of many small packets, which overwhelmed my router. The obvious solution here is to buy a better router. (it was 200Mbps of rather small packets on a 1000Mbps pipe.) I ended up just having my upstream blackhole all traffic to the target IP, which is sad, as it 'finishes the job' for the attacker, but on a $12/month account, what do you want?

Now, for what I provide (VPS hosting) an application-level DoS, slowlaris or just hitting your app in an expensive way isn't really my problem. That's on the application programmer, and as this is mostly a form for application programmers, it does make sense that the highest modded comments on this page describe mostly application level DoS mitigation strategies.

vegasbrianc
With security it is always cat and mouse and the target is always moving due to software updates, new OS's, and applications. I agree with Troels that you need a shard security sysadmin with a plan in place before an attack ever happens. If a company is scrambling at the last second trying to put security measures in place or hiring security staff to assist. It is already to late. Preparation , testing, and regular updates is key to security
jacquesm
The best way is to have an attack mitigation strategy worked out in advance. Once the traffic is hitting your servers you are very much too late, the best spot to filter it out is as far upstream as you can get. Getting to know your upstream providers and having contact information is probably the single most effective thing you can do in terms of ROI when it comes to dealing with large scale DDoS.

Chances are they already have that big Cisco or Force10 router complete with FPGA based filtering magic.

dazzawazza
This is great advice. At a poker startup I worked at the most important thing was our relationship with the pipe owner.

It can be expensive to get them to act but for a poker site it's a lot cheaper than loosing hands per minute.

We also paid a lot of money for boxes at our end that did real time traffic analysis for various attack vectors. Expensive but again cheaper than loosing money.

wlievens
Was it necessary at some point? I'm not being sarcastic, just interested.
jacquesm
Typically anybody that makes more than a few million per month on the internet and is not under the protection of the authorities for stuff like this has by now found out about the dark side of the net.
chopsueyar
What about streaming video sites? Has ww seen any distributed attacks on 1935?

I am running Wowza currently, and worry about this.

jacquesm
We have not seen any attacks on the video infrastructure but we're small fry. Let me ask a much larger site using wowza and get back to you on that one.
chopsueyar
Thanks! Look forward to it.
jacquesm
Ok, here is the answer:

"If there has been we haven’t noticed. Taking down one wowza server wouldn’t cause us any problems. They would have to take down 5+ servers for us to notice."

dazzawazza
At first it was all completely unnecessary no one had ever heard of the product.

However as it started to climb the poker ranks and get more players the early start we had made meant that the network guys were ahead of the hackers/criminals from the start. We had developed a good relationship with our providers (of bandwidth and hardware) and internally stress tested the system.

By the time I left it was WELL worth it. A reasonable sized poker site can easily rake a million dollars a week which is a revenue stream worth defending.

My advice would be that if you think your are going to be vulnerable invest early, make mistakes early, when the big boys come at best you have hours to respond. Investing doesn't have to mean buying the most expensive hardware but planning for how much you can afford to loose will put things into perspective. Deciding how much to spend of defence is important and must be continually reviewed as hardware/software change and new threats pose themselves.

It's also important to remember that the big attacks like DDos are annoying but can normally be defended against. Intrusion detection is as important but much harder to defend against.

[deleted]
(comment deleted)
wyclif
losing
dazzawazza
thank you, unfortunately I can't edit the post now. You know what I meant though.
catlike
One of the most effective answers is not "buy a stinkin' Cisco firewall" but rather "buy a stinkin' Arbor"

http://www.arbornetworks.com/

If you want to survive large scale DDOS you need equipment that can scrub the incoming unvalidated data in real-time and keep up.

Combining Source Based Remotely Triggered Black Holing (RTBH) with uRPF affords you the ability in a sophisticated network to drop a large amount of undesired traffic (especially if it's from simpler DDOS strategies). If you do in fact have the $$/need for the Arbor then inside of black-holing the traffic you send it to the Arbor and let it scrub the packets.

tptacek
I was the lead developer on Arbor Peakflow DoS (not the traffic scrubber they acquired from Ellacoya; the NetFlow analysis engine) and then a product manager at Arbor. This is fine advice if you're an Internet service provider (except that virtually every ISP has already taken that advice). But if you're buying connectivity, the Arbor box isn't going to do you any good; your links are going to be saturated before the scrubber can do anything about it.
tzs
One of the calls I saw for the current round of DDoS attacks listed Amazon as one of the targets. It seems to be having no effect. Can I infer from this that Amazon knows how to handle DDoS attacks?

If so, would one reasonable answer to the question then be "host with Amazon" and let them deal with it for you? Or at least have a backup ready at Amazon to switch to? (With the recent announcement that Amazon's hosting is now PCI level 1 compliant, hosting at Amazon becomes possible for many for whom it was not an option).

meatmanek
This works great until a Senator pressures Amazon to drop your site.
mfukar
The same way all CDNs handle DDoS attacks.
lsc
>One of the calls I saw for the current round of DDoS attacks listed Amazon as one of the targets. It seems to be having no effect. Can I infer from this that Amazon knows how to handle DDoS attacks?

keep in mind that you are paying for that DDoS traffic. $0.08 per gigabyte means you are paying something north of $350/hr for a 10Gps DDoS.

Now, I guess if you only expect the traffic to continue for a short while that's not a bad deal at all, but if it goes on for any period of time, that can get expensive fast.

For comparison, one of my providers has offered me a 10 gigabit transit link for around $10.5K/month, so assuming I could use that to DDoS you, (and really, that sort of thing would be... difficult. Far easier to hire out a botnet.) I'd be spending about a dollar to cost you twenty dollars.

dedward
Amazon operates on a scale where they can handle this type of thing I don't mean automaticlaly, but they have the access points and gear in place to do it, and the levearage with the right ISPs.

Buying a 10 gig link woulnd't be a distributed denial of service attack, and itwould be far too easy for the providers to not only find out who it was and filter it out, but to find out who to sue for all the wasted time and bandwidth and other criminal charges.

chopsueyar
First, you are assuming both parties are in the same country, and second, you think the provider would notice instantly, or charge ~$6 per minute until filtered.
borism
and lsc is assuming in his calculations that he'd be able to attack for a month.
lsc
>Amazon operates on a scale where they can handle this type of thing

My point is that by "handle this sort of thing" in the context of ec2, means "charge you for the traffic and pass it through" which some times is just fine; but quite often it still means a denial of service, if it doesn't bring down your servers, it will eventually bring down your bank account. I mean, I don't know about you, but at north of $350 an hour, my bank account isn't going to last long.

vladd
0) Be prepared

Be proactive. Plan for the solutions below in advance, not when traffic is hitting your servers and SSH along with everything else is unresponsive.

1) Availability for investigation

Renice sshd to make sure you can get to your servers when they are hit with CPU usage due to the DDOS. Reserve some connections for the root user on your SQL server (Postgresql already defaults to 3 for this) so you can still connect, investigate and analyse what's happening.

2) Pattern matching

Every solution against DDOS can be described more or less as "pattern matching". There are two steps involved:

-> identify a pattern associated with DDOS requests.

-> match them as early as possible (iptables, upstream provider, hardware firewall) and reject them.

Easiest is to match offending IPs, those that make repeated queries (i.e. once every second). Set the threshold in the right spot - too high and you'll ban valid users, to low and you fail to ban enough in order to restore service availability.

Other things to watch for are at the connection level: either SYN flood behaviour ( http://en.wikipedia.org/wiki/SYN_flood , deploy http://en.wikipedia.org/wiki/SYN_cookies ) or KeepAlive connections that keep httpd busy (always reserve in httpd.conf for example a number of connections that aren't KeepAlive enabled).

Identify pages on your website that are slow to load or generate large, time-consuming SQL-queries. Those are the main target during DDOS attacks; pattern-match their URLs and disable them during attacks if they're not critical for your website, or have a fall-back for them.

Check for errors in the attackers' requests: either they have the same weird-looking user-string, or lack the referrer, or something else that you might be able to find. Pattern-match and reject based on that.

Pattern-match clients that keep requesting the .html without loading the corresponding .css or images -- most probably they are attackers calling the GET API (corresponding to wget-ing the HTML) but don't parse and download the corresponding images on the page. Sure, you'll ban text-browser users but that's a small price to pay in order to restore your website.

Have a low-resource version of the website where you can turn off the requests for CSS, images and so on and just serve plain functional HTML. Pattern-match, if any, those IPs that keep requesting CSS and images previously associated with non-cache-able HTML pages.

And lastly... be proactive, be prepared, have a plan.

robotkad
Most clients participating in DDoS attacks are plain dumb.

They will continually perform a GET request on a single page and not parse the response. This means they wont respond to javascript, images, cookies or redirects correctly.

You can devise a test that identifies attacking clients and then blacklist the IP addresses for a while.

While I'm sure this strategy isn't perfect, it has been sufficient for the two attacks I have been subjected to.

Edit: vladd's reply articulates this idea better than I was able to :)

xentronium
Effectively banning robots is the last thing you want to do.
bdonlan
It's better than having the site be completely inaccessible. You'd only deploy this when you come under load. Additionally, many of the more important robots can be identified reliably (eg, googlebot has a DNS handshake that can positively identify legitimate googlebots) - and even if you get a false positive, if you filter out all their packets, they're likely to assume a temporary failure and come back later.
pbhjpbhj
Could you counter equally naively, does a single IP need to hit your site more than a couple of times in 10mins, say?

Still even an simple test could be too slow to counter a big enough attack.

Travis
The only problem with this (note: I speak from no experience, just reading other comments) is that it's at the application level. For the serious DDoS stuff (Anon, extortion rings), you have to filter at the TCP level (from what I know). That gets more complicated when the attacks originate from a botnet (especially if that botnet has infected IP ranges similar to what your customers use -- can't very well go off and ban Time Warner, if you're in the US)
carl_
You only mitigate attacks you don't stop them.

You either get massive uplinks (multi 10GE) and cisco nexus (or similar) or proxy/spread/diversify across multiple facilities separate to the core and only forward legit traffic (or null route targetted proxies).

robotkad
I also forgot to mention in my earlier reply the one thing you should DEFIANTLY not do: pay an attacker to stop.

I often hear of attacks that are accompanied by an email demanding thousands of dollars via Western Union.

Attackers like this pray on small, non-technical businesses (eg blogs, small retailers) who often feel they have no option but to pay the ransom.

Guess what happens when you pay for the attack to stop? The attack stops, only to start again in 8 weeks time with a new, higher ransom email.

dedward
Never negotiate with terrorists, right?

THe thing is, this all depends on scale. If you are small scale, look at you cost per day. They won't attack you forever - their resources are limited.

So if you're in that situation - okay, it sucks. First, calculalte the cost of doing business. If they're asking for 10 grand and you're thinking of paying because you'll lose that much in a few days, you're better off getting hooked up with a real DDOS protection provider - it will cost a lot less than that in the long run.

If you make tons of money daily (some affiliate sites, etc) then it may (may) be worth doing both - if one day's business will cost you a quarter million in profits, and you havent' already set yourself up for DDOS, it might be worth letting hte FBI know, paying them off anyway, and then getting protection services in place so it never happens again (cause now you know)

It's the small fish who just can't afford DDOS protection services and might have sites who make just enough money to live on off their site that are in the tough spot, and could be a prime target for the extortionist-type DDOS'ers. I wonder how common that is.

phillian
As a web publisher and affiliate marketer, I've had domains DDoSed at least 4 times in FY10. The worst of it being twice in two days on two separate domains both targeting the same campaign by the same botnet based in Turkey.

Each one cost me money, but the two in two days cost me thousands. I was hosting a VPS on Rackspace and did everything I could to mitigate it but ultimately the attack stopped because they wanted those resources directed elsewhere.

After the first one, I hooked up with a smart guy on Twitter (http://twitter.com/sempersecurus) that tracks botnets. He allowed me to sinkhole the domain so that he could collect and aggregate the data.

To stay defensive, I put some .htaccess blocks in for countries that are clearly well outside normal buyer patterns. In each case, I know a user (read: a human) saw the campaign and then put the DDoS in place so I hoped, by the .htaccess blocks, to prevent them from seeing the site altogether. But I know I really need to be a much more knowledgable coder to be proficient at deflecting these in the future.

dedward
AThere isa point where coding won't help you at all, and sheer bandwidth wins out. These guys can saturate 10gigs without worrying about it.

If your business is really worth at least $1000/day then, a real-life protection service that proxies your traffic at a high bandwidth location and sends you back only clean traffic would seem easily within your budget, and worth it ot protect your assett. Let them worry about the constantly changing attack patterns and let you worry about increasing the value of your site.

chopsueyar
Can you provide links to the domains in question?
Keyframe
Send a submarine to cut off submarine communications cables.
pbhjpbhj
I'd have thought you could counter-attack? Assuming that the bots have poor security and so have been compromised - I guess that the botnet could be hardened before use to mitigate against this.

Do any ISPs profile their clients to test for bots?

[dead] c00p3r
man tcpdump man iptables

^_^

tptacek
I have what Patrick would call "a wee bit" of experience with this problem; I led a dev team on a successful anti-DDoS product from 2001-2003. For now, I have two pieces of advice:

Advice #1:

Every major ISP has a tier of network engineer that is equipped to handle the DDoS problem. They're the ones with access to the traffic analysis tools, they're the ones with the scripts to deploy ACLs, they're the ones that can reroute traffic to a regional scrubbing center. If you're dealing with a real ISP, they have an anti-DDoS product (probably one I've very familiar with) deployed.

There aren't many of them and they are never the person answering the phone when you call the ISP. Nobody at your ISP has any incentive to escalate you to that person. Your ISP may deny that the person exists (I've seen that happen at ISPs where I know the right person by name).

Find that person. Be persistent. If I knew who mine was right now I'd send brownies and belgian ale. Weekly. Advance planning and cheap insurance can't hurt.

Advice #2:

Nobody knows what "I'm under DDoS attack" means. It doesn't mean anything. You have to be able to describe the attack precisely in technical terms. That doesn't mean "it's a SYN flood!"; it means, "My link is saturated, I'm getting N million packets per second, an unusually high number of connections sourced from TCP ports 15030 - 19012, it started 9 minutes ago, and doesn't coincide with a spike in requests to my DNS server."

In all likelihood, nothing you have deployed today is going to generate that information for you, so your job today is to get that infrastructure set up. I recommend getting NetFlow turned on and using Argus, which is free; NetFlow also happens to be a language ISP network engineers speak readily. You have other alternatives, like ntop. Just have something that can characterize traffic and ideally tell you (either directly, or via graphs) when things are out of the ordinary.

You will have much better luck getting help from your upstreams if you can write the ACL for them and make it easier to find what places in the network need it.

bradleyland
I can vouch for nTop. Really cool project. I did a bandwidth monitoring deployment for a customer who runs their own layer-3 network atop a carrier ethernet solution, but didn't have any plan in place for monitoring what goes over the wire. We found all kinds of fun stuff once we installed a few nTop nodes. Among the things we found: a Halo server to which employees from several different states were connecting and playing on a daily basis.
lhnn
Please tell me you didn't shut that down. Work/life balance! Stress relief!
swalberg
I've dealt with a couple over two different employers, both had a similar pattern. The same page was being requested over and over, but there was no discernible patterns in the headers.

On the first one, I already had a Squid reverse proxy in front of the servers, so I configured that page to cache and all was good.

On the second one my load balancer was a POS, so we ended up getting the developers to configure the page to 302 to a new page, and the bot didn't follow the redirect.

In both cases, the thing requesting the page wasn't a real web browser, so it didn't take cookies or follow redirects. I ended up getting an F5 shortly after and wrote an iRule to do a cookie check should another DDOS come by, but never had to use it. I was also fortunate that we had enough bandwidth to serve the requests and that it was a resource starvation type attack.

snorkel
On your outward-facing servers

* turn down the TCP handshake timeouts * turn on TCP SYN cookies * watch incoming network connections and traffic logs and add firewall rules to block each suspected attacker subnet.

In a real DDoS you have to block the attack traffic at the TCP level, not at the application level (the web server) which means adding firewall rules or iptables.

You will end up blocking a lot of legitimate traffic but better than being 100% down.

Still getting hammered? On to phase 2:

* Launch mirror servers, change your DNS records to point traffic at the mirror. (Most attack bots don't follow DNS changes) * If it's an application-level attack then put your site in static delivery mode.

Still getting hammered? That'll teach you to hate on Justin Bieber.

tabman
It cost a lot of money to get the necessary equipment to defend against an attack, but DOSarrest can handle the job rather well. They defend against any kind of DDOS attack and they do not care about the size of the attack. I had a forum which came under attack one day and I did all kinds of things to try and defend against the attack. I wrote my own script to grab IPs, use automated attack detection and block tools, bought expensive cisco hardware, and still, I had a LOT of problems. I finally went and looked for experts in DDOS protection which is when I came across DOSarrest. They where nice and helped get everything set up and optimized for my site. I gave them my IP address and a payment, within 5 minutes I was given an IP to point my DNS to and then my site was back up as soon as my DNS migrated. I turned my TTL down in anticipation for this, so it was only about 10 minutes from the time I gave them my IP to the time I was back online. Then to be safe I changed my IP to a secondary IP, had them update their system which took less then 5 minutes, and then blocked all traffic but theirs coming to my system.

These 4Chan guys attacked MPAA which went to DOSarrest as well and within minutes of being set up there they where back online. They protect a bunch of high end clients who have come under attack and bring on new people all the time.

They also have a nice customer panel where I can log in and change all kinds of settings for my site and view the statistics of the attack. I saw the attack get up to over 500mb/s and have no effect what so ever. I highly recommend them.

karl9231
I too had to go to a specialist I was attacked last year and my provider Time Warner could do nothing in our case. We went to Dosarrest.com and they fixed everything in about an hour.....if your stuck its the fastest way to get back online.
monological
hmm what if you 301 redirected (if http) the ddos flood back at 4chan, would that end up disabled their site?
monological
but that would mean that you only redirect packets based on filtering out the botnet IPs, which would be pointless because if you could distinguish between botnet IPs and valid users, then you'd have the ddos problem solved