Ask HN: A botnet may be attacking our site right now, how should we respond?
It seems like our ecommerce site is under attack from a botnet, causing our site to go down. We're receiving 50-100 requests per second (far surpasses normal traffic). Some requests are for outdated URLs not even normally accessible from the site.
Here's IP data related to requests: https://gist.github.com/panabee/9595411. Can post more if more is needed for diagnosis.
Questions:
1) How do we confirm if the site is under attack?
2) If the site is under attack, how can we ward off the attack and prevent future ones?
3) We seem to have a ton of sleeping MySQL connections that continue to crop up even after we try killing them. Is this symptomatic of a botnet attack?
We appreciate any help or guidance anyone can offer.
Thanks!
43 comments
[ 3.0 ms ] story [ 95.6 ms ] threadYou might just be getting crawled by an overly aggressive search engine, by the way. (Probably not one of the big guys.) From your perspective that is isomorphic to being hit with a botnet, but the nice thing about that is the search engine won't attempt to get around the mitigation steps you put in place. (Well, typically.)
We also suspect it might be an aggressive search engine. Our next step is to pull some IP data. If it turns out to be a search engine, what can we do beyond editing robots.txt?
If it's not a search engine, how can we confirm a botnet attack ourselves. We don't want to wait for these network guys to wake up and drink their morning coffee.
Thanks again!
I'm not a sysadmin, and I have nothing but respect for you and your team with the following recommendation: if it isn't obviously how to implement "drop that C block at the firewall" then find yourself a competent contract sysadmin, pay him for one hour of work today, then pay him for 10 hours of work over the next few weeks beefing up your infrastructure. It will be cheap at the price.
If you're absolutely determined to do it yourself the magic word is "iptables" and may God have mercy on your soul, because iptables is the most obtuse piece of software I've ever had the misfortune of trying to configure.
In the alternative, if you're actually facing a botnet, the expected behavior is lots of requests from widely distributed IPs. You'd be able to verify this with your access logs (which record IPs and timestamps) and a 10 or 20 line shell script to put a histogram of unique requests per IP. If it's a few requests from thousands of IPs per hour, then you have a bit of a challenge to address it on your end of things. My first totally-not-a-sysadmin advice would be to dial down dynamic content within the app for non-logged-in users and start aggressively caching pages, which might or might not be enough for you to weather the storm.
Best of luck and skill.
If you're looking for advice on how to mitigate this yourself:
- apachetop is a useful tool for this sort of thing, and since the activity is occurring on a single site, it should be easy to use. (Assuming of course you use apache.)
- If it's an actual botnet, you'll need to pick some honeypot URLs -- links that regular customers aren't hitting but the botnet is -- and then use something like Fail2Ban to pipe accesses to those URLs into iptables bans.
- If it's a crawler, it should be from a limited set of IPs and a quick iptables rule addition will fix you up. For reference, here are two easy-to-use rules:
The first rule will add a "drop" command to your INPUT table for any traffic coming from <source_ip_address>. The second rule will allow you to quickly drop a range of IP addresses; <source_ip_range> should look something like "10.0.0.1-10.0.1.255", and "iprange" is a literal.That's not a very good ISP. You should consider switching away from them as soon as you can.
Step 1: Get a new provider.
Anyway, best way to figure out what's going on, is to generate a distinct list of IPs originating the bad traffic from your logs, and block those addresses at your firewall. If this doesn't stop the strange requests in 15 minutes or so, you may want to get some expert help.
Your hosting provider may be able to be of some assistance here, but that greatly depends on their level of clue: many budget providers will just drop you at the first sign of denial-of-service related trouble, so do keep that in mind.
Signing up for a specialized service like Cloudflare is probably your best bet, and should be pretty affordable as well (their web site lists their business plan with "advanced denial of service attack mitigation" as $200/month, and they suggest that if you call them they can have you protected pretty much immediately).
Unfortunately the advanced DDoS support is only in their upper $$ plans. Might be worth it to keep your site revenue going though.
It was created by kickstarter to fend off these kind of attacks.
To fight off the attack immediately I would recommend switching your DNS over to CloudFlare's Pro or Business plan (that depends a lot though on your site's current configuration, the size of your budget, etc.) with certain settings, then configuring Sucuri CloudProxy with CloudFlare, and lastly implementing some additional security for your server on Single Hop.
In terms of preventing future ones, you basically want to be proactive as possible in terms of preventing them, but that won't stop everything. The configuration I recommended above should be pretty good for most sites, but you might want to consider other DDOS mitigation companies as well and you could need something completely different depending on your business etc. Then you want to have a plan and several other things in place in case it was to get past the infrastructure you implemented to prevent it, so you can minimize the downtime you have etc.
It is hard to tell whether that is a possible issue just based on the information you gave in terms of the sleeping MySQL connections.
Lastly, depending on what the issue exactly is, it probably would be a good idea to make your site as static as possible, but that is difficult to do since your site is down right now. Once you get the site back up though and stable, you might consider doing this for a little bit. Lastly, having a separate site that you can deploy in the event of an attack is a good idea as well.
If you want more help, shoot me an email at caleb.lane4@gmail.com. I hope that helped and you get this solved quickly.
It could be some kind of attack, but it's so weak it barely qualifies. 50-100 requests is so low it's probably a search engine spider or a log-spamming bot. If request rates like these bring your site down, you have a serious software problem. You really should be able to handle at least a few hundred requests per second, especially if they're bogus requests going to non-existent URLs.
> 2) If the site is under attack, how can we ward off the attack and prevent future ones?
If this was a real attack, you'd be dealing with thousands of requests per second and a real spike in data volume.
As a basic protection against things like these, get behind a reverse proxy right now. Cloudflare offers a good free plan, but it doesn't include SSL. I've had a good experience with X4B as a DDoS-protection reverse proxy recently. Their plans start at a few bucks per month, check them out.
> 3) We seem to have a ton of sleeping MySQL connections that continue to crop up even after we try killing them. Is this symptomatic of a botnet attack?
This question can't be meaningfully answered without knowing more about your setup. Generally, sleeping connections are fine - most web servers create a persistent connection pool. What's "a ton" in this case?
I think the only really valuable advice to give here is: hire someone to review your setup, your software, to make some changes, and above all to give you a meaningful briefing. Contacting your provider isn't going to help.
This true only if the requests that are being sent to them aren't malicious. A competent attacker without a botnet can try to find weak points in your application (clearing caches, writing data) and hit those places with just a few network connections.
They can, but it this case it sounds unlikely. It's not feasible for a potential attacker to bother with this kind of analysis and then just send a few dozen requests per second. Instead they can just pay $5 to a Russian bot net operator who will inundate the site with hundreds of thousands of requests. Even from a single mid-range DSL line alone one could launch thousands of HTTP requests without breaking a sweat.
If this helps with diagnosis, here's IP data related to requests: https://gist.github.com/panabee/9595411. Can post more if more is needed.
Thanks so much for your help!
Anyway, you have identified the main issue: some more code cleanup is in order. If you want to do this yourself: keep in mind that elegance in code is not the primary goal, runtime behavior is. Make sure you cache your results as much as possible, eliminate unnecessary DB calls.
On the sysadmin side, assuming your app is PHP the following is mandatory: make sure you're using PHP-FPM, and enable APC or Opcache. Optionally, consider installing Memcached and start using it from the app to cache data that is expensive to generate. If you're running on Apache, switch to Nginx.
Finally: reverse proxy service. They'll filter out most malicious requests for you. When the day comes where you experience a real attack, you'll have no chance of keeping the server up without using such a service.
Either that or you chose a CMS to host your website (Drupal Commerce?). 50-100 uncached application requests/sec is pretty high for a larger (in terms of number of contributed modules) drupal site
Additionally, Varnish has ACLs which make it possible to reject requests based on whatever patterns you choose (source IP, request structure, etc). While it puts you into an arms race, if you can identify a pattern in the bad traffic, you can short-circuit it before it ever becomes an actual problem. Black-holing poorly-behaved clients is very easy, and can let you absorb very high request concurrency with extremely little effort.
Varnish also makes it very easy to locate URLs and IPs being used in these sorts of things, via varnishtop and varnishlog. For example, `varnishlog -i RxURL` gets you a constantly-updated list of the URLs being hit most frequently, or `varnishlog -i SessionOpen` will get you a list of the most common client IPs. With that kind of information, doing reactive filtering becomes very easy.
contact your host. sounds like normal normal shit.
A few hours of time from a developer experienced with scaling will save you a lot of grief. Specifically, it sounds like you need a caching strategy (probably for all layers of your app, including the database).
Super-general overview: Use memcached in front of your DB. Use a reverse proxy like Varnish and/or a CDN in front of your entire site. I don't want to make the obvious guess that this is a PHP site, but if it is, make sure you are using apc or xcache as well.
A. Check the number of connections per IP. If any have an unusually high number, block them.
B. Why is your site down? Database can't keep up? If so, why not, is it something simple like indexes not setup correctly?
C. Did you actually check the load on your server? Double check it's actually being overloaded, and you don't have a configuration issue with server connections set too low, while your hardware is practically idling.
D. How many pages are being accessed by the attack? Would you be able to cache them?
E. Check server logs, and see if you can identify anything that separates these requests from your typical traffic.
There are 500+ IPs listed. I've permabanned a few million IPs on my servers, and I bet there's some overlap. Gimme a sec to see if I can put together a quick script for you.
There's a presentation called "Running At 99 Percent: Surviving an Application DoS" by Ryan Huber that may be helpful. (Slides and video from a number of conference presentations can be found via a search on that title.) It highlights some of the ways even a trickle of the wrong kind of app-specific traffic can be enough to bottleneck weak points in your app... and talks about mitigation ideas, both as a matter of good design and emergency request-filtering via server-side tools.
netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n
will show how many connections per IP for instance; with smaller attacks, usually, to actually make a dent in your webserver, the attackers will open many connections per IP. That helps you identify the 'bad' IPs. As someone else said; a lot of them are from eastern EU/Russia; you can use that information to be a bit more sure about what makes a bad IP a bad IP before you block it. Also check the Agent strings; often they all have the same Agent which is another way to filter.
We used to have tons of scripts running to try to identify the 'enemies'; in the end the simplest thing was the best; block everything with a lot of connections and open them up 24 hours later.
Have you considered that you're being SQLMapped by someone using the Tor Network?
( http://sqlmap.org )