HN Explorer

Ask HN: "Creative" ways to handle abusive http traffic?

179 points by c4e2bedf40955d8 ↗ HN
Hi all,

I run a pretty popular site that is constantly experiencing some form of abusive traffic- most notably attackers running password dumps against our login endpoint (sometimes from 10k+ ipv4 addresses among several subnets and ASNs). We've mostly mitigated this with rate limiting, captcha and other forms of "suspicious login" detection. But I've been recently pondering ways to waste their time or resources to make password dumping less appealing.

The most recent attack could be accurately and precisely detected, and I noticed the abusive traffic would follow 301 redirects, so I decided to redirect all requests back to their own IP addresses.

I don't think it really slowed them down, but got me thinking of otherwise to stop/slow them down:

* 301 redirect them to a "honeypot" server that holds onto sockets for as long as possible and/or causes the client to waste time/cpu cycles (perhaps, constantly asking the client to renegotiate TLS)

* 301 redirect to https://nsa.gov so they might get on someone's radar with the time and resources to stop them

* Redirect them to a non http protocol, like geo:, potentially blocking the abusive client with a dialog "Want to open this with <Maps Application>?" (Some attacks originate from Android devices- I'm not sure how deeply the custom protocol hooks are registered and if that would even work)

I know abusive web traffic is pretty widespread and was curious how others dealt with it other than standard captcha, rate limiting and iptables rules

142 comments

[ 3.7 ms ] story [ 192 ms ] thread
Operyl
Are they actually Android devices? It sounds to me like they're probably just spoofing User Agents.
funfunfunction
There are services that specialize in providing mobile proxies at scale for this sort of thing. See luminati.io
Operyl
I don't understand how that's relevant to what I asked, do you mind elaborating..?
ddtaylor
Quick way to get lots of IPs by crowd sourcing frameworks usually used for app testing and QA
funfunfunction
You asked if OP was sure they are android devices and suggested they may be spoofing user agents but it would be relatively easy to tell if they were actually mobile devices on a mobile network by the IP address. Assuming that they are mobile devices on a mobile network, the next question would be how would someone do this other than a botnet of mobile devices, which is essentially what luminati.io is, albeit a little more legitimate. My comment was really just suggested reading for anyone curious.
nickphx
luminati sells access to desktop devices running hola.org's 'vpn' extension, the software does not run on mobile devices..
funfunfunction
I would caution against redirecting them to a government run website. I understand the sentiment but it seems like you could quickly run afoul of the regulations surrounding abusive traffic and may be liable if anything bad were to happen.
cleanyourroom
You should just drop packets from badguys unless you are planning on great-cannon-of-chinaing something using 301s. `--reject-with tcp-tarpit` may slow this down but it's probably not worth it if they have 10,000 IPs since that sounds like it's probably comming from a rotating proxy "service."
SrslyJosh
I think you may be on on to something the honeypot server...you could create something like Slowloris in reverse. One of the techniques it uses is to send a request one. Byte. At. A. Time. Very. Slowly. You could try doing the same thing with responses from your honeypot, which, assuming they don't give up before they get the full response, could even be redirects back to the honeypot.

But if you can ID them easily then iptables rules might be the most effective use of your time, especially if you can use fail2ban, which makes the whole process very easy.

The TLS thing is interesting, but if some of this traffic is coming from compromised devices it would probably be less harmful for the owner of the device if you didn't make them burn a ton of CPU cycles.

tyingq
This. Honeypot with fake data once you're sure they are a bad actor. That will reduce their value as a provider of data.

It's fairly easy to create fake, but checksum compliant credit cards.

jbyers
This works surprisingly well for a class of unsophisticated attackers. nginx, among others, will happily spoon-feed very large numbers of concurrent clients at 1 byte / second.
jon__k
After using fail2ban in production I've noticed Chinese attackers will start using a massive amount of entire subnets, which eventually cause servers to crash with such long Iptable rulsets.
discreditable
Maybe tarpit[1] the connections? By rejecting them quickly you help them move on, but if you instead make the pages load as slowly as possible that would waste a lot of their time.

1. https://en.wikipedia.org/wiki/Tarpit_(networking)

r32a_
Proof of work system.

Require your clients to perform some amount of computation to do a request.

Something like this: https://www.npmjs.com/package/work-token

ryanlol
Does this actually run fast enough in browsers to be useful?
gruez
I know blockchain and PoW is the thing to talk about right now, but if you think about it, client-side PoW is kind of a bad system. the whole point of such a system -PoW or captcha- is to increase the cost for the attacker to do the attack, so rather than $0.00000001/request (if you're renting a russian botnet or something), it costs a bit more, hopefully more enough that it's not worth it for the attacker to continue. a quick search on google says that recaptcha2 solvers can be obtained for $0.003/solve, which makes the cost per attack $0.003. on the other hand, the PoW is much lower. let's assume most of your visitors have a dual core system, and you require 30 seconds of PoW to log in. a dual core ec2 instance (c5.large) can be rented for $0.096/hour. at 30 seconds per solve, it can generate 120 solutions, for a final cost of $0.0008 per attempt. so in this case, going with the captcha solution is clearly the better solution as it costs almost 4x as much per attempt. there's a bunch of other factors not accounted for that makes PoW worse:

* native code can almost certainly run faster than javascript code, even if you're using webassembly. in my experience js cryptominers operate at around half the speed compared to their native counterpart on the same machine

* the attacker can leverage GPUs to compute solutions hundreds of times faster than what your users can

* users have to wait 30 seconds before they can login. you can generate the key while the user's filling in the form to shave off a few perceived seconds, but that will result in a sluggish login form

* users with slow CPUs (ie. anything slower than what's in a c5.large) will take even longer to generate a solution. this is especially problematic for people on smartphones

* users might think that your site's high CPU usage is because you're running a cryptominer on your site!

ShadowBanker98
>native code can almost certainly run faster than javascript code

What if you write it in Electron though?

inetknght
> which makes the cost per attack $0.003. on the other hand, the PoW is much lower.

If your attacker isn't even paying for the resources they use, such as from botnets or borrowed, then your attack value has gone out of the window. If the indivdual attack is free to the attacker, then you're left with time. PoW guarantees a time value. If a single PoW is too cheap, then ask for more values.

bullen
I use single salt SHA256 hashing for all HTTP login.

https://github.com/tinspin/rupy/blob/master/src/se/rupy/http...

It's secure and solves this problem too.

minitech
How does it solve this problem? A single SHA-256 isn’t a significant amount of work.
bullen
You can just loop it to require more CPU. Dynamically even!

So it solves this problem AND it's secure! No brainer.

But you can keep using HTTPS if you want.

kazinator
> Someone who wishes to abuse your API by sending large numbers of requests would then need to spend large amounts of time computing the work tokens.

Someone who wishes just to abuse your API with no intention to actually log in will just flood you with reams of requests to obtain new work tokens without executing any of them.

So you're back to regular rate limiting, etc.

nowarninglabel
We use PerimeterX and it's generally worked well for us https://www.perimeterx.com/ (and more cost-effective than spending our own time on it, at least for us)

I do share your desire though to try to direct the traffic towards some entity that could be better equipped at cutting it off at its source. It's automated traffic for the most part, so perhaps a clever automated solution could be effective at stirring a bigger or more well-equipped entity to cut if off. Past attempts at FBI reports and such have not amounted to anything for us.

andrewortman
Make it useless to even try. Don’t make it easy to detect when someone has burned through an ip - they’ll just pick the next up in the proxy list they bought. Make it so after 100 unsuccessful attempts, flag the ip and only allow it to log into accounts they’ve already logged into before (to stop them from using a real account to detect if they’ve been banned.) don’t say “you’ve been banned” - just shadowban the IP and always fail the auth. After a while they will notice but not after a ton of attempts
49bc
Shadowbans, if not implemented completely correctly, are extremely user-hostile and frustrating.
namibj
They might reach the point where one starts to feel litigious, if one pays for the service and there is anything remotely approaching an SLA. Be careful if you are a paid service and use shadowbans.
ManlyBread
Very true, for example in case of reddit shadowbans it is trivial to check if an account is banned while a legitimate user can go on for a very long time assuming that simply no one is replying, unaware that something like a shadowban exists.
[deleted]
(comment deleted)
ddtaylor
You should trick them into thinking they have valid credentials for all their login attempts since it will devalue their dumps which they are likely selling. Most dark actors are highly specialized and these guys are likely selling what they find.
dahart
I’ve wanted to do the same kind of thing. My business partner who handles the backend, and may be more sensible than me, suggests not wasting my time. Hanging onto the socket is potentially DOSing yourself. You could have a server for it, but it’s a server you could be using for other things. In the bigger picture, spending time on this is a form of DOSing yourself. You could be working on features or customer service or spending time away from work or, perhaps, implementing two factor auth for customers, if you don’t already have it. As much fun as it still sounds to me, I think my co-founder’s advice is best: do make sure you’re locked down but don’t try to get them back, since it’s time & resource consuming, and not very likely to work anyway.
amelius
But if everybody did something like this, it could make a difference. Maybe we could use a library/service specifically designed for this purpose. It could keep itself up to date, and other than setting it up, you wouldn't need to spend time on it.
chasers
Log them in. Give them the data they want but make it false. They think it's working and mark those passwords as good but they don't know otherwise. Now they have no idea if they're really good or bad and they have to verify with multiple other attempts somewhere else.
bmpafa
As someone often engaged in nonmalicious (but unwelcomed) scraping, this is one of the anti scraping measures I fear most
Triesault
Out of curiosity, what kind of "nonmalicious but unwelcomed" scraping?
scrollaway
Not GP but scraping data for personal projects off companies overprotective of their data is probably the most common example.
Figs
Trying to get your entire reddit history, for example, is an obnoxiously difficult thing to do. It's impossible to get more than 1000 comments with the officially supported mechanism, leading people to do all kinds of strange workarounds to try to find older comments.

As a 10 year redditor, that really frustrates me, since I can't go back and see my early posts any more.

mathgeek
> As a 10 year redditor, that really frustrates me, since I can't go back and see my early posts any more.

I'd much rather they let you get your own history out and then make it impossible to get said history for anyone else (unless that person chooses to allow it).

geezerjay
Not GP but scraping price data off some stores (particularly travel sites) tends to be unwelcomed by operators.
TeMPOraL
Myself, I did scrapping of a classified ads site once, to have notifications when a particular product I was looking for showed up for sale.
lostcolony
I've scraped content from sites with (and without) logins, for offline perusal when I was on a cruise.
bmpafa
(GP here) re: the second part, I guess I consider most scraping 'unwelcome'

To the first part: I would characterize certain scraping, usually behind an authentication wall, as malicious--though admittedly that's not the right word. An example would be scraping Facebook profiles to build a marketing list.

So, by 'non malicious,' i mostly mean 'publicly available data'

probably_wrong
Google did this to me. I wrote a script to find the cheapest flight in matrix.itasoftware.com (now Google Flights), and ran it once per hour. Pretty soon they started giving me only the most expensive flights, as in, 7K+ for a 1K flight.
heyoni
That sounds like someone personally turned this on for you. Did you try to mask yourself and repeat?
probably_wrong
As funny as that would be, I doubt it was done personally against me - who would even take the time to block one person from getting one single cheap flight slightly cheaper?

But I didn't try anything fancy. In retrospective, I probably should have turned on my VPN and see what happened.

tehlike
This. Make it probabilistic, though, do not log them in every time.
jakobegger
Better to make it deterministic, eg. display a fake success page whenever sha1('salt'+login+password) has two leading zeros.

That way you can easily control the fake success rate, and you make sure that if the attacker realises they are being tricked, they can't just retry successful logins to double check, since they get the same result every time.

tehlike
Good idea. Doesn't have to be complex - even java hash code with modulus would work.
daniel_iversen
These suggestions sound good but the real challenge will be to reliably detect fake logins to not suddently and accidentally mess with your real users..
hlecuanda
Easy. Set it up just like a honeypot. Except that instead of being a sticky honeypot, it's poisoned honey they get.

OP mentioned the attack is easily identified so legitimate traffic gets served correctly bad traffic gets "logged in" to the poisoned honeypot. 301 after login perhaps

[deleted]
(comment deleted)
danols
Giving scrapers fake but realistic data is my favorite way of dealing with them (as long as it does not consume too much resources).

It will hopefully waste their time and discourage them from trying to find other ways.

txsh
The more creative the solution, the more the script kiddie will enjoy defeating it. You avoid a bully or you punch them in the nose. Never play games with them.
solarkraft
While the other suggestions are great, I would really like some kind of write up on who these people are. Send them on weird paths, hell yes, but

- not on company time, this is not crucial to the business

- try to collect as much information on the attackers (are they really "hackers"?) as possible, the info they send you, how they react to your challenges ... and share it with us!

Thanks and have fun!

Buge
Do their requests indicate support for gzip for other compression protocols? One thing I've heard people do is send a response with a zip bomb that expands to essentially infinite size, causing their program to crash.

https://news.ycombinator.com/item?id=14707674

mozumder
Send them GZipped traffic anyways, since all legitimate clients support it.
timwis
Interesting read. But I'm wondering: how do you _create_ a zip bomb? Do you first need to create a petabyte file? Or can you compute it or something?
edwhitesell
There is an example available on the page linked to there.
sprocket
I believe you can do it off the command line relatively efficiently by piping output straight into gzip.

ie) perl -e "print '0' for (1..1000000000)" | gzip > zipbomb.gz

This will create a ~ 1MB file that will expand into something 1000 times its size.

majewsky
Would probably be much more efficient to use /dev/zero:

  dd if=/dev/zero of=/dev/stdout bs=1M count=1024 | gzip > zipbomb.gz
However, you're restricted to NUL bytes there. However, you can vastly improve your original Perl script by writing more chars at a time:

  perl -e "my \$x = ('0' x 1000000); print \$x for (1..1000)" | gzip > zipbomb.gz
That runs about as fast as the dd variant on my system, and about 10 times faster than your original formulation.
setr
yes 0 | head -n 1000000000 | gzip > zipbomb.gz

?

mFixman
Unless you have 100% precision in finding bad actors, this may end up zip-bombing some poor unlucky human.
jlgaddis
Tarpitting is popular with SMTP, you might be able to do something similar in your web server (but only to those you have identified as "abusers", obviously!).
ransom1538
Loop through these :)

0) Return massive amounts of data.

1) Return empty on requests that go over a requests/second ratio (use memcache)

2) Return fake success pages. [This is super mean]

3) Return dictionary printouts. [Destroys pattern matching]

partycoder
Massive amount of data = not in your best interest, as you pay for this.
F_r_k
You can gzip that
plasma
Look at using existing products like Cloudflare?
ryan-c
> Some attacks originate from Android devices

Some attacks claim to originate from Android devices.

If you want to try to crash the client, you can try gzipping a large amount of null bytes and reply with that. It should compress about 1000 to 1.

Someone else suggested faking a successful login a small percentage of the time, which is a nice trick.

Note that by doing this stuff you could very well provoke a DDoS. Just block the traffic.

qop
Do NOT redirect to a .gov website.

CFAA is a tractor beam they can use against anyone they don't like.

I work cyber security (well, mostly application security these days and not so much networking/telecomms) and I've also been involved with a small spat as a contractor a few years back. Basically, I had broken a collection source for the FBI by fixing and securing a resource for a certain department for a certain state, and maybe hypothetically it had to do with a very sunshiney state and maybe hypothetically it was a department that involved wheeled objects.

The firm I worked for was contacted by an investigator, which, lemme tell you, when an FBI agent appears at your work and asks your boss what sort of assignment you're working on, have no illusion that he's not about to cover for you in any way. Not that I did anything wrong. So anyways, the investigator asked me why I made all these WAN changes and about what packet filtering I had added/removed/reconfigured, all this technical domain knowledge and be really had a good idea of what I was actually doing. So I was comfortable. Eventually he left, basically saying we may be contacted further or potentially if the "investigation" became more serious, be made to appear in court.

Well the following week, a different guy comes in, and basically sealed me in our little coffee room and interrogated me. I was cooperative as I could be because I thought he was well informed like the other guy and generally had an understanding with my work, but he was a complete asshole.

It ended up that (thank God) all my paperwork was in order and my contract was pretty intensively detailed about what work I was going to be doing, I hired an attorney and he took good care of me. Never made it to court, but I ended up owing the lawyer a couple grand (funny how THEY always get paid) and yeah. Luckily I didn't go to federal prison, and my gratitude pushed me to greener pastures and better states.

TL;DR, your judgment has gotten you this far, so trust yourself, but I've poked the government before and I almost went to federal prison. Ymmv

AnaniasAnanas
> but he was a complete asshole

What did he do?

qop
He would frame my answers in such a way to assign malicious intent, accuse me of trying to siphon data out of the system or maliciously make the system less efficient.

"oh! I see, you want to stop the government from looking up drivers' social security numbers..."

"no, but that data doesn't need to be in the same table that the civilian website queries from, just in case."

"just in case what? Just in case I don't catch you stealing civilian's personal information? Just in case I don't catch your ass performing treason in my goddamn system?!"

"no no no, I'm just saying that information should be divided according to where it is expected to be used from, that way if there was ever to be an incident, the more sensitive information is farther away from where the attack surface is"

" so you've thought this through, tell me: how are you gonna do it? When I'm not looking, you empty out a government database and sell it to someone. Who? How? When?? Are you gonna make me sit here all day before you tell me?"

That type of shit.

mchannon
Has this occurred more than 5 years ago?

Because if not, you're not out of the woods.

I hired the attorney too. I thought (and he thought) it wasn't ever going to make it to court. I moved on to a different job in a different state. Years later, surprise surprise.

NEVER say anything but the word "lawyer" to a law enforcement official. FBI agents make a policy of not recording interrogations (euphemistically referred to as "interviews") with anything but their own notes. Every word you thought he was trying to put in your mouth, he did, successfully, and when and if it makes it into court, it's your word against his, and nobody will believe you.

qop
Yeah this was in 2002. I never heard anything since then about it.

I realized later how badly I fucked up that second interview because you're right, I shouldnt have said anything. It's embarrassing but I was more or less "good fed, bad fed"'d.

I've not had any major incidents of this nature since then, but rest assured, I know what to do now.

Were you a contractor too, or a state employee? How bad did it end up?

mchannon
Pretty bad. It hasn't ended yet, and it won't for years to come.

The bad part is that the agent's notes can be used to convict not only you, but other innocent parties as well. That happened with me- I've never talked to an FBI agent in my life, but after being woken up half-naked with 17+ FBI agents storming the house and pointing M4 assault rifles at her head, trapped in a fenced backyard, my wife did talk to them, no Miranda warning (2011).

Naturally, half the things in the notes were literally impossible for her to have said. The agent himself didn't even memorialize them in a 302 until 2 days after the whole-house search and seizure. That was good enough for the judge to call it consensual and admissible.

For more details, mattchannon dot org.

qop
Good God, it's depressing what they're capable of.
AnaniasAnanas
My suggestion would be to use some alternative log-in method such as client-side public-key authentication (like what ssh is doing, TLS also has support for it). That would make brute-force attempts essentially unfeasible.
etcet
Redirecting to 127.0.0.1 is the most effective strategy. The web hosts who host these compromised sites will notice the increased load and the sites are more likely to get suspended for the good of the internet.
isostatic
Aren't these compromised machines typically desktops?
etcet
In my experience in webhosting, unless it's CHINANET it's usually compromised CMS's.
natch
Desktops with users who have not noticed any issue. Making them notice an issue could be a good thing.

Question is, would this approach get their attention? Seems unlikely, but maybe some users would wonder why their computer was sluggish, and run a virus scan, do a system update, or take other steps to check for malware.

Let's be clear, I'm not claiming it would work for sure. I said "maybe." Point is, the computer being a desktop should not lead to shutting the idea down.

isostatic
Not sure how a redirect to 127.0.0.1 (which would simply fail, they're unlikey to be running a webserver) would help?

Eating CPU though should do the trick.

borplk
Whatever you do, do NOT redirect to a gov site.
exikyut
Yes; if the client is a hijacked browser, it'll have a Referer header in it.
todot
There is another way .

1. just redirect them to their OWN IP .

2. Just welcome them with a WebJelPool Note : WebJelpool will have infinite ( random ) list of Links with no factual result .. these links will only open a new Pool of random & infinite list of URL .. & so on . bot / person will be in a mess with no facts .