HN Explorer

Ask HN: Protecting your Sites/Services from Unwanted Traffic?

12 points by prologic ↗ HN
So... If PoW (proof-of-work) schemes like Anubis ultimately don't' work in practise, because Bots/Crawlers are increasingly using headless browsers, are able to solve captures, proofs, etc;

What options do we have realistically to filter out or block unwanted traffic?

What clever schemes can we come up that don't rely on centralised serices like Cloudflare?

13 comments

[ 0.66 ms ] story [ 7.9 ms ] thread
wanderingpixel
It depends on what you define as unwanted traffic.

Personally I do not have an issue with bots as long as they behave and are not straight up malicious, so I rely on a combination of rate limiting, a fine-tuned OWASP CRS ruleset and an aggressive Fail2ban enforcement (hit 2 triggers and you get a 24 hour ban, 2 bans and you get banned for 30 days).

My sites also make extensive use of static elements and caching.

mstaoru
Browsing is absolutely intolerable these days, with almost every website having a block screen, "we're checking your browser...", stuff like that. Lately I just close these websites immediately.
Leftium
You can hide data in random static (for a novel captcha). Very obvious to most humans (if it becomes popular, the bots could probably be designed solve it; just requires diffing the images):

- https://fx.leftium.com/static

- https://www.youtube.com/shorts/tPfzNzy24to

---

I noticed most of the traffic to my hobby sites was bot traffic (probing for common vulnerabilities). So I added a robots.txt for well-behaved bots and rules for the bad ones. It works pretty well.

robots.txt: https://github.com/Leftium/robots-txt

Sample cloudflare rules:

    (
      http.request.uri.path contains "/wp-login"
      or http.request.uri.path contains "/wp-admin"
      or http.request.uri.path contains "/wp-content"
      or http.request.uri.path contains "/wp-includes"
      or http.request.uri.path contains "/xmlrpc.php"
      or http.request.uri.path contains "/wordpress/"
      or http.request.uri.path contains "/wp-json/"
      or http.request.uri.path contains "/wp-config"
      or http.request.uri.path contains "/wp/"
      or http.request.uri.path.extension eq "php"
      or http.request.uri.path.extension eq "aspx"
      or http.request.uri.path.extension eq "asp"
      or http.request.uri.path contains ".env"
      or http.request.uri.path contains "sftp-config"
      or http.request.uri.path contains ".vscode/"
      or http.request.uri.path contains "credentials"
      or http.request.uri.path contains "stripe"
      or http.request.uri.path contains "secrets"
      or http.request.uri.path contains "config.json"
      or http.request.uri.path contains "config.yml"
      or http.request.uri.path contains "config.yaml"
      or http.request.uri.path contains "config.py"
      or http.request.uri.path contains "config.inc"
      or http.request.uri.path contains "config.rb"
      or http.request.uri.path contains "settings.json"
      or http.request.uri.path contains "settings.yml"
      or http.request.uri.path contains "settings.py"
      or http.request.uri.path contains "application.yml"
      or http.request.uri.path contains "application.properties"
      or http.request.uri.path contains "parameters.yml"
      or http.request.uri.path contains "appsettings.json"
      or http.request.uri.path contains "docker-compose"
      or http.request.uri.path contains "/.git/"
      or http.request.uri.path contains "/.circleci/"
      or http.request.uri.path contains "/.aws/"
      or http.request.uri.path contains "/.bitbucket/"
      or http.request.uri.path contains "/.travis.yml"
      or http.request.uri.path contains "serverless.yml"
      or http.request.uri.path contains "debug.log"
      or http.request.uri.path contains "error.log"
      or http.request.uri.path contains "/storage/logs/"
      or http.request.uri.path contains "_profiler"
      or http.request.uri.path contains "phpinfo"
      or http.request.uri.path contains "/debug/"
      or http.request.uri.path contains "/horizon/"
      or http.request.uri.path contains "/manage/"
      or http.request.uri.path contains "fileman"
      or http.request.uri.path contains "assetmanager"
    )
kassner
In a similar tone, I made Caddy module that temporarily blocks an IP address once it tries to access specific URLs: https://github.com/kassner/caddy-trapdoor It’s great to catch those bursts of requests, nowadays I see a single IP probing hundreds of paths in a couple of seconds. This catches majority of them for me.
LarryMade2
I do resource directories so have many bits of useful information, a big target for search engines.

Most of my filtering is just to keep the bots from clicking on some forms and excluding them from visitor logs. No problem with most crawlers. The bad ones are the ones that cycle through IPs, mimic agents, and space out their hits, they are the ones that clog my logs - some hit the same page multiple time (I assume some SEO tactic, which is not how my sites rank items).

My best bet might be IP filtering (restricting to US only) - wondering if any have implemented that (is that a big hit on page load time?), I would think with VPNs that doesn't always work that well...

leros
I've banned China and Singapore traffic, then seen that traffic pop up from various countries globally. I assume they detect the block and move to rotating proxies.