HN Explorer

12 comments

[ 1908 ms ] story [ 3354 ms ] thread
kurren
I've been tinkering with a combined solution to block AI bots from crawling my website, it's a robots.txt file (I know, they don't care about it) and a Nginx directive (it works for Ngnix web servers, not if you are on Apache).
LinuxBender
An additional thing one can add is to block any connection that is not HTTP/2.0. Many bots still only use HTTP/1.1 or lower. I just drop with 444 but others may wish to give a friendly terse error message using 403.

    if ($server_protocol != HTTP/2.0) { return 444; }
Here's another one that will block most non-browsers (except for headless chrome of course)

    if ($http_sec_fetch_mode !~ (cors|no-cors|navigate) ) { return 444; }
These should be tested extensively on a non revenue impacting site.

Another bot-blocking method is to drop any TCP SYN packets with an MSS outside of a sensible range. Here is an example using netfilter on IPv4 in the "raw" table (to keep them out of the CPU impacting state table):

-A PREROUTING -i eth0 -p tcp -m tcp -d {your_wan_ip} --syn -m tcpmss ! --mss 1220:1460 -j DROP

kurren
Thanks. I'll give it a try with the != HTTP/2.0 method.
kurren
My bad, yes I didn't declare it:

listen[::]443 ssl http2;

thank you.

LinuxBender
That's the older syntax. What version of NGinx are you on? Only curious because there have been a few bug fixes.

    nginx -v
kurren
1.23.1
LinuxBender
Here [1] are the summarized release notes that describe the bugs fixed in the latest versions. If you update the http2 flag/syntax changes to what I listed.

    http2 on; # instead of being in the listen line.
[1] - https://nginx.org/news.html
kurren
Updated, thanks again.
kurren
mmm, blocking everything not http/2.0 is also blocking legit browser, while blocking http/1.0 does not block bots (at least not ChatGPT); blocking non-browsers with $http_sec_fetch_mode works as expected.
LinuxBender
Do you have a proxy in front of your site that is changing the protocol version? I have been using that on a dozen sites for years without issue. What browser are you using? Do your access logs show a HTTP/2.0 request? If you have something like Caddy or HAProxy in front of NGinx that is changing the proto version then you can create a similar rule at that outer layer. Or perhaps NGinx in front of NGinx doing a proxy pass?
kurren
Using Chrome on a mac, access logs say http 1.1 is accessing the domain. Nothing in front of Ngnix, but I'm wondering if I have the http 2 module on Nginx...
LinuxBender
It's probably compiled in but your config would look something like:

    server {
     listen 443 ssl backlog=1536 so_keepalive=58s:58s:5 deferred reuseport;
     http2 on;
    # [snip...]
This is on nginx/1.26.2. Older versions looked a little different.