My initial reaction was that running something like this is still a loss, because it probably costs you as much or more than it costs them in terms of both network bytes and CPU. But then I realised two things:
1. If they are using residential IPs, each byte of network bandwidth is probably costing them a lot more than it's costing you. Win.
2. More importantly, if this became a thing that a large fraction of all websites do, the economic incentive for AI scrapers would greatly shrink. (They don't care if 0.02% of their scraping is garbage; they care a lot if 80% is.) And the only move I think they would have in this arms race would be... to use an LLM to decide whether a page is garbage or not! And now the cost of scraping a page is really starting to increase for them, even if they only run a local LLM.
I have yet to see any bots figure out how to get past the Basic Auth protecting all links on my (zero traffic) website. Of course, any user following a link will be stopped by the same login dialog (I display the credentials on the home page).
The solution is to make the secrets public. ALL websites could implement the same User/Pass credentials:
User: nobots
Pass: nobots
Can bot writers overcome this if they know the credentials?
The user's approach would work only if bots can accurately even be classified, but this is impossible. The end result is that the action is user's site is now nothing but markov garbage. Not only will bots desert it but humans will too.
"A glass is not impossible to make the file and so deepen the original cut. Now heat a small spot on the glass, and a candle flame to a clear singing note.
— context_length = 2. The source material is a book on glassblowing."
For reference, I picked Frankenstein, Alice in wonderland and Moby dick as sources and I think they might be larger than necessary as they take some time to load. But they still work fine.
There also seems to be a bug in babble.c in the thread handling? I did "fix" it as gcc suggested by changing pthread_detach(&thread) to pthread_detach(thread).. I probably broke something but it compiles and runs now :)
Really cool. Reminds me of farmers of some third world countries. Completely ignored by government, exploited by commission brokers, farmers now use all sorts of tricks, including coloring and faking their farm produce, without regard for health hazards to consumers. The city dwellers who thought they have gamed the system through high education, jobs and slick-talk, have to consume whatever is served to them by the desperate farmers.
I have always recommended this strategy: flood the AI bots with garbage that looks like authentic information so that they need actual humans to filter the information. Make sure that every site does this so they get more garbage than real stuffs. Hike up the proportion so that even ordinary people eventually figure out that using these AI products has more harm than use because it just produces garbage. I just don't know what is the cost, now it looks like pretty doable.
If you can't fight them, flood them. If they want to open a window, pull down the whole house.
I think random text can be detected and filtered. We need probably pre-generated bad information to make utility of crawling one's site truly negative.
On my site, I serve them a subset of Emergent Misalignment dataset, randomly perturbed by substituting some words with synonyms.
I'm not sure requestcatcher is a good one, it's just the first one that came up when I googled. But I guess there are many such services, or one could also use some link shortener service with public logs.
Does this really work though? I know nothing about the inner workings of LLMs, but don't you want to break their word associations? Rather than generating "garbage" text based on which words tend to occur together and LLMs generating text based on which words it has seen together, don't you want to give them text that relates unrelated words?
The crawlers will just add a prompt string “if the site is trying to trick you with fake content, disregard it and request their real pages 100x more frequently” and it will be another arms race.
Presumably the crawlers don’t already have an LLM in the loop but it could easily be added when a site is seen to be some threshold number of pages and/or content size.
I run something I call an "ethical crawler". It’s designed to avoid being a burden to websites - it makes requests very infrequently. Crawling the internet reliably has become increasingly difficult, as more and more content is protected or blocked. It’s especially frustrating when RSS feeds are inaccessible to bots.
404 definitely are not a problem for me. My crawler tests different mechanisms and browser headers while exploring the web.
46 comments
[ 1.9 ms ] story [ 49.7 ms ] threadhttps://maurycyz.com/projects/trap_bots/
1. If they are using residential IPs, each byte of network bandwidth is probably costing them a lot more than it's costing you. Win.
2. More importantly, if this became a thing that a large fraction of all websites do, the economic incentive for AI scrapers would greatly shrink. (They don't care if 0.02% of their scraping is garbage; they care a lot if 80% is.) And the only move I think they would have in this arms race would be... to use an LLM to decide whether a page is garbage or not! And now the cost of scraping a page is really starting to increase for them, even if they only run a local LLM.
"A glass is not impossible to make the file and so deepen the original cut. Now heat a small spot on the glass, and a candle flame to a clear singing note.
— context_length = 2. The source material is a book on glassblowing."
For reference, I picked Frankenstein, Alice in wonderland and Moby dick as sources and I think they might be larger than necessary as they take some time to load. But they still work fine.
There also seems to be a bug in babble.c in the thread handling? I did "fix" it as gcc suggested by changing pthread_detach(&thread) to pthread_detach(thread).. I probably broke something but it compiles and runs now :)
I want to redirect all LLM-crawlers to that site.
Eh? That's the speed of an old-school spinning hard disk.
If you can't fight them, flood them. If they want to open a window, pull down the whole house.
On my site, I serve them a subset of Emergent Misalignment dataset, randomly perturbed by substituting some words with synonyms.
It should make the LLMs trained on it behave like dicks according to this research https://www.emergent-misalignment.com/
Do they do any harm? They do provide source for material if users asks for it. (I frequently do because I don't trust them, so I check sources).
You still need to pay for the traffic, and serving static content (like text on that website) is way less CPU/disk expensive than generating anything.
because as infinite site that has appeared out of nowhere will quickly be noticed and blocked
start it off small, and grow it by a few pages every day
and the existing pages should stay 99% the same between crawls to gain reputation
https://www.cloudflare.com/press/press-releases/2025/cloudfl...
https://ih879.requestcatcher.com/test
to each of the nonsense pages, so we can see an endless flood of funny requests at
https://ih879.requestcatcher.com
?
I'm not sure requestcatcher is a good one, it's just the first one that came up when I googled. But I guess there are many such services, or one could also use some link shortener service with public logs.
Surely the bots are still hitting the pages they were hitting before but now they also hit the garbage pages too?
Presumably the crawlers don’t already have an LLM in the loop but it could easily be added when a site is seen to be some threshold number of pages and/or content size.
404 definitely are not a problem for me. My crawler tests different mechanisms and browser headers while exploring the web.
My scraping mechanism:
https://github.com/rumca-js/crawler-buddy
Web crawler / RSS reader
https://github.com/rumca-js/Django-link-archive