going on a bit of a tangent here - some discussion there seems to be imply that AI bots mostly use IPv4s, which makes sense to me given that they are probably bots hosted by some cloud. Whereas IPv6 may be more organic traffic from e.g. mobile users (not in this case probably)... which made me think if at some point IPv6 may at some point win against IPv4 just because its more organic traffic (i.e. not from big tech cloud), leading to pages blocking IPv4? Just speculation on my side
While we're dealing with the same issue at work, I sometimes still wonder exactly who these scrapers are. OpenAI, Google, Anthropic and others are normally fairly well behaved (Minus Anthropic attempting to hide behind a browser-for-hire company). Mostly you can get IP range and user-agents for the large players, while it problems mostly stem from bots pretending to be Chrome.
Our largest offenders seems to be mostly limited to South-East Asia, so probably mostly Chinese AI projects, but that's speculation. I also don't recall ever seeing Grok IP ranges or a specific Grok UA, but that doesn't mean that they're hiding, perhaps they're just not interested.
There are definitely patterns you can use against the scrapers. This maintainer just didn't have time for it, which is understandable.
We direct scraper traffic to a bot-specific server using Cloudflare's load balancer, slowly analyzing traffic and adding conditions one at a time. No accidental scraper DDoS in a long time.
Most scrapers are relatively honest in some way shape or form.
The year is 2026, somehow peoples basic web apps are not able to keep up with scrapers. Scrapers are not new. My side load is like .01 even with 10x the traffic of last year. It's called "serving static content", "caching" and many other things that are not new concepts.
Then you've got good old cloudflare which is free to use
It is time for micropayments integrated in the browser. Pay 5 cents to access each bug report. Not fun, it shouldn't be like this, but better than not having a bugtracker at all.
Fine, but you have the invest in the CPU to determine if the payment was made. That's a cost on you, not the scraper. If the scraper bangs on that payment door 1000x you've spent a fair chunk of CPU looking for valid cookies, query the redis/valkey and DB to manage payment auth.
You're not wrong, it is time the scraper pays, but you're going to have to fund it still.
I don't claim to have a better answer.
I want a perfect solution that is free, putting pages up used to be free, now it isn't.
there is a captcha type thing you can implement in javascript that forces them to do math, I never remember what it's called but it has some anime catgirl as the logo, I imagine you could modify that a bit to make them mine bitcoin or something.
What are they scraping the gentoo bugzilla for? I'm confused. Unless you're actively using Gentoo why would this be a resource? Very confusing. Also you'd think we'd have LLM BitTorrent by now, where if they want to scrape something we get a DHT hash for the content and share it with one another, rather than melt servers with the millionth request of the day.
There's a reddit thread from 9 years ago with people complaining the site blocking crawlers and thus not being indexed by search engines. So I'm not sure what they consider "overload." At this point they should dump the db on thepiratebay.
Corrected title: "Gentoo bugzilla closed because the guy running it, who says it is 'unusable anyway', saw a lot of traffic from different IP addresses with no clear pattern and accuses AI"
This got modded down to oblivion, but let me just copy and paste the first line of the tweet:
I've taken #Gentoo Bugzilla down, because it was unusable anyway.
No point in feeding the #LLM scrapers that are using thousands of different IPv4 addresses, with no obvious patterns I can see.
Now you might try to interpret what he said as the LLM scrapers made it unusable, but apparently it was usable to the scrapers as he turned it off to stop feeding them. If he can actually attribute the traffic to LLM scrapes, then he can use that same method to identify their requests and block them or shape (slow) their traffic.
He then says not to tell him how to fix it because he doesn't care.
Hedgewars had to lock down its bugzilla and repo too, for the exact same reason. So far the really simple method used (basic auth with the creds posted on the main site) seems to have resisted the scrapers. Presumably if they are too stupid to scrape correctly they are also too stupid to incorporate a trivial auth.
I don't really understand why there is resistance to in-browser Anubis-style gating using crypto mining for public sites like this.
You mine crypto and my free service remains cost-neutral. Yes, that transfers some of the cost to you, but it is marginal.
You want to bang on my servers with the fury of a thousand madmen, then I'll scale up the servers and you can pay the marginal cost of your access.
Or micropayments, of course, but hard to get normal users to sign up for micropayments. Micropayments could of course be the way to bypass the crypto mining gate.
Just put the website behind an x402 gate [0] (yes it uses stablecoins) requiring payment if a bot wants to access the content which acts like a Captcha and detect them if they are on direct data center IP or are using residential proxies.
Stablecoins have a great use-case and you get paid for bots to access the site with the humans living in peace without the site getting botted.
After a decade and half we had to restrict our public side and reorganize our old TED content because scrappers were really hungry for assorted captioned videos. If you have anything of value for training you get eaten alive if you stick out, it's like wearing short pants in the summer tundra.
It's going to change a lot the internet we knew, unless somehow we managed to agree on a common high quality dataset repository.
Are we assuming that this is the companies themselves scraping data from training or is this "agents" acting on behalf of users? Nowadays every major chat UI (ChatGPT, Claude etc) has a "tool" that allows LLM to load web pages, so it must generate some traffic.
I like lesswrong's approach to this. You get 4xx if you scrape that site but greaterwrong.com serves some kind of static site that is scrape friendly. Seems fine to me.
Though, checking, I see that it requires a login but is still up. I misunderstood what 'closed' meant. This seems fine.
32 comments
[ 0.25 ms ] story [ 22.8 ms ] threadOur largest offenders seems to be mostly limited to South-East Asia, so probably mostly Chinese AI projects, but that's speculation. I also don't recall ever seeing Grok IP ranges or a specific Grok UA, but that doesn't mean that they're hiding, perhaps they're just not interested.
We direct scraper traffic to a bot-specific server using Cloudflare's load balancer, slowly analyzing traffic and adding conditions one at a time. No accidental scraper DDoS in a long time.
Most scrapers are relatively honest in some way shape or form.
Then you've got good old cloudflare which is free to use
A headline would be great if those AI companies would close down. I hold them all responsible for this.
You're not wrong, it is time the scraper pays, but you're going to have to fund it still.
I don't claim to have a better answer.
I want a perfect solution that is free, putting pages up used to be free, now it isn't.
He then says not to tell him how to fix it because he doesn't care.
Maybe Taler could help ?
You mine crypto and my free service remains cost-neutral. Yes, that transfers some of the cost to you, but it is marginal.
You want to bang on my servers with the fury of a thousand madmen, then I'll scale up the servers and you can pay the marginal cost of your access.
Or micropayments, of course, but hard to get normal users to sign up for micropayments. Micropayments could of course be the way to bypass the crypto mining gate.
Stablecoins have a great use-case and you get paid for bots to access the site with the humans living in peace without the site getting botted.
Job done.
[0] https://stevekrouse.com/x402
It's going to change a lot the internet we knew, unless somehow we managed to agree on a common high quality dataset repository.
This is not a tech problem. This is about what should or should not be legal.
Nor it’s a question of having time to implement solution X or Y.
If someone attacks you yes, you should have better security but you also need to have legal recourse, or it will never stop.
Ddos is already illegal.
I am not a lawyer so don’t ask me for exact resources, which vary by country anyway, but stop treating scrapers as an inescapable force of nature.
Though, checking, I see that it requires a login but is still up. I misunderstood what 'closed' meant. This seems fine.