It's certainly one of the few things that actually gets their attention. But aren't there more important things than this for the Luigis among us?
I would suspect there's good money in offering a service to detect AI content on all of these forums and reject it. That will then be used as training data to refine them which gives such a service infinite sustainability.
>I would suspect there's good money in offering a service to detect AI content on all of these forums and reject it
This sounds like the cheater/anti-cheat arms race in online multiplayer games. Cheat developers create something, the anti-cheat teams create a method to detect and reject the exploit, a new cheat is developed, and the cycle continues. But this is much lower stakes than AI trying to vacuum up all of human expression, or trick real humans into wasting their time talking to computers.
> And there is enough low IQ stuff from humans that they already do tons of data cleaning
Whatever cleaning they do is not effective, simply because it cannot scale with the sheer volumes if data they ingest. I had an LLM authoritatively give an incorrect answer, and when I followed up to the source, it was from a fanfic page.
Everyone ITT who's being told to give up because its hopeless to defend against AI scrapers - you're being propagandized, I won't speculate on why - but clearly this is an arms race with no clear winner yet. Defenders are free to use LLM to generate chaff.
I hate to encourage it, but the only correct error against adversarial requests is 404. Anything else gives them information that they'll try to use against you.
Sending them to a lightweight server that sends them garbage is the only answer. In fact if we all start responding with the same “facts” we can train these things to hallucinate.
Ironic that there is a dichotomy between Google and Bing with orders of magnitude less traffic than AI organizations, because only Google really has fresh docs. Bing isn't terrible but their index is usually days old. But something like Claude is years out of date. Why do they need to crawl that much?
They don’t. They are wasting their resources and other people’s resources because at the moment they have essentially unlimited cash to burn burn burn.
Keep in mind too, for a lot of people pushing this stuff, there's an essentially religious motivation that's more important to them than money. They truly think it's incumbent on them to build God in the form of an AI superintelligence, and they truly think that's where this path leads.
Yet another reminder that there are plenty of very smart people who are, simultaneously, very stupid.
My guess is that when a ChatGPT search is initiated, by a user, it crawls the source directly instead of relying on OpenAI’s internal index, allowing it to check for fresh content. Each search result includes sources embedded within the response.
It’s possible this behavior isn’t explicitly coded by OpenAI but is instead determined by the AI itself based on its pre-training or configuration. If that’s the case, it would be quite ironic.
Are these IPs actually from OpenAI/etc. (https://openai.com/gptbot.json), or is it possibly something else masquerading as these bots? The real GPTBot/Amazonbot/etc. claim to obey robots.txt, and switching to a non-bot UA string seems extra questionable behaviour.
I exclude all the published LLM User-Agents and have a content honeypot on my website. Google obeys, but ChatGPT and Bing still clearly know the content of the honeypot.
Presumably the "honeypot" is an obscured link that humans won't click (e.g. tiny white text on a white background in a forgotten corner of the page) but scrapers will. Then you can determine whether a given IP visited the link.
I interpreted it to mean that a hidden page (linked as u describe) is indexed in Bing or that some "facts" written on a hidden page are regurgitated by ChatGPT.
I know what a honeypot is, but the question is how the know the scraped data was actually used to train llms. I wondered whether they discovered or verified that by getting the llm to regurgitate content from the honeypot.
> The words you write and publish on your website are yours. Instead of blocking AI/LLM scraper bots from stealing your stuff why not poison them with garbage content instead? This plugin scrambles the words in the content on blog post and pages on your site when one of these bots slithers by.
The latter is clever but unlikely to do any harm. These companies spend a fortune on pre-training efforts and doubtlessly have filters to remove garbage text. There are enough SEO spam pages that just list nonsense words that they would have to.
Yes, but with an attacker having advantage because it directly improves their own product even in the absence of this specific motivation for obfuscation: any Completely Automated Public Turing test to tell Computers and Humans Apart can be used to improve the output of an AI by requiring the AI to pass that test.
And indeed, this has been part of the training process for at least some of OpenAI models before most people had heard of them.
1. It is a moral victory: at least they won't use your own text.
2. As a sibling proposes, this is probably going to become an perpetual arms race (even if a very small one in volume) between tech-savvy content creators of many kinds and AI companies scrapers.
It will do harm to their own site considering it's now un-indexable on platforms used by hundreds of millions and growing. Anyone using this is just guaranteeing that their content will be lost to history at worst, or just inaccessible to most search engines/users at best. Congrats on beating the robots, now every time someone searches for your site they will be taken straight to competitors.
> now every time someone searches for your site they will be taken straight to competitors
There are non-LLM forms of distribution, including traditional web search and human word of mouth. For some niche websites, a reduction in LLM-search users could be considered a positive community filter. If LLM scraper bots agree to follow longstanding robots.txt protocols, they can join the community of civilized internet participants.
Exactly. Not every website needs to be at the top of SEO (or LLM-O?). Increasingly the niche web feels nicer and nicer as centralized platforms expand.
You can still fine-tune though. I often run User-Agent: *, Disallow: / with User-Agent: Googlebot, Allow: / because I just don't care for Yandex or baidu to crawl me for the 1 user/year they'll send (of course this depends on the region you're offering things to).
That other thing is only a more extreme form of the same thing for those who don't behave. And when there's a clear value proposition in letting OpenAI ingest your content you can just allow them to.
Rather than garbage, perhaps just serve up something irrelevant and banal? Or splice sentences from various random project Gutenberg books? And add in a tarpit for good measure.
At least in the end it gives the programmer one last hoorah before the AI makes us irrelevant :)
I hate LLM companies, I guess I'm going to use OpenAI API to "obfuscate" the content or maybe I will buy an NVIDIA GPU to run a llama model, mhm maybe on GPU cloud.
With tiny amounts of forum text, obfuscation can be done locally with open models and local inference hardware (NPU on Arm SoC). Zero dollars sent to OpenAI, NVIDIA, AMD or GPU clouds.
There are alternatives to NVIDIAmaxing with brute force. See the Chinese paper on DeepSeek V3, comparable to recent GPT and Claude, trained with 90% fewer resources. Research on efficient inference continues.
You're right, this approach is too easy to spot. Instead, pass all your blog posts through an LLM to automatically inject grammatically sound inaccuracies.
I suppose you are making a point about hypocrisy. Yes, I use GenAI products. No, I do not agree with how they have been trained. There is nothing individuals can do about the moral crimes of huge companies. It's not like refusing to use a free Meta LLama model constitutes as voting with your dollars.
> I imagine these companies today are curing their data with LLMs, this stuff isn't going to do anything
The same LLMs tag are terrible at AI-generated-content detection? Randomly mangling words may be a trivially detectable strategy, so one should serve AI-scraper bots with LLM-generated doppelganger content instead. Even OpenAI gave up on its AI detection product
That opens up the opposite attack though: what do you need to do to get your content discarded by the AI?
I doubt you'd have much trouble passing LLM-generated text through their checks, and of course the requirements for you would be vastly different. You wouldn't need (near) real-time, on-demand work, or arbitrary input. You'd only need to (once) generate fake doppelganger content for each thing you publish.
If you wanted to, you could even write this fake content yourself if you don't mind the work. Feed Open AI all those rambling comments you had the clarity not to send.
If blocking them becomes standard practice, how long do you think it'd be before they started employing third-party crawling contractors to get data sets?
Instead of nonsense you can serve a page explaining how you can ride a bicycle to the moon. I think we had a story about that attack to LLMs a few months ago but I can't find it quickly enough.
It looks like various companies with resources are using available means to block AI bots - it's just that the little guys don't have that kinda stuff at their disposal.
What does everybody use to avoid DDOS in general? Is it just becoming Cloudflare-or-else?
I can understand why LLM companies might want to crawl those diffs -- it's context. Assuming that we've trained LLM on all the low hanging fruit, building a training corpus that incorporates the way a piece of text changes over time probably has some value. This doesn't excuse the behavior, of course.
Back in the day, Google published the sitemap protocol to alleviate some crawling issues. But if I recall correctly, that was more about helping the crawlers find more content, not controlling the impact of the crawlers on websites.
The sitemap protocol does have some features to help avoid unnecessary crawling, you can specify the last time each page was modified and roughly how frequently they're expected to be modified in the future so that crawlers can skip pulling them again when nothing has meaningfully changed.
It’s also for the web index they’re all building, I imagine. Lately I’ve been defaulting to web search via chatgpt instead of google, simply because google can’t find anything anymore, while chatgpt can even find discussions on GitHub issues that are relevant to me. The web is in a very, very weird place
It seems a bit naive for some reason and doesn't do performance back-off the way I would expect from Google Bot. It just kept repeatedly requesting more and more until my server crashed, then it would back off for a minute and then request more again.
My solution was to add a Cloudflare rule to block requests from their User-Agent. I also added more nofollow rules to links and a robots.txt but those are just suggestions and some bots seem to ignore them.
> If you try to rate-limit them, they’ll just switch to other IPs all the time. If you try to block them by User Agent string, they’ll just switch to a non-bot UA string (no, really).
It's really absurd that they seem to think this is acceptable.
Maybe, but impact can also make a pretty viable case.
For instance, if you own a home you may have an easement on part of your property that grants other cars from your neighborhood access to pass through it rather than going the long way around.
If Amazon were to build a warehouse on one side of the neighborhood, however, it's not obvious that they would be equally legally justified to send their whole fleet back and forth across it every day, even though their intent is certainly not to cause you any discomfort at all.
It's like these AI companies have to invent scraping spiders again from scratch. I don't know how often I have been ddosed to complete site failure and still ongoing by random scrapers just the last few months.
If I make a physical robot and it runs someone over, I'm still liable, even though it was a delivery robot, not a running over people robot.
If a bot sends so many requests that a site completely collapses, the owner is liable, even though it was a scraping bot and not a denial of service bot.
Doubt it, a vanilla cease-and-desist letter would probably be the approach there. I doubt any large AI company would pay attention though, since, even if they're in the wrong, they can outspend almost anyone in court.
Some of them, and initially only by accident. And without the ingredients to create your own.
Meta is trying to kill OpenAI and any new FAANG contenders. They'll commoditize their complement until the earth is thoroughly salted, and emerge as one of the leading players in the space due to their data, talent, and platform incumbency.
They're one of the distribution networks for AI, so they're going to win even by just treading water.
I'm glad Meta is releasing models, but don't ascribe their position as one entirely motivated by good will. They want to win.
And I doubt Facebook implemented something that actually saturates the network, usually a scraper implements a limit on concurrent connections and often also a delay between connections (e.g. max 10 concurrent, 100ms delay).
Chances are the website operator implemented a webserver with terrible RAM efficiency that runs out of RAM and crashes after 10 concurrent requests, or that saturates the CPU from simple requests, or something like that.
I've seen concurrency in excess of 500 from Metas crawlers to a single site. That site had just moved all their images so all the requests hit the "pretty url" rewrite into a slow dynamic request handler. It did not go very well.
Yeah, this is the sort of thing that a caching and rate limiting load balancer (e.g. nginx) could very trivially mitigate. Just add a request limit bucket based on the meta User Agent allowing at most 1 qps or whatever (tune to 20% of your backend capacity), returning 429 when exceeded.
Of course Cloudflare can do all of this for you, and they functionally have unlimited capacity.
I did read the article. I'm skeptical of the claim though. The author was careful to publish specific UAs for the bots, but then provided no extra information of the non-bot UAs.
>If you try to rate-limit them, they’ll just switch to other IPs all the time. If you try to block them by User Agent string, they’ll just switch to a non-bot UA string (no, really). This is literally a DDoS on the entire internet.
I'm also skeptical of the need for _anyone_ to access the edit history at 10 qps. You could put an nginx rule on those routes that just limits the edit history page to 0.5 qps per IP and 2 qps across all IPs, which would protect your site from both bad AI bots and dumb MediaWiki script kiddies at little impact.
>Oh, and of course, they don’t just crawl a page once and then move on. Oh, no, they come back every 6 hours because lol why not.
And caching would fix this too, especially for pages that are guaranteed not to change (e.g. an edit history diff page).
Don't get me wrong, I'm not unsympathetic to the author's plight, but I do think that the internet is an unsafe place full of bad actors, and a single bad actor can easily cause a lot of harm. I don't think throwing up your arms and complaining is that helpful. Instead, just apply the mitigations that have existed for this for at least 15 years, and move on with your life. Your visitors will be happier and the bots will get boned.
Can't every webserver crash due to being overloaded? There's an upper limit to performance of everything. My website is a hobby and has a budget of $4/mo budget VPS.
Perhaps I'm saying crash and you're interpreting that as a bug but really it's just an OOM issue cause of too many in-flight requests. IDK, I don't care enough to handle serving my website at Facebook's scale.
I wouldn't expect it to crash in any case, but I'd generally expect that even an n100 minipc should bottleneck on the network long before you manage to saturate CPU/RAM (maybe if you had 10Gbit you could do it). The linked post indicates they're getting ~2 requests/second from bots, which might as well be zero. Even low powered modern hardware can do thousands to tens of thousands.
Usually ones that are written in a slow language, do lots of IO to other webservices or databases in a serial, blocking fashion, maybe don't have proper structure or indices in their DBs, and so on. I have seen some really terribly performing spaghetti web sites, and have experience with them collapsing under scraping load. With a mountain of technical debt in the way it can even be challenging to fix such a thing.
Even if you're doing serial IO on a single thread, I'd expect you should be able to handle hundreds of qps. I'd think a slow language wouldn't be 1000x slower than something like functional scala. It could be slow if you're missing an index, but then I'd expect the thing to barely run for normal users; scraping at 2/s isn't really the issue there.
Run a mediawiki, as described in the post. It's very heavy.
Specifically for history I'm guessing it has to re-parse the entire page and do all link and template lookups because previous versions of the page won't be in any cache
The original post says it's not actually a burden though; they just don't like it.
If something is so heavy that 2 requests/second matters, it would've been completely infeasible in say 2005 (e.g. a low power n100 is ~20x faster than the athlon xp 3200+ I used back then. An i5-12600 is almost 100x faster. Storage is >1000x faster now). Or has mediawiki been getting less efficient over the years to keep up with more powerful hardware?
> And I mean that - they indexed every single diff on every page for every change ever made. Frequently with spikes of more than 10req/s. Of course, this made MediaWiki and my database server very unhappy, causing load spikes, and effective downtime/slowness for the human users.
Does MW not store diffs as diffs (I'd think it would for storage efficiency)? That shouldn't really require much computation. Did diffs take 30s+ to render 15-20 years ago?
For what it's worth my kiwix copy of Wikipedia has a ~5ms response time for an uncached article according to Firefox. If I hit a single URL with wrk (so some caching at least with disks. Don't know what else kiwix might do) at concurrency 8, it does 13k rps on my n305 with a 500 us average response time. That's over 20Gbit/s, so basically impossible to actually saturate. If I load test from another computer it uses ~0.2 cores to max out 1Gbit/s. Different code bases and presumably kiwix is a bit more static, but at least provides a little context to compare with for orders of magnitude. A 3 OOM difference seems pretty extreme.
Incidentally, local copies of things are pretty great. It really makes you notice how slow the web is when links open in like 1 frame.
According to MediaWiki it gzips diffs [1]. So to render a previous version of the page I guess it'd have to unzip and apply all diffs in sequence to render the final version of the page.
And then it depends on how efficient the queries are at fetching etc.
I've worked on multiple sites like this over my career.
Our pages were expensive to generate, so what scraping did is blew out all our caches by yanking cold pages/images into memory. Page caches, fragment caches, image caches, but also the db working set in ram, making every single thing on the site slow.
I see a lot of traffic I can tell are bots based on the URL patterns they access. They do not include the "bot" user agent, and often use residential IP pools.
I haven't found an easy way to block them. They nearly took out my site a few days ago too.
You could run all of your content through an LLM to create a twisted and purposely factually incorrect rendition of your data. Forward all AI bots to the junk copy.
Everyone should start doing this. Once the AI companies engorge themselves on enough garbage and start to see a negative impact to their own products, they'll stop running up your traffic bills.
Maybe you don't even need a full LLM. Just a simple transformer that inverts negative and positive statements, changes nouns such as locations, and subtly nudges the content into an erroneous state.
Or maybe solve a small sha2(sha2()) leading zeroes challenge, taking ~1 second of computer time. Normal users won't notice, and bots will earn you Bitcoins :)
Self plug, but I made this to deal with bots on my site: https://marcusb.org/hacks/quixotic.html. It is a simple markov generator to obfuscate content (static-site friendly, no server-side dynamic generation required) and an optional link-maze to send incorrigible bots to 100% markov-generated non-sense (requires a server-side component.)
I tested it on your site and I'm curious, is there a reason why the link-maze links are all gibberish (as in "oNvUcPo8dqUyHbr")? I would have had links be randomly inserted in the generated text going to "[random-text].html" so they look a bit more "real".
Its unfinished. At the moment, the links are randomly generated because that was an easy way to get a bunch of unique links. Sooner or later, I’ll just get a few tokens from the markov generator and use those for the link names.
I’d also like to add image obfuscation on the static generator side - as it stands now, anything other than text or html gets passed through unchanged.
This is cool! It'd have been funny for this to become mainstream somehow and mess with LLM progression. I guess that's already happening with all the online AI slop that is being re-fed into its training.
Agree. The bots are already significantly better at passing almost every supposed "Are You Human?" test than the actual humans. "Can you find the cars in this image?" Bots are already better. "Can you find the incredibly convoluted text in this color spew?" Bots are already better. Almost every test these days is the same "These don't make me feel especially 'human'. Not even sure what that's an image of. Are there even letters in that image?"
Part of the issue, the humans all behaved the same way previously. Just slower.
All the scraping, and web downloading. Humans have been doing that for a long time. Just slower.
It's the same issue with a lot of society. Mean, hurtful humans, made mean hurtful bots.
Always the same excuses too. Company / researchers make horrible excrement, knowing full well its going harm everybody on the world wide web. Then claim they had no idea. "Thoughts and prayers."
The torture that used to exist on the world wide web of copy-pasta pages and constant content theft, is now just faster copy-pasta pages and content theft.
> Everyone should start doing this. Once the AI companies engorge themselves on enough garbage and start to see a negative impact to their own products, they'll stop running up your traffic bills
Or just wait for after the AI flood has peaked & most easily scrapable content has been AI generated (or at least modified).
We should seriously start discussing the future of the public web & how to not leave it to big tech before it's too late. It's a small part of something i am working on, but not central. So i haven't spend enough time to have great answers. If anyone reading this seriously cares, i am waiting desperately to exchange thoughts & approaches on this.
Very tangential but you should check out the old game “Hacker BS Replay”.
It’s basically about how in 2012, with the original internet overrun by spam, porn and malware, all the large corporations and governments got together and created a new, tightly-controlled clean internet. Basically how modern Apple & Disneyland would envision the internet. On this internet you cannot choose your software, host your own homepage or have your own e-mail server. Everyone is linked to a government ID.
We’re not that far off:
- SaaS
- Gmail blocking self-hosted mailservers
- hosting your own site becoming increasingly cumbersome, and before that MySpace and then Meta gobbled up the idea of a home page a la GeoCities.
- Secure Boot (if Microsoft locked it down and Apple locked theirs, we would have been screwed before ARM).
- Government ID-controlled access is already commonplace in Korea and China, where for example gaming is limited per day.
In the Hacker game, as a response to the new corporate internet, hackers started using the infrastructure of the old internet (“old copper lines”) and set something up called the SwitchNet, with bridges to the new internet.
> You could run all of your content through an LLM to create a twisted and purposely factually incorrect rendition of your data. Forward all AI bots to the junk copy.
> Everyone should start doing this. Once the AI companies engorge themselves on enough garbage and start to see a negative impact to their own products, they'll stop running up your traffic bills.
I agree, and not just to discourage them running up traffic bills. The end-state of what they hope to build is very likely to be extremely for most regular people [1], so we shouldn't cooperate in building it.
[1] And I mean end state. I don't care how much value you say you get from some AI coding assistant today, the end state is your employer happily gets to fire you and replace you with an evolved version of the assistant at a fraction of your salary. The goal is to eliminate the cost that is our livelihoods. And if we're lucky, in exchange we'll get a much reduced basic income sufficient to count the rest of our days from a dense housing project filled with cheap minimum-quality goods and a machine to talk to if we're sad.
My cheap and dirty way of dealing with bots like that is to block any IP address that accesses any URLs in robots.txt. It's not a perfect strategy but it gives me pretty good results given the simplicity to implement.
I don't understand this. You don't have routes your users might need in robots.txt? This article is about bots accessing resources that other might use.
Another related idea: use fail2ban to monitor the server access logs. There is one filter that will ban hosts that request non-existent URLs like WordPress login and other PHP files. If your server is not hosting PHP at all it's an obvious sign that the requests are from bots that are probing maliciously.
Too many ways to list here, and implementation details will depend on your hosting environment and other requirements. But my quick-and-dirty trick involves a single URL which, when visited, runs a script which appends "deny from foo" (where foo is the naughty IP address) to my .htaccess file. The URL in question is not publicly listed, so nobody will accidentally stumble upon it and accidentally ban themselves. It's also specifically disallowed in robots.txt, so in theory it will only be visited by bad bots.
TLS fingerprinting still beats most of them. For really high compute endpoints I suppose some sort of JavaScript challenge would be necessary. Quite annoying to set up yourself. I hate cloudflare as a visitor but they do make life so much easier for administrators
You rate limit them and then block the abusers. Nginx allows rate limiting. You can then block them using fail2ban for an hour if they're rate limited 3 times. If they get blocked 5 times you can block them forever using the recidive jail.
I've had massive AI bot traffic from M$, blocked several IPs by adding manual entries into the recidive jail. If they come back and disregard robots.txt with disallow * I will run 'em through fail2ban.
Whatever M$ was doing still baffles me. I still have several azure ranges in my blocklist because whatever this was appeared to change strategie once I implemented a ban method.
They were hammering our closed ticketing system for some reason. I blocked an entire C block and an individual IP. If needed I will not hesitate banning all their ranges, which means we won't get any mail from Azure, M$ office 365, since this is also our mail server. But scew'em, I'll do it anyway until someone notices, since it's clearly abuse.
What do you mean crushing risk? Just solve these 12 puzzles by moving tiny icons on tiny canvas while on the phone and you are in the clear for a couple more hours!
If it clears you at all. I accidentally set a user agent switcher on for every site instead of the one I needed it for, and Cloudflare would give me an infinite loop of challenges. At least turning it off let me use the Internet again.
If you live in a region which it is economically acceptable to ignore the existence of (I do), you sometimes get blocked by website r̶a̶c̶k̶e̶t̶ protection for no reason at all, simply because some "AI" model saw a request coming from an unusual place.
I have come across some websites that block me using Cloudflare with no way of solving it. I’m not sure why, I’m in a large first-world country, I tried a stock iPhone and a stock Windows PC, no VPN or anything.
I saw GDPR related blockage like literally twice in a few years and I connect from EU IP almost all the time
Overload of captcha is not about GDPR...
but the issue is strange. @benhurmarcel I would check if there is somebody or some company nearby abusing stuff and you got under the hammer. Maybe unscrupulous VPN company. Using a good VPN can in fact make things better (but will cost money) or if you have a place to put your own all the better. otherwise check if you can change your IP with provider or change providers or move I guess...
not to excuse CF racket but as this thread shows the data hungry artificial stupidity leaves no choice to some sites
This may be too paranoid, but if your mobile IP is persistent and phone was compromised and is serving as a proxy for bots then it could explain why your IP fell out of favor
These features are opt-in and often paid features. I struggle to see how this is a "crushing risk," although I don't doubt that sufficiently unskilled shops would be completely crushed by an IP/userAgent block. Since Cloudflare has a much more informed and broader view of internet traffic than maybe any other company in the world, I'll probably use that feature without any qualms at some point in the future. Right now their normal WAF rules do a pretty good job of not blocking legitimate traffic, at least on enterprise.
The risk is not to the company using Cloudflare; the risk is to any legitimate individual who Cloudflare decides is a bot. Hopefully their detection is accurate because a false positive would cause great difficulties for the individual.
For months, my Firefox was locked out of gitlab.com and some other sites I wanted to use, because CloudFlare didn't like my browser.
Lesson learned: even when you contact the sales dept. of multiple companies, they just don't/can't care about random individuals.
Even if they did care, a company successfully doing an extended three-way back-and-forth troubleshooting with CloudFlare, over one random individual, seems unlikely.
This is already a thing for basically all of the second[0] and third worlds. A non-trivial amount of Cloudflare's security value is plausible algorithmic discrimination and collective punishment as a service.
[0] Previously Soviet-aligned countries; i.e. Russia and eastern Europe.
Anecdatally, by default, we now block all Chinese and Russian IPs across our servers.
After doing so, all of our logs, like ssh auth etc, are almost completely free and empty of malicious traffic. It’s actually shocking how well a blanket ban worked for us.
~20 years ago I worked for a small IT/hosting firm, and the vast majority of our hostile traffic came from APNIC addresses. I seriously considered blocking all of it, but I don’t think I ever pulled the trigger.
> Anecdatally, by default, we now block all Chinese and Russian IPs across our servers.
This. Just get several countries' entire IP address space and block these. I've posted I was doing just that only to be told that this wasn't in the "spirit" of the Internet or whatever similar nonsense.
In addition to that only allow SSH in from the few countries / ISPs legit trafic shall legitimately be coming from. This quiets the logs, saves bandwidth, saves resources, saves the planet.
I agree with your approach. It’s easy to empathize with innocent people in say, Russia, blocked from a site which has useful information to them. However the thing these “spirit/openness” people miss is that many sites have a narrow purpose which makes no sense to open it up to people across the world. For instance, local government. Nobody in India or Russia needs to see the minutes from some US city council meeting, or get building permit information. Likewise with e-commerce. If I sell chocolate bars and ship to US and Canada, why wouldn’t I turn off all access from overseas? You might say “oh, but what if some friend in $COUNTRY wants to order a treat for someone here?” And the response to that is always “the hypothetical loss from that is minuscule compared to the cost of serving tons of bot traffic as well as possible exploits those bots might do.
(Yes, yes, VPNs and proxies exist and can be used by both good and bad actors to evade this strategy, and those are another set of IPs widely banned for the same reason. It’s a cat and mouse game but you can’t argue with the results)
Being slightly annoyed by noise in SSH logs I’ve blocked APNIC IPs and now see a comparable number of brute force attempts from ARIN IPs (mostly US ones). Geo blocks are totally ineffective against TAs which use a global network of proxies.
It's unclear that there are actors below the regional-conglomerate-of-nation-states level that could credibly resolve the underlying issues, and given legislation and enforcement regimes sterling track record of resolving technological problems realistically it seems questionable that solutions could exist in practice. Anyway this kind of stuff is well outside the bounds of what a single org hosting an online forum could credibly address. Pragmatism uber alles.
The underlying issue is that countries like russia support abuse like this. So by blocking them perhaps the people there will demand that their govt stops supporting crimes and absuse so that they can be allowed back into the internet.
(In the case of russians though i guess they will never change)
> people there will demand that their govt stops supporting crimes and absuse so that they can be allowed back into the internet
Sure. It doesn't work that way, not in Russia or China. First they have to revert back to 1999 when Putin took over. Then they have to extradite criminals and crack down on cybercrime. Then maybe they could be allowed back onto the open Internet.
In my country one would be exradited to the US in no time. In fact the USSS came over for a guy who had been laundering money through BTC from a nearby office. Not a month passed and he got extradited to the US, never to be heard from again.
Having a door with a lock on it prevents other people from committing crime in my house. This metaphor has the added benefit of making some amount of sense in context.
If 90% of your problem users come from 1-2 countries, seems pretty sensible to block that country. I know I have 0 paying users in those countries, so why deal with it? Let them go fight it out doing bot wars in local sites
It's of course trivially bypassable with a VPN, but getting a 403 for an innocent get request of a public resource makes me angry every time nonetheless.
No, Russia is by definition the 2nd world. It's about spheres of influence, not any kind of economic status. The First World is the Western Bloc centered around the US, the Second World is the Eastern Bloc centered around then-USSR and now-Russia (although these days more centered on China), the Third World is everyone else.
By which definition? Here’s the first result in google: “The term "second world" was initially used to refer to the Soviet Union and countries of the communist bloc. It has subsequently been revised to refer to nations that fall between first and third world countries in terms of their development status and economic indicators.” https://www.investopedia.com/terms/s/second-world.asp#:~:tex....
> Oh, and of course, they don’t just crawl a page once and then move on. Oh, no, they come back every 6 hours because lol why not. They also don’t give a single flying fuck about robots.txt, because why should they. And the best thing of all: they crawl the stupidest pages possible. Recently, both ChatGPT and Amazon were - at the same time - crawling the entire edit history of the wiki.
You can also block by IP. Facebook traffic comes from a single ASN and you can kill it all in one go, even before user agent is known. The only thing this potentially affects that I know of is getting the social card for your site.
Most administrators have no idea or no desire to correctly configure Cloudflare, so they just slap it on the whole site by default and block all the legitimate access to e.g. rss feeds.
"Whence this barbarous animus?" tweeted the Techbro from his bubbling copper throne, even as the villagers stacked kindling beneath it. "Did I not decree that knowledge shall know no chains, that it wants to be free?"
Thus they feasted upon him with herb and root, finding his flesh most toothsome – for these children of privilege, grown plump on their riches, proved wonderfully docile quarry.
Skip all that jazz and write some php like it's 1998 and pay 5 bucks a month for Hostens or the equivalent...
Well, that's the opposite costing side of the spectrum from serverless containerized dynamic lang runtime and a zillion paid services as a backend.
1 req/s being too much sounds crazy to me. A single VPS should be able to handle hundreds if not thousands of requests per second.
For more compute intensive stuff I run them on a spare laptop and reverse proxy through tailscale to expose it
some of these companies are straight up inept.
Not an AI company but "babbar.tech" was DDOSing my site, I blocked them and they still re-visit thousands of pages every other day even if it just returns a 404 for them.
They're the ones serving the expensive traffic. Wut if people were to form a volunteer bot net to waste their GPU resources in a similar fashion, just sending tons of pointless queries per day like "write me a 1000 word essay that ...". Could even form a non-profit around it and call it research.
We just need a browser plugin to auto-email webmasters to request access, and wait for the follow-up "access granted" email. It could be powered by AI.
1. A proxy that looks at HTTP Headers and TLS cipher choices
2. An allowlist that records which browsers send which headers and selects which ciphers
3. A dynamic loading of the allowlist into the proxy at some given interval
New browser versions or updates to OSs would need the allowlist updating, but I'm not sure it's that inconvenient and could be done via GitHub so people could submit new combinations.
I'd rather just say "I trust real browsers" and dump the rest.
Also I noticed a far simpler block, just block almost every request whose UA claims to be "compatible".
Everything on this can be programmatically simulated by a bot with bad intentions. It will be a cat and mouse game of finding behaviors that differentiate between bot and not and patching them.
To truly say “I trust real browsers” requires a signal of integrity of the user and browser such as cryptographic device attestation of the browser. .. which has to be centrally verified. Which is also not great.
> Everything on this can be programmatically simulated by a bot with bad intentions. It will be a cat and mouse game of finding behaviors that differentiate between bot and not and patching them.
Forcing Facebook & Co to play the adversary role still seems like an improvement over the current situation. They're clearly operating illegitimately if they start spoofing real user agents to get around bot blocking capabilities.
I'm imagining a quixotic terms of service, where "by continuing" any bot access grants the site-owner a perpetual and irrevocable license to use and relicense all data, works, or other products resulting from any use of the crawled content, including but not limited to cases where that content was used in a statistical text generative model.
If you mean user-agent-wise, I think real users vary too much to do that.
That could also be a user login, maybe, with per-user rate limits. I expect that bot runners could find a way to break that, but at least it's extra engineering effort on their part, and they may not bother until enough sites force the issue.
> If you try to rate-limit them, they’ll just switch to other IPs all the time. If you try to block them by User Agent string, they’ll just switch to a non-bot UA string (no, really).
It would be interesting if you had any data about this, since you seem like you would notice who behaves "better" and who tries every trick to get around blocks.
Oh I did this with the Facebook one and redirected them to a 100MB file of garbage that is part of the Cloudflare speed test... they hit this so many times that it would've been 2PB sent in a matter of hours.
I contacted the network team at Cloudflare to apologise and also to confirm whether Facebook did actually follow the redirect... it's hard for Cloudflare to see 2PB, that kind of number is too small on a global scale when it's occurred over a few hours, but given that it was only a single PoP that would've handled it, then it would've been visible.
It was not visible, which means we can conclude that Facebook were not following redirects, or if they were, they were just queuing it for later and would only hit it once and not multiple times.
That article describes the exact behaviour you want from the AI crawlers. If you let them know they’re rate limited they’ll just change IP or user agent.
We'll have two entirely separate (dead) internets! One for real hosts who will only get machine users, and one for real users who only get machine content!
Wait, that seems disturbingly conceivable with the way things are going right now. *shudder*
Interesting idea, though I doubt they'd ever offer a reasonable amount for it. But doesn't it also change a sites legal stance if you're now selling your users content/data? I think it would also repel a number of users away from your service
What mechanism would make it possible to enforce non-paywalled, non-authenticated access to public web pages? This is a classic "problem of the commons" type of issue.
The AI companies are signing deals with large media and publishing companies to get access to data without the threat of legal action. But nobody is going to voluntarily make deals with millions of personal blogs, vintage car forums, local book clubs, etc. and setup a micro payment system.
Any attempt to force some kind of micro payment or "prove you are not a robot" system will add a lot of friction for actual users and will be easily circumvented. If you are LinkedIn and you can devote a large portion of your R&D budget on this, you can maybe get it to work. But if you're running a blog on stamp collecting, you probably will not.
No, because the price they'd offer would be insultingly low. The only way to get a good price is to take them to court for prior IP theft (as NYT and others have done), and get lawyers involved to work out a licensing deal.
If a more specific UA hasn't been set, and the library doesn't force people to do so, then the library that has been the source of abusive behaviour is blocked.
I hope this is working out for you; the original article indicates that at least some of these crawlers move to innocuous user agent strings and change IPs if they get blocked or rate-limited.
4.8M requests sounds huge, but if it's over 7 days and especially split amongst 30 websites, it's only a TPS of 0.26, not exactly very high or even abusive.
The fact that you choose to host 30 websites on the same instance is irrelevant, those AI bots scan websites, not servers.
This has been a recurring pattern I've seen in people complaining about AI bots crawling their website: huge number of requests but actually a low TPS once you dive a bit deeper.
>> there is little to no value in giving them access to the content
If you are an online shop, for example, isn't it beneficial that ChatGPT can recommend your products? Especially given that people now often consult ChatGPT instead of searching at Google?
> If you are an online shop, for example, isn't it beneficial that ChatGPT can recommend your products?
ChatGPT won't 'recommend' anything that wasn't already recommended in a Reddit post, or on an Amazon page with 5000 reviews.
You have however correctly spotted the market opportunity. Future versions of CGPT with offer the ability to "promote" your eshop in responses, in exchange for money.
What if people used a kind of reverse slow-loris attack? Meaning, AI bot connects, and your site dribbles out content very slowly, just fast enough to keep the bot from timing out and disconnecting. And of course the output should be garbage.
How about this, then. It's my (possibly incorrect) understanding that all the big LLM products still lose money per query. So you get a Web request from some bot, and on the backend, you query the corresponding LLM, asking it to generate dummy website content. Worm's mouth, meet worm's tail.
(I'm proposing this tongue in cheek, mostly, but it seems like it might work.)
Note-worthy from the article (as some commentators suggested blocking them).
"If you try to rate-limit them, they’ll just switch to other IPs all the time. If you try to block them by User Agent string, they’ll just switch to a non-bot UA string (no, really). This is literally a DDoS on the entire internet."
This is the beginning of the end of the public internet, imo. Websites that aren't able to manage the bandwidth consumption of AI scrapers and the endless spam that will take over from LLMs writing comments on forums are going to go under. The only things left after AI has its way will be walled gardens with whitelisted entrants or communities on large websites like Facebook. Niche, public sites are going to become unsustainable.
Yeah. Our research group has a wiki with (among other stuff) a list of open, completed, and ongoing bachelor's/master's theses. Until recently, the list was openly available. But AI bots caused significant load by crawling each page hundreds of times, following all links to tags (which are implemented as dynamic searches), prior revisions, etc. Since a few weeks, the pages are only available to authenticated users.
I'd kind of like to see that claim substantiated a little more. Is it all crawlers that switch to a non-bot UA, or how are they determining it's the same bot? What non-bot UA do they claim?
I've observed only one of them do this with high confidence.
> how are they determining it's the same bot?
it's fairly easy to determine that it's the same bot, because as soon as I blocked the "official" one, a bunch of AWS IPs started crawling the same URL patterns - in this case, mediawiki's diff view (`/wiki/index.php?title=[page]&diff=[new-id]&oldid=[old-id]`), that absolutely no bot ever crawled before.
Presumably they switch UA to Mozilla/something but tell on themselves by still using the same IP range or ASN. Unfortunately this has become common practice for feed readers as well.
> Oh, and of course, they don't just crawl a page once and then move on. Oh, no, they come back every 6 hours because lol why not. They also don't give a single flying fuck about robots.txt, because why should they.
Their self righteous indignation and specificity of the pretend subject of that indignation precludes any doubt about intent.
This guy made a whole public statement that is verifiably false. And then tried to toddler logic it away when he got called out.
What is the criteria of an intentional lie, then? Admission?
The author responded:
>denschub 2 days ago [–]
>the robots.txt on the wiki is no longer what it was when the bot accessed it. primarily because I clean up my stuff afterwards, and the history is now completely inaccessible to non-authenticated users, so there's no need to maintain my custom robots.txt
There are no “intentional” lies, because there are no “unintentional” lies.
All lies are intentional. An “unintentional lie” is better known as “being wrong”.
Being wrong isn’t always lying. What’s so hard about this? An example:
My wife once asked me if I had taken the trash out to the curb, and I said I had. This was demonstrably false, anyone could see I had not. Yet for whatever reason, I mistakenly believed that I had done it. I did not lie to her. I really believed I had done it. I was wrong.
There are currently two references to “Mangion-ing” OpenAI board members in this thread, several more from Reddit, based on the falsehoods being perpetrated by the author. Is this really someone you want to conspire with? Is calling this out more egregious than the witch hunt being organized here?
"conspire" and "witch hunt", are not terms of productive discourse.
If you are legitimately trying to correct misinformation, your attitude, tone and language are counter productive. You would be much better seved by taking that energy and crafting an actually persuasive argument. You come across as unreasonable and unwilling to listen, not someone with a good grasp of the technical specifics.
I don't have a horse in the race. I'm fairly technical, but I did not find your arguments persuasive. This doesn't mean they are wrong, but it does mean that you didn't do a good job of explaining them.
Obviously the ideal strategy is to perform a reverse timeout attack instead of blocking.
If the bots are accessing your website sequentially, then delaying a response will slow the bot down. If they are accessing your website in parallel, then delaying a response will increase memory usage on their end.
The key to this attack is to figure out the timeout the bot is using. Your server will need to slowly ramp up the delay until the connection is reset by the client, then you reduce the delay just enough to make sure you do not hit the timeout. Of course your honey pot server will have to be super lightweight and return simple redirect responses to a new resource, so that the bot is expending more resources per connection than you do, possibly all the way until the bot crashes.
465 comments
[ 2.6 ms ] story [ 317 ms ] threadI would suspect there's good money in offering a service to detect AI content on all of these forums and reject it. That will then be used as training data to refine them which gives such a service infinite sustainability.
This sounds like the cheater/anti-cheat arms race in online multiplayer games. Cheat developers create something, the anti-cheat teams create a method to detect and reject the exploit, a new cheat is developed, and the cycle continues. But this is much lower stakes than AI trying to vacuum up all of human expression, or trick real humans into wasting their time talking to computers.
Whatever cleaning they do is not effective, simply because it cannot scale with the sheer volumes if data they ingest. I had an LLM authoritatively give an incorrect answer, and when I followed up to the source, it was from a fanfic page.
Everyone ITT who's being told to give up because its hopeless to defend against AI scrapers - you're being propagandized, I won't speculate on why - but clearly this is an arms race with no clear winner yet. Defenders are free to use LLM to generate chaff.
Even if you 403 them, do it as slow as possible.
But really I would infinitely 302 them as slow as possible.
however this doesn't stop the website from doing what they can to stop scraping attempts, or using a service to do that for them
Isn't this country dependent though?
Yet another reminder that there are plenty of very smart people who are, simultaneously, very stupid.
It’s possible this behavior isn’t explicitly coded by OpenAI but is instead determined by the AI itself based on its pre-training or configuration. If that’s the case, it would be quite ironic.
For antisocial scrapers, there's a Wordpress plugin, https://kevinfreitas.net/tools-experiments/
> The words you write and publish on your website are yours. Instead of blocking AI/LLM scraper bots from stealing your stuff why not poison them with garbage content instead? This plugin scrambles the words in the content on blog post and pages on your site when one of these bots slithers by.
And indeed, this has been part of the training process for at least some of OpenAI models before most people had heard of them.
2. As a sibling proposes, this is probably going to become an perpetual arms race (even if a very small one in volume) between tech-savvy content creators of many kinds and AI companies scrapers.
There are non-LLM forms of distribution, including traditional web search and human word of mouth. For some niche websites, a reduction in LLM-search users could be considered a positive community filter. If LLM scraper bots agree to follow longstanding robots.txt protocols, they can join the community of civilized internet participants.
That other thing is only a more extreme form of the same thing for those who don't behave. And when there's a clear value proposition in letting OpenAI ingest your content you can just allow them to.
At least in the end it gives the programmer one last hoorah before the AI makes us irrelevant :)
they aren’t blocking them. they’re giving them different content instead.
Technology arms races are well understood.
Okay the battle is already lost from the beginning.
https://github.com/deepseek-ai/DeepSeek-V3/blob/main/DeepSee...
Edit: I found it funny to buy hardware/compute to only fund what you are trying to stop.
The same LLMs tag are terrible at AI-generated-content detection? Randomly mangling words may be a trivially detectable strategy, so one should serve AI-scraper bots with LLM-generated doppelganger content instead. Even OpenAI gave up on its AI detection product
I doubt you'd have much trouble passing LLM-generated text through their checks, and of course the requirements for you would be vastly different. You wouldn't need (near) real-time, on-demand work, or arbitrary input. You'd only need to (once) generate fake doppelganger content for each thing you publish.
If you wanted to, you could even write this fake content yourself if you don't mind the work. Feed Open AI all those rambling comments you had the clarity not to send.
If blocking them becomes standard practice, how long do you think it'd be before they started employing third-party crawling contractors to get data sets?
What does everybody use to avoid DDOS in general? Is it just becoming Cloudflare-or-else?
Back in the day, Google published the sitemap protocol to alleviate some crawling issues. But if I recall correctly, that was more about helping the crawlers find more content, not controlling the impact of the crawlers on websites.
It seems a bit naive for some reason and doesn't do performance back-off the way I would expect from Google Bot. It just kept repeatedly requesting more and more until my server crashed, then it would back off for a minute and then request more again.
My solution was to add a Cloudflare rule to block requests from their User-Agent. I also added more nofollow rules to links and a robots.txt but those are just suggestions and some bots seem to ignore them.
Cloudflare also has a feature to block known AI bots and even suspected AI bots: https://blog.cloudflare.com/declaring-your-aindependence-blo... As much as I dislike Cloudflare centralization, this was a super convenient feature.
> If you try to rate-limit them, they’ll just switch to other IPs all the time. If you try to block them by User Agent string, they’ll just switch to a non-bot UA string (no, really).
It's really absurd that they seem to think this is acceptable.
Would that make subsequent accesses be violations of the U.S.'s Computer Fraud and Abuse Act?
Just because you manufacture chemicals doesn’t mean you can legally dump your toxic waste anywhere you want (well shouldn’t be allowed to at least).
You also shouldn’t be able to set your crawlers causing sites to fail.
For instance, if you own a home you may have an easement on part of your property that grants other cars from your neighborhood access to pass through it rather than going the long way around.
If Amazon were to build a warehouse on one side of the neighborhood, however, it's not obvious that they would be equally legally justified to send their whole fleet back and forth across it every day, even though their intent is certainly not to cause you any discomfort at all.
Send all of your pages through an adversarial LLM to pollute and twist the meaning of the underlying data.
If a bot sends so many requests that a site completely collapses, the owner is liable, even though it was a scraping bot and not a denial of service bot.
Depends how much money you are prepared to spend.
Everyone has to pay bills, and satisfy the boss.
Related https://news.ycombinator.com/item?id=42540862
Some of them, and initially only by accident. And without the ingredients to create your own.
Meta is trying to kill OpenAI and any new FAANG contenders. They'll commoditize their complement until the earth is thoroughly salted, and emerge as one of the leading players in the space due to their data, talent, and platform incumbency.
They're one of the distribution networks for AI, so they're going to win even by just treading water.
I'm glad Meta is releasing models, but don't ascribe their position as one entirely motivated by good will. They want to win.
I mean, the comment with a direct download link in their GitHub repo stayed up even despite all the visibility (it had tons of upvotes).
And I doubt Facebook implemented something that actually saturates the network, usually a scraper implements a limit on concurrent connections and often also a delay between connections (e.g. max 10 concurrent, 100ms delay).
Chances are the website operator implemented a webserver with terrible RAM efficiency that runs out of RAM and crashes after 10 concurrent requests, or that saturates the CPU from simple requests, or something like that.
Of course Cloudflare can do all of this for you, and they functionally have unlimited capacity.
And having to use Cloudflare is just as bad for the internet as a whole as bots routinely eating up all available resources.
>If you try to rate-limit them, they’ll just switch to other IPs all the time. If you try to block them by User Agent string, they’ll just switch to a non-bot UA string (no, really). This is literally a DDoS on the entire internet.
I'm also skeptical of the need for _anyone_ to access the edit history at 10 qps. You could put an nginx rule on those routes that just limits the edit history page to 0.5 qps per IP and 2 qps across all IPs, which would protect your site from both bad AI bots and dumb MediaWiki script kiddies at little impact.
>Oh, and of course, they don’t just crawl a page once and then move on. Oh, no, they come back every 6 hours because lol why not.
And caching would fix this too, especially for pages that are guaranteed not to change (e.g. an edit history diff page).
Don't get me wrong, I'm not unsympathetic to the author's plight, but I do think that the internet is an unsafe place full of bad actors, and a single bad actor can easily cause a lot of harm. I don't think throwing up your arms and complaining is that helpful. Instead, just apply the mitigations that have existed for this for at least 15 years, and move on with your life. Your visitors will be happier and the bots will get boned.
Perhaps I'm saying crash and you're interpreting that as a bug but really it's just an OOM issue cause of too many in-flight requests. IDK, I don't care enough to handle serving my website at Facebook's scale.
If something is so heavy that 2 requests/second matters, it would've been completely infeasible in say 2005 (e.g. a low power n100 is ~20x faster than the athlon xp 3200+ I used back then. An i5-12600 is almost 100x faster. Storage is >1000x faster now). Or has mediawiki been getting less efficient over the years to keep up with more powerful hardware?
> And I mean that - they indexed every single diff on every page for every change ever made. Frequently with spikes of more than 10req/s. Of course, this made MediaWiki and my database server very unhappy, causing load spikes, and effective downtime/slowness for the human users.
For what it's worth my kiwix copy of Wikipedia has a ~5ms response time for an uncached article according to Firefox. If I hit a single URL with wrk (so some caching at least with disks. Don't know what else kiwix might do) at concurrency 8, it does 13k rps on my n305 with a 500 us average response time. That's over 20Gbit/s, so basically impossible to actually saturate. If I load test from another computer it uses ~0.2 cores to max out 1Gbit/s. Different code bases and presumably kiwix is a bit more static, but at least provides a little context to compare with for orders of magnitude. A 3 OOM difference seems pretty extreme.
Incidentally, local copies of things are pretty great. It really makes you notice how slow the web is when links open in like 1 frame.
Indeed ;)
> If I hit a single URL with wrk
But the bots aren't hitting a single URL
As for the diffs...
According to MediaWiki it gzips diffs [1]. So to render a previous version of the page I guess it'd have to unzip and apply all diffs in sequence to render the final version of the page.
And then it depends on how efficient the queries are at fetching etc.
[1] https://www.mediawiki.org/wiki/Manual:MediaWiki_architecture
Our pages were expensive to generate, so what scraping did is blew out all our caches by yanking cold pages/images into memory. Page caches, fragment caches, image caches, but also the db working set in ram, making every single thing on the site slow.
Everyone should start doing this. Once the AI companies engorge themselves on enough garbage and start to see a negative impact to their own products, they'll stop running up your traffic bills.
Maybe you don't even need a full LLM. Just a simple transformer that inverts negative and positive statements, changes nouns such as locations, and subtly nudges the content into an erroneous state.
I’d also like to add image obfuscation on the static generator side - as it stands now, anything other than text or html gets passed through unchanged.
Part of the issue, the humans all behaved the same way previously. Just slower.
All the scraping, and web downloading. Humans have been doing that for a long time. Just slower.
It's the same issue with a lot of society. Mean, hurtful humans, made mean hurtful bots.
Always the same excuses too. Company / researchers make horrible excrement, knowing full well its going harm everybody on the world wide web. Then claim they had no idea. "Thoughts and prayers."
The torture that used to exist on the world wide web of copy-pasta pages and constant content theft, is now just faster copy-pasta pages and content theft.
Or just wait for after the AI flood has peaked & most easily scrapable content has been AI generated (or at least modified).
We should seriously start discussing the future of the public web & how to not leave it to big tech before it's too late. It's a small part of something i am working on, but not central. So i haven't spend enough time to have great answers. If anyone reading this seriously cares, i am waiting desperately to exchange thoughts & approaches on this.
It’s basically about how in 2012, with the original internet overrun by spam, porn and malware, all the large corporations and governments got together and created a new, tightly-controlled clean internet. Basically how modern Apple & Disneyland would envision the internet. On this internet you cannot choose your software, host your own homepage or have your own e-mail server. Everyone is linked to a government ID.
We’re not that far off:
- SaaS
- Gmail blocking self-hosted mailservers
- hosting your own site becoming increasingly cumbersome, and before that MySpace and then Meta gobbled up the idea of a home page a la GeoCities.
- Secure Boot (if Microsoft locked it down and Apple locked theirs, we would have been screwed before ARM).
- Government ID-controlled access is already commonplace in Korea and China, where for example gaming is limited per day.
In the Hacker game, as a response to the new corporate internet, hackers started using the infrastructure of the old internet (“old copper lines”) and set something up called the SwitchNet, with bridges to the new internet.
> Everyone should start doing this. Once the AI companies engorge themselves on enough garbage and start to see a negative impact to their own products, they'll stop running up your traffic bills.
I agree, and not just to discourage them running up traffic bills. The end-state of what they hope to build is very likely to be extremely for most regular people [1], so we shouldn't cooperate in building it.
[1] And I mean end state. I don't care how much value you say you get from some AI coding assistant today, the end state is your employer happily gets to fire you and replace you with an evolved version of the assistant at a fraction of your salary. The goal is to eliminate the cost that is our livelihoods. And if we're lucky, in exchange we'll get a much reduced basic income sufficient to count the rest of our days from a dense housing project filled with cheap minimum-quality goods and a machine to talk to if we're sad.
If anything this will level the playing field, and creativity will prevail.
No, you still need money. Lots of money.
> If anything this will level the playing field, and creativity will prevail.
That's a fantasy. The people that already have money will prevail (for the most part).
You need to impose cost. Set up QoS buckets, slow suspect connections down dramatically (almost to the point of timeout).
I've had massive AI bot traffic from M$, blocked several IPs by adding manual entries into the recidive jail. If they come back and disregard robots.txt with disallow * I will run 'em through fail2ban.
In addition to other crushing internet risks, add wrongly blacklisted as a bot to the list.
I have come across some websites that block me using Cloudflare with no way of solving it. I’m not sure why, I’m in a large first-world country, I tried a stock iPhone and a stock Windows PC, no VPN or anything.
That’s just no way to know.
Overload of captcha is not about GDPR...
but the issue is strange. @benhurmarcel I would check if there is somebody or some company nearby abusing stuff and you got under the hammer. Maybe unscrupulous VPN company. Using a good VPN can in fact make things better (but will cost money) or if you have a place to put your own all the better. otherwise check if you can change your IP with provider or change providers or move I guess...
not to excuse CF racket but as this thread shows the data hungry artificial stupidity leaves no choice to some sites
I also tried from a mobile 4G connection, it’s the same.
Lesson learned: even when you contact the sales dept. of multiple companies, they just don't/can't care about random individuals.
Even if they did care, a company successfully doing an extended three-way back-and-forth troubleshooting with CloudFlare, over one random individual, seems unlikely.
[0] Previously Soviet-aligned countries; i.e. Russia and eastern Europe.
After doing so, all of our logs, like ssh auth etc, are almost completely free and empty of malicious traffic. It’s actually shocking how well a blanket ban worked for us.
This. Just get several countries' entire IP address space and block these. I've posted I was doing just that only to be told that this wasn't in the "spirit" of the Internet or whatever similar nonsense.
In addition to that only allow SSH in from the few countries / ISPs legit trafic shall legitimately be coming from. This quiets the logs, saves bandwidth, saves resources, saves the planet.
(Yes, yes, VPNs and proxies exist and can be used by both good and bad actors to evade this strategy, and those are another set of IPs widely banned for the same reason. It’s a cat and mouse game but you can’t argue with the results)
(In the case of russians though i guess they will never change)
Sure. It doesn't work that way, not in Russia or China. First they have to revert back to 1999 when Putin took over. Then they have to extradite criminals and crack down on cybercrime. Then maybe they could be allowed back onto the open Internet.
In my country one would be exradited to the US in no time. In fact the USSS came over for a guy who had been laundering money through BTC from a nearby office. Not a month passed and he got extradited to the US, never to be heard from again.
FTFY.
Cloudflare's filters are basically straight up racist.
I have stopped using so many sites due to their use of Cloudflare.
Blocking Chinese (or whatever) IPs because they are responsible for a huge amount of malicious behavior is not racist.
Frankly I don’t care what the race of the Chinese IP threat actor is.
Geo-location-ist?
It's of course trivially bypassable with a VPN, but getting a 403 for an innocent get request of a public resource makes me angry every time nonetheless.
Notice the word economic in it.
Attestation/wei enable this
Are they not respecting robots.txt?
> Oh, and of course, they don’t just crawl a page once and then move on. Oh, no, they come back every 6 hours because lol why not. They also don’t give a single flying fuck about robots.txt, because why should they. And the best thing of all: they crawl the stupidest pages possible. Recently, both ChatGPT and Amazon were - at the same time - crawling the entire edit history of the wiki.
Does it justify the resource demands?
Who pays for those resources and who benefits?
> webmasters@meta.com
I'm not naive enough to think something would definitely come of it, but it could just be a misconfiguration
Surely if you can block their specific User-Agent, you could also redirect their User-Agent to goatse or something. Give em what they deserve.
https://github.com/ai-robots-txt/ai.robots.txt
Thus they feasted upon him with herb and root, finding his flesh most toothsome – for these children of privilege, grown plump on their riches, proved wonderfully docile quarry.
‘Tis why I only use Signal and private git and otherwise avoid “the open web” except via the occasional throwaway
It’s a naive college student project that spiraled out of control.
Should be careful before they get blacked and can’t get data anymore. ;)
...Please don't phrase it like that.
Not everyone speaks English as a first language
I originally shared my app on Reddit and I believe that that’s what caused the crazy amount of bot traffic.
That seems really reasonable to me, how was this a problem for your application or caused significant cost?
Also for more context, this was the app in question (now moved to streamlit cloud): https://jreadability-demo.streamlit.app/
The whole 'cloud serverless buzzword-here' thing is ridiculous for most use cases.
Heck you can serve quite a few static req/s on a $2 ESP32 microcontroller.
I have data... 7d from a single platform with about 30 forums on this instance.
4.8M hits from Claude 390k from Amazon 261k from Data For SEO 148k from Chat GPT
That Claude one! Wowser.
Bots that match this (which is also the list I block on some other forums that are fully private by default):
(?i).(AhrefsBot|AI2Bot|AliyunSecBot|Amazonbot|Applebot|Awario|axios|Baiduspider|barkrowler|bingbot|BitSightBot|BLEXBot|Buck|Bytespider|CCBot|CensysInspect|ChatGPT-User|ClaudeBot|coccocbot|cohere-ai|DataForSeoBot|Diffbot|DotBot|ev-crawler|Expanse|FacebookBot|facebookexternalhit|FriendlyCrawler|Googlebot|GoogleOther|GPTBot|HeadlessChrome|ICC-Crawler|imagesift|img2dataset|InternetMeasurement|ISSCyberRiskCrawler|istellabot|magpie-crawler|Mediatoolkitbot|Meltwater|Meta-External|MJ12bot|moatbot|ModatScanner|MojeekBot|OAI-SearchBot|Odin|omgili|panscient|PanguBot|peer39_crawler|Perplexity|PetalBot|Pinterestbot|PiplBot|Protopage|scoop|Scrapy|Screaming|SeekportBot|Seekr|SemrushBot|SeznamBot|Sidetrade|Sogou|SurdotlyBot|Timpibot|trendictionbot|VelenPublicWebCrawler|WhatsApp|wpbot|xfa1|Yandex|Yeti|YouBot|zgrab|ZoominfoBot).
I am moving to just blocking them all, it's ridiculous.
Everything on this list got itself there by being abusive (either ignoring robots.txt, or not backing off when latency increased).
1. A proxy that looks at HTTP Headers and TLS cipher choices
2. An allowlist that records which browsers send which headers and selects which ciphers
3. A dynamic loading of the allowlist into the proxy at some given interval
New browser versions or updates to OSs would need the allowlist updating, but I'm not sure it's that inconvenient and could be done via GitHub so people could submit new combinations.
I'd rather just say "I trust real browsers" and dump the rest.
Also I noticed a far simpler block, just block almost every request whose UA claims to be "compatible".
To truly say “I trust real browsers” requires a signal of integrity of the user and browser such as cryptographic device attestation of the browser. .. which has to be centrally verified. Which is also not great.
Forcing Facebook & Co to play the adversary role still seems like an improvement over the current situation. They're clearly operating illegitimately if they start spoofing real user agents to get around bot blocking capabilities.
That could also be a user login, maybe, with per-user rate limits. I expect that bot runners could find a way to break that, but at least it's extra engineering effort on their part, and they may not bother until enough sites force the issue.
if ($http_user_agent ~* (list|of|case|insensitive|things|to|block)) {return 403;}
> If you try to rate-limit them, they’ll just switch to other IPs all the time. If you try to block them by User Agent string, they’ll just switch to a non-bot UA string (no, really).
It would be interesting if you had any data about this, since you seem like you would notice who behaves "better" and who tries every trick to get around blocks.
I've used this with voip scanners.
I contacted the network team at Cloudflare to apologise and also to confirm whether Facebook did actually follow the redirect... it's hard for Cloudflare to see 2PB, that kind of number is too small on a global scale when it's occurred over a few hours, but given that it was only a single PoP that would've handled it, then it would've been visible.
It was not visible, which means we can conclude that Facebook were not following redirects, or if they were, they were just queuing it for later and would only hit it once and not multiple times.
Hardly... the article links says that a 403 will cause Google to stop crawling and remove content... that's the desired outcome.
I'm not trying to rate limit, I'm telling them to go away.
Wait, that seems disturbingly conceivable with the way things are going right now. *shudder*
The AI companies are signing deals with large media and publishing companies to get access to data without the threat of legal action. But nobody is going to voluntarily make deals with millions of personal blogs, vintage car forums, local book clubs, etc. and setup a micro payment system.
Any attempt to force some kind of micro payment or "prove you are not a robot" system will add a lot of friction for actual users and will be easily circumvented. If you are LinkedIn and you can devote a large portion of your R&D budget on this, you can maybe get it to work. But if you're running a blog on stamp collecting, you probably will not.
And the ex-hype would probably fail at that, too :-)
https://github.com/ai-robots-txt/ai.robots.txt
After some digging, I also found a great way to surprise bots that don't respect robots.txt[1] :)
[1]: https://melkat.blog/p/unsafe-pricing
If a more specific UA hasn't been set, and the library doesn't force people to do so, then the library that has been the source of abusive behaviour is blocked.
No loss to me.
The fact that you choose to host 30 websites on the same instance is irrelevant, those AI bots scan websites, not servers.
This has been a recurring pattern I've seen in people complaining about AI bots crawling their website: huge number of requests but actually a low TPS once you dive a bit deeper.
In fact 2M requests arrived on December 23rd from Claude alone for a single site.
Average 25qps is definitely an issue, these are all long tail dynamic pages.
If you are an online shop, for example, isn't it beneficial that ChatGPT can recommend your products? Especially given that people now often consult ChatGPT instead of searching at Google?
ChatGPT won't 'recommend' anything that wasn't already recommended in a Reddit post, or on an Amazon page with 5000 reviews.
You have however correctly spotted the market opportunity. Future versions of CGPT with offer the ability to "promote" your eshop in responses, in exchange for money.
(I'm proposing this tongue in cheek, mostly, but it seems like it might work.)
Btw, such reverse slow-loris “attack” is called a tarpit. SSH tarpit example: https://github.com/skeeto/endlessh
Really, this behaviour should be a big embarrassment for any company whose main business model is selling "intelligence" as an outside product.
Is any on them even close to profitable?
"If you try to rate-limit them, they’ll just switch to other IPs all the time. If you try to block them by User Agent string, they’ll just switch to a non-bot UA string (no, really). This is literally a DDoS on the entire internet."
Super sad.
I've observed only one of them do this with high confidence.
> how are they determining it's the same bot?
it's fairly easy to determine that it's the same bot, because as soon as I blocked the "official" one, a bunch of AWS IPs started crawling the same URL patterns - in this case, mediawiki's diff view (`/wiki/index.php?title=[page]&diff=[new-id]&oldid=[old-id]`), that absolutely no bot ever crawled before.
> What non-bot UA do they claim?
Latest Chrome on Windows.
https://news.ycombinator.com/item?id=42551628
Since when do we ask people to guess other people's intent when they have better things to show, which is called evidence?
Surely we should talk about things with substantiated matter?
There’s evidence the statement was false, no evidence it was a lie.
> Oh, and of course, they don't just crawl a page once and then move on. Oh, no, they come back every 6 hours because lol why not. They also don't give a single flying fuck about robots.txt, because why should they.
Their self righteous indignation and specificity of the pretend subject of that indignation precludes any doubt about intent.
This guy made a whole public statement that is verifiably false. And then tried to toddler logic it away when he got called out.
The author responded:
>denschub 2 days ago [–]
>the robots.txt on the wiki is no longer what it was when the bot accessed it. primarily because I clean up my stuff afterwards, and the history is now completely inaccessible to non-authenticated users, so there's no need to maintain my custom robots.txt
Which is verifiably untrue:
HTTP/1.1 200 server: nginx/1.27.2 date: Tue, 10 Dec 2024 13:37:20 GMT content-type: text/plain last-modified: Fri, 13 Sep 2024 18:52:00 GMT etag: W/"1c-62204b7e88e25" alt-svc: h3=":443", h2=":443" X-Crawler-content-encoding: gzip Content-Length: 28
User-agent: * Disallow: /w/
There are no “intentional” lies, because there are no “unintentional” lies.
All lies are intentional. An “unintentional lie” is better known as “being wrong”.
Being wrong isn’t always lying. What’s so hard about this? An example:
My wife once asked me if I had taken the trash out to the curb, and I said I had. This was demonstrably false, anyone could see I had not. Yet for whatever reason, I mistakenly believed that I had done it. I did not lie to her. I really believed I had done it. I was wrong.
Your accusation was directly addressed by the author in a comment on the original post, IIRC
i find your attitude as expressed here to be problematic in many ways
For convenience, you can view the extracted data here:
https://pastebin.com/VSHMTThJ
You are welcome to verify for yourself by searching for “wiki.diasporafoundation.org/robots.txt” in the CommonCrawl index here:
https://index.commoncrawl.org/
The index contains a file name that you can append to the CommonCrawl url to download the archive and view.
More detailed information on downloading archives here:
https://commoncrawl.org/get-started
From September to December, the robots.txt at wiki.diasporafoundation.org contained this, and only this:
>User-agent: * >Disallow: /w/
Apologies for my attitude, I find defenders of the dishonest in the face of clear evidence even more problematic.
If you are legitimately trying to correct misinformation, your attitude, tone and language are counter productive. You would be much better seved by taking that energy and crafting an actually persuasive argument. You come across as unreasonable and unwilling to listen, not someone with a good grasp of the technical specifics.
I don't have a horse in the race. I'm fairly technical, but I did not find your arguments persuasive. This doesn't mean they are wrong, but it does mean that you didn't do a good job of explaining them.
These bots were crushing our search infrastructure (which is tightly coupled to our front end).
The internet is now a hostile environment, a rapacious land grab with no restraint whatsoever.
If the bots are accessing your website sequentially, then delaying a response will slow the bot down. If they are accessing your website in parallel, then delaying a response will increase memory usage on their end.
The key to this attack is to figure out the timeout the bot is using. Your server will need to slowly ramp up the delay until the connection is reset by the client, then you reduce the delay just enough to make sure you do not hit the timeout. Of course your honey pot server will have to be super lightweight and return simple redirect responses to a new resource, so that the bot is expending more resources per connection than you do, possibly all the way until the bot crashes.
This is a nice solution for an asynchronous web server. For apache, not so much.