The people doing the work aren't paying for AI, they're getting paid by AI.
The people who are having the costs of users not paying "their fair share" (this phrase has officially jumped the shark) are the investors who are subsidizing these companies.
I've been noticing page views in the past several months with a much wider span of origin on my Blogger stats. Before I would get a few from several countries, but now I am getting views from tiny countries and obscure or outdated browsers and operating systems, which leads me to think scrapers could be using VPN services in various countries along with header anonymizers that mask the device that they are using.
Given the nature of git, wouldn't all that HTML be highly cacheable? I get that's not free either, but it's got to be a lot less intensive than having cgit generate it every single time.
Or, html generation could be very, very fast in the first place. For example, serving static files is usually just as fast as a reverse-proxy cache.
Interestingly enough, cgit bills itself as "a hyperfast web frontend for git repositories" [0]. Alas, nothing is fast enough for the bots, it would appear.
using the current solution and a cache pattern doesnt solve it, but it is an "affordable" solvable problem. but first we need to ignore OP said its ok for now...
The issue is there's far more pages than could reasonably be stored: cgit is an old-school server-side renderer and there's a combinatorial explosion of possible pages it could render. The article calculates there's about a quadrillion different pages that could be 'crawled'
Most of those pages are going to be identical if not for some very minor differences. You can make it much much cheaper to put together these pages as opposed to the expensive operations it has to do right now.
i think its only 7gb of data total, therefore there should be a strategy to fully optimize for this deterministic scenario, like with a compiled streaming renderer... but he said its ok for now.. and anyway why should they need to fix anything, the slurp is what sucks. pun intended
My first thought was put varnish in front of it as well. But given how wide the url scheming is, the valid urls seem to be in the quintillions, if i counted correctly, its unlikely a naive caching solution would work.
But given that it's only a few million actual commits it may be possible to come up with an intelligent caching system that at full capacity require only a few TB.
> So, you'd think that something that pretends to be “Artificial Intelligence” would use the most efficient way of using our data for training purposes, right? Clone the repos, walk every commit. Done.
If repos like this were 10%+ of what they crawled, having a special case for clone-able repos would be smart, but if you're crawling everything, you're not going to do an efficiency tweak for each special case that has a more parse-able option.
I guess GitHub is in a similar bunch of sources, it should be also more efficient to crawl by cloning.
Anyway, isn’t it the whole sales pitch that it generates tailored solutions fast?
This tick about "if AI smart how come crawler dumb" is in most complaints I've read about AI crawlers and I've started to find it pretty annoying. The crawlers might be written using AI but they're evidently not actually running AI inference over the pages they get back--besides being able to tell this from the behavior, if this is pretraining input, that's enormous scale, so it'd mean a large increase in effective training cost. Naively assume inference costs are equal to pretraining costs (probably not true but maybe right order-of-magnitude) and it's a doubling.
This ends up tacitly turning a very legitimate complaint (ill-behaved crawlers) into a justification for head-in-the-sand AI denialism.
There's nothing "denialist" about recognizing the utter stupidity of systems that are being mislabeled as "AI". You judge a tool by its results, and the results have been very poor indeed. The only heads in the sand are those whose owners continually refuse to recognize the proofs before their very eyes that there's zero intelligence here.
That's head-in-the-sand stuff. AI is certainly very capable of being dumb (as are humans). But:
> “The problem was in need of a new real idea, which this new result seems to provide,” says James Maynard, a mathematician at the University of Oxford. “It seems that the AI has made a genuinely interesting mathematical contribution.”
Nobody a decade ago would have said "oh yeah solving a bunch of open problems in research mathematics, and finding a bunch of zero days in Chrome and Firefox, and winning literature prizes, are things that don't require intelligence."
>AI is certainly very capable of being dumb (as are humans).
I wonder how exactly the average scraper got to be so inefficient on kernel.org.
Did someone prompt a SotA model to write the most generic scraper possible?
Did someone prompt an old local model on their laptop to write a kernel.org scraper?
Perhaps no LLMs were involved in the first place. Seems to me there isn't much relation between how good a random scraper is and how usable/effective Mythos/Sol's outputs can be.
These crawlers (in contrast to e.g. googlebot and similar better behaved crawlers) are not very smart: they seem to make very little effort to avoiding crawling useless deep trees of generated pages. About the only thing they seem to put a lot of effort into is avoiding blocking.
(I'll note that while these are generally attributed to AI data gathering because of the timing of when they took off, it's not actually obvious who's running these bots. The big players all have crawlers that identify themselves and are reasonably well behaved, but I don't know if anyone has managed to positively attribute these other ones to any particular group)
> it's not actually obvious who's running these bots
This is fascinating to me. It's a large enough phenomenon that it's affecting the entire Internet and yet nobody seems to know yet who's actually doing it. Which isn't surprising, of course, it's hard to trace back to a source through all these proxies and it's probably a bunch of distinct groups anyways, but still!
Personally I have to wonder how much of it is "scrapers for training data" vs just tool-use LLMs. Even if you use chatgpt in thinking mode you can clearly see it searching and visiting a bunch of different websites to answer a question, presumably faster than any human would. That's got to add up. It's got me wondering why everyone seemingly discounts that as an option
I allow Google, Bing, Yahoo, Kagi, and Apple on my personal blog... who else do I need to allow? Claude, OpenAI, Alibaba, and perplexity can go to hell.
This is bad enough that I'm going to stop serving cgit. I've been doing cvsweb, then subversion, then cgit over my home server for many many years and for the first time ever this is annoying my own bw usage.
It's ridiculous also how you ban an IP then 1 second later another one picks up from where the first one left on.
i wonder what they're all up to. i imagine some are scraping datasets for pre-training, others are probably real-time scrapers looking for security bugs, even more still are agents working on coding tasks and looking at the kernel. also interesting to think about solutions: does everything need to be optimized now for weird access patterns that proliferated ai creates? do the ais need to have behavior trained in to be better netizens? is this the end of anonymous browsing and the beginning of an era where one has to attach an identity to all requests? or the end of community hosted free information services more broadly?
High Anubis difficulty is annoying the hell out of me for several sites. And it's starting to not block LLM bots anymore?
> 33% are now solving the math and getting through to the main site — because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge.
Visions of vast data centers surrounded by fields of browning grass, in which aging, rusting, formerly extremely expensive hardware is spending billions of compute cycles looking at anime catgirls
Unfortunately we're apt to run into some kind of Jeavons Paradox where the hardware gets so much faster in those few years will be able to slurp massive amounts of data cheaply so the problem never really ends.
It's very likely the last few years of bot behavior is the consequence of the residential proxy business booming. This is indirectly due to AI company crawling, but the fact that they are as cheap and available as they are changes the incentives for anyone using them toward reckless and unsustainable request behavior, as there is no risk of burning your IPs, and very small chances of seeing any consequences of essentially DDoS:ing a website.
And the residential proxy business was created by Cloudflare, who was created by us using Cloudflare.
I've been on all three sides (user of RPs, getting paid to run an RP, and trying to block RPs from my site). Residential proxy service is nice. You can scrape anything, even with the dumbest curl command, and only get a Cloudflare block maybe 15% of the time, in which case you just try again. That's less often than I get a cloudflare block from using a privacy browser from a non-proxy address. Cloudflare does not stop bots, it stops humans.
Wait, so you’ve been the person who wanted to keep people from scraping your site, the person who’s trying to scrape your site, and the person getting paid to help someone scrape your site? Brother, what are you doing with your life?
Contingency exists, I get that, and if you're truly in that state, sure, I don't judge necessity. A lot of our cohort seems to confuse a studied disinterest in looking beyond the end of their own nose for the sage wisdom of adulthood, though.
How will you know anything about a system you stubbornly only look at one side of? Like saying planes are terrible because they're loud. They are, but have you never traveled to a different city?
Has anyone considered having Anubis perform more valuable hashing?
Like, maybe you can't stop the LLM bots, but you can use them as one-off Bitcoin pool mining pool participants. You have to assume that making them find hash values with N leading zeroes has led to finding hash values with more than N leading zeroes. Maybe run a Bitcoin node under there and let each visitor take a couple swings for you with their pickaxes.
> High Anubis difficulty is annoying the hell out of me for several sites.
I just close the website if I see Anubis. Some have it set at reasonable difficulties (like 2)… others have it where I need to wait for like 30 seconds, I'm not wasting 30 seconds of my life for that.
I maintain a formally popular gaming website, and it used to have hundreds of legitimate requests per second. The load would be especially high during popular event times. So, it’s always been running on a dedicated server.
It also has an “online users” counter, which attempted to count real user sessions of unauthenticated user which still maintained a session, which lets them comment, or modify certain filter and display options. It never counted the Google bot.
Over the last few years this counter went from 100-200 users online to thousands. I have been very hands-off with it for many years, doing minor upgrades and backups. However, the site also has gotten quite slow, these sessions were obviously impacting it. So, I finally investigated these crawlers, and yes, it turns out it’s an insane amount of traffic that is entirely artificial, the site has just a handful of real users, and thousands of these crawling sessions that actually try to do everything they can, click every button. It doesn’t help that sort and search were implemented using GET links.
I fixed the counter to exclude the crawlers, but I have a bit of a dilemma. I don’t want to stop the bots from updating their knowledge based on all the content.
The best solution I could find is the new CloudFlare feature where they might charge the crawlers for every request, or otherwise block them. I think that’s a fantastic idea for the internet, at large. I signed up for the beta access, but haven’t heard from them again. I do think it’s unfortunate that this requires CloudFlare and the middleman.
Overall, it seems like the LLM are really straining the internet economy, the openness of it. Email spam used to be the worst, but the organized trillionaire labs sucking up the entire internet is going to break something if we don’t preempt them better.
It’s too bad the copyright and public internet systems are not acting quick enough. And I think there is no reason to act like this race really has to be at such a breakneck speed.
It makes sense doesn't it? If you own all the pipes, you can effectively block bad actors. But in reality what they do is to just block everyone who blocks advertising tracking and profiling tech (such as canvas fingerprint etc)... and because all vendors use cf, they all block those same clients, and they think there's some winner takes all magic happening with finely tuned blocklists. Don't fall for the marketing!
Try it yourself. Block those fingerprinting tech and try to access their own user forum. you get blocked. Now allow fingerprinting (even if emulated in a headless chrome!) and access by the most toxic vpn you can think off, access granted.
All cloudflare does is act like a gatekeep for monetizable-by-creepy-advertising-networks traffic, not bots. If the bots pay the small headless chrome tax, you get bombarded with their requests just fine.
Yes. Any number of methods work. From requiring a cookie to requiring CSS to be loaded to a PoW to a JavaScript redirect. Most scraper bots are really dumb. If you do it cookie based, you can also put a poison link on every page that invalidates their cookie and starts returning poison on every page after that, so they're damned if they do and damned if they don't.
I have tried cookie based sessions and I have tried multiple things. I have been battling DDOS and script kiddies longer than CF has been a thing. You’re wrong in that you think I could continue to maintain the site without CF in front of it, I would just shut it down. It’s not a profitable forum, it’s just a hobby.
If I recall by browsing the docs, there's a lot of fine grained client control over the blocking params. Which makes sense since the main product of cf is bot blocking.
So you can definitely block by IP, by fingerprint, both or neither
the point is if their solution is as magical because of the monopoly they have. which is what their marketing hint at (and everyone here internalized, judging by this thread).
why do you feel like discussing features was relevant?
Cloudflare is also great at playing both sides, and they're trying pretty hard to push for pay-to-crawl because they'll probably get a 30% cut along the way.
We don't even need a new law. What they're doing is already a felony, and there's a process already in place to go to a court and request unmasking of 10000 IP addresses.
If these were actual legitimate bots using a known user-agent and coming from the same subnet so you could throttle their requests it would be fine.
But I’m seeing requests coming in from random IP addresses from all over the world using randomised user-agents. But somehow these are trying URLs in sequential order. A while ago this would be classed as a DDoS cyber attack.
All anti-bot measures are inherently flawed, and if you overdo it, your false positives cause real harm by excluding actual people from using your services.
Not as in “1 out of 1000 legit requests get blocked” but “you’re going to block 100% of some person’s requests.”
HoustonArchitecture.com, a site that posts about new development projects in the Houston area, had the exact same thing happen to them. (They also got hacked; don't remember the details.) They now pay the Cloudflare tax.
It feels inevitable that many systems will have to go to a login/trusted ip source type system. Its just not feasible to continue to operate with 99% of your traffic being fake.
As the article states, this phenomenon may be documented, but there is no consensus that it describes any practical reality.
The predicted consequences have now had time to manifest, and have not done so. This makes the claim either false or overstated. Perhaps there will be issues in the future, but to date there have been many claims that AI development will stall (for a variety of reasons). If they were the critical weaknesses they have been portrayed as, models would not have advanced to the level they are today.
If you have a hypothesis, make a clear prediction based upon it. If you start pushing the date forward after each failed prediction, you end up looking like a hapless doomsday cult. If your hypothesis is correct however, your prediction should actually happen. Then provided you have not made so many predictions to get one right by chance, people will take what you have to say seriously.
You state it as if its only the quality of the hypothesis that matters, but you are ignoring an important part of it, timing. During the 08 financial crisis Burry had a hypothesis that was correct, however he almost went bankrupt still because he thought it would happen earlier than it did because of the government bailouts. He was pushing the day forward, and was looking like a "hapless doomsday cult". His hypothesis still turned out correct
If you can't say when something will happen you are just playing with statistics on another domain.
Seems like this chaps problem was that the actions were based upon a different hypothesis than the one he stated. X will cause Y by the end of the year is considerably different to X will eventually cause Y.
Side note: why are shallow clones evil? I always thought they were cheaper, but I guess that’s really just for my disk space. (since the server has to compute what blobs to give you instead of just “everything”?)
Normal clones can reuse delta-compressed data the server stored on disk. Shallow clones impose a negative constraint: do not transfer data outside the requested commit depth. Pre-computed delta chains than contain unrequested data become unusable and the server must do delta compression on the fly to satsify the shallow clone.
What I find remarkable is that for at least a decade, i.e., long before LLM scrapers were a thing, GitHub engineers have been reaching out to popular package manager projects, asking them to do away with shallow clones [0] [1]. They basically used the same reasoning as your comment did.
If I'm doing a shallow clone it isn't because I only want to receive a specific commit, it's because I don't want to burn a giant amount of disk space and network traffic on a full history.
In most use cases it would be perfectly acceptable for the server to send additional data. The client doesn't care about it because it is meaningless to them, but if it results in a significant load reduction on the server's side they don't really mind receiving it either. A 100MB shallow checkout coming with 400MB of garbage still beats cloning an entire 5GB history!
> So now, because of bad actors, I need to pay for the privilege of watching a website or using a service that was meant to be free? No, thanks.
First of all - I suppose it should be very cheap. So, real humans will not pay much.
Second - why do u think that websites are meant to be free? They provide some service, so its a rather strange that the internet is so free (in both senses). I think, this freeiness is allowed to greatly speed up popularization. But for me is obvious that it can demand payment for service.
And third - service owner really meant it to be free, I don't see any problems with this in my idea. It can still provide free service.
I'm not saying websites can't demand payments for service, I'm just saying it's bad if it's a necessary fix for "scrapers are destroying the basic social contract of the web".
The internet is already like that, but for some reason the payment only extends as far as the recipient's ISP, not the actual recipient. Most senders pay a flat rate, but their ISP doesn't.
You just take a sample of the crawling to whichever police department knows how to deal with DDoS, they subpoena the ISPs involved and trace who's using the proxies?
so true. the article authors wishing crawlers will use git instead is so funny because the crawlers don't care at all. they are scrapping everything with brute force. they don't care about your content or effective alternatives, and one more site driving their real users crazy with Anubis is nothing more than a new blip in their dashboard. the crawler operators will not even look at the url.
Seeing the exact same thing on (somewhat high profile) open data sites I run.
The crawlers get stuck in a loop requesting the dataset listing page with every. single. combination. of. facets. At essentially as fast as it can be pumped out or blocked.
Had one bot super interested in a single organization to the tune of 1000 r/min for 24+ hours. Realistic rotating user agents, realistic sec-* headers, no ip address seen more than a couple times in 10 minutes. The only commonality was the route.
Once detected, don't block them because they'll just change strategies automatically, but you can toy with them, like returning a page full of random numbers instead of real data.
I’d love to, but I’m a bit limited in what I can do from a reputational damage POV. They’re my sites, in that I’m responsible, but they aren’t something where I can return incorrect responses.
To protect your sites, once you know the requester is abusive the correct response is to make the abuse more expensive, like with a tarpit. I feel obliged to return correct responses to visits at a human-like rate and volume, and obliged to fuck with the others.
WordPress powered a huge number of websites. Yet the crawlers all go straight for the HTML of those sites rather than the more efficient and structured JSON API which all WordPress sites have.
If these crawlers are so smart, why aren't they following the rel="alternate" which is provided explicitly for them?
It never really cohered as a solution. High-powered scrapers are better equipped to handle proof-of-work challenges than end users. Proof of work makes sense for a password hash, where any one guess at a password provides zero marginal utility. But every request from a scraper is productive to the scraper.
The implication here is that the proxy fridge forwards the Anubis challenge to a dedicated rig controlled by the scraper who efficiently solves it and returns the answer.
That's still a notable step up in completely and resource investment for the crawler
See also how captchas continued being effective for years despite services like anti-captcha offering to solve them for you for a fifth of a cent each by farming the work out to India. It took advances in AI that made it viable to reliably solve them on-device to bring the end of the captcha
It's weird to believe data center based, Internet-scale scraping operations will be less able to allocate compute to proof-of-work challenges than individual users. This is design problem with things like Anubis: proof-of-work depends on a cost asymmetry between attacker and defender. But in scraping, both legitimate users and scrapers get the same value out of a transaction.
For PoW to work, you don't need scrapers to have less compute than users, you only need their compute advantage to be less than the load-generation advantage they would otherwise have. And unlike legitimate users, scrapers aren't carefully picking the URLs they request, so the average value they get out of it is probably rather low. They have to make it up in volume.
I just checked, and these many months later after installing Anubis, I'm getting about 14K requests per day. It's unreasonably effective at cutting crawler abuse. It should not be able to reduce my traffic by 98%, and yet in practice that's exactly what it's doing.
I think that the most abusive scrapers are operated by people who absolutely suck at programming. If they didn't, they wouldn't be crawling every permutation of "show me the git blame for every file in this repo for every commit in its history". Were they actually competent, they'd be investing in smarts such that they weren't crawling a web frontend to get that information.
The problem is that the scrapers are not sophisticated enough to use things like cgit and Git Forges efficiently (e.g. by git clone), while still being sophisticated enough to bypass widespread controls they are implemented to bypass.
An author of a scraper would need to trade off how much arbitrary compute they'll run against how much risk they take that someone will use them for free compute (inverting their role from exploiter to victim). They can work around that somewhat by analysing common enough code (to justify the LLM or human expense) and deciding to run it.
PoW challenges work when they have low enough adoption that it isn't worth scrapers' while to have a custom countermeasure for. To scale them once adoption is big enough, it probably takes some bespoke engineering per group of sites (coupled perhaps with some scraper traps that take advantage of scraper CPU for something useful, and aren't easily distinguished from the normal challenge).
It doesn’t have to be fully sound, it’s there to provide _some_ degree of interim protection.
Evidently, it makes _enough_ of a difference for the people running it. Criticising it is just letting the perfect be the enemy of the good.
The alternative to Anubis isn’t “something better that isn’t being picked”, it’s “completely unprotected”, and the people criticising it for being imperfect could helpfully move the conversation forward by actively suggesting a concrete alternative.
Sure, but then you’re essentially just doing versions of the old “what is 2+2” text captchas with extra computational steps, so why not save yourself and your users those?
As gp said, it does provide some protection. If it didn't, nobody would use it and there would be nothing to discuss. Making snarky comparisons doesn't do anything.
What makes you think they even run a browser engine on the devices they're hosted on?
All they need to do is forward traffic and launder its IP origin. They don't even need to be able to (and would actually be well advised not to) decrypt TLS streams.
Until you actually do the math and realize that it is not meaningful at all. It’s equivalent to the blogs that have a custom “bot protector” that asks you “what’s 2+2” every time you submit a comment. It might work temporarily as an inconvenience, but nothing more.
> Many of these boxes may not even have the spare CPU power ...
I don't think that's generally how these networks use residential exit proxies. There are at least a dozen well-developed frameworks out there for decoupling the crawler from the network exit point.
Most res proxy exits are just slinging bytes for clients using SOCKS, or another tunneling protocol.
If nothing else, a modern scraper will want better control over their TLS fingerprints, and you can't get that if you're depending on the on-device TLS libraries alone.
If you want to argue that there's excessive enthusiasm about anubis, which is neither a permanent solution nor a solution against all bots run for all purposes (some bot authors are cleverer than others, and you don't have to block everything, just enough to reduce resource use to tolerable levels)... fine.
However, the refrain from naysayers isn't that there's too much enthusiasm. It's that anubis doesn't make sense as a theoretical barrier against bots. Who cares as long as it works empirically, even briefly? Are you suggesting people let their sites continue to be degraded or unusable until they find a better solution, even though PoW challenges can improve the situation today?
Not everyone has the luxury of writing their own web app designed to handle massive bot traffic at minimal cost, gutting functionality to meet that requirement if necessary.
I noticed a couple of weeks ago that 1-2 orders of magnitude more bots were getting past anubis challenges. However, it would be incorrect to conclude that it's doing nothing and now worthless aside from heating up visitors' cpus. If I turn it off, I see >2x bots.
The next salvo in the war of attrition will probably be when anubis switches to wasm hashx, which the author suggested, in previous HN threads, is on the roadmap.
If bots quickly adapt (Claude vibe-coded JIT wasm-to-native-transpiler for hashx challenge algorithms?), that may be the end of anubis and efforts like it. If they don't, I expect more HN threads with people complaining that such compiler efforts are possible and therefore PoW-gated sites are doing it wrong... even if those sites stay usable.
If you just want to block the bottom of the barrel scrapers you don't need to do any proof of work. It's overkill and wastes user's times for such a purpose.
Yep, all security is about making attack more expensive than defense. If they have a farm of other people's devices, as alleged in the article, then it costs them less to calculate PoW than it costs ordinary users. Ordinary users have to sit there waiting, whereas bots are somebody else's problem.
The kernel folks likely have a good profile on what page people trigger Anubis on (i.e. what page people hit first). From that they could make heuristics about what pages are likely to be useful deep links.
Keep in mind that Anubis will rarely inconvenience an actual user who visits the site often; it’s meant to keep out first-time scrapers trying to grab a few pages from their queue.
another thing smaller sites benefit from -- where the load induced by crawlers tends to be bursty (e.g. as they discover new expensive endpoints to crawl) -- is to adjust the difficulty up/down to maintain a steady system load.
They're the exception, not the rule. They get crawled like any other site, but happen to host git repositories. It's not obvious that these are targeted crawls and they likely may just end up in crawling queues a lot generally
Maybe you could have a system that heuristicially detects when an crawler is making the request and then feeds them a modified page, itself generated from an LLM, that injects vulnerabilities and bad code and discussion and such.
> But no, let's in fact choose the stupidest possible way of doing it — by rendering everything as HTML commit by commit and then parsing it.
I feel like I'm at work.
We had some web crawler using Selenium to make queries and scrape the data instead of just downloading the whole file.
Every day it seems like we have some people that know just enough to be dangerous creating things like that. And then of course it's our fault that things are slow, or we won't give them infinite system resources, etc
555 comments
[ 0.21 ms ] story [ 12.1 ms ] threadThe people who are having the costs of users not paying "their fair share" (this phrase has officially jumped the shark) are the investors who are subsidizing these companies.
Interestingly enough, cgit bills itself as "a hyperfast web frontend for git repositories" [0]. Alas, nothing is fast enough for the bots, it would appear.
[0] https://git.zx2c4.com/cgit/about/
But given that it's only a few million actual commits it may be possible to come up with an intelligent caching system that at full capacity require only a few TB.
If repos like this were 10%+ of what they crawled, having a special case for clone-able repos would be smart, but if you're crawling everything, you're not going to do an efficiency tweak for each special case that has a more parse-able option.
This ends up tacitly turning a very legitimate complaint (ill-behaved crawlers) into a justification for head-in-the-sand AI denialism.
That's head-in-the-sand stuff. AI is certainly very capable of being dumb (as are humans). But:
> “The problem was in need of a new real idea, which this new result seems to provide,” says James Maynard, a mathematician at the University of Oxford. “It seems that the AI has made a genuinely interesting mathematical contribution.”
https://www.scientificamerican.com/article/no-ai-didnt-just-...
Nobody a decade ago would have said "oh yeah solving a bunch of open problems in research mathematics, and finding a bunch of zero days in Chrome and Firefox, and winning literature prizes, are things that don't require intelligence."
I wonder how exactly the average scraper got to be so inefficient on kernel.org.
Did someone prompt a SotA model to write the most generic scraper possible?
Did someone prompt an old local model on their laptop to write a kernel.org scraper?
Perhaps no LLMs were involved in the first place. Seems to me there isn't much relation between how good a random scraper is and how usable/effective Mythos/Sol's outputs can be.
Thats a good enough explanation in my opinion.
Source: someone who reviews AI generated PR's on a daily basis
(I'll note that while these are generally attributed to AI data gathering because of the timing of when they took off, it's not actually obvious who's running these bots. The big players all have crawlers that identify themselves and are reasonably well behaved, but I don't know if anyone has managed to positively attribute these other ones to any particular group)
This is fascinating to me. It's a large enough phenomenon that it's affecting the entire Internet and yet nobody seems to know yet who's actually doing it. Which isn't surprising, of course, it's hard to trace back to a source through all these proxies and it's probably a bunch of distinct groups anyways, but still!
Personally I have to wonder how much of it is "scrapers for training data" vs just tool-use LLMs. Even if you use chatgpt in thinking mode you can clearly see it searching and visiting a bunch of different websites to answer a question, presumably faster than any human would. That's got to add up. It's got me wondering why everyone seemingly discounts that as an option
GPTBot → model improvement / training crawl
OAI-SearchBot → search index / ChatGPT Search
ChatGPT-User → fetch initiated by a user's ChatGPT interaction
OAI-AdsBot → advertising landing-page crawl
(No second chances, I blocked their whole IP range.)
It's ridiculous also how you ban an IP then 1 second later another one picks up from where the first one left on.
> 33% are now solving the math and getting through to the main site — because apparently what we have to offer is worth spending a ton of cycles to calculate the Anubis challenge.
I've been on all three sides (user of RPs, getting paid to run an RP, and trying to block RPs from my site). Residential proxy service is nice. You can scrape anything, even with the dumbest curl command, and only get a Cloudflare block maybe 15% of the time, in which case you just try again. That's less often than I get a cloudflare block from using a privacy browser from a non-proxy address. Cloudflare does not stop bots, it stops humans.
Like, maybe you can't stop the LLM bots, but you can use them as one-off Bitcoin pool mining pool participants. You have to assume that making them find hash values with N leading zeroes has led to finding hash values with more than N leading zeroes. Maybe run a Bitcoin node under there and let each visitor take a couple swings for you with their pickaxes.
I just close the website if I see Anubis. Some have it set at reasonable difficulties (like 2)… others have it where I need to wait for like 30 seconds, I'm not wasting 30 seconds of my life for that.
It also has an “online users” counter, which attempted to count real user sessions of unauthenticated user which still maintained a session, which lets them comment, or modify certain filter and display options. It never counted the Google bot.
Over the last few years this counter went from 100-200 users online to thousands. I have been very hands-off with it for many years, doing minor upgrades and backups. However, the site also has gotten quite slow, these sessions were obviously impacting it. So, I finally investigated these crawlers, and yes, it turns out it’s an insane amount of traffic that is entirely artificial, the site has just a handful of real users, and thousands of these crawling sessions that actually try to do everything they can, click every button. It doesn’t help that sort and search were implemented using GET links.
I fixed the counter to exclude the crawlers, but I have a bit of a dilemma. I don’t want to stop the bots from updating their knowledge based on all the content.
The best solution I could find is the new CloudFlare feature where they might charge the crawlers for every request, or otherwise block them. I think that’s a fantastic idea for the internet, at large. I signed up for the beta access, but haven’t heard from them again. I do think it’s unfortunate that this requires CloudFlare and the middleman.
Overall, it seems like the LLM are really straining the internet economy, the openness of it. Email spam used to be the worst, but the organized trillionaire labs sucking up the entire internet is going to break something if we don’t preempt them better.
It’s too bad the copyright and public internet systems are not acting quick enough. And I think there is no reason to act like this race really has to be at such a breakneck speed.
they have one of the worst bot detection in the industry. but because everyone uses them, their huge false positive numbers won't show up anywhere.
It makes sense doesn't it? If you own all the pipes, you can effectively block bad actors. But in reality what they do is to just block everyone who blocks advertising tracking and profiling tech (such as canvas fingerprint etc)... and because all vendors use cf, they all block those same clients, and they think there's some winner takes all magic happening with finely tuned blocklists. Don't fall for the marketing!
Try it yourself. Block those fingerprinting tech and try to access their own user forum. you get blocked. Now allow fingerprinting (even if emulated in a headless chrome!) and access by the most toxic vpn you can think off, access granted.
All cloudflare does is act like a gatekeep for monetizable-by-creepy-advertising-networks traffic, not bots. If the bots pay the small headless chrome tax, you get bombarded with their requests just fine.
So you can definitely block by IP, by fingerprint, both or neither
why do you feel like discussing features was relevant?
And Cloudflare is no exception, the "hello world" of bots is Cloudflare bypass.
And I only stand to gain from pay-to-crawl, so I don't really mind that play.
I feel like it's the same issue that public resources are facing.
Step 1: public resource is freely available
Step 2: asshole shows up, drains the resource completely just because they can
Step 3: either resource is taken away, or it has massive security behind it
We don't even need a new law. What they're doing is already a felony, and there's a process already in place to go to a court and request unmasking of 10000 IP addresses.
Why not? At this point, any attempt to try to meet the bots halfway is doomed.
But I’m seeing requests coming in from random IP addresses from all over the world using randomised user-agents. But somehow these are trying URLs in sequential order. A while ago this would be classed as a DDoS cyber attack.
Not as in “1 out of 1000 legit requests get blocked” but “you’re going to block 100% of some person’s requests.”
Is it foolish of me to have expected more from a blog post on kernel.org?
The predicted consequences have now had time to manifest, and have not done so. This makes the claim either false or overstated. Perhaps there will be issues in the future, but to date there have been many claims that AI development will stall (for a variety of reasons). If they were the critical weaknesses they have been portrayed as, models would not have advanced to the level they are today.
If you have a hypothesis, make a clear prediction based upon it. If you start pushing the date forward after each failed prediction, you end up looking like a hapless doomsday cult. If your hypothesis is correct however, your prediction should actually happen. Then provided you have not made so many predictions to get one right by chance, people will take what you have to say seriously.
Seems like this chaps problem was that the actions were based upon a different hypothesis than the one he stated. X will cause Y by the end of the year is considerably different to X will eventually cause Y.
[0]: https://github.com/Homebrew/brew/pull/9383
[1]: https://github.com/CocoaPods/CocoaPods/issues/4989#issuecomm...
If I'm doing a shallow clone it isn't because I only want to receive a specific commit, it's because I don't want to burn a giant amount of disk space and network traffic on a full history.
In most use cases it would be perfectly acceptable for the server to send additional data. The client doesn't care about it because it is meaningless to them, but if it results in a significant load reduction on the server's side they don't really mind receiving it either. A 100MB shallow checkout coming with 400MB of garbage still beats cloning an entire 5GB history!
First idea - there should be some cost for sending traffic somewhere. And the server owner also should receive pay - not only the internet provider.
So, in with this idea, the server owner can potentially increase the amount of computing power to satisfy all requests.
So now, because of bad actors, I need to pay for the privilege of watching a website or using a service that was meant to be free? No, thanks.
I don't have a solution, but "break how the web currently works" is not one I would accept all willy nilly.
EDIT: yes, I realize we already broke the web (with Anubis, cloudflare, recaptcha etc) but I think we should resist slowly breaking it further.
First of all - I suppose it should be very cheap. So, real humans will not pay much.
Second - why do u think that websites are meant to be free? They provide some service, so its a rather strange that the internet is so free (in both senses). I think, this freeiness is allowed to greatly speed up popularization. But for me is obvious that it can demand payment for service.
And third - service owner really meant it to be free, I don't see any problems with this in my idea. It can still provide free service.
> [four phases, each of which has historically been hard-to-impossible for highly organized international coalitions]
Never going to happen.
Interesting to crawlers is not a narrow scope. We have the same problem on a B2B car wash site.
The crawlers get stuck in a loop requesting the dataset listing page with every. single. combination. of. facets. At essentially as fast as it can be pumped out or blocked.
Had one bot super interested in a single organization to the tune of 1000 r/min for 24+ hours. Realistic rotating user agents, realistic sec-* headers, no ip address seen more than a couple times in 10 minutes. The only commonality was the route.
If these crawlers are so smart, why aren't they following the rel="alternate" which is provided explicitly for them?
https://news.ycombinator.com/item?id=44962529
It never really cohered as a solution. High-powered scrapers are better equipped to handle proof-of-work challenges than end users. Proof of work makes sense for a password hash, where any one guess at a password provides zero marginal utility. But every request from a scraper is productive to the scraper.
See also how captchas continued being effective for years despite services like anti-captcha offering to solve them for you for a fifth of a cent each by farming the work out to India. It took advances in AI that made it viable to reliably solve them on-device to bring the end of the captcha
I just checked, and these many months later after installing Anubis, I'm getting about 14K requests per day. It's unreasonably effective at cutting crawler abuse. It should not be able to reduce my traffic by 98%, and yet in practice that's exactly what it's doing.
I think that the most abusive scrapers are operated by people who absolutely suck at programming. If they didn't, they wouldn't be crawling every permutation of "show me the git blame for every file in this repo for every commit in its history". Were they actually competent, they'd be investing in smarts such that they weren't crawling a web frontend to get that information.
An author of a scraper would need to trade off how much arbitrary compute they'll run against how much risk they take that someone will use them for free compute (inverting their role from exploiter to victim). They can work around that somewhat by analysing common enough code (to justify the LLM or human expense) and deciding to run it.
PoW challenges work when they have low enough adoption that it isn't worth scrapers' while to have a custom countermeasure for. To scale them once adoption is big enough, it probably takes some bespoke engineering per group of sites (coupled perhaps with some scraper traps that take advantage of scraper CPU for something useful, and aren't easily distinguished from the normal challenge).
Until they discover this neat trick [1] and solve challenges orders of magnitudes more efficiently than legitimate users.
The game theory of Anubis is not sound. It makes fundamentally less sense than Captchas, and even those have been on the way out for a while.
[1] https://news.ycombinator.com/item?id=49500040
Evidently, it makes _enough_ of a difference for the people running it. Criticising it is just letting the perfect be the enemy of the good.
The alternative to Anubis isn’t “something better that isn’t being picked”, it’s “completely unprotected”, and the people criticising it for being imperfect could helpfully move the conversation forward by actively suggesting a concrete alternative.
And I really wasn't trying to be snarky; people had "what is i + j" type of captchas for a long time and I still see them every once in a while!
All they need to do is forward traffic and launder its IP origin. They don't even need to be able to (and would actually be well advised not to) decrypt TLS streams.
I don't think that's generally how these networks use residential exit proxies. There are at least a dozen well-developed frameworks out there for decoupling the crawler from the network exit point.
Most res proxy exits are just slinging bytes for clients using SOCKS, or another tunneling protocol.
If nothing else, a modern scraper will want better control over their TLS fingerprints, and you can't get that if you're depending on the on-device TLS libraries alone.
However, the refrain from naysayers isn't that there's too much enthusiasm. It's that anubis doesn't make sense as a theoretical barrier against bots. Who cares as long as it works empirically, even briefly? Are you suggesting people let their sites continue to be degraded or unusable until they find a better solution, even though PoW challenges can improve the situation today?
Not everyone has the luxury of writing their own web app designed to handle massive bot traffic at minimal cost, gutting functionality to meet that requirement if necessary.
I noticed a couple of weeks ago that 1-2 orders of magnitude more bots were getting past anubis challenges. However, it would be incorrect to conclude that it's doing nothing and now worthless aside from heating up visitors' cpus. If I turn it off, I see >2x bots.
The next salvo in the war of attrition will probably be when anubis switches to wasm hashx, which the author suggested, in previous HN threads, is on the roadmap.
If bots quickly adapt (Claude vibe-coded JIT wasm-to-native-transpiler for hashx challenge algorithms?), that may be the end of anubis and efforts like it. If they don't, I expect more HN threads with people complaining that such compiler efforts are possible and therefore PoW-gated sites are doing it wrong... even if those sites stay usable.
Keep in mind that Anubis will rarely inconvenience an actual user who visits the site often; it’s meant to keep out first-time scrapers trying to grab a few pages from their queue.
Anubis has a fairly capable "policy" system. you can place something like this in your policy.json:
``` { "bots": [ { "action": "WEIGH", "expression": "path.startsWith(\"/expensive/endpoint\")", "name": "scrutinize-expensive-endpoints", "weight": { "adjust": 20 } } ] } ```
another thing smaller sites benefit from -- where the load induced by crawlers tends to be bursty (e.g. as they discover new expensive endpoints to crawl) -- is to adjust the difficulty up/down to maintain a steady system load.
``` { "bots": [ { "action": "WEIGH", "expression": "load_15m <= 16.0", "name": "sustained-low-load", "weight": { "adjust": -10 } }, { "action": "WEIGH", "expression": "load_5m >= 24.0", "name": "intermittent-high-load", "weight": { "adjust": 10 } }, ] } ```
I feel like I'm at work.
We had some web crawler using Selenium to make queries and scrape the data instead of just downloading the whole file.
Every day it seems like we have some people that know just enough to be dangerous creating things like that. And then of course it's our fault that things are slow, or we won't give them infinite system resources, etc