In the same time it’s so practical to ask a question and it opens 25 pages to search and summarize the answer. Before that’s more or less what I was trying to do by hand. Maybe not 25 websites because of crap SEO the top 10 contains BS content so I curated the list but the idea is the same no ?
I'm absolutely pro AI-crawlers. The internet is so polluted with garbage, compliments of marketing. My AI agent should find and give me concise and precise answers.
> "I don't know what this actually gives people, but our industry takes great pride in doing this"
> "unsleeping automatons that never get sick, go on vacation, or need to be paid health insurance that can produce output that superficially resembles the output of human employees"
> "This is a regulatory issue. The thing that needs to happen is that governments need to step in and give these AI companies that are destroying the digital common good existentially threatening fines and make them pay reparations to the communities they are harming."
I run a symbol server, as in, PDB debug symbol server. Amazon's crawler and a few others love requesting the ever loving shit out of it for no obvious reason. Especially since the files are binaries.
I just set a rate-limit in cloudflare because no legitimate symbol server user will ever be excessive.
Isn't there a class action lawsuit coming from all this? I see a bunch of people here indicating these scrapers are costing real money to people who host even small niche sites.
Is the reason these large companies don't care because they are large enough to hide behind a bunch of lawyers?
A bit off-topic but wtf is this preview image of a spider in the eye?
It’s even worse than the clickbait title of this post.
I think this should be considered bad practice.
This article and the "report" look like a submarine ad for Fastly services. At no point does it mention the human/bot/AI bot ratio, making it useless for any real insights.
I recently, for pretty much the first time ever in 30 years of running websites, had to blanket ban crawlers. I now whitelist a few, but the rest (and all other non-UK visitors) have to pass a Cloudflare challenge [1].
AI crawlers were downloading whole pages and executing all the javascript tens of millions of times a day - hurting performance, filling logs, skewing analytics and costing too much money in Google Maps loads.
My book discovery website shepherd.com is getting hammered every day by AI crawlers (and crashing often)... my security lists in CloudFlare are ridiculous and the bots are getting smarter.
AI is going to damage society not in fancy sci-fi ways but by centralizing profit made at the expense of everyone else on the internet, who is then forced to erect boundaries to protect themselves, worsening the experience for the rest of the public. Who also have to pay higher electricity bills, because keeping humans warm is not as profitable as a machine which directly converts electricity into stock price rises.
I wonder if we're doing the wrong thing blocking them with invasive tools like cloudflare?
If all you're concerned about is server load, wouldn't it be better to just offer a tar file containing all of your pages they can download instead? The models are months out of date, so a monthly dumb would surely satisfy them. There could even be some coordination for this.
They're going to crawl anyway. We can either cooperate or turn it into some weird dark market with bad externalities like drugs.
My worst offender for scraping one of my sites was Anthropic. I deployed an ai tar pit (https://news.ycombinator.com/item?id=42725147) to see what it would do it with it, and Anthropic's crawler kept scraping it for weeks. I calculated the logs and I think I wasted nearly a year of their time in total, because they were crawling in parallel. Other scrapers weren't so persistent.
about 18 months ago, our non-Google / Bing bot traffic went from single digits per cent to over 99.9% bot traffic. We tried some home-spun solutions at first, but eventually threw in the towel and put Cloudflare in front of all our publicly accessible pages. On a long term basis, this was probably the right move for us, but we felt forced into this. And the Cloudflare Managed Ruleset definitely blocks some legit traffic such that it requires a fair amount of manual tuning.
Why is this not a violation of the CFAA, and why aren't SWEs and directors going to prison over it?
As long as I have an EULA or a robots.txt or even a banner that forbids this sort of access, shouldn't any computerized access be considered abuse? Something, something, scraping JSTOR?
That's the moment that you remember that years ago in self hosted you could have sustained millions request per second on a single low end server for quite nothing.
But now you are "on the cloud", with lambdas because "who cares" and hiring a proper part-time sysadmin is too complicated and so now you are pounded with crazy costs for moderate loads...
I am under attack mode right now because I am being attacked from hundreds of chinese proxied bots taking my PHP response time from its usual 0.2 to 2+second response times. fucking ridiculous.
One thing I don't fully understand in all this is how the IP address stuff works. Like I keep hearing people saying somebody can get 10 gazillion residential IPs so they become unblockable, but how? This article also mentions crawlers should publish there IP ranges. Like, yeah? What if using more than X number of IPs to crawl was a criminal offense unless you got a permit, which would require you to identify and publish all those IPs up front?
Some companies publish the IP ranges of their crawler (OpenAI [1], Mistral [2] for example) but many like Anthropic don’t.
Not sure those lists can be fully trusted though. Perplexity, for instance, was caught using IPs outside of their declared list [3].
29 comments
[ 0.20 ms ] story [ 58.6 ms ] thread> "I don't know what this actually gives people, but our industry takes great pride in doing this"
> "unsleeping automatons that never get sick, go on vacation, or need to be paid health insurance that can produce output that superficially resembles the output of human employees"
> "This is a regulatory issue. The thing that needs to happen is that governments need to step in and give these AI companies that are destroying the digital common good existentially threatening fines and make them pay reparations to the communities they are harming."
<3 <3
I just set a rate-limit in cloudflare because no legitimate symbol server user will ever be excessive.
Is the reason these large companies don't care because they are large enough to hide behind a bunch of lawyers?
AI crawlers were downloading whole pages and executing all the javascript tens of millions of times a day - hurting performance, filling logs, skewing analytics and costing too much money in Google Maps loads.
Really disappointing.
[1] https://developers.cloudflare.com/cloudflare-challenges/
I wish there were a better way to solve this.
AI is going to damage society not in fancy sci-fi ways but by centralizing profit made at the expense of everyone else on the internet, who is then forced to erect boundaries to protect themselves, worsening the experience for the rest of the public. Who also have to pay higher electricity bills, because keeping humans warm is not as profitable as a machine which directly converts electricity into stock price rises.
The same incentives to do this already existed for search engine operators.
If all you're concerned about is server load, wouldn't it be better to just offer a tar file containing all of your pages they can download instead? The models are months out of date, so a monthly dumb would surely satisfy them. There could even be some coordination for this.
They're going to crawl anyway. We can either cooperate or turn it into some weird dark market with bad externalities like drugs.
Maybe they are paying big bucks for people who are actually very bad at their jobs?
Why would the CEOs tolerate that? Do they think it's a profitable/strategic thing to get away with, rather than a sign of incompetence?
When subtrees of the org chart don't care that they are very bad at their jobs, harmed parties might have to sue to get the company to stop.
As long as I have an EULA or a robots.txt or even a banner that forbids this sort of access, shouldn't any computerized access be considered abuse? Something, something, scraping JSTOR?
But now you are "on the cloud", with lambdas because "who cares" and hiring a proper part-time sysadmin is too complicated and so now you are pounded with crazy costs for moderate loads...
[1] https://platform.openai.com/docs/bots
[2] https://docs.mistral.ai/robots/
[3] https://blog.cloudflare.com/perplexity-is-using-stealth-unde...