Getting hit with "Prove you are a human" at every other website

What the people who like Anubis don't get is that the cost for honest people is very real, the cost for crawlers is basically $0 because: most of the people who run crawlers are just idiots and aren't paying attention to what's happening and get crap data and don't even know they are getting bounced and probably don't even look at the crap data they collect, the worst of the crawlers are running botnets using stolen CPU, the real AI companies don't have any financial discipline and wouldn't care it if anubis wasted a CPU*hour per page.

Or rather, those "click on the bus" images raise your blood pressure but are close to trivial for a bot to automate... It's all performative... We're good open source people who prove we are good open source people because we put Anubis on our page to shame those bad AI people.

Feels like the whole internet is hitting that hockey stick growth... towards becoming shit.

Buster: Captcha Solver for Humans [0]

NopeCHA: CAPTCHA Solver [1]

OF COURSE, these are band-aids on an internet under full assault.

P.S. Paywall busters are a nuisance to run through. Often, a paywalled article is syndicated (MSN does this a lot) or available at arxiv, or an author's institution, so "Use the web search, Luke."

[0] https://github.com/dessant/buster

[1] https://nopecha.com/

Whatever magic is built into the CloudFlare checkbox and my motion to click on it, I really wish was just part of the browser and my motion to click on everything else.

This is the internet in 2026 and it sure as hell won't get any better.

Kinda disappointing that this doesn't get daily attention on HN of all places.

Most of the time a web page reports an error or seems to be down, or it gives you another captcha no matter how many times you correctly answer it, it's just denying you access, but it's also trying to hide that from you. Whether there's captchas or an outright denial, those pages are guarded by bots that only allow other bots to proceed to the page. We've known for several years that captchas only deter humans (https://www.newscientist.com/article/2384228-bots-are-better...) so clearly anyone currently using them doesn't want humans visiting the page.

Most of the time, the Gemini summary that Google includes in their search results isn't particularly useful, but extracting data from those bot-only pages is one thing it excels at. So, whenever a search result links to a page guarded by a bot, and it senses that I am a human and denies access, I go back to the Google search results page, and ask Gemini to scrape the page and provide the data I want from it.

The original model for web didn't include REST style API calls.

A new model designed holistically for content distribution, with a hardened standardized spec for the API calls part, something that by design entirely shirks today's daily challenges, is necessary.

_

> Or just come back read the comments on hn for that article.

Same lol

Another one on the list now is Amazon adding amazon.com redirects around IMDB pages. I can't even follow IMDB links anymore without getting trapped in broken redirects, because my Amazon cookies are in another session and whatever challenge they are running on amazon.com is bad at completing redirects and keeps redirecting brokenly encoded addresses that are 404 pages instead. I have to search things from IMDB in an Amazon container to get it to take me to the same addresses the direct link should have and used to in any container.

- https://docs.sweeting.me/s/ai-scraping-will-force-us-towards...

- https://docs.sweeting.me/s/so-you-want-to-build-a-social-net...

I've worked at multiple AI browser companies at this point and I think the best way forward is to start paying for content again honestly. Brave and BAT were too early but we need more things like that with lower friction and more primitives baked into the browser. The AI bots that create value for people and have money will happily pay to get access to the content, keeping it alive in the process and paying for the maintenance cost of allowing good bots.

DBSC, DPoP, and Google's latest "Google Cloud Fraud Defense" reCAPTCHA using phone are another approach where you use hardware secure enclaves to prevent cookies from leaving devices and attest to 1 visitor = 1 physical device. I don't like these because I want my personal agents to be able to access the things I can on my behalf. If I misbehave then block me & bots, not all bots, this is not as impossible as companies make it sound.

The worst possible approach is to require biometric or government ID, that's the one we all need to push hard against, as it's bad for everyone involved.

Because often these reports are from people using weird browsers, privacy plugins, VPNs, javascript off, etc

It does happen to "normal" browsers but are you doing anything that look suspicious?

Alternatively, I use Firefox with uBlock Origin and Privacy Badger - so it might be that.

Secondly, depending on what continent and country you use the internet from, you get a vastly different experience. If you're unlucky enough to live in Brazil, Peru, Algeria, Cambodia or hundreds of other countries, you'll be served challenge after challenge, regardless of how many times you continue to pass them. Cloudflare will in some cases just straight up reject traffic from you, and many times just the fact that websites set too short timesouts to HTTP requests, means you effectively can't browse their websites, even if you'd be ready to wait 2 seconds for each request to be done.

So you end up segregating internet users per country, because of gatekeepers like Cloudflare and others, who've somehow made people feel much more comfortable with just blocking all traffic from a whole country because of some attack they sustained in the past originating from that country.

Not sure what the solution is, but I've definitely felt the effects of these things too, making the internet I'm used to borderline impossible to use because of the country I was in. I'm not sure most people realize how fragmented it all already is, and been for decades already, not least because of language.

This has been particularly annoying for me because I disable NixOS's binary cache and build all my software from source. So when I run `nix build` I end up fetching source code tarballs from code forges. Invariably, the build will fail because what was supposed to be thesourcecode.tar ended up being the html for the anime girl page, leading to a hash mismatch.

I haven't yet figured out the incantation to download the source files myself and inject them into the nix store. `nix-store --add-fixed sha256 <file>` always seems to result in a different nix store path than the one `nix build` was downloading to.

I totally understand the desire to stem the fire hose of AI crawlers. I self-host my code forge and so I am no stranger to the devastation those locusts leave in their wake. But this is also hurting innocent non-crawling humans who are just trying to compile your software.

You're having to prove you're human all the time because there's likely something you're doing that makes you look like a bot.

I think for web sites you really love and care about, you should write to them and let them know that the user experience sucks. If enough people complain they may fix the captchas like we did!