90 comments

[ 3.6 ms ] story [ 92.2 ms ] thread
I would love that vision to become reality but what Cloudflare is doing is unfortunately necessary atm.
I would love to get off Cloudflare but there are no real good alternatives
I have zero issue with Ai Agents, if there's a real user behind there somewhere. I DO have a major issue with my sites being crawled extremely aggressively by offenders including Meta, Perplexity and OpenAI - it's really annoying realising that we're tying up several cpu cores on AI crawling. Less than on real users and google et al.
Maybe the title means something more like "The web should not have gatekeepers (Cloudflare)". They do seem to say as much toward the end:

>We need protocols, not gatekeepers.

But until we have working protocols, many webmasters literally do need a gatekeeper if they want to realistically keep their site safe and online.

I wish this weren't the case, but I believe the "protocol" era of the web was basically ended when proprietary web 2.0 platforms emerged that explicitly locked users in with non-open protocols. Facebook doesn't want you to use Messenger in an open client next to AIM, MSN, and IRC. And the bad guys won.

But like I said, I hope I'm wrong.

>We need protocols, not gatekeepers

The funny thing is that this blog post is complaining about a proposed protocol from Cloudflare (one which will identify bots so that good bots can be permitted). The signup form is just a method to ask Cloudflare (or any other website owner/CDN) to be categorized as a good bot.

It's not a great protocol if you're in the business of scraping websites or selling people bots to access websites for them, but it's a great protocol for people who just want their website to work without being overwhelmed by the bad side of the internet.

The whitelist approach Cloudflare takes isn't good for the internet, but for website owners who are already behind Cloudflare, it's better than the alternative. Someone will need to come up with a better protocol that also serves the website owners' needs if they want Cloudflare to fail here. The AI industry simply doesn't want to cooperate, so their hand must be forced, and only companies like Cloudflare are powerful enough to accomplish that.

I understand the concerns around a central gatekeeper but I'm confused as to why this specifically is viewed negatively. Don't website owners have to choose to enable cloudflare and to opt-in to this gate that the site owners control?

If this was cloudflare going into some centralized routing of the internet and saying everything must do X then that would be a lot more alarming but at the end of the day the internet is decentralized and site owners are the ones who are using this capability.

Additionally I don't think that I as an individual website owner would actually want / be capable of knowing which agents are good and bad and cloudflare doing this would be helpful to me as a site owner as long as they act in good faith. And the moment they stop acting in good faith I would be able to disable them. This is definitely a problem right now as unrestricted access to the bots means bad bots are taking up many cycles raising costs and taking away resources from real users

Site owners are tricked and scared (by Cloudflare) into using Cloudflare when they don't need to. Cloudflare feels the increase in customer growth and the rest of us feel the pain.
I recently ran a test on the page load reliability of Browserbase and I was shocked to see how unreliable it was for a standard set of websites - the top 100 websites in the US by traffic according to SimilarWeb. 29% of page load requests failed. Without an open standard for agent identification, it will always be a cat and mouse game to trap agents, and many agents will predictably fail simple tasks.

https://anchorbrowser.io/blog/page-load-reliability-on-the-t...

Here's to working together to develop a new protocol that works for agents and website owners alike.

I pretty much use Perplexity exclusively at this point, instead of Google. I'd rather just get my questions answered than navigate all of the ads and slowness that Google provides. I'm fine with paying a small monthly fee, but I don't want Cloudflare being the gatekeeper.

Perhaps a way to serve ads through the agents would be good enough. I'd prefer that to be some open protocol than controlled by a company.

This has been my experience more recently as well, I've finally migrated from google to Brave Search since google was just slow for me.

I also appreciate the AI search results a bit when im looking for something very specific (like what the yaml definition for a docker swarm deployment constraint looks like) because the AI just gives me the snippet while the search results are 300 medium blog posts about how to use docker and none of them explain the variables/what each does. Even the official docker documentation website is a mess to navigate and find anything relevant!

Perplexity is the problem Cloudflare and companies like it are trying to solve. The company refuses to take no for an answer and will mislead and fake their way through until they've crawled the content they wanted to crawl.

The problem isn't just that ads can't be served. It's that every technical measure to attempt to block their service produces new ways of misleading website owners and the services they use. Perplexity refuses any attempt at abuse detection and prevention from their servers.

None of this would've been necessary if companies like Perplexity would've just acted like a responsible web service and told their customers "sorry, this website doesn't allow Perplexity to act on your behalf".

The open protocol you want already exists: it's the user agent. A responsible bot will set the correct user agent, maybe follow the instructions in robots.txt, and leave it at that. Companies like Perplexity (and many (AI) scrapers) don't want to participate in such a protocol. They will seek out and abuse any loopholes in any well-intended protocol anyone can come up with.

I don't think anyone wants Cloudflare to have even more influence on the internet, but it's thanks to the growth of inconsiderate AI companies like Perplexity that these measure are necessary. The protocol Cloudflare proposes is open (it's just a signature), the problem people have with it is that they have to ask Cloudflare nicely to permit website owners to track and prevent abuse from bots. For any Azure-gated websites, your bot would need to ask permission there as well, as with Akamai-gated websites, and maybe even individual websites.

A new protocol is a technical solution. Technical solutions work for technical problems. The problem Cloudflare is trying to solve isn't a technical problem; it's a social problem.

>but I don't want Cloudflare being the gatekeeper

Cloudflare is not the gatekeeper, it's the owner of the site that blocks Perplexity that's "gatekeeping" you. You're telling me that's not right?

I can see a future where I don't use the internet at all.
Your ideas are intriguing to me and wish to subscribe to your newsletter.

Joking aside, I think the ideas and substance are great and sorely needed. However, I can only see the idea of a sort of token chain verification as running into the same UX problems that plagued (plagues?) PGP and more encryption-focused processes. The workflow is too opaque, requires too much specialized knowledge that is out of reach for most people. It would have to be wrapped up into something stupid simple like an iOS FaceID modal to have any hope of succeeding with the general public. I think that's the idea, that these agents would be working on behalf of their owners on their own devices, so it has to be absolutely seamless.

Otherwise, rock on.

Well, if you have a better way to solve this that’s open I’m all ears. But what Cloudflare is doing is solving the real problem of AI bots. We’ve tried to solve this problem with IP blocking and user agents, but they do not work. And this is actually how other similar problems have been solved. Certificate authorities aren’t open and yet they work just fine. Attestation providers are also not open and they work just fine.
Are they? Until Let's Encrypt came along and democratise the CA scene, it was a hell hole. Web Security was depending on how deep your pockets are. One can argue that the same path is being laid in front us until a Let's Encrypt comes along and democratise it? And here as it's about attestation, how are we going to prevent gatekeeper's doing "selective attestations with arguable criteria"? How will we prevent political forces?
I'm not sure if things are as fine as you say they are. Certificate authorities were practically unheard of outside of corporate websites (and even then mostly restricted to login pages) until Let's Encrypt normalized HTTPS. Without the openness of Let's Encrypt, we'd still be sharing our browser history and search queries with our ISPs for data mining. Attestation providers have so far refused to revoke attestation for known-vulnerable devices (because customers needing to replace thousands of devices would be an unacceptable business decision), making the entire market rather useless.

That said, what I am missing from these articles is an actual solution. Obviously we don't want Cloudflare from becoming an internet gatekeeper. It's a bad solution. But: it's a bad solution to an even worse problem.

Alternatives do exist, even decentralised ones, in the form of remote attestation ("can't access this website without secure boot and a TPM and a known-good operating system"), paying for every single visit or for subscriptions to every site you visit (which leads to centralisation because nobody wants a subscription to just your blog), or self-hosted firewalls like Anubis that mostly rely on AI abuse being the result of lazy or cheap parties.

People drinking the AI Kool-Aid will tell you to just ignore the problem, pay for the extra costs, and scale up your servers, because it's *the future*, but ignoring problems is exactly why Cloudflare still exists. If ISPs hadn't ignored spoofing, DDoS attacks, botnets within their network, """residential proxies""", and other such malicious acts, Cloudflare would've been an Akamai competitor rather than a middle man to most of the internet.

> Well, if you have a better way to solve this that’s open I’m all ears.

Regulation.

Make it illegal to request the content of a webpage by crawler if a website operator doesn't explicitly allows it via robots.txt. Institute a government agency that is tasked with enforcement. If you as a website operator can show that traffic came from bots, you can open a complaint with the government agency and they take care of shaking painful fines out of the offending companies. Force cloud hosts to keep books on who was using what IP addresses. Will it be a 100% fix, no, will it have a massive chilling effect if done well, absolutely.

This is sort of like how email is based on Internet standards but a large percentage of email users use Gmail. The Internet standards Cloudflare is promoting are open, but Cloudflare has a lot of power due to having so many customers.

(What are some good alternatives to Cloudflare?)

Another way the situation is similar: email delivery is often unreliable and hard to implement due to spam filters. A similar thing seems to be happening to the web.

Everyone loves the dream of a free for all and open web.

But the reality is how can someone small protect their blog or content from AI training bots? E.g.: They just blindly trust someone is sending Agent vs Training bots and super duper respecting robots.txt? Get real...

Or, fine what if they do respect robots.txt, but they buy the data that may or may not have been shielded through liability layers via "licensed data"?

Unless you're reddit, X, Google, or Meta with scary unlimited budget legal teams, you have no power.

Great video: https://www.youtube.com/shorts/M0QyOp7zqcY

(comment deleted)
You can't trust everyone will be polite or follow "standards".

However, you can incentivize good behavior. Let's say there's a scraping agent, you could make a x402 compatible endpoint and offer them a discount or something.

Kinda like piracy; if you offer a good, simple, cheap service people will pay for it versus go through the hassle of pirating.

> But the reality is how can someone small protect their blog or content from AI training bots? E.g.: They just blindly trust someone is sending Agent vs Training bots and super duper respecting robots.txt? Get real...

baking in hashcash into http 1.0/1.1/1.2/2/3, smtp, imap, pop3, tls and ssh. then this will all to expensive for spammers and training bots. but IETF is infiltrated by government and corporate interests..

Why do you have to protect it? Have you suffered any actual problem, or are you being overly paranoid? I think only a few people have actually received DDoS-level traffic, and the rest are being paranoid.
Yeah where does this absolute bullshit comes from and why would anyone target your blog?
The problem's cause really is about _who_ has to pay for the traffic, and currently that's the hosting end. If you turn that model around, suddenly AI web scrapers have to behave and all the issues that we currently have are kind of solved(?), because there is no incentive to scrape datasets anymore that were put together by others, and there automatically will be a payment incentive instead to buy high-quality datasets.
But I don't want to make human users of my website pay for the traffic just like I also donate to real world charities that I believe in.
How can someone small protect their IP? It's called copyright law and it's been around for a long ass time just for some reason big tech gets a pass and can steal and control whatever the fuck they want without limit
This isn't even an issue, you have made that problem up. I host a blog and there are some AI bots coming around. Big deal. Most of them do respect a robots.txt. Some don't. Not a big deal as well.

In contrast trying to change the infrastructure of the net, which previously was quite resistant to censorship is quite a big deal.

This sounds exactly like a crazy preacher warning about the dangers of rock music. A completely made up threat. And we need the protection of god against these evil AI bots.

Wow, a bot that disrespected a robots.txt. How can the internet survive...

Also, OpenAI already has the data. You want to ensure they will never get competitors by putting up barriers now. It makes no sense...

This is like saying companies don't need security gates and checkpoints. Unfortunately the world is filled with bad people, and you need security to keep them off your property.
We have far too many gatekeepers as it is. Any attempt to add any more should be treated as an act of aggression.

Cloudflare seems very vocal about its desire to become yet another digital gatekeeper as of late, and so is Google. I want both reduced to rubble if they persist in it.

Several companies are looking to provide a solution for the AI bot problem. Cloudflare stands to make a lot of money if people pick their solution. But Cloudflare backing down won't make the problem go away, and someone else's bad solution will be chosen instead.

The gatekeeping described here is gatekeeping a website owner chooses. It's an alternative to pay walls, bespoke bot detection, or some kind of ID verification. Cloudflare already provides a service, but standardising the service will open up the market (at the cost of competitors adopting Cloudflare's standard).

The freedom of the open web also extends to the owners of the websites people visit.

You could use the same argument against antitrust laws. All monopolies were "chosen" by their customers, that doesn't mean we should give them a free pass.
What do you mean Google "desires" to become a gatekeeper? They have been a gatekeeper for years, since they control the browser everyone uses, and Firefox usage is now in the noise. Google just steers the www where they want it to go. Killing ublock, pushing .webp trash, etc.
I think about this as a startup founder building a 'proof-of-human' layer on the Internet.

One of the hard parts in this space is what level of transparency should you have. We're advancing the thesis that behavioral biometrics offers robust continuous authentication that helps with bot/human and good/bad, but people are obviously skeptical to trust black-box models for accuracy and/or privacy reasons.

We've defaulted to a lot of transparency in terms of publishing research online (and hopefully in scientific journals), but we've seen the downside: competitors fake claims about their own best in-house behavioral tools that is behind their company walls in addition to investors constantly worried about an arms race.

As someone genuinely interested (and incentivized!) to build a great solution in this space, what are good protocols/examples to follow?

> An allowlist run by ONE company?

An allowlist run by one company that site owners chose to engage with. But the irony of taking an ideological stance about fairness while using AI generated comics for blog posts…

> An allowlist run by one company that site owners chose to engage with. The allow/blocklist is pre-filled by CloudFlare.

The bots/crawlers/browsers are pre-categorized by CloudFlare.

Defaults matter and how CloudFlare categorizes your privacy-focuses or agentic browser would impact your experience on a good chunk of the web.

I think it shouldn't require registering /with/ cloudflare. cloudflare should just look up the .well-known referenced and double check for impersonation, and keep score on how well behaved each one is.
Using completely automated means would leave open the possibility to set up a new signature for every single request, or for batches of requests. The manual step is to cut down on the amount of automated abuse.
Brought to you by substack. ;-) Seriously though, great post and a great conversation starter.
as a Cloudflare customer, I am happy with their proposition. I personally do not want companies like Perplexity that fake their user-agent and ignore my robots.txt to trespass.

and isn't this why people sign up with Cloudflare in the first place? for bot protection? to me, this is just the same, but with agents.

i love the idea of an open internet, but this requires all party to be honest. a company like Perplexity that fakes their user-agent to get around blocks disrespects that idea.

my attitude towards agents is positive. if a user used an LLM to access my websites and web apps, i'm all for it. but the LLM providers must disclose who they are - that they are OpenAI, Google, Meta, or the snake oil company Perplexity

> When I’m driving, I hand my phone to a friend and say, “Reply ‘on my way’ to my Mom.” They act on my behalf, through my identity, even though the software has no built-in concept of delegation. That is the world we are entering.

That is a very small part of the world we're entering.

The other vast majority of use cases will come from even more abusive bots than we have today, filling the internet with spam, disinformation, and garbage. The dead internet is no longer a theory, and the future we're building will make the internet for bots, by bots. Humans will retreat into niche corners of it, and those who wish to participate in the broader internet will either have to live with this, or abide by new government regulations that invade their privacy and undermine their security.

So, yes, confirming human identity is the only path forward if we want to make the internet usable by humans, but I do agree that the ideal solution will not come from a single company, or a single government, for that matter. It will be a bumpy ride until we figure this out.

I use uncommon web browsers that don't leak a lot of information. To Cloudflare, I am indistingushable from a bot.

Privacy cannot exist in an environment where the host gets to decide who access the web page. I'm okay with rate limiting or otherwise blocking activity that creates too much of a load, but trying to prevent automated access is impossible withou preventing access from real people.

And god forbid you live in an authoritarian country and must use VPN to protect your freedom. Internet becomes captcha hell run by 2-3 companies.

I've had far fewer issues with my own bots that access cloudflare protected websites, than during my regular browsing with privacy respecting browsers and a VPN.

As a side note: I'm at least thankful Microsoft isn't behind web gatekeeping. Try and solve any microsoft captcha behind a VPN - its like writing a thesis, you gotta dedicate like 5 minutes, full attention.

Do you currently get blocked a lot by Cloudflare/turnstile a lot then? Sorry, I think you implied that, just want to be clear.
Thank you. I've been telling people this for years and mostly get treated like I'm crazy or told to try other browsers, but browsers are pretty personal and lived-in, I consider it quite uncomfortable to use a different one even to temporarily access a given site, and it always leaves a bad taste in my mouth. I am very comfortable in qutebrowser and I feel a bit disabled when I have to load up LibreWolf or Ungoogled Chromium. The vim addons don't help, they're not even half as good. Plus most sites I use regularly work fine, so when something I don't use often doesn't work right, I'm definitely blaming cloudflare or the site operator, not my browser that works fine otherwise.

Honestly, just let the bots or whatever through. It's absolutely ridiculous locking out real people who did nothing wrong.

I suppose it’s time AI proselytization rediscovered the tragedy of the commons.