Such a cool idea. I have https://chess.maxmcd.com/ and (before blocking most of them) many bots played thousands of moves deep. I remember bingbot was very active.
If it’s been this way since February, how have AI crawlers not “caught up” yet?
The internet is big, but it isn’t that big. I’d expect to see a sudden dropoff as they start re-checking content that hasn’t changed, with some sort of exponential backoff.
Instead, my takeaway is that they are AI crawlers aren’t indexing to store in a way we’re used to with typical search engines, and unilaterally blocking these crawlers across the board would result in quite the “effect”.
The "Generative AI services popularity" [1] chart is surprising. ChatGPT is being #1 makes sense, but Character.AI being #2 is surprising, being ahead of Anthropic, Perplexity, and xAI. I suspect this data is strongly affected by the services DNS caching strategies.
The other interesting chart is "Workers AI model popularity" [2]. `llama-3-8b-instruct` has been leading at 30% to 40% since April. That makes it hands the most popular weights available small "large language model". I would have expected Meta's `m2m100-1.2b` to be more used, as well as Alphabet's `Gemma 3 270M` starting to appear. People are likely using the most powerful model that fits on a CF worker.
As shameless plug, for more popularity analysis, check out my "LLM Assistant Census" [3].
Feels like Cloudflare are positioning themselves as the gatekeepers of "good bots". The fact there is an "In Progress" state at all is telling: for everyone else, the answer is "No", but for OpenAI, the answer is "we're not doing it yet, but we've told CF that we plan to".
Honestly, I am shocked there hasn't already been an anti-trust case against cloudflare. They are so dominant, I rarely meet a customer that doesn't have an implementation utilizing their reverse proxy or other ZTNA functionality.
My main learning is that character.ai is consistently in the top four, along with ChatGPT (always #1) and Claude. I didn't even know it was in the running.
This data is incredibly valuable for both AI companies and publishers. CF gets unprecedented visibility into who's crawling what, when, and how much. Wouldn't be surprised if this becomes a premium product - 'pay for priority bot verification' or 'detailed crawl analytics.
Very interesting data, particularly the AI rankings based on DNS requests. They appear to be off by one day because switching to a 4 week period, character AI is consistently #2 on weekends and Claude is #3 and they switch weekdays. But it’s shows the switch for Sunday and Monday. Probably a US time vs UTC issue.
If I use Anthropic’s api for search, but then send user traffic directly to websites after showing the user the link, there’s no way for cloudflare to attribute that search to Anthropic.
That makes the ratios of crawl to referrals shown suspect.
My experience disagrees with the 'Respects robots.txt' column for most of the bots listed. Would love to see more details of how they determine that metric.
One way that Cloudflare is gatekeeping is by declaring which bots are AI Bots. Common Crawl's CCBot is used for a lot of stuff -- it's an archive, there are more than 10,000 research papers citing common crawl, mostly not AI -- but Cloudflare deems CCBot to be an "AI Bot", and I suspect most website owners don't have any idea what the list of AI Bots is and how they were chosen.
Perhaps this data could provide a useful example for Apple and OpenAI in their defence against Elon's laughable lawsuit. It's funny how xAI is almost at the bottom.
37 comments
[ 2.7 ms ] story [ 61.9 ms ] thread[1] https://deep.43z.one
The internet is big, but it isn’t that big. I’d expect to see a sudden dropoff as they start re-checking content that hasn’t changed, with some sort of exponential backoff.
Instead, my takeaway is that they are AI crawlers aren’t indexing to store in a way we’re used to with typical search engines, and unilaterally blocking these crawlers across the board would result in quite the “effect”.
The "Generative AI services popularity" [1] chart is surprising. ChatGPT is being #1 makes sense, but Character.AI being #2 is surprising, being ahead of Anthropic, Perplexity, and xAI. I suspect this data is strongly affected by the services DNS caching strategies.
The other interesting chart is "Workers AI model popularity" [2]. `llama-3-8b-instruct` has been leading at 30% to 40% since April. That makes it hands the most popular weights available small "large language model". I would have expected Meta's `m2m100-1.2b` to be more used, as well as Alphabet's `Gemma 3 270M` starting to appear. People are likely using the most powerful model that fits on a CF worker.
As shameless plug, for more popularity analysis, check out my "LLM Assistant Census" [3].
[1] https://radar.cloudflare.com/ai-insights#generative-ai-servi...
[2] https://radar.cloudflare.com/ai-insights?dateRange=24w#worke...
[3] https://aleyan.com/blog/2025-llm-assistant-census/
> Verified via WebBotAuth: In Progress
Feels like Cloudflare are positioning themselves as the gatekeepers of "good bots". The fact there is an "In Progress" state at all is telling: for everyone else, the answer is "No", but for OpenAI, the answer is "we're not doing it yet, but we've told CF that we plan to".
> Firerox 3.8%
This is sad.
https://radar.cloudflare.com/adoption-and-usage
but around 2010-ish, chrome got way better and superior in every way. even I cant ignore that and switch to chrome
until they recently nerf adblock and I use dual browser, good thing firefox is still there. but I cant say the same for 20 years in the future
Web Bot Auth
https://news.ycombinator.com/item?id=45055452
That makes the ratios of crawl to referrals shown suspect.
I sincerely hope this initiative fails and no one bends over for CloudFlare on this.
So interesting they are orders of magnitude worse than the others with the crawl:user-request ratio... noted