Cloudflare are setting themselves up as the arbiter who will decide which requests are a) human, b) authorized AI bots, c) illicit/banned bots.
Given the number of people on HN who report massive problems from scrapers and other bots, it sounds like if Cloudflare doesn't do this, someone else will need to. I might have thought bandwidth was cheap enough now for it not to matter, but I guess the bots are costing some sites a lot of money.
Cloudflare doesn't block bots. It's trivial to use residential proxies and your very obvious bot will only get blocked maybe 5% of the time when using rotating IPs.
- block larger cohorts of traffic, affect many real users
- babysit the rules to get them just right, lose time doing that
In my experience the residential proxies exist but are not that common and many aren't trying as hard as they could. It's really a war of which side wants to spend more attention on the problem.
It isn't bandwidth that causes problems. It is the various types of load that they can add to your servers. Which is why no centralized vendor can decide the proper caching or what bots should be blocked, allowed, rate limited, etc. Those things do not have standard answers - it depends on what your apps do, your audience, their usage patterns, and sometimes the regulatory environment in which you run.
I've been using the web search in OpenRouter, which is similar in that it's a wrapper around other search engine providers. It's really convenient to be able to experiment with new models and new search engines without having to go through corporate hoops to subscribe to a new service.
I think Cloudflare is (for companies already using it) approaching the status of trusted main cloud supplier (which usually would be AWS, GCP, Azure) via which the majority of cloud costs are billed (so you don't have to go through a fresh procurement process).
It means "hi spending approver, I'm going to add $100 to our CF account" instead of "hi accounting+management+security, please initiate the process of evaluating new third party vendor Foo for use in my project, I hope we can get it approved and integrated into SSO sometime next month".
There is nothing wrong with “inserting yourself in the middle of everybody’s business”. That is how you reduce friction between parties and make optimizations that are only possible by being able to manage both sides of the connection. It’s also a really good way to make money.
I think their strategy is: "AI coding means we can build everything. Our infrastructure approach is incredibly quick to build upon, so why not build it all ourselves and then anyone with half a brain will move their stuff to Cloudflare and leave AWS in the dust."
This practice of having the one provider should be eliminated. Companies self-inflict lock-in to large platform providers, prevent their own teams from using better technology options and stifle innovation. It's crazy that even with a pile of SOC/ISO/PCI/HIPAA/NIS certificates, procurement is still a months-long process, it should be much easier to do business.
Replace trusted with convenient. They're glowing pretty hard giving out all that stuff very cheap in exchange for being the middle man on everything. Not that I mind for my trivial use case.
9. None of this matters, because personal apps will use none of that because individual personal customers are way cheaper than organizations, and will use whatever is included in their subscriptions (e.g. ChatGPT Sites).
Weird choice by CloudFlare, would been great if they have shared why it was created.
I use CloudFlare developer platform and quite happy with tools, but I didn’t use the gateway API and always used OpenRouter which does support web search.
I can see it useful for those who didn’t do any integrations or like to keep logs at one place, but did customers actually ask for this?
A related Q: why are their products so popular? I get why CDN/DDOS protection is but what about everything else? I have never ever found a use for stuff like Workers. (Sincerely asking, not dismissing them as useless)
Pages is a very convenient way of deploying static websites/SPAs with a generous free tier. You just need to find the tiny links in their dashboard to avoid accidentally using workers instead (which is supposed to supersede it but is clearly worse for this usecase).
Workers have a nice and easy deployment model (when it's not broken) compared to AWS lambda, so I get why people are tempted. It's one simple file compared to 4 separate pieces of infra. But yes, please, use anything else that doesn't pay for the CloudFlare protection racket. For example there's https://bunny.net/edge-scripting/
The workers paid tier is $5/month and you can do a LOT with that. Once you drink the cloudflare kool-aid regarding workers they let you build very scalable apps while jumping through many fewer hoops than you would on AWS/GCP, at a fraction of the cost.
Note - you should not be hosting purely static sites with Workers, if you can avoid it, give CF Pages is actually free and unlimited (assuming you don't use "Functions", which are just workers in disguise).
I think they've been merged, is my point. It's managed through the same interface now.
On the Pages docs website there is even a header that encourages you to use workers (though I don't think you're given a choice at this point), like pages in general are being deprecated.
---
Are you sure you want to use Pages?
Workers supports most Pages use cases and offers a broader feature set. It is Cloudflare's primary platform for building applications. Start new projects with Workers.
Pages are now essentially deprecated and Workers have dedicated ways to host static files and are the recommended path going forward for static and dynamic websites
> Workers supports most Pages use cases and offers a broader feature set. It is Cloudflare's primary platform for building applications. Start new projects with Workers.
And I see that the assets[0] wrangler config key can also be setup so that hits it prefers hitting static files, so that they can still be served in a free and unlimited manner.
Although I imagine an official deprecation probably is a long ways off, and if they do it I hope they'd handle a large-scale migration for each page to its own worker with assets preserved.
We will not force users to migrate from old Pages to Pages-in-Workers manually. We will either implement an automatic migration, or just keep supporting old Pages forever.
Workers is compute-on-demand and particularly only pay what you use. Especially in the current days, something like Workers is infinitely appealing if you don't want to manage or pay for a dedicated server, assuming the service you want to run can be completely hosted (or a replacement vibe-coded) for the Service Workers API / deployed to CF's platform.
You also don't pay for actual data transfer, so the billing is overall simpler - AWS and GCP have similar per-request compute options, but every part of the platform has extra fees (like per-gb data transfer billing, sometimes you need a VPC to interconnect services, secrets being an extra charge, etc).
I suspect also to do with internal Slack etc. where employees vote on launch posts (a lot of them on HN since long). Not a scam or accusing anyone but this probably propels a lot.
For those developers out there, the best is still Gemini Flash Lite 2.5 believe it or not. It gives you 1000 google searches per day for free. Compare to Flash Lite 3.x which is 5k PER MONTH and then a few pennies PER SEARCH. Nuts. Didn’t realize search was so expensive.
Perhaps realizing all of this, Google hasn’t yet deprecated 2.5, bit limits access to it to “those who have used it before.”
That’s the enterprise agent one, still available for API. Oh and I got it wrong… it’s actually 1500 searches per day free! Gemini Flash 2.5 also has this btw but I prefer Flash Lite to reduce costs. It’s nuts when you compare cost and features to any of the 3.x models. https://ai.google.dev/gemini-api/docs/pricing
So I wish I could use Google for https://veruscite.com/, but the number of Google searches are a hard cap on the account! So yes that is fine for agentic coding, but for an app that relies on web-search is not sufficient.
I am currently using Perplexity fast search and fetch, and I am happy with that. I would try our Ceramic.ai, but I need to be able to fetch the pages as well (I do not want summaries).
(I work at Linkup.) We do both: search returns raw results, no summaries, and there's a separate fetch endpoint that returns the full page as markdown, with optional JS rendering. You should compare us against your Perplexity setup - you might some value in switching
Put in the todo list to check out. If you cut web search costs even further for flash/fast search (similar to Perplexity, which is $1 per 1000 for fast search) would make it more competitive (at least for my app!)
Can Gemini Flash Lite 2.5 be made to return raw search results. Some 'search' providers I looked at returned summaries, or vector relevance matches (of presumably a smaller page set).
1. Google News API now returns only Google links that don't resolve to anything in code.
2. Google Search results are atrocious and only unearth non-authoritative blogspam and aggregator sites.
Tried one query on ceramic.ai (the default provider for cloudflare web search api): "qwen-3.8 flash next and rtx 5090 best inference setup" ... 0 results ... same query on google and ddg both yield proper results.
Then shortened the query to just "qwen-3.8 flash next" ... results came.. all unrelated. In fact, these were almost all paper links .... no relation to actual search term.
And I had thought that I finally had found a cheaper search alternative.
I made a zero ads SERP using one of these "AI first" search providers: https://github.com/scosman/froogle (live version https://froogle.fyi). In this case Keenable.ai. Generally the same pattern: it's not usable.
It's okay but not great. I'm always annoyed grokipedia outranks wikipedia. And some queries return the definitive link at position 5/6 behind grokipedia and other slop.
You know the craziest part? This time I searched for their own website address: "ceramic.ai" .. results came... none pointing to the website or any page on it.
Then searched for "Cloudflare OHTTP Gateway" .. this text is literally in the title ... but zero link for this page.. the closest it yielded was this link: "https://developers.cloudflare.com/privacy-gateway/" ... it seems cloudflare updated this 2 days back.. the original content was last updated in 2022 ... so that's what the cutoff index seems to be.
I guess I'm not understanding the value here - to compete with Google and the likes, the scale, cost and complexity would be huge. Appreciate new entrants in an existing field but not seeing this one.
>"to compete with Google and the likes, the scale, cost and complexity would be huge."
CloudFlare's entire business is scale, cost, and complexity. They are powering like half the web at this point. Wouldn't really call them a "new entrant".
My coding agent uses the hister cli, i.e. a local index. That often requires me to seed it manually as a downside. The upside is that it caches website contents via browser plugin, which is a nice workaround for bot blocking.
What do you mean by seeding it manually? Like do you programmatically "browse" to bolster your hister index? Asking because I started using hister a month or so ago and have been really liking it, and I'm curious how others are using it.
Yes, I browse around, open a dozen tabs so they get indexed, and close them without reading. Then a coding agent is pretty good in composing a report from that index. Generated these recently: https://qznc.github.io/sloppy_research/en/
Hister tells me my index is currently 39208 pages.
Let's not conflate crawlers with the traffic that bot protection services block. A crawler that respects robots.txt is a good internet citizen and can provide a vital service.
That doesn't match my experience. For example, I recently had Meta scrape robots.txt disallowed paths. On some page they claimed they respect it but they don't. And yes, all the IPs they used to connect to me (they used a different /64 network for each connection) were owned by Meta, it was not someone pretending to be them.
The era of robots.txt files being reasonable is gone too. Usually they either block everything, or block everything except Googlebot. If you want to make a search engine at all, you have to ignore robots.txt, ignore noindex, ignore nofollow, and so on. If we had your way, Google would have a legally enforced search engine monopoly.
However, so-called AI crawlers are not the same as crawlers of yore. They hit live pages every time a user prompt triggers a web search.
This Web Search API, unlike an AI crawler, only fetches periodically. It feels like a step in the right direction for managing resource strain across the internet. If only the LLM giants could do something similar.
The "LLM giants" do both. They have to create an index to know which pages to even fetch in the first place. A simple web search does not actually necessarily hit any pages it's only when it does a page fetch based on that search
It's peak capitalist enshittification. The more you sell the problem, the more you can sell the solution. If something makes you money, you should do it.
Yeah, I keep wondering if I should switch to something cheaper, but I'm too lazy to evaluate service qualities across providers, and I don't really use enough for it to matter. If they are the most expensive because they're the best, I'm fine with that, but have no idea.
I wish non-fuzzy searching was still trendy. It sucks when I know what I need and remember a bunch of keywords from the page, but search engines return either 0 results or a bunch of results that don't even contain the keywords.
Does anyone have insights on the quality differences?
Web search API pricing for AI agent usecases has always felt so expensive for what it is, but I have no grounding on the economics of running a web index.
I was just looking into these as DeepSeek Harness w/ Qwen3.8-27B relies heavily on search. I was going to go with Serper.dev[0] $1 per 1000 (or lower in quantity).
The providers[1] behind this Web Search API have very different rates:
Ceramic.ai: $0.25 per 1,000 requests
Linkup: $5.00 per 1,000 requests
Exa: $7.00 per 1,000 requests
261 comments
[ 0.25 ms ] story [ 21.8 ms ] threadGiven the number of people on HN who report massive problems from scrapers and other bots, it sounds like if Cloudflare doesn't do this, someone else will need to. I might have thought bandwidth was cheap enough now for it not to matter, but I guess the bots are costing some sites a lot of money.
- let more traffic in, eat the compute cost
- block larger cohorts of traffic, affect many real users
- babysit the rules to get them just right, lose time doing that
In my experience the residential proxies exist but are not that common and many aren't trying as hard as they could. It's really a war of which side wants to spend more attention on the problem.
It means "hi spending approver, I'm going to add $100 to our CF account" instead of "hi accounting+management+security, please initiate the process of evaluating new third party vendor Foo for use in my project, I hope we can get it approved and integrated into SSO sometime next month".
1. Due to AI, most of the software is going to be personal apps.
2. For personal apps, (even most commercial SaaS) SQLite is all you need.
3. The Javascript ecosystem is huge in terms of components, libraries. ShadCN and what not.
4. Cloudflare has workers, durable objects which is isolated SQLite, D1 which is isolated SQLite
5. You can run crons, long running jobs, cloudlflare has emails.
6. Cloudlflare has best in class open weigh AI models as API for cheap.
7. Cloudflare has amazing CLI wrnagler but upcoming cf is purpose built for agentic workflows.
8. The built in AI assistant is really really good in guiding you how to setup, architecture and work around limitations etc.
Combine all the above, in next 5 years Cloureflare is going to get stronger with more workloads deployed to it then all three hyper clouds combined.
Why would I use bonsai? Why not use ElasticSearch directly?
The absence of the Perplexity Search API is to be expected though, knowing how much these two companies despise each other.
I use CloudFlare developer platform and quite happy with tools, but I didn’t use the gateway API and always used OpenRouter which does support web search.
I can see it useful for those who didn’t do any integrations or like to keep logs at one place, but did customers actually ask for this?
however now i just have agent do it with CLI and that works well and I don't have to find the secret link
Running your stuff, even private stuff, through a tunnel is great so that you don't have to expose your VPS' IPv4.
On the Pages docs website there is even a header that encourages you to use workers (though I don't think you're given a choice at this point), like pages in general are being deprecated.
---
Are you sure you want to use Pages?
Workers supports most Pages use cases and offers a broader feature set. It is Cloudflare's primary platform for building applications. Start new projects with Workers.
---
https://developers.cloudflare.com/pages/
Pages are now essentially deprecated and Workers have dedicated ways to host static files and are the recommended path going forward for static and dynamic websites
> Workers supports most Pages use cases and offers a broader feature set. It is Cloudflare's primary platform for building applications. Start new projects with Workers.
And I see that the assets[0] wrangler config key can also be setup so that hits it prefers hitting static files, so that they can still be served in a free and unlimited manner.
Although I imagine an official deprecation probably is a long ways off, and if they do it I hope they'd handle a large-scale migration for each page to its own worker with assets preserved.
0: https://developers.cloudflare.com/workers/static-assets/bill...
You also don't pay for actual data transfer, so the billing is overall simpler - AWS and GCP have similar per-request compute options, but every part of the platform has extra fees (like per-gb data transfer billing, sometimes you need a VPC to interconnect services, secrets being an extra charge, etc).
Perhaps realizing all of this, Google hasn’t yet deprecated 2.5, bit limits access to it to “those who have used it before.”
It’s really really good for low cost search!
(Was your comment a joke? Or did google announce this through different channels?)
0: https://ai.google.dev/gemini-api/docs/deprecations
https://docs.cloud.google.com/gemini-enterprise-agent-platfo...
I am currently using Perplexity fast search and fetch, and I am happy with that. I would try our Ceramic.ai, but I need to be able to fetch the pages as well (I do not want summaries).
1. Google News API now returns only Google links that don't resolve to anything in code. 2. Google Search results are atrocious and only unearth non-authoritative blogspam and aggregator sites.
[1]: https://aistudio.google.com
Edit to add that my ai scientists currently use google’s vertex for web search and grounding as well as firecrawl.
But does CloudFlare itself commit to zero data retention? If not, this isn’t too meaningful.
Then shortened the query to just "qwen-3.8 flash next" ... results came.. all unrelated. In fact, these were almost all paper links .... no relation to actual search term.
And I had thought that I finally had found a cheaper search alternative.
Then searched for "Cloudflare OHTTP Gateway" .. this text is literally in the title ... but zero link for this page.. the closest it yielded was this link: "https://developers.cloudflare.com/privacy-gateway/" ... it seems cloudflare updated this 2 days back.. the original content was last updated in 2022 ... so that's what the cutoff index seems to be.
CloudFlare's entire business is scale, cost, and complexity. They are powering like half the web at this point. Wouldn't really call them a "new entrant".
Thanks asciimoo for https://github.com/asciimoo/hister
Hister tells me my index is currently 39208 pages.
The original request via the MCP is somehow blocked?
does it? I am running it but was under the impression that it did not cache the content I am viewing, unlike SinglePage.
4.1G ~/.config/hister/
3.1G ~/.config/hister/data/
But that's meaningless because 99% of AI crawlers are "bad bots" which ignore robots.txt and use domestic IPs to circumvent blocks.
Some other reports of this: https://github.com/TecharoHQ/anubis/issues/1565
I ended up just fully closing connections with no response from these assholes on any URL.
This Web Search API, unlike an AI crawler, only fetches periodically. It feels like a step in the right direction for managing resource strain across the internet. If only the LLM giants could do something similar.
1: https://kagi.com/api/docs/openapi
I noticed that my API quota resets every month. Have not been charged once.
What's your plan? I'm on Duo and I get charged for every MCP hit. If an API quota is included with Ultimate, that might be worth my upgrading.
ceramic.ai - $0.25 per 1,000 requests
Exa - $7.00 per 1,000 requests
Linkup - $5.00 per 1,000 requests
Does anyone have insights on the quality differences?
Web search API pricing for AI agent usecases has always felt so expensive for what it is, but I have no grounding on the economics of running a web index.
EDIT: formatting
They have 2,500 free which I used, it seemed good.
I'm unaffiliated--actually a clanker told me about it so I told it "go ahead"
And then on the providers page:
The providers[1] behind this Web Search API have very different rates:
[0] https://serper.dev/[1] https://developers.cloudflare.com/web-search/providers/
[0] https://serpbase.dev/blog/serpbase-vs-serper-dev-serp-api-pr...