1 comment

[ 3.0 ms ] story [ 10.0 ms ] thread
Cloudflare has launched a private beta feature called Pay per Crawl, whose sole purpose is to let a website owner charge an AI crawler a fixed fee each time the crawler requests a page. The feature addresses a common operational gap: currently, a publisher can either leave all content open to automated collection or block crawlers entirely, and any paid arrangement must be negotiated manually.