Cloudflare has launched a private beta feature called Pay per Crawl, whose sole purpose is to let a website owner charge an AI crawler a fixed fee each time the crawler requests a page. The feature addresses a common operational gap: currently, a publisher can either leave all content open to automated collection or block crawlers entirely, and any paid arrangement must be negotiated manually.
1 comment
[ 3.0 ms ] story [ 10.0 ms ] thread