[–] cratermoon 2y ago ↗ To the extent that robots.txt still works, see Block the Bots that Feed “AI” Models by Scraping Your Website: https://neil-clarke.com/block-the-bots-that-feed-ai-models-b...You can also block user agents directly, for example in nginx. https://www.xmodulo.com/block-specific-user-agents-nginx-web...The UAs I'm aware of: CCBot ChatGPT-User GPTBot Does anyone know of a resource that tracks these crawlers' UA strings?
1 comment
[ 3.9 ms ] story [ 10.1 ms ] threadYou can also block user agents directly, for example in nginx. https://www.xmodulo.com/block-specific-user-agents-nginx-web...
The UAs I'm aware of:
Does anyone know of a resource that tracks these crawlers' UA strings?