Ask HN: List of known AI dataset, training, and access crawlers

2 points by verygoode ↗ HN
Looking to see if anyone has created or found a well-maintained list of known AI dataset and training crawlers.

This information can be useful if you want to keep stats or attempt to limit or block training.

E.g., OpenAI's user-agent for their bot is `GPTBot`

Common Crawl's is `CCBot`.

If you aren't aware of such a list, would you find one useful?

1 comment

[ 2.5 ms ] story [ 39.8 ms ] thread