Ask HN: List of known AI dataset, training, and access crawlers

2 points by verygoode ↗ HN
Looking to see if anyone has created or found a well-maintained list of known AI dataset and training crawlers.

This information can be useful if you want to keep stats or attempt to limit or block training.

E.g., OpenAI's user-agent for their bot is `GPTBot`

Common Crawl's is `CCBot`.

If you aren't aware of such a list, would you find one useful?

1 comment

[ 495 ms ] story [ 311 ms ] thread