Some companies publish the IP ranges of their crawler (OpenAI [1], Mistral [2] for example) but many like Anthropic don’t. Not sure those lists can be fully trusted though. Perplexity, for instance, was caught using IPs…
Marco made a review of John Siracusa review ;) http://www.marco.org/2012/07/25/siracusa-mountain-lion-revie...
Some companies publish the IP ranges of their crawler (OpenAI [1], Mistral [2] for example) but many like Anthropic don’t. Not sure those lists can be fully trusted though. Perplexity, for instance, was caught using IPs…
Marco made a review of John Siracusa review ;) http://www.marco.org/2012/07/25/siracusa-mountain-lion-revie...