What do you mean by "rule"? Depending on the exact circumstances, violating terms of service or ignoring robots.txt may not be a violation of criminal law or create any civil liability. In particular, scraping public data is generally legal under the CFAA regardless of robots.txt content.
That's a cool article to read. It explicitly wonders whether a robots.txt is enough to revoke authorization. And it seems like the DoJ does allow itself to consider the different blocking mechanisms used (including robots.txt) on whether to prosecute.
The DoJ is explicit in saying that something like a Cease and Desist is enough, so if for example the NYTs found OpenAI's bot then that would likely be prosecutable.
6 comments
[ 3.8 ms ] story [ 15.3 ms ] threadhttps://newmedialaw.proskauer.com/2022/05/24/doj-revises-pol...
As a practical matter, if web site owners don't like particular HTTP requests then they can just ignore them or return errors or junk responses.
The DoJ is explicit in saying that something like a Cease and Desist is enough, so if for example the NYTs found OpenAI's bot then that would likely be prosecutable.