AI crawlers in robots.txt: a study of the top 10k sites (2026) (hitmind.ai) 1 points by mimimile 8h ago ↗ HN
1 comment
[ 6.0 ms ] story [ 33.4 ms ] thread