2 comments

[ 4.9 ms ] story [ 17.8 ms ] thread
Wait, Common Crawl is part of Automattic? I had no idea!
It's not. The example shows them crawling a page from 102jamzorlando.cbslocal.com, which is hosted by Wordpress.com. Apparently Automattic inserts that recruitment ad into the headers of every site they host. (At least, all the ones I've checked so far.)