Ask HN: Crawl the entire web?

2 points by nullundefined 8y ago ↗ HN

If you were tasked with crawling the entire web, how would you go about doing it?

2 comments

[ 3.3 ms ] story [ 16.6 ms ] thread

I'd grab the CommonCrawl data and keep moving on what my actual problem is.

[dead] onkdinlsi 8y ago ↗