Ask HN: Crawl the entire web?

2 points by nullundefined 9y ago ↗ HN

If you were tasked with crawling the entire web, how would you go about doing it?

2 comments

[ 3.2 ms ] story [ 8.2 ms ] thread

I'd grab the CommonCrawl data and keep moving on what my actual problem is.

[dead] onkdinlsi 9y ago ↗