[–] spoondocz 9y ago ↗ What's the largest crawl you can handle? meaning your upper limit... [–] mixnode 9y ago ↗ We've done crawls of over 250TB and 3 billion web pages. If you need to crawl more than this you can simply shoot us an email and we'll provision resources for your mega crawl.
[–] mixnode 9y ago ↗ We've done crawls of over 250TB and 3 billion web pages. If you need to crawl more than this you can simply shoot us an email and we'll provision resources for your mega crawl.
3 comments
[ 3.9 ms ] story [ 145 ms ] thread