Having worked with/on two large-scale web crawling systems at FAST Search & Transfer; The C-Crawler that powered alltheweb.com (owned now by Yahoo) and a python-based crawler that was a workhorse for FAST…
Having worked with/on two large-scale web crawling systems at FAST Search & Transfer; The C-Crawler that powered alltheweb.com (owned now by Yahoo) and a python-based crawler that was a workhorse for FAST…