HN Explorer

Ask YC: How to web scrape 100s of sites/forms?

1 points by bdouglas ↗ HN
hi...

a potential research app requires the scraping of a few hundred websites/forms and diving into the child links to obtain the linked/parent structure ie. company->dept->title->name.

in this case, this would involve going 4 levels deep, and getting the required information.

so, does anyone know of a method/app/company that can be used to accomplish this. orm am i going to have to figure out how to get a number of cheap guys to write a bunch of python scripts!!

thanks

5 comments

[ 3.0 ms ] story [ 20.2 ms ] thread
nreece
...cheap guys to write a bunch of python scripts

You know what will be 'cheap'. Writing it yourself.

olefoo
Or get one expensive guy to write you a script that

writes the scripts to scrape the sites by scraping the sites to read the structure to write the scripts to scrape the sites.

Anon84
Check out the search.wikia.org project. They make their crawler (and crawl data) available. Maybe you can get away with using theirs. That would really be cheap!
qhoxie
Libraries like mechanize and hpricot are shrinking the curve for scraping tasks. That's not to say it is easy, but it should not take a bunch of people working on it. One good developer with proper experience would be ample in my opinion.
gaius
I am unable to think of an application for this technology other than spamming. Care to provide more details before we shoot ourselves in the face by helping you?