Crawly – Never write another web scraper (crawly.diffbot.com)

Link to more of an overview / documentation without having to fork over an email address? Silly.

0. Arrived at site

1. Entered HN URL

2. Entered email reluctantly

3. Checked email, nothing received

4. Started writing comment on HN, frustrated

5. Checked email again, still nothing

6. Posted comment, disappointed

This won't work when I need to scrape 100,000 pages in an hour.

I got a reply within 3 minutes.

The email said, "We set our crawler loose on <site>, and WOW did we find some interesting results."

The resulting CSV file? 0 bytes. I guess that's "interesting" in its own way.

www.ncbi.nlm.nih.gov };)

CloudScrape works