HN Explorer

10 comments

[ 5.0 ms ] story [ 30.7 ms ] thread
thesehands
Remember to scrape responsibly, check terms of service, robots.txt etc.
eccfcco15
Why? Isn’t it better to use a vps or two (or even just a vpn), and strong rate limiting? I.e. try not to be noticed regardless of the robots.txt.
slaymaker1907
A lot of robots.txt exist to help crawlers automatically know which URLs are safe and which ones are mutable (i.e. the sign up link).

You should only ignore the robots.txt if you are very careful about what you are doing and have a very good reason for doing so.

slezyr
Just in time. That XPath example is really useful for me.

I didn't finished the article yet, but I have one question. Is there ability in Scrapy to append new data after initial scrapping?

For example site at the moment of scrapping:

  Article 1  
  Article 2
Then we run scrapy again after some time and site will look like this:

  Article 1  
  Article 2  
  Article 3  
  Article 4
Can it just append article 3 and 4 to file/db table and stop scraping?
jonotime
Scrapy is great. I have been using it at work to populate our solr index for our site search page. It is very extensible and powerful, but helps to understand Twisted if you need to do anything advanced.
CamelCaseName
I used Scrapy extensively last term in my non-CS co-op. Really loved how quickly I could make beautiful dashboards with a bit of Python / VBA.

My problem with Scrapy was that there was a lot of manual work involved for each site, but perhaps that speaks more about my proficiency rather than Scrapy.

What's the next step from here in terms of gathering, cleaning, and analyzing data?