Using Sitemaps in Web Scraping (webrobots.io)

Good article, useful general tips that work for many scraping tools and platforms.

Very useful for some big eshops if they list their sitemaps. However not all stores have sitemaps and even robots.txt site...