This article has a lot of useful information. I've developed some web scrapers myself and have noticed a couple of things:
A lot of developers will write scrapers that download a page, parses whatever information it needs, discards the page, then moves on. A better practice is to save the contents of the page in case a piece of data was missed or parsed incorrectly. It saves a lot of time and bandwidth over re-scraping the pages, though that would have to be accounted for during development.
The second thing is that the article mentions using concurrent requests. This will definitely speed things up, but it can be dangerous. Some domains use protection like Cloudflare to block aggressive requests, so a high number of requests to one domain from one source may result in the bot's requests being blocked or the bot's IP being blacklisted entirely.
1 comment
[ 2.2 ms ] story [ 14.0 ms ] threadA lot of developers will write scrapers that download a page, parses whatever information it needs, discards the page, then moves on. A better practice is to save the contents of the page in case a piece of data was missed or parsed incorrectly. It saves a lot of time and bandwidth over re-scraping the pages, though that would have to be accounted for during development.
The second thing is that the article mentions using concurrent requests. This will definitely speed things up, but it can be dangerous. Some domains use protection like Cloudflare to block aggressive requests, so a high number of requests to one domain from one source may result in the bot's requests being blocked or the bot's IP being blacklisted entirely.