How to build a DIY web scraper in any language (levelup.gitconnected.com) 49 points by kld87 6y ago ↗ HN
[–] KasianFranks 6y ago ↗ Make crawlers not scrapers. [–] fredley 6y ago ↗ Make open APIs, not scrapers. [–] bitfhacker 6y ago ↗ The problem is that most companies/developers still aren't "Open API first"... So, people must build scrapers. [–] bransonf 6y ago ↗ Depends on your use case. If you just need specific data from a known location, build a scraper.If you’re looking to index as much data as possible, build a crawler.The complexity of a crawler is significantly more than a scraper, so it’s not always the right choice.
[–] fredley 6y ago ↗ Make open APIs, not scrapers. [–] bitfhacker 6y ago ↗ The problem is that most companies/developers still aren't "Open API first"... So, people must build scrapers.
[–] bitfhacker 6y ago ↗ The problem is that most companies/developers still aren't "Open API first"... So, people must build scrapers.
[–] bransonf 6y ago ↗ Depends on your use case. If you just need specific data from a known location, build a scraper.If you’re looking to index as much data as possible, build a crawler.The complexity of a crawler is significantly more than a scraper, so it’s not always the right choice.
[–] kld87 6y ago ↗ I've been told I should've shared the friend link to remove the paywall (my bad), here it is: https://levelup.gitconnected.com/how-to-build-a-diy-web-scra...
5 comments
[ 3.2 ms ] story [ 29.7 ms ] threadIf you’re looking to index as much data as possible, build a crawler.
The complexity of a crawler is significantly more than a scraper, so it’s not always the right choice.