HN Explorer

Ask HN: What are the best tools for web scraping in 2022?

313 points by pablohoffman ↗ HN
Last time this question was asked on HN was in 2017 (https://news.ycombinator.com/item?id=15694118), a lot has changed in the last 5 years in the world of web scraping (legal landscape, antibot unblockers, data type specific APIs, etc), so I thought it may be a good idea to refresh this question and see what are the most popular tools used by the HN community these days.

159 comments

[ 2.1 ms ] story [ 209 ms ] thread
SQL2219
powershell
dyml
Can you expand on this? I’ve always found powershell hard to do right.
majkinetor 

    $x = iwr https://google.com
    foreach ($link in $x.Links.href) { iwr $link -OutFile foo ...}
snehesht
In the world of SPA (single page applications), headless browser API is super helpful, playwright[1] and puppeteer[2] are very good choices.

[1] https://github.com/microsoft/playwright

[2] https://github.com/puppeteer/puppeteer

mynameismon
Also is the ability to open the Networks panel, to snoop on requests and find the exact API call that you might need to perform your task, instead of having to pull in all of HTML/JS/CSS crap. As a lot of SPAs have essentially pushed everything behind JSON APIs, all information is usually one (authenticated) API call away.
XzAeRosho
Most content heavy websites that tend to be scrapped, usually use server side rendering for this exact same reason, and put many obstacles in the way to make sure that data doesn't get scrapped easily. See: product price, stock, delivery information.
lofatdairy
Highly recommend playwright (if I'm not mistaken most of the big developers from puppeteer were hired by MS to work on playwright). I run into significantly less await/async problems with playwright than I did with puppeteer and the codegen tool is super helpful as a first pass option.
snehesht
Playwright integrates with lot of different browsers compared to puppeteer which just uses chrome.
snehesht
If you're interested in running the puppeteer in containers, take a look at chrome-aws-lambda[1] and browserless docker container[2]

Not affiliated with browserless, but they do have a free/paid cloud service. https://www.browserless.io

[1] https://github.com/alixaxel/chrome-aws-lambda

[2] https://github.com/browserless/chrome

btown
https://chrome.browserless.io/ is perhaps the best technical demo I've ever seen, and shows off Browserless's capabilities amazingly. An incredibly high-quality service and codebase.
[deleted]
(comment deleted)
lwthiker
curl-impersonate[1] is a curl fork that I maintain and which lets you fetch sites while impersonating a browser. Unfortunately, the practice of TLS and HTTP fingerprinting of web clients has become extremely common in the past ~1 year, which means a regular curl request will often return some JS challenge and not the real content. curl-impersonate helps with that.

[1] https://github.com/lwthiker/curl-impersonate

bunnyfoofoo
Oh wow, thank you! I had no idea something like this existed. I was manually compiling against BoringSSL.
sph
As someone that has built and maintained a few scraper tools in my career: hand-written logic and patience because your scraper will break any time upstream changes their HTML. It's an infinite game of whack-a-mole outside your control.

Scrapers are very simple, effective and probably one of the least fun things to build.

mouse-linguist
Do you have recommendations for platforms that monitor scraping fleets?

I occasionally code scrapers for quick data aggregation, but have trouble running anything long-term because it can be a chore to monitor. I've been looking into various options for self-hosting some sort of monitor/dashboard that can send alerts but haven't found anything satisfying yet.

breno
We released estela for this and other purposes, check it out, maybe it will suit your needs:

https://github.com/bitmakerla/estela

Only Scrapy support atm, but additional scraping frameworks/language are on the roadmap. It would be great to have feedback to consider it when prioritizing some over others :-)

real-dino
I'd create a simple health check based on the integrity of the data your retrieve.

Think about giving it a score based on how the data is shaped. If it's missing prices for example, then it immediately goes down to zero, doesn't update the database and sends an alert.

hbrn
I would recommend against building a score and always stick to booleans whenever possible. Having a numeric score means you're trying to extract nice-to-have data points, and in my experience this always leads messy codebases and false assumptions.

If you allow "slightly unhealthy" scrapers in production, almost inevitably the state of your scrapers will always be "slightly unhealthy". Save yourself the trouble and always treat it as either "it works" or "it doesn't work", no in between. Your first iterations will probably break every day, but eventually you'll get to a happy place.

[deleted]
(comment deleted)
adriancooney
Apologies for the shameless self-promotion here but it was this very problem that I built puppeteer-heap-snapshot. It decouples the HTML from the scraper and instead we inspect the booted app’s memory. Not near as performant but a lot more reliable. I wrote about it here: https://www.adriancooney.ie/blog/web-scraping-via-javascript...
robk
This is amazing thanks for sharing!!
btzs
Hi! Your application looks interesting! I have a question regarding your YouTube example: Where do you get property names like channelId,viewCount,keywords from? Thanks
jamescampbell
Hard agree here. Stopped taking scraping projects because the sites always change. No matter how general you design it.
larrydag
The polite package using R is intended to be a friendly way of scraping content from the owner. "The three pillars of a polite session are seeking permission, taking slowly and never asking twice."

https://github.com/dmi3kno/polite

epberry
I've thought https://www.scrapingbee.com/ looked great, especially their auto rotation of IP addresses.
XzAeRosho
ScrappingBee is a fantastic paid solution that makes most of the pain of modern web scraping go away.

The prices are not very friendly though...

asdadsdad
Can I piggyback on the question and ask what are people scraping these days?
shmoogy
Competitor pricing, stock, leadtimes, promotions.
XzAeRosho
Also product information updates/mistakes. Some companies like to make sure that the data that is being shown out there is accurate.
olvy0
Recently I wrote a small firefox extension to scrape only the Hacker News submissions which I upvoted, and also my own submissions, and create browser bookmarks from them.

It also scrapes all my comments I upvoted and if those have links inside them it creates bookmarks from them too.

That's because I often find myself searching for some submission I upvoted but can't find it, especially if there were many similar ones, whereas Firefox bookmarks manager has a nifty search feature...

I had to scrape since the HN API doesn't expose ability to get information about upvoted submissions/comments. The extension assumes you are logged in, it doesn't ask for your username or password.

It's mostly for my own use and not very polished, but it works. I uploaded it to the Firefox extensions gallery, and you can probably find it there, but I don't think it's ready for public consumption yet...

b1476
During Covid lockdown online food shopping in my area (UK) became increasingly difficult to book delivery slots as they would fill up as soon as they became available. For most people who either owned a car or lived nearby a supermarket this was probably just a mild inconvenience, however at the time I didn't have a car and the nearest store was miles away from my house. Writing a bunch of Python scripts to check for free delivery slots over several of the big supermarkets and then pinging me when one became available was a lifesaver (maybe a bit dramatic, but it helped me secure regular food deliveries without having to spend hours hitting the refresh button). I learned a lot in the process, the lengths at which some online stores go to try and deter automated scripts was quite interesting.
asdadsdad
I feel you. I've wondered many times why anti-bot scripts are deployed at all, save some cases like airline booking, where airlines are penalized for abnormal look-book ratios.
NDizzle
SEC database… 10-K, 10-Q, 20-F.
[deleted]
(comment deleted)
na85
US Treasury rates
shmoogy
Has anybody created anything similar to Portia for scraping? I'd love to self host or pay a nominal fee to allow my team to create / adjust scrapers via a UI
smashah
I've been using puppeteer as it's got a very established ecosystem. There are also puppeteer plugins that make it very powerful against captchas/detection/etc.

The worst thing about Puppeteer is chrome and it's bad memory management so I'm going to give playwright a spin soon.

nathancahill
It's been a while, but last time I used it Puppeteer had a headless Firefox backend?
mccorrinall
You might be mistaking playwright with puppeteer
minimaul
Puppeteer can do both Chrome & Firefox, you're right. I use both for a scraping project I do.
altilunium
Beautiful Soup gets the job done. I made several app by using it.

[1] https://github.com/altilunium/wistalk (Scrap wikipedia to analyze user's activity)

[2] https://github.com/altilunium/psedex (Scrap goverment website to get list of all registered online services in Indonesia)

[3] https://github.com/altilunium/makalahIF (Scrap university lecturer's web page to get list of papers)

[4] https://github.com/altilunium/wi-page (Scrap wikipedia to get most active contributors that contribute to a certain article)

[5] https://github.com/altilunium/arachnid (Web scraper, optimized for wordpress and blogger)

Buttons840
I've found lxml to be more powerful. The lxml library supports xpaths, which I don't believe Beautiful Soup does?

In other words, consider lxml as well.

yifanl
lxml is supported (mostly) out of the box for BeautifulSoup, so you can it as a parser behind BS4's nicer interface, which I believe the OP does in the linked codebases.
traverseda
I reach for selectolax first if I'm doing relatively tame stuff. Also css selectors are nice.
tomp
Python, requests, BeautifulSoup, lxml, BrightData proxy provider. If necessary, async (if you’re scraping multiple pages) and Pyppeteer (if scraping JS-heavy pages).
fersarr
Selenium via Python is really useful too if you need to do a bit more (e.g clicks) than just fetching the html from the page.
vikR0001
Selenium can be very difficult to install when it comes to specific browser libraries.. Playwright, just as one example, is very easy to install.
sireat
Isn't it just pip install selenium then download the correct driver version for your browser (place it in a path or supply path when initializing client)?

I use Selenium every few months so I have to update the drivers but otherwise it is pretty painless.

Selenium is much slower than BS4 which is much preferred for static sites.

simplecto
i hate to be that guy, but “it depends”

scrapy is still king for me (scrapy.org). there are even packages to use headless browsers for those awful javascript heavy sites

however, APIs and RSS are still in play, and that does not require a heavy scraper. I am building vertical industry portals, and many of my data rollups consume APIs and structured XML/RSS feeds from social and other sites.

breno
estela is an elastic web scraping cluster running on Kubernetes. It provides mechanisms to deploy, run and scale web scraping spiders via a REST API and a web interface.

It is a modern alternative to the few OSS projects available for such needs, like scrapyd and gerapy. estela aims to help web scraping teams and individuals that are considering moving away from proprietary scraping clouds, or who are in the process of designing their on-premise scraping architecture, so as not to needlessly reinvent the wheel, and to benefit from the get-go from features such as built-in scalability and elasticity, among others.

estela has been recently published as OSS under the MIT license:

https://github.com/bitmakerla/estela

More details about it can be found in the release blog post and the official documentation:

https://bitmaker.la/blog/2022/06/24/estela-oss-release.html

https://estela.bitmaker.la/docs/

estela supports Scrapy spiders for the moment being, but additional frameworks/languages are on the roadmap.

All kinds of feedback and contributions are welcome!

Disclaimer: I'm part of the development team behind estela :-)

gkfasdfasdf
If the content you need is static, I like using node + cheerio [0] as the selector syntax is quite powerful. If there is some javascript execution involved however, I will fall back to puppeteer.

[0] - https://cheerio.js.org/

lioeters
Node.js and cheerio is what came to my mind too.

I heard the team behind Puppeteer moved from Google to Microsoft, and started the project Playwright, which has a more ergonomic API and better cross-browser support (Chromium, WebKit, and Firefox).

https://playwright.dev/

benibela
I wrote my own webscraper: https://videlibri.de/xidel.html

The main purpose was to submit HTML forms. You just say in which input fields something should be written and then it does the other things (i.e. download the page, find all other fields and their default values, build a HTTP request from all of them and send that ).

The last 5 years, I spent updating the XPath implementation to XPath/XQuery 3.1. The W3C has put a lot new stuff in the new XPath versions like JSON support or higher order functions, for some reason they decided to turn XPath into a Turing-complete functional programming language.

shubhamjain
Unpopular opinion, but Bash/Shell Scripting. Seriously, it's probably the fastest way to get things done. For fetching, use cURL. Want to extract particular markup? Use pup[1]. Want to process csv? Use cskit[2]. Or JSON? Use jq[3]. Want to use DB? Use psql. Once you get the hang of shell scripting, you can create simple scrapers by wiring up these utilities in a matter of minutes.

The only thing I wish was present was better support for RegExes. Bash and most unix tools don't support PCRE which can severely limiting. Plus, sometimes you want to process text as a whole vs line-by-line.

I would also recommend Python's sh[4] module if Shell scripting isn't your cup of tea. You get best of both worlds: faster dev work with Bash utils, and a saner syntax.

[1]: https://github.com/ericchiang/pup

[2]: https://csvkit.readthedocs.io/en/latest/

[3]: https://stedolan.github.io/jq/

[4]: https://pypi.org/project/sh/

hhthrowaway1230
Also a fan. Usually I generate indexes/urls, and then just wget and scrape the content offline once all is downloaded.
plainnoodles
My main qualms with bash as a scripting language are that its syntax is not only kind of bonkers (no judgement, I know it's an old tool) but also just crazily unsafe. I link to a few high-profile things whenever people ask me why my mantra is "the time to switch your script from bash to python is when you want to delete things".

>rm -rf /usr /lib/nvidia-current/xorg/xorg

https://github.com/MrMEEE/bumblebee-Old-and-abbandoned/commi...

>rm -rf "$STEAMROOT/"*

https://github.com/valvesoftware/steam-for-linux/issues/3671

It's just too easy to shoot your foot.

shubhamjain
There are couple of flags you can use to mitigate the safety risks. `set -u`, for instance, will thrown an error if an unbound variable is used. I always start my scripts with

> set -euo pipefail

Here's a detail explaination of all the switches: https://gist.github.com/mohanpedala/1e2ff5661761d3abd0385e82....

I do agree though, it's not the best tool. But combining CLI utilities tends to be fast.

simonw
My shot-scraper CLI tool was designed to support this kind of workflow but with a real headless browser inserted into the mix. Means you can do things like this:

    shot-scraper javascript \
      "https://news.ycombinator.com/from?site=simonwillison.net" "
    Array.from(document.querySelectorAll('.itemlist .athing')).map(el => {
      const title = el.querySelector('a.titlelink').innerText;
      const points = parseInt(el.nextSibling.querySelector('.score').innerText);
      const url = el.querySelector('a.titlelink').href;
      const dt = el.nextSibling.querySelector('.age').title;
      const submitter = el.nextSibling.querySelector('.hnuser').innerText;
      const commentsUrl = el.nextSibling.querySelector('.subtext a:last-child').href;
      const id = commentsUrl.split('?id=')[1];
      const numComments = parseInt(
        Array.from(
          el.nextSibling.querySelectorAll('.subtext a[href^=item]')
        ).slice(-1)[0].innerText.split()[0]
      ) || 0;
      return {id, title, url, dt, points, submitter, commentsUrl, numComments};
    })
    " | jq '. | map(.numComments) | add'
That example scrapes a page on Hacker News by running JavaScript inside headless Chromium, outputs the results as JSON to stdout, then pipes them into jq to add them up. It outputs "1274".

https://simonwillison.net/2022/Mar/14/scraping-web-pages-sho...

(Fun side note: I figured out the jq recipe I'm using in this example using GPT-3: https://til.simonwillison.net/gpt3/jq )

jamescampbell
Badass as always Simon. I prefer my custom cloudflare killer chrome headless python code. But this is cool for quick things.
simonw
For things like regular expressions, it's useful to know that Python has a "-c" option which can be passed a multi-line string as part of a CLI pipeline. You can do something like this:

    curl 'https://news.ycombinator.com/' | python -c '
    import sys, re, json
    html = sys.stdin.read()
    r = re.compile("<a href=\"(.*)\"")
    print(json.dumps(r.findall(html), indent=2))
    '
This outputs JSON which you can then pipe to other tools.
[deleted]
(comment deleted)