In terms of NodeJS vs Python, specifically for web scraping, would you choose NodeJS? If so, why?
I'm more familiar with NodeJS but I'm working with a team that is leaning towards using Python for web scraping, so that's why I'm asking. They said spinning up multiple processes in Python is easier so at scale it will work better.
I know you can use the Cluster module to have child processes in NodeJS, but in my experience it's a bit of a pain to use, although it's not always required to use anyway, at least when only using NodeJS as a web server (as long as you have multiple NodeJS instances, in case one goes down). Web scraping is a bit different though.
I worked on a project which required some medium scale web scraping (less than 100 million pages), and went with node primarily because of puppeteer.
The system had a couple dozen worker processes doing the scraping, and one coordinator which maintained a queue of pages which needed to be scraped. There was some logic to balance requests between sites, so we weren't making more than a request/s to any in particular. The coordinator just had a rest api endpoint, which the workers would hit to get their next job and to return w/e data.
Each worker process was ran on a separate aws instance, believe it was a t2 with unlimited cpu enabled. These are only a few dollars a day, and it was necessary to have as many ip addresses as possible (at least 5% of the sites we were scraping had some preventative measures in place, but they all seemed to be ip based)
I’ve never done web scraping professionally, but I can relay that any time I’ve tried to use Python for anything involving concurrency, except in the most trivial cases, it has been pure pain.
NodeJS (V8) is faster than CPython, and NodeJS was built for this precise use case. I wouldn't use it over Python personally though, I don't use NodeJS and would sooner reach for Rust if performance mattered.
Concurrency is an important limitation as you've noticed, but it's already a problem for CPython. You would be able to squeeze out more req/s from NodeJS than CPython, up to a point where you would need to bring in something extra to scale to all the cores of one machine (multiprocessing in Python, something like Cluster in NodeJS) which you wouldn't need in Go/Rust/Java.
Then of course scaling further, you would need a system to run jobs across machines, and your choice of Go over Python wouldn't necessarily matter so much. The difference in performance wouldn't limit what you can do, it would just change what you pay for compute. If your compute costs more but your devs can implement features faster, performance is usually unimportant.
Elixir. Parallelism is 99% free, it's super easy to make almost any task parallel. And it's one of the fastest dynamic languages too, surpassing Node.JS in places.
Many people still don't understand you can run millions of threads on Linux on a beefy server. The bottleneck with python will always be the CPU doing something with those requests.
So it's a single thread event loop? which means 1 cpu utilization?
I get a lot of people have gotten mileage out of event loop scaling over the last 10-15 years (that's the trick of nginx, node.js, etc, right?), but even raspberry pis are quad core?
Some C extensions run even when they don't have the lock. Also, the OS gets to run.
If you have a lot of parallel computation, Python may not be a great choice. However, if you're managing a lot of IO with not much computation, Python is perfectly reasonable.
I wonder where the time is going. I suspect that something like atop would show that the CPU is idle for much of the time, which suggests that the problem is not Python or even the GIL.
If that's true, there's something ineffective about the way that the requests are being made/handled.
AFAIK, the author did not try "make a bunch of requests before waiting for answers". If many of the requests are going to the same server, blasting a bunch of requests down a single socket before waiting for a response should be faster, if only because there's only one ssl handshake, even if the remote server doesn't pipeline request processing. (I suspect that "too many threads" didn't work because that reduced connection reuse.)
There's a fairly simple monkey patch to http\httpsconnection that enables request pipelining.
It might also be useful to use something like epoll to wait for the sockets.
The author very much did try to do that, doing requests with async APIs is often the most performant way to do them because under the hood they use IO async primitives like epoll/kqueue/iocp depending on the environment they run on.
If you check the code in the article, there is an innocent looking call to decode the response as JSON; that's most likely where the CPU time is going, most JSON decoders are just not very fast. I remember back in the day having to write my own decoder for a JSON API because I was getting capped on around 250req/s on a windows machine (using c# and iocp which is pretty fast). I think I almost doubled performance by taking some decoding shortcuts when the JSON structure was known ahead of time.
TL;DR: network I/O is fast, json is a really slow format to parse.
27 comments
[ 3.1 ms ] story [ 94.8 ms ] threadI'm more familiar with NodeJS but I'm working with a team that is leaning towards using Python for web scraping, so that's why I'm asking. They said spinning up multiple processes in Python is easier so at scale it will work better.
I know you can use the Cluster module to have child processes in NodeJS, but in my experience it's a bit of a pain to use, although it's not always required to use anyway, at least when only using NodeJS as a web server (as long as you have multiple NodeJS instances, in case one goes down). Web scraping is a bit different though.
Curious if you have any thoughts on this.
The system had a couple dozen worker processes doing the scraping, and one coordinator which maintained a queue of pages which needed to be scraped. There was some logic to balance requests between sites, so we weren't making more than a request/s to any in particular. The coordinator just had a rest api endpoint, which the workers would hit to get their next job and to return w/e data.
Each worker process was ran on a separate aws instance, believe it was a t2 with unlimited cpu enabled. These are only a few dollars a day, and it was necessary to have as many ip addresses as possible (at least 5% of the sites we were scraping had some preventative measures in place, but they all seemed to be ip based)
I wonder if these kind of processes are cheaper on Lambda.
Concurrency is an important limitation as you've noticed, but it's already a problem for CPython. You would be able to squeeze out more req/s from NodeJS than CPython, up to a point where you would need to bring in something extra to scale to all the cores of one machine (multiprocessing in Python, something like Cluster in NodeJS) which you wouldn't need in Go/Rust/Java.
Then of course scaling further, you would need a system to run jobs across machines, and your choice of Go over Python wouldn't necessarily matter so much. The difference in performance wouldn't limit what you can do, it would just change what you pay for compute. If your compute costs more but your devs can implement features faster, performance is usually unimportant.
I’ve heard V8 is fairly optimised, but single thread (regardless of async) is a fairly big hit over Go’s green threads right?
> pool = ThreadPool(40)
All but one of the runnable threads are waiting on the GIL. The "not runnable" threads are waiting on some lock or a blocking system call.
I get a lot of people have gotten mileage out of event loop scaling over the last 10-15 years (that's the trick of nginx, node.js, etc, right?), but even raspberry pis are quad core?
If you have a lot of parallel computation, Python may not be a great choice. However, if you're managing a lot of IO with not much computation, Python is perfectly reasonable.
If that's true, there's something ineffective about the way that the requests are being made/handled.
AFAIK, the author did not try "make a bunch of requests before waiting for answers". If many of the requests are going to the same server, blasting a bunch of requests down a single socket before waiting for a response should be faster, if only because there's only one ssl handshake, even if the remote server doesn't pipeline request processing. (I suspect that "too many threads" didn't work because that reduced connection reuse.)
There's a fairly simple monkey patch to http\httpsconnection that enables request pipelining.
It might also be useful to use something like epoll to wait for the sockets.
If you check the code in the article, there is an innocent looking call to decode the response as JSON; that's most likely where the CPU time is going, most JSON decoders are just not very fast. I remember back in the day having to write my own decoder for a JSON API because I was getting capped on around 250req/s on a windows machine (using c# and iocp which is pretty fast). I think I almost doubled performance by taking some decoding shortcuts when the JSON structure was known ahead of time.
TL;DR: network I/O is fast, json is a really slow format to parse.
Epoll to decide which socket to read is just one thing - efficiently using sockets also matters, hence my suggestion of communication pipelining.