Any good Node.js parallelization libraries?

1 points by thatxliner 3y ago ↗ HN

Or do I have to use something along the lines of Web Workers or NodeJS Worker Threads? I would prefer it if I had an API similar to Python’s concurrent.futures. What I really want is apply a function to an array in parallel (i.e. some parallel version of .map)

5 comments

[ 3.4 ms ] story [ 20.0 ms ] thread

normalhappy 3y ago ↗

https://www.npmjs.com/package/paralleljs

thatxliner 3y ago ↗

Tried it, seemed to actually DECREASE my program’s performance. I think it’s because of the overhead of creating thousands of threads…

PaulHoule 3y ago ↗

It is a single threaded runtime so to parallelize a map you will need to serialize the items and ship them to another node process. A bit like dask in Python.

rektide 3y ago ↗

Some ground info, Node has worker_threads for this purpose. https://nodejs.org/api/worker_threads.html

I haven't had cause to use it yet, but I very much trust/respect the team behind: https://www.npmjs.com/package/piscina . It's so popular it has well maintained forks, such as tinypool, https://www.npmjs.com/package/tinypool .

Also seeing good npm popularity metrics around: https://www.npmjs.com/package/synckit

SvenL 3y ago ↗

We used piscina at it was quite good: https://github.com/piscinajs/piscina

While it may not provide parallel map out of the box it will help to implement it a little bit easier than using nodes api.