Some algorithms aren't easily parallelisable (at least I haven't worked out how to paralellise them!). My original attempts at paralellising A* didn't work because of communication overhead.
But I sharded my data that it searches - each thread searches a different subset of neighbours as A* is a graph search. This lets me paralellise, with 8 threads (when I boosted the repl.it from 2 vcpu to 8vcpu) It would run in 20-40 milliseconds as opposed to 100-300 milliseconds.
Program synthesis through optimal control or directly through algorithms like A* sounds fun and interesting. If I may, I would suggest to either comment your code or add more details to your README.md, because it is currently quite difficult to understand what exactly is being done here and how.
It's path finding between output states of a program. I was inspired by path finding in video games. The path is the discovered program. It's like having a game map where each neighbour of a cell is the potential things you can do at that point in time. For this reason it can be used for unit AI.
I think of assembly (and regular) programming as a sliding image puzzle game. My design is based on this intuitive idea that most programming is just moving things around - it's logistics. How much time do you spend marshalling data? REST APIs? To get registers to the value for a CALL instruction, the calling convention, is just moving the right data into the right locations.
Imagine you have a start state and an end state. It's a turing machine. State is processor register values and memory values in each memory location. There is a series of state transitions between the start state and end state which is created by an instruction, these are intermediary states. Instructions can be MOV instructions or CALL instructions (function calls)
I dynamically generate neighbours of starting from the start state which are potential instructions we can run that get us nearer to the end state. The program searches the space of possible instructions. It infers hidden states caused by functions.
The distance heuristic is the number of values that in registers and memory that match the end state.
My dream: if you knew the type of every API call, you could provide a start state and an end state and the computer would work out the program based on exploring potential instructions that get us to the target state. This is a much simpler form of program synthesis compared to LLMs.
I'll update the README and document the code.
I also have a dream of a real time strategy game where you program the computer by telling unit what to do. In most RTS games you can only build, attack but we can do more.
When I need to process sometimes millions of documents from a production mongoDB instance I normally push all the objectID into rabbitMQ then spin up N workers that each fetch a single message at a time.
Is there a reason why you use rabbitmq instead of just using the whats available in the language (I understand if those primitives are hard to work with)
As said people using multiple ec2s to process the data but the biggest reason is so I don't have to run it all in one go. Rabbit is used by a few of our services so is always up, means I can run for a few hours then come back to it another day.
Also error handling, if one fails I just don't ack the message, after everything is processed I only have failed ones. Can slowly process the messages to see what is wrong
Coincidence: I literally read that doc-page and wrote some ThreadPoolExecutor code 5 minutes ago to workaround the lack of a specific async operation in asyncio.
Agreed ! Plus, the ability submit a bunch of tasks, and to block until _one_ task completed (akin to `epoll_wait` of tokio's `select`) makes it quite useful.
I don't know of a use case or `mutiprocessing.Pool` which is not covered by `concurrent.futures.ProcessPoolExecutor`; so I wonder why both exist
In general, the Executor or even Queue abstractions are much better and safer ways of doing multithreading and multiprocessing. These days, I rarely ever directly create threads. It's fine for situations when you must but parallelizing work can usually be done better, safer, and easier with executors.
Yeah, that's struck me as well. "Let's get down to some work" without explaining why it should (or should not!) be done. And in this case it's crucial.
It's low quality content marketing for Red Hat. Equivalent to all those Medium articles written by boot camp grads, except it's a Red Hat ad instead of a "hire me" ad.
I don't think I can do all of those justice right this moment but let's get a few here so people can link to an HN thread instead of the red hat article ;)
MP is good when the GIL would hamper your thread concurrency, ie your problem is likely CPU bound rather than IO/network bound (where Cython is good about releasing the GIL).
Cost is that there's process overhead of each new instance of Python running the worker code and pickling any required data to/from the worker across a process boundary (rather than between threads). Benefit is mostly the last answer: can saturate all available CPUs.
Few options but generally something like `concurrent.future`'s `.map` will keep tasks with order while a `.submit` and then checking with `.as_completed` will be tasks out of order (but if you return an ID of what you were working on you could reorder after and that may be worthwhile if the workloads are highly variable).
Exceptions:
Capture all in your worker and make available to the main via event or queue and check that signal periodically in your main and take action as needed.
For the other (Ctrl+C in your main) have your workers periodically check a signal from main as often as needed for the responsiveness desired and have the worker cleanup/quit on Interrupt signals.
Data transmission feels too problem-dependent to give a single answer to but if you're processing say, files, don't read and pass the files bytes to a worker, pass the file's location and let the worker read the file and return/write results.
Thanks. That's more or less how I solved these problems too. And the questions are what I had to painstakingly answer myself at work.
BTW, I haven't touched MP in Py for couple of years now, but remember there's some misalignment in concurrent futures between pool.map and pool.submit.
And for exceptions in background processes, I had a shared bool var called "emergency brake", and an error queue.
Glad to help, it was similar for me with a lot of effort to synthesize best practices and really appreciate all the concurrent options (asyncio, threads, MP) for where they each are valuable.
I mean, hopefully why you might need multiprocessing in python is clear?
If you have a python task that is highly parallelizable on a single machine with multiple cores, then multiprocessing is probably the right tool to quickly see if you can dramatically speed up your code with parallelism with basically no code overhead or investment in distributed solutions (there are edge cases where it is not, but it takes very little time to test if you are an edge case).
I encounter this situation in my data science workflow routinely. It is an easy way to impress product / managers and say "hey, I made this batch algorithm 50x faster, so now it runs in 10 minutes instead of 500."
As a simple example, Threading in Python works good for i/o-bound operations like scraping a website, whereas Multiprocessing works best for CPU-bound operations like result hashing/transforming the data you just scraped.
> I mean, hopefully why you might need multiprocessing in python is clear?
My experience, after first getting into python, was:
I needed to do something concurrently on one set of data.
Python threading doesn't provide concurrent execution, so my program slowed down when I used threads.
So, I tried multiprocessing. My program slowed down even more, because any communication between processes uses pickle. I was trying to process one dataset in parallel, and pass big chunks back, for a final processing.
So, I saved it to disk, loaded the dataset into each process, multiplying my memory usage by 16x.
I then threw it all out and wrote the performant bits in C++, using swig to automagically make the python interface for it.
So, knowing why (concurrency) isn't necessarily enough.
I mean, of course? Concurrency != parallelism, so that makes sense.
Where multiprocessing shines is when you have an algorithm that can be fully parallelized and represented in a baby map-reduce framework, where the data being sent to each process isn't too big. The idiom I often reuse is literally the first example in the python docs:
from multiprocessing import Pool
def f(x):
return x\*x
if __name__ == '__main__':
with Pool(5) as p:
print(p.map(f, [1, 2, 3]))
# to give more of a map reduce flavor:
print(sum(p.map(f, [1, 2, 3])))
This can be modified to fit a pretty big range of tasks in my use cases and chop hours off my workflows. It's so much faster in DS workflows to just have extra cores and use them than to spin up a cluster for distributed compute.
That's the perfect situation many parallelization packages in python ecosystem take for granted. In my case, I had big datasets that wouldn't fit into memory (or I didn't want to demand this peak RAM usage on a server), and I had to write a parallelizer that read data in chunks and fed it to workers -- something very simple but unavailable in Pandas ecosystem (most packages assume you have swallowed the entire dataset, which isn't that large, and it's easy to throw the data to process-based workers)
I've just left the question open, but the best learning material for me on this was this lecture. Other points down the list, I had to figure out just by trial and error.
This is a decent article, but for me the best discussion on parelleism in Python is Raymond Hettinger's talk "Keynote on Concurrency, PyBay 2017" https://www.youtube.com/watch?v=Bv25Dwe84g0
It includes a very important observation that lots of problems are small enough that they can be solved with a single CPU core, and that lots of problems are so big you need full on Cloud Computing Distributed Processing with effectively unlimited number of cores, and there are relatively few problems that are too big for 1 core but can be made tractable by going up to around 10 cores. (This is the "Martelli Model of Scalability")
Also this...
Some people, when confronted with a problem, think "I know, I'll use parallelism."
Hettinger is a great teacher but I think he's been part of core python's status-quoism, that quote is just in line with that. And that culture is being changed now (faster cpython and maybe even nogil!)
One observation I'd like to make is that feeding a thread farm from a single input stream demands some understanding of locks, queues and mutex.
The examples are a bit piss weak/simple. One master and one slave. Really I expected 3-4 consumers so you can understand how they interact over q.put() and q.get() and q.join() and q.task_done()
On a large(ish) memory and core host using dnspython I've been getting significantly more dns throughout over log file processing and a dict() cache shared amongst the threads avoiding dns stalls. 20+ threads. It's still buggy, better examples would be helpful.
I know: threads not processes. They're documented as neighbour concepts. I'd have done process forking if I could find good lock and shared mem/mmap use Documented.
Well, it's complicated as I'm not doing multiprocessing with Numpy, actually :)
One part of my program uses Numpy, and the other part does multiprocessing. But Python's multiprocessing module causes interference between them (which took quite some time and effort to figure out).
The models for curve fitting in my lab were using lambdas, which cannot be serialized by multiprocessing/pickle (at least that was the state about 5 years ago). The pathos.multiprocessing module [1] served as a drop-in replacement, which was able to handle lambda with its own dill serializer. Saved me quite a bit of refactoring.
Does anyone have a good overview of threading on AWS? Every time I read about it I get confused, it seems like there are wildly different rules depending on the context. Lambda is different than ECS which is different from EC2
careful you don't ever accidentally try to pass a large result (for example, a pandas dataframe...) back from the workers to the controller process :-) The communication happens via pickle and the memory usage is huuuge. If you can, think map/reduce, and return the smallest-sized answer.
I needed to pass big dataframes between workers and the main thread. Turned out that the easiest way for me was having them write parquet files and reading them in the main thread.
I like using multiprocessing pools whenever I need to plow through something quickly, or if I don't already have a task queue like celery set up.
However, this article feels off to me. Everything I read from redhat always feels a bit wrong for some reason. I don't know what it is exactly. It might just be that they used "a" instead of "the" in the title. But Ive had the same feeling with other manuals and whatnot from redhat.
Just use Erlang or Go. Even Java... Python doesn't hold a candle to any of those languages when it comes to multitasking.
The most important advise anyone could give on using Python's anything, especially stuff that has to do with concurrency is to stock up on mood enhancing drugs, else they'll probably cry themselves to death out of despair.
I don't understand why a company like RHEL would publish such a low-effort article. But, if you genuinely needed and advice on how to use Python's multiprocessing, and for some screwed-up reason you cannot use a decent language... well, never use plain multiprocessing.Process. Use multirpocessing.pool.Pool: https://docs.python.org/3/library/multiprocessing.html#multi...
And if you wonder why would you do that... well, obviously the later builds on the former. And, trust me, it doesn't improve the former in any way. But, it saves you some work. Virtually any task you will end up doing will be done with process pool in less code. Most importantly though, it's a "recognizable pattern" -- it's something that someone who ate from the garbage pile known as Python has learned to recognize in code and to treat accordingly. Code randomly peppered with mention of Process or, worse yet, extending that class is a sign of lack of experience, and produces an urge to delete that nonsense code and write it afresh because nobody's got the time to deal with novice's nonsense.
> Just use Erlang or Go. Even Java... Python doesn't hold a candle to any of those languages when it comes to multitasking.
1000%. Pretty much any other language.
> But, if you genuinely needed and advice on how to use Python's multiprocessing, and for some screwed-up reason you cannot use a decent language
Sometimes the drag of dealing with Python multiprocessing is easier and incurs less debt than splitting the project across multiple languages and dealing with (de)serialization of complicated data structures in way that doesn't introduce bottlenecks. And there are at least a few domains where there isn't really a viable alternative to Python.
But a good rule of thumb is that if you're not going to spend at least a couple full days dealing with the issues listed above -- or if you don't absolutely need to use a big chunk of complicated Python code -- you're probably better off picking a different language.
55 comments
[ 2.7 ms ] story [ 95.3 ms ] threadhttps://replit.com/@Chronological/SlidingPuzzle3 (the code shows multiprocessing Processes connected by queues, the README.md explains what the program does)
Some algorithms aren't easily parallelisable (at least I haven't worked out how to paralellise them!). My original attempts at paralellising A* didn't work because of communication overhead.
But I sharded my data that it searches - each thread searches a different subset of neighbours as A* is a graph search. This lets me paralellise, with 8 threads (when I boosted the repl.it from 2 vcpu to 8vcpu) It would run in 20-40 milliseconds as opposed to 100-300 milliseconds.
mgaunard [0] recently told me that there is something called an algorithmic skeleton for parelllisation problems. I've not used any though. https://en.wikipedia.org/wiki/Algorithmic_skeleton
I have no exposure to high performance computing unfortunately. Just a deep interest in parallelism.
[0]: https://news.ycombinator.com/item?id=36792796
It's path finding between output states of a program. I was inspired by path finding in video games. The path is the discovered program. It's like having a game map where each neighbour of a cell is the potential things you can do at that point in time. For this reason it can be used for unit AI.
I think of assembly (and regular) programming as a sliding image puzzle game. My design is based on this intuitive idea that most programming is just moving things around - it's logistics. How much time do you spend marshalling data? REST APIs? To get registers to the value for a CALL instruction, the calling convention, is just moving the right data into the right locations.
Imagine you have a start state and an end state. It's a turing machine. State is processor register values and memory values in each memory location. There is a series of state transitions between the start state and end state which is created by an instruction, these are intermediary states. Instructions can be MOV instructions or CALL instructions (function calls)
I dynamically generate neighbours of starting from the start state which are potential instructions we can run that get us nearer to the end state. The program searches the space of possible instructions. It infers hidden states caused by functions.
The distance heuristic is the number of values that in registers and memory that match the end state.
My dream: if you knew the type of every API call, you could provide a start state and an end state and the computer would work out the program based on exploring potential instructions that get us to the target state. This is a much simpler form of program synthesis compared to LLMs.
I'll update the README and document the code.
I also have a dream of a real time strategy game where you program the computer by telling unit what to do. In most RTS games you can only build, attack but we can do more.
Also error handling, if one fails I just don't ack the message, after everything is processed I only have failed ones. Can slowly process the messages to see what is wrong
I played about with Kafka consumer groups to get the same effect.
And how do you do that?
Because...that is also what the multiprocessing module does.
[0] https://docs.python.org/3/library/concurrent.futures.html#pr...
It doesn't answer
* why you need multiprocessing (rather than threads)
* what are costs and benefits of them
* how to process a sequence of tasks maintaining results in order
* ...or without maintaining the order
* how to handle exceptions in the processes
* how to gracefully exit by Ctrl+C or other exceptions in the main process
* how to minimize data transmission between processes, because it's costy
etc. None of this is covered.
MP is good when the GIL would hamper your thread concurrency, ie your problem is likely CPU bound rather than IO/network bound (where Cython is good about releasing the GIL).
Cost is that there's process overhead of each new instance of Python running the worker code and pickling any required data to/from the worker across a process boundary (rather than between threads). Benefit is mostly the last answer: can saturate all available CPUs.
Few options but generally something like `concurrent.future`'s `.map` will keep tasks with order while a `.submit` and then checking with `.as_completed` will be tasks out of order (but if you return an ID of what you were working on you could reorder after and that may be worthwhile if the workloads are highly variable).
Exceptions:
Capture all in your worker and make available to the main via event or queue and check that signal periodically in your main and take action as needed.
For the other (Ctrl+C in your main) have your workers periodically check a signal from main as often as needed for the responsiveness desired and have the worker cleanup/quit on Interrupt signals.
Data transmission feels too problem-dependent to give a single answer to but if you're processing say, files, don't read and pass the files bytes to a worker, pass the file's location and let the worker read the file and return/write results.
BTW, I haven't touched MP in Py for couple of years now, but remember there's some misalignment in concurrent futures between pool.map and pool.submit.
And for exceptions in background processes, I had a shared bool var called "emergency brake", and an error queue.
Glad to help, it was similar for me with a lot of effort to synthesize best practices and really appreciate all the concurrent options (asyncio, threads, MP) for where they each are valuable.
If you have a python task that is highly parallelizable on a single machine with multiple cores, then multiprocessing is probably the right tool to quickly see if you can dramatically speed up your code with parallelism with basically no code overhead or investment in distributed solutions (there are edge cases where it is not, but it takes very little time to test if you are an edge case).
I encounter this situation in my data science workflow routinely. It is an easy way to impress product / managers and say "hey, I made this batch algorithm 50x faster, so now it runs in 10 minutes instead of 500."
My experience, after first getting into python, was:
I needed to do something concurrently on one set of data.
Python threading doesn't provide concurrent execution, so my program slowed down when I used threads.
So, I tried multiprocessing. My program slowed down even more, because any communication between processes uses pickle. I was trying to process one dataset in parallel, and pass big chunks back, for a final processing.
So, I saved it to disk, loaded the dataset into each process, multiplying my memory usage by 16x.
I then threw it all out and wrote the performant bits in C++, using swig to automagically make the python interface for it.
So, knowing why (concurrency) isn't necessarily enough.
Where multiprocessing shines is when you have an algorithm that can be fully parallelized and represented in a baby map-reduce framework, where the data being sent to each process isn't too big. The idiom I often reuse is literally the first example in the python docs:
This can be modified to fit a pretty big range of tasks in my use cases and chop hours off my workflows. It's so much faster in DS workflows to just have extra cores and use them than to spin up a cluster for distributed compute.https://realpython.com/python-concurrency/
Raymond Hettinger, Keynote on Concurrency, PyBay 2017 https://www.youtube.com/watch?v=9zinZmE3Ogk
It includes a very important observation that lots of problems are small enough that they can be solved with a single CPU core, and that lots of problems are so big you need full on Cloud Computing Distributed Processing with effectively unlimited number of cores, and there are relatively few problems that are too big for 1 core but can be made tractable by going up to around 10 cores. (This is the "Martelli Model of Scalability")
Also this...
Some people, when confronted with a problem, think "I know, I'll use parallelism."
problems
two
they
two
Now
two
two
two
two
two....
The examples are a bit piss weak/simple. One master and one slave. Really I expected 3-4 consumers so you can understand how they interact over q.put() and q.get() and q.join() and q.task_done()
On a large(ish) memory and core host using dnspython I've been getting significantly more dns throughout over log file processing and a dict() cache shared amongst the threads avoiding dns stalls. 20+ threads. It's still buggy, better examples would be helpful.
I know: threads not processes. They're documented as neighbour concepts. I'd have done process forking if I could find good lock and shared mem/mmap use Documented.
(This will be fixed in future Python versions, 3.14 maybe.)
I can somehow only reliably do multiprocessing with numpy when I set:
export OPENBLAS_NUM_THREADS=1
Of course, this is a workaround and not a true solution.
One part of my program uses Numpy, and the other part does multiprocessing. But Python's multiprocessing module causes interference between them (which took quite some time and effort to figure out).
[1] https://github.com/uqfoundation/pathos
https://github.com/tedivm/quasiqueue
I've built a few multiprocess programs with it and it's been really useful, so I thought I'd share.
However, this article feels off to me. Everything I read from redhat always feels a bit wrong for some reason. I don't know what it is exactly. It might just be that they used "a" instead of "the" in the title. But Ive had the same feeling with other manuals and whatnot from redhat.
The most important advise anyone could give on using Python's anything, especially stuff that has to do with concurrency is to stock up on mood enhancing drugs, else they'll probably cry themselves to death out of despair.
I don't understand why a company like RHEL would publish such a low-effort article. But, if you genuinely needed and advice on how to use Python's multiprocessing, and for some screwed-up reason you cannot use a decent language... well, never use plain multiprocessing.Process. Use multirpocessing.pool.Pool: https://docs.python.org/3/library/multiprocessing.html#multi...
And if you wonder why would you do that... well, obviously the later builds on the former. And, trust me, it doesn't improve the former in any way. But, it saves you some work. Virtually any task you will end up doing will be done with process pool in less code. Most importantly though, it's a "recognizable pattern" -- it's something that someone who ate from the garbage pile known as Python has learned to recognize in code and to treat accordingly. Code randomly peppered with mention of Process or, worse yet, extending that class is a sign of lack of experience, and produces an urge to delete that nonsense code and write it afresh because nobody's got the time to deal with novice's nonsense.
1000%. Pretty much any other language.
> But, if you genuinely needed and advice on how to use Python's multiprocessing, and for some screwed-up reason you cannot use a decent language
Sometimes the drag of dealing with Python multiprocessing is easier and incurs less debt than splitting the project across multiple languages and dealing with (de)serialization of complicated data structures in way that doesn't introduce bottlenecks. And there are at least a few domains where there isn't really a viable alternative to Python.
But a good rule of thumb is that if you're not going to spend at least a couple full days dealing with the issues listed above -- or if you don't absolutely need to use a big chunk of complicated Python code -- you're probably better off picking a different language.