> Since a new instance of Python VM is running the code, there is no GIL and you get parallelism running on multiple cores.
Has the potential to be misinterpreted by new Python devs. As I understand it, each process gets its own interpreter with its own GIL. So its not that there is no GIL, just that there are now 3 GILs each managing separate execution spaces.
Also, at a higher level of discussion, this is just a rehash of multi-processing vs. multi-threading so perhaps it would be more illuminating to start from there and move down to the library examples.
The casual reference to multi-threading in the quote from the POSIX fork definition also adds to potential reader confusion.
The flip side is a lot of people think "oh the GIL, I can't even run Python code in a server with more than 1 CPU"
No you can, just run lots of processes and make lots of GILS. Ends up still using less memory than Java in many cases. Gil will be the least of your worries.
That's a fair point: you can take advantage of multiple CPUs with Python, but you have to eat the overhead of multi-processing vs multi-threading.
Multi-processing breaks down when you have to communicate with high frequency between various logical runners.
I think there are vastly more problems that can be solved with multi-processing than those that require multi-threading but they do exist.
There is also another layer of simulated parallelism via coroutines...but I was just trying to point out that there was potential for confusion in this piece.
You can of course use multithreading to get parallelism, eg. if the expensive calculations are not done in Python itself but rather some other library, be it cryptography, compression, numpy, serdes'ing stuff and so on. I/O works in parallel as well.
C extensions called from Python are still subject to the GIL of the calling interpreter[1].
In fact, this guaranteed thread safety has led to the creation of many non-thread safe Python C extensions which is now a critical blocker to efforts to removing the GIL[2], although there are also a number of other issues as well.
As noted in the StackOverflow, you can release the GIL but there are some tricks to doing so safely[3].
But threads of a C extensions library are not necessarily pinned to a single core by the GIL like regular Python byte code.
Although many performance oriented libraries like numpy[4] forsake multi-threading by default in many cases and rely on parallelization by the user (programmer)[5].
I wish I had a thousand upvotes to give this comment. This is the single most misunderstood concept in async python. I started and still contribute to hendrix (a "Twisted Django") and I can't believe how bright a lightbulb this is for people when they realize it.
I have coded services in Java that would never JIT before a reboot. Python can often beat un-JITed Java.
For an IO bound tasks that is using less than 100% CPU.. again Python could be identical performance or at least very similar performance to Java.. Think of a simple rest service getting 10 hits per second. Maybe the Java service will be a few micros faster, but to a client across the internet you won't care.
Now yes you are right if you are having 900 requests per second hitting your API, the Java API will crush the Python API. Not every app or service hits that though, and I would sure rather code in Python over Java.
A memory consumption thing that you have to be careful of when doing that that we ran into recently. When you call fork Linux doesn't actually have to copy any of the existing memory until it's changed thanks to the magic of copy-on-write. But often Python will quickly increment or decrement the reference counters on all those bits of memory forcing copies; increasing memory usage and slowing things down. So either being careful with that or turning off garbage collection for the process (if it's short lived) are things to consider there.
"Less memory than Java" I can make a JVM use little memory at the cost of slowing it down. I can also give it all the memory on my server and it will translate that into performance. Java's memory usage is extremely within your control so this comparison doesn't make sense.
> "Less memory than Java" I can make a JVM use little memory at the cost of slowing it down.
Maybe? You also need to assign about 2x live memory to the heap in order for GC to work properly. If you give it less, you will blow your heap and crash the JVM
> I can also give it all the memory on my server and it will translate that into performance.
No, it won't.
CMS is realistically limited to about 16 GB of ram. Past that and you are going to have routine 2, 3, 4 second pauses for full GCs. G1GC is limited a bit higher, maybe 64 GB.. but again, beyond that and you are screwed. In a high performance app, routinely you limit heap usage to around 2 GB, which means you have about 1 GB of actual memory to use.
> Java's memory usage is extremely within your control so this comparison doesn't make sense.
Honestly, it sounds like you have never had to tune the JVM or run a Java app in production, so I disagree strongly with all you have said.
Latest Chrome on MacOS. If you hit back after going to that page you have to click through about 4-5 other articles before you ultimately get back to this comment thread.
I used this module for a fractal rendering program, years ago. The APIS are (were?) a little annoying - last I checked, the relevant code had to be picklable. But it used all my CPUs and scaled just fine.
I had a really rough time finding practical/real world examples of this stuff. After digging around and piecing together a bunch of tips/guides/pointers, I managed to get what I wanted working.
Also if anyone has a ton of experience with task management/scheduling/multiprocessing I'd love to buy you a coffee and pick your brain -- reach out! :)
If you need multi-core for performance, first you should rewrite in a compiled language instead of Python. Java/C/C++/Go/FORTRAN/whatever will run 10 to 20 times faster before even worrying about running multi-core.
Python can be compiled, interpreted, or translated into smoke signals and reconstituted on the other side of the valley. It is not per se any of these things.
On a mid-range desktop PC, the horsepower breakdown is something like: 100% everything, 75% GPU, 25% all CPUs, 5% one CPU including SIMD, 1% a single CPU running scalar code.
I love Python. But, it's good to remember sometimes that it runs at about 1% efficiency compared to well-optimized C. So, you paid for a whole machine and you are utilizing 0.01%. That's fine for a lot of tasks; especially tasks that are 99% I/O-bound or tasks that are just coordinating big C libraries to do 99% of the work. Just good to remember sometimes...
Hedging around these statements is important, but far too often I see people convinced they are in the 1% of tasks that truly need top level performance. Which is to say, I often see people who are wrong.
It is mostly done already, if you consider numpy. But the top-layer glue code, which is Python, needs to allow it to tun in parallel, where it's not automatic.
It's the equivalent of: (edit: actually an improvement over)
from threading import Thread
def blocking_function(arg):
....
pool = []
for arg in args[:16]:
f = lambda: blocking_function(arg)
thread = Thread(target=f)
thread.start()
pool.append(thread)
Edit: see below, this doesn't include job queueing but rather, hard limits your input to maximum 16 args. The point is, the code is a nice, easy way to start a number of threads working on a list of arguments.
Not quite. You can submit as many jobs as you'd like to the pool and it will delegate them to 16 threads. The pool maintains a queue of work to be done and the fixed number of threads consume from the queue.
Almost but as quinnftw mentioned it does a bit more, specifically it limits the number of threads it runs to 16. I can pass 1000 url or arguments into it and it won't spawn 1000 threads but keep it at 16 max. Also I think it does a bit a better job with exception propagation.
I would recommend you use coroutines rather than multiprocessing.dummy for balancing blocking operations (like IO). I believe multiprocessing.dummy is just a wrapper for threading which has context switching overhead [1].
Python3 has asyncio as part of the stdlib but for Python2 there are a variety of coroutine libraries out there (I think the most widely recognized being gevent).
Switching code-base to Python 3 or installing gevent/eventlet is certainly more than a 3 line hack.
> I believe multiprocessing.dummy is just a wrapper for threading which has context switching overhead [1].
Yap. But I am fetching and waiting for data to come from servers half-way across the country. Not worried about too much thread switching overhead. The point it was still faster than a for loop with a fetch. It was replaced by 3 lines which was 10x faster.
> Python releases the GIL during I/O right, isn't that desirable?
Python releases GIL during IO so that's why thread-based pool works for IO parallelism and there is no need to fork. Was that what you mentioned? Sorry, I don't think I completely followed.
Some good alternatives, for when you just want quick and easy parallelization:
Joblib[0]: 'Embarrassingly parallel for loops'. Basically just write a generator and get multicore processing on it. Pretty straightforward.
Multiprocess[1]: This is a fork of the multiprocessing module. The biggest benefit to me (the one time I used it) is that it's easier to create shared data structures for multiprocessing (which I couldn't do with multiprocessing.Pool). Last week I had to do a really long graph calculation that only needed to return a result for a very, very small fraction of the arguments. Joblib stored all of the null results as 'None', which tore through my RAM after billions of relatively fast calculations before crashing. But using a shared dictionary and multiprocess.Pool and imap_unordered, I was able to use mutlicore processing, adding items to the dict only if the right conditions were met, and discarding the 'None's. RAM use was very minimal.
48 comments
[ 4.9 ms ] story [ 61.8 ms ] thread> Since a new instance of Python VM is running the code, there is no GIL and you get parallelism running on multiple cores.
Has the potential to be misinterpreted by new Python devs. As I understand it, each process gets its own interpreter with its own GIL. So its not that there is no GIL, just that there are now 3 GILs each managing separate execution spaces.
Also, at a higher level of discussion, this is just a rehash of multi-processing vs. multi-threading so perhaps it would be more illuminating to start from there and move down to the library examples.
The casual reference to multi-threading in the quote from the POSIX fork definition also adds to potential reader confusion.
No you can, just run lots of processes and make lots of GILS. Ends up still using less memory than Java in many cases. Gil will be the least of your worries.
Multi-processing breaks down when you have to communicate with high frequency between various logical runners.
I think there are vastly more problems that can be solved with multi-processing than those that require multi-threading but they do exist.
There is also another layer of simulated parallelism via coroutines...but I was just trying to point out that there was potential for confusion in this piece.
C extensions called from Python are still subject to the GIL of the calling interpreter[1].
In fact, this guaranteed thread safety has led to the creation of many non-thread safe Python C extensions which is now a critical blocker to efforts to removing the GIL[2], although there are also a number of other issues as well.
As noted in the StackOverflow, you can release the GIL but there are some tricks to doing so safely[3].
But threads of a C extensions library are not necessarily pinned to a single core by the GIL like regular Python byte code.
Although many performance oriented libraries like numpy[4] forsake multi-threading by default in many cases and rely on parallelization by the user (programmer)[5].
[1] http://stackoverflow.com/questions/651048/concurrency-are-py...
[2] https://youtu.be/P3AyI_u66Bw
[3] https://docs.python.org/3/c-api/init.html#thread-state-and-t...
[4] http://www.numpy.org/
[5] http://stackoverflow.com/questions/16617973/why-isnt-numpy-m...
It will run slower in any case, and if I really care about startup speed, I will just use one of the third party JDKs and AOT compile to native code.
I have coded services in Java that would never JIT before a reboot. Python can often beat un-JITed Java.
For an IO bound tasks that is using less than 100% CPU.. again Python could be identical performance or at least very similar performance to Java.. Think of a simple rest service getting 10 hits per second. Maybe the Java service will be a few micros faster, but to a client across the internet you won't care.
Now yes you are right if you are having 900 requests per second hitting your API, the Java API will crush the Python API. Not every app or service hits that though, and I would sure rather code in Python over Java.
https://engineering.instagram.com/dismissing-python-garbage-...
Maybe? You also need to assign about 2x live memory to the heap in order for GC to work properly. If you give it less, you will blow your heap and crash the JVM
> I can also give it all the memory on my server and it will translate that into performance.
No, it won't.
CMS is realistically limited to about 16 GB of ram. Past that and you are going to have routine 2, 3, 4 second pauses for full GCs. G1GC is limited a bit higher, maybe 64 GB.. but again, beyond that and you are screwed. In a high performance app, routinely you limit heap usage to around 2 GB, which means you have about 1 GB of actual memory to use.
> Java's memory usage is extremely within your control so this comparison doesn't make sense.
Honestly, it sounds like you have never had to tune the JVM or run a Java app in production, so I disagree strongly with all you have said.
Even happens if you open in a new tab.
In fact, I can't find a single browser/OS combo it doesn't happen on.
Don't have my Ubuntu machine to test it there though.
If you're looking for some real world examples of using multiprocessing (v3), check here: https://github.com/dpgailey/asteria/blob/master/asteria-v3/c...
I love Python. But, it's good to remember sometimes that it runs at about 1% efficiency compared to well-optimized C. So, you paid for a whole machine and you are utilizing 0.01%. That's fine for a lot of tasks; especially tasks that are 99% I/O-bound or tasks that are just coordinating big C libraries to do 99% of the work. Just good to remember sometimes...
It doesn't have true CPU parallelism like multiple processes would have, but it works for IO code.
It's a little gold nuggest in the multiprocessing lib.
Python3 has asyncio as part of the stdlib but for Python2 there are a variety of coroutine libraries out there (I think the most widely recognized being gevent).
[1] https://www.youtube.com/watch?v=Obt-vMVdM8s
> I believe multiprocessing.dummy is just a wrapper for threading which has context switching overhead [1].
Yap. But I am fetching and waiting for data to come from servers half-way across the country. Not worried about too much thread switching overhead. The point it was still faster than a for loop with a fetch. It was replaced by 3 lines which was 10x faster.
Python releases GIL during IO so that's why thread-based pool works for IO parallelism and there is no need to fork. Was that what you mentioned? Sorry, I don't think I completely followed.
Joblib[0]: 'Embarrassingly parallel for loops'. Basically just write a generator and get multicore processing on it. Pretty straightforward.
Multiprocess[1]: This is a fork of the multiprocessing module. The biggest benefit to me (the one time I used it) is that it's easier to create shared data structures for multiprocessing (which I couldn't do with multiprocessing.Pool). Last week I had to do a really long graph calculation that only needed to return a result for a very, very small fraction of the arguments. Joblib stored all of the null results as 'None', which tore through my RAM after billions of relatively fast calculations before crashing. But using a shared dictionary and multiprocess.Pool and imap_unordered, I was able to use mutlicore processing, adding items to the dict only if the right conditions were met, and discarding the 'None's. RAM use was very minimal.
[0]: https://pythonhosted.org/joblib/index.html [1]: https://pypi.python.org/pypi/multiprocess
[0] http://www.celeryproject.org/