Why would the inability to write multi-threaded programs that are worth a damn be a troll talking point? It meaningfully lowers the appropriateness of using python for server code.
There's more than one way to do it. If you're thinking about hundreds of threads doing chunks of work, then my comment will not be meaningful but if you're talking about maximizing CPU utilization for a particular task, then multiple processes is a valid and proven approach (Tornado does just that for example, forking as many processes as you have CPUs and putting them to work).
No, the GIL is simply not relevant for real work. True Pythonistas, when performance is critical, just drop into C for the crucial section and use Python to control it. When C code is running, the Python runtime isn't blocked - you can kick off a hundred native pthreads in the background if you want to.
This is also true in OCaml BTW, but it's trickier to make the call as when running single-threaded, OCaml code is as fast as C anyway. A Python/OCaml hybrid using C as glue is probably the best of both worlds.
The main problem is the perception that code would linearly scale if the GIL was removed. Eg on a 4 core machine code would run 4 times as fast. And that the GIL has no value. Both statements are completely false. (A bigger issue is the reference counting anyway. Changing that should be a precursor to GIL removal.)
I'm an author of a C extension (apsw) and the GIL is fantastic. Without it the code would be considerably more complicated, longer and likely to have race conditions and similar gnarly bugs.
As a Python developer I use:
* Multiple processes usually by executing a new process which allows me to easily substitute testing processes and using stdin/stdout
* Use networking for talking between parts (allows spreading over multiple machines, failover, redundancy, load balancing etc)
* multiprocessing for pure Python work
* Threads for I/O bound work or work that waits on other things
* C extensions that release the GIL
I'm actually interested to know what the HN community does to achieve parallelism in Python? Do you use multiprocessing module, Stackless python, PP module (parallel python)?
Either multiprocessing or shifting the bits that really benefit from parallelization out to C/C++ tend to cover the vast majority of my needs. Also numpy and numexpr does a lot of the multiprocessing stuff for me, so in many cases I don't even have to think about it.
For sufficiently new version of Python (3.2+), concurrent.futures[1] is pretty nice for its use-case, but otherwise multiprocessing has been quite good.
I'm using QThread for PyQt GUI applications.
However the GIL makes it impossible to execute _Python_ code in parallel. I can understand the reasons for that, however it doesn't change the fact that it makes my application inherently slower and is therefore something I dislike.
Same here. As a nice side effect, QThread somehow makes signals work again, so you can Ctrl-C a MT Python program (you still have to set a handler for SIGINT but at least it works).
For this reason, I actually use QThread in console apps aswell.
Well I guess it is "unnecessary" to the extent it is unnecessary to be able to Control-C your programs. Personally I like my programs to respond to signals.
Qt's split up into separate libraries; QtCore is pretty much a general portable utility library like glib or APR, and it's perfectly reasonable for a console app to depend on it.
That's usually the point at which I decide the project could really use being rewritten in C.
If I need performance badly enough that I'm considering making major structural changes in order to utilise multi-process or find some other way around the GIL, I probably ought to be considering using a language that supports what I'm trying to do.
Of course my python projects tend to be fairly small.
If my projects were large enough to make that a more attractive option, sure, that's a great plan.
Usually when I've hit this situation I'm writing something simple but processor intensive, and it's easier for me (12 years commercial parallel/distributed C programming, amateur python enthusiast) to go back to home ground at this point.
If the problem at hand isn't easily solved with a multi-process approach then this would probably be the time I'd start taking a hard look at another language.
Scala, Go, Clojure etc make this so much easier that I think a language jump is reasonable.
Python objects only need to be thread-safe if they are accessed by multiple threads. A proven solution that could preserve Python's atomicity guarantees is actor-based concurrency like Erlang or Go. When Python objects are queued to an actor's channel, a deep copy (or copy-on-write reference) could be sent to avoid data sharing.
> You will rewrite your code to not make use of the guarantees.
I suppose a problem is that unlike Java, Python is not fast enough to implement low level data structures in the language itself, so you can't easily implement non-thread-safe data structures without losing memory safety at the C level, which is unacceptable.
Still, I have the feeling that with heavy use of biased locking, you could stick locks everywhere without a lot of overhead in the non-contended case. Or maybe that famous patch just needs to be revisited, tweaked to more efficiently address the areas of highest contention... see http://dabeaz.blogspot.com/2011/08/inside-look-at-gil-remova... for some context.
If you tag objects as being either "owned" by a thread or "shared" then you could (in the python implementation) do something like:
if (obj.owner == current_thread):
# just get in and mess with it
elsif (obj.owner == shared):
# get in and mess with it under a lock
else
throw "not your obj - make this obj shared if you need that"
From a language perspective, you'd just need a way to declare that an object was 'shared'. All other objs would be owned by their creating thread.
No, that doesn't work. Java has something like this (synchronized methods) but they are cumbersome (I find programming nicer with plain old locks and conditions, the Java synchronized model feels just a layer of cruft but its all lock&cond inside).
But Python is different than Java. The Python interpreter has to guard the internal data structures of the interpreter itself, and that is a lot more tricky than it seems on the surface.
32 comments
[ 3.4 ms ] story [ 72.2 ms ] threadThis is also true in OCaml BTW, but it's trickier to make the call as when running single-threaded, OCaml code is as fast as C anyway. A Python/OCaml hybrid using C as glue is probably the best of both worlds.
I'm an author of a C extension (apsw) and the GIL is fantastic. Without it the code would be considerably more complicated, longer and likely to have race conditions and similar gnarly bugs.
As a Python developer I use:
* Multiple processes usually by executing a new process which allows me to easily substitute testing processes and using stdin/stdout * Use networking for talking between parts (allows spreading over multiple machines, failover, redundancy, load balancing etc) * multiprocessing for pure Python work * Threads for I/O bound work or work that waits on other things * C extensions that release the GIL
Actually getting rid of the GIL wouldn't help.
Often I have use-cases where my own shared memory system works well http://williamedwardscoder.tumblr.com/post/13363076806/buffc...
> I was writing a custom video editing program in Python (interesting choice of language for that problem) and I wanted to cache decoded frames
[1]: http://docs.python.org/py3k/library/concurrent.futures.html
Apart from that I really like Python.
For this reason, I actually use QThread in console apps aswell.
If I need performance badly enough that I'm considering making major structural changes in order to utilise multi-process or find some other way around the GIL, I probably ought to be considering using a language that supports what I'm trying to do. Of course my python projects tend to be fairly small.
Usually when I've hit this situation I'm writing something simple but processor intensive, and it's easier for me (12 years commercial parallel/distributed C programming, amateur python enthusiast) to go back to home ground at this point.
Scala, Go, Clojure etc make this so much easier that I think a language jump is reasonable.
@jagw: ok fair, but that is going to be the very first example of things thou shalt not do when using threads in any article, book, or course
I suppose a problem is that unlike Java, Python is not fast enough to implement low level data structures in the language itself, so you can't easily implement non-thread-safe data structures without losing memory safety at the C level, which is unacceptable.
Still, I have the feeling that with heavy use of biased locking, you could stick locks everywhere without a lot of overhead in the non-contended case. Or maybe that famous patch just needs to be revisited, tweaked to more efficiently address the areas of highest contention... see http://dabeaz.blogspot.com/2011/08/inside-look-at-gil-remova... for some context.
(some forms of biased locking are patented.)
But Python is different than Java. The Python interpreter has to guard the internal data structures of the interpreter itself, and that is a lot more tricky than it seems on the surface.