I'm a bit confused. This article seems to indicate that Python is moving in the direction of better support for parallelism. Note the line about "in this age of multicore processors" at the end of the first paragraph.
And yet the GIL is still around. It seems to me that something's amiss here.
I looked in "What's New In Python 3.2" [1]. The "Multi-threading" section says the GIL has been improved, and refers to a message on the python-dev list [2] for details. That message says, in the second paragraph:
> There still is a Global Interpreter Lock ... so Python doesn't become really better at extracting computational parallelism out of several cores.
So does the author of this article not understand the difference between parallelism and concurrency (and the related performance issues for multi-core processors)? Or is there some wonderful tidbit of information that I'm missing?
Using multiple processes in Python (rather than multiple threads) avoids many of the limitations of the GIL, and the new concurrent package (according to the PEP) can use the processing module or the threading module.
Processes in Python does have a different set of limitations in it's own right, but for most applications they aren't too disastrous.
The shared-nothing approach (for example, implemented by using processes instead of threads) is also more scalable.
After all, if you design your parallelism architecture to not rely on a shared memory space, you can later parallelize over multple machines as well as cores.
The only thing I wonder about is that processes might be more heavy-weight than threads with respect to memory usage. Then again, if you only create a fixed amount of processes, equal to the number of cores, as you should for "perfect" parallelism, the overhead is pretty much zero.
The traditional GIL implementation caused a multi-threaded program to run slower on a multi-core processor than on a single-core processor, so if Python's new GIL can make multi-core processors more competitive with single-core processors, I'd say that's a win for parallelism.
Unfortunately, I'm afraid the post might be excessively optimistic. PyPy doesn't have a GIL, does it? I really hope it becomes the standard Python at some point, especially with all the progress they've been making lately...
The PyPy FAQ[1] says: "Operating system-level threads basically work. If you enable the thread module then PyPy will get support for GIL based threading."
I had hoped that because PyPy uses a more sophisticated garbage-collector than CPython, it wouldn't need locking to update reference counts and hence wouldn't need a GIL (or wouldn't use it as much); I guess that's not the case (at least yet).
I think they are referring to the new concurrent.futures module, which abstracts the notions of task and executor.
The executors can then be thread pools, process pools, event loops, etc...
"Concurrent Development" sounds like "multiple people working on the same code at the same time". For which you need a good version control system, not a feature of the language.
If the code is concurrent, not the development, could we please have that in the title?
9 comments
[ 4.6 ms ] story [ 37.1 ms ] threadAnd yet the GIL is still around. It seems to me that something's amiss here.
I looked in "What's New In Python 3.2" [1]. The "Multi-threading" section says the GIL has been improved, and refers to a message on the python-dev list [2] for details. That message says, in the second paragraph:
> There still is a Global Interpreter Lock ... so Python doesn't become really better at extracting computational parallelism out of several cores.
So does the author of this article not understand the difference between parallelism and concurrency (and the related performance issues for multi-core processors)? Or is there some wonderful tidbit of information that I'm missing?
[1] http://docs.python.org/dev/py3k/whatsnew/3.2.html
[2] http://mail.python.org/pipermail/python-dev/2009-October/093...
Processes in Python does have a different set of limitations in it's own right, but for most applications they aren't too disastrous.
After all, if you design your parallelism architecture to not rely on a shared memory space, you can later parallelize over multple machines as well as cores.
The only thing I wonder about is that processes might be more heavy-weight than threads with respect to memory usage. Then again, if you only create a fixed amount of processes, equal to the number of cores, as you should for "perfect" parallelism, the overhead is pretty much zero.
(note: this post may be excessively cynical)
I had hoped that because PyPy uses a more sophisticated garbage-collector than CPython, it wouldn't need locking to update reference counts and hence wouldn't need a GIL (or wouldn't use it as much); I guess that's not the case (at least yet).
[1]: http://codespeak.net/pypy/dist/pypy/doc/faq.html#do-threads-...
If the code is concurrent, not the development, could we please have that in the title?