This strategy should be more common! In the source tarball, there should be one main program which you start. The compiling should be an implementation detail of the main program.
The same for web applications. I never liked the idea of reintroducing a separate build (concat/minify/compress) process whenever the sources change. Instead, the production mode should be as simple as the debug mode: When the main page is requested per HTTP, first rebuild the client JS app as needed, then deliver it to the browser.
Caching the build result should be no different from any other caching of automatically-generated resources.
I use this approach for almost all new software that I write and I'm quite satisfied with that. The main program (e.g. "bin/mytool") is a wrapper shell script or Python script that runs the build as needed, then starts the program. In the simplest case, this is a wrapper around "make && ./run_tool".
Of course, the program should indicate the build with a small message to stderr, especially if the build takes some seconds or more. Moreover, it should be possible to trigger the build individually such that distros can build packages from that. On the other hand, running the program with "-h" or "--help" is a kind of build-only process, too.
It's practical, but on the other hand, that goes against the security practice of not allowing the executable zone (be it memory or filesystem) to be writeable.
I personally prefer to precompile stuff, even running "python -m compileall" to create .pyc files, to avoid having to keep that path writeable by the user running the program.
Good point! However, if you go down that route in a web application, don't forget to pre-compile templates, and do the same for all other cached things that are "code-like".
More generall, I find W^X hard to enforce in typical web applications. Sure you can mount directory read-only, disable access to /tmp and observe what happens. But then they might use the database as template cache, or whatever.
In most (web) application I'd be happy if they had a central directory where they write stuff into (bonus points for making that directory configurable), instead of scattering generated files all across the source tree.
I like to write my code to assume that it will run the same on dev and prod, but then precompile/minify prod code as part of the deployment.
That is, I like to set it up like you say, but then not rely on it in production. Mainly because I want as few moving parts as possible on prod (especially if it means I don't need dev tools/libs on the production server).
Cython, Numpy, Numba (and others) are what make me skeptical that any of the numerical computing competitors to Python (Julia, or to a lesser extent Lua/JS/Clojure even compiled Scala etc) can displace it at. Why would you abandon this wonderfully friendly, malleable language with easily the largest set of options for anything you might like to do, for something nominally faster on an artificial benchmark (which inevitably ignores the above 3 libraries), when by dipping into any of these you get C-speed in such an accessible and friendly way, most of the time beating the competitors easily on performance.
The only real threats to Python to me come would from a properly parallelized language that would use the GPU/multicore natively in vector form. I don't see that anywhere yet.
I've seen some very impressive Julia libraries that perform almost at the level of C code, so I'm not sure if the gap that Julia needs to fill is that large.
Also, a quick question: which one is preferred nowadays, Cython or Numba?
Cython is much more mature, works on basically all python code and basically never leads to slowdowns compared to pure python. Unfortunately non of that can really be said about Numba.
That being said, when Numba works it's great and is much easier to work with than cython.
I'll echo @dagw's comments. Cython has been rock solid for me for a long time and it is my go to for any sort of external c/c++ library interfacing. That said, I find myself using Numba more and more in the places that I can as Numba has gotten significantly better in the last 6 months or so. It's still not a complete replacement for cython (and I don't think it ever will or intends to be), but for hot spot numerical calculations it's really nice since it's much faster to test things out since it doesn't involve the boiler plate required by Cython, and doesn't require the (often slow) compilation times.
In Numba 0.21.0, on-disk caching of jit'd code was also introduced, which was one of the major sticking points for us to put Numba into production in areas where we needed faster start-up times. Before we could really only use Cython because we required the start-up times available only from AOT compilation.
That all said, Numba has been a bit buggy for me at times, although these get squashed pretty quickly. I've only found a single bug in Cython in all of the years I've been using it, and it's amazing how quickly Robert Bradshaw or Stefan Behnel respond and fix things considering Cython is not their full time job.
Regarding the GPU, you might find an interest for MGL-MAT[1]:
"MAT is a linear algebra library for working with multi-dimensional arrays which supports efficient interfacing to foreign and CUDA code. BLAS and CUBLAS bindings are available.". The author's blog is quite interesting to read (see for example [2]).
The problem with the Python ecosystem is that there exist a lot of libraries for scientific computation mostly incompatible with each other (blaze, numpy, numba, numexpr, dask...). Some of them try to reintroduce types in order to compile to native code, thereby losing the advantages of dynamic typing. In short, it is a mess, and everyone is trying to add on this because the existing ecosystem is big and the cost of rewriting everything is deemed too high, so people try to fix python limitations at the library level, sometimes making python more cumbersome to write tha a conventional statically typed language (I am looking at theano).
I don't think that Lua is that much of an improvement, and I do not have big hopes for the JVM. The growing number of projects trying to improve the situation with value types, off-heap allocations and C compatibility are a sign that the VM is not suitable for scientific computing and will not be much better for a while.
I instead hope that Nim will take a leading role for scientific computation. It has an easy syntax and is flexible enough to write libraries such as numpy. At the same time it is fast and C compatibility is trivial. Of course, there is a library ecosystem to build, but I think that this is more doable than trying to work against the language itself. I am trying to start this effort with https://github.com/unicredit/linear-algebra but the road is very long and I hope it takes off.
I cannot really comment on Julia as I do not know it welll enough
> The problem with the Python ecosystem is that there exist a lot of libraries for scientific computation mostly incompatible with each other (blaze, numpy, numba, numexpr, dask...).
I actually disagree completely with your specific examples. These projects have mostly orthogonal goals and are actually quite compatible with each other -- they all speak NumPy arrays. In fact, three of them (Blaze, Numba and Dask) are sponsored by the same company (Continuum Analytics).
Yes, the SciPy ecosystem is a complex beast. There are lots of complementary projects and its not always clear what the best tool for the job is. There are certainly improvements we could make for easier cross-compatibility between libraries (and there are likely projects that could be consolidated), but the number of options you have for scientific computing in Python is an indication of a very robust ecosystem.
Dask requires you to encode computations as a graph explicitly, essentially forcing you to write an AST manually. This means that you are sidestepping the normal language mechanisms to encode program flow, and doing so is essentially incompatible with anything else. Moreover, it does not use numpy arrays - it uses its own internal format that you can convert to a numpy array when needed as explained the overview: http://dask.pydata.org/en/latest/array-overview.html
Numba is another cool project, but - again - it deviates from the mainstream Python. Not every Python function is compilable by Numba, and arithmetic is fixed-size. This means that existing Python code may or may not be compilable by Numba, and even if it works one has to carefully check that arbitrary precision arithmetic is not used. So, it is nice when it works, but is little more than a way to write C-like code with a Pythonish syntax (in which case I greatly prefer Nim, that has actual dispatching on types, generics and so on).
Numexpr requires you to write your code as strings, so it is essentially a separate interpreter. It is nice that it is fast, but it does not play with normal Python code.
Not a single project of these works on PyPy, which is the only fast interpreter for non-numeric Python code, nor on Jython, which is the only multithreaded Python interpreter.
Do not misunderstand me: I enjoy Python for many things, and internally we use it a lot. But I am frustrated that a lot of effort seems to be dedicated to fix things at the language level starting from a powerful library ecosystem, where I would like to see something solid at the language level that evolves libraries instead
I appreciate where you are coming from, but on many of these projects you are just misinformed.
The entire point of dask is to enable parallel and bigger than memory computations that are not be possible with NumPy arrays. It uses an internal graph representation for deferred computation because deferred computation is basically the only sensible way to do these computations. But in fact, a major point of dask is that users should not need to write these task graphs explicitly. The dask.array API is actually designed to be almost a perfect match for NumPy. The docs do a nice job of explaining the internal abstraction, but understanding it is by no means necessary to use it. (Dask is not my project, but I have made quite a few contributions to it.)
Numba certainly does deviate from mainstream Python, but in my experience it works very much in line with the needs and expectations of scientific python users.
As for Blaze, it's changed a long way since Travis wrote that blog post in 2012. This website provides a better overview of what Blaze is today: http://blaze.github.io/. Confusingly, the name is used for both an ecosystem and one of its major components more specifically. Dask is a component of the larger Blaze project.
> Jython, which is the only multithreaded Python interpreter.
I am intrigued by Nim. Does it have a REPL though? I think an official REPL (not a third party hack) is mandatory for scientific/numerical computing. Ocaml is the only compiled language I can think of which has a credible REPL, officially supported.
It currently does not have a REPL, and that is its main limitation right now. I think it is still suitable for more complex workflows where you would not do it in the REPL anyway - think training a large neural network. I hope something will come after 1.0 is out...
I disagree, and see the proliferation of all these performance-focused libraries/compilers as proving that CPython's slow speed is a huge problem. I haven't used Numba, but after my experience with others I conclude that you just can't totally escape the fact that CPython is slow. Having to use tools like Cython is a big tax that's cost me weeks or months of time.
Firstly numpy/scipy don't help if you can't write your algorithms in terms of operations on multidimensional arrays, and numpy overheads (creating and accessing ndarrays) are actually prohibitively large if the arrays aren't large, easy to be slower than straight Python.
First I tried writing code (scientific stuff) in type-annotated Cython. But it turns out that data structures are the bottleneck if your algorithms need to read/write something other than a bunch of numpy ndarrays. If you try to use lists, dicts, etc, you still go through the Python runtime so get little speed benefit over Python. (Cython optimises ndarray accesses.)
So I ended up writing C++ code and interfacing to it using Cython. But now I have to write a huge amount of code to translate between the Python/Numpy and C++ datastructures. And it's bug prone due to memory allocation and ownership. Using multiple poorly compatible languages is a miserable experience. Julia sounds fantastic. Don't get me wrong though, I love Python, but it wasn't designed for scientific computing. And most of the time, numpy, scipy & friends are all that you need or want.
Your second point is spot on. I realized this when I tried to optimize my numpy code that does millions of dot products between 3x1 vectors and 3x3 matrices to no avail. I realized there is an awesome little library called tinyarray, that doesn't have the overhead of numpy, and is compatible with the basic numpy syntax. I exchanged all my numpy arrays for tinyarrays and got the easiest 10x speed up ever.
If you look in the numpy source you can begin to understand why the overheads are so large: for every operation it's first necessary to apply rules for broadcasting between different sizes and types, plus often being callable with a number of different function signatures (that includes __getitem__). And in many cases all of that is implemented in Python.
fwiw - the best you can do here, is probably implement the dots yourself as cdef functions. If your arrays get larger, you can get a C pointer to the blas routine from scipy and call it from cython (which can be faster than the dot you write when your arrays are larger)
I don't see why you would use Cython when you are writing C/C++ anyway. All the trivial data types and wrappers are easily handled by SWIG. Just don't try to wrap any non-trivial types with SWIG, instead declare them as `PyObject *` and use the Python/C API for those. Demo of this technique: https://github.com/martinxyz/python/tree/master/realistic
The main advantage of using Cython for glue code is that you don't need to touch the Python/C API, it generates the necessary code for you (in particular it handles the reference counting). (I like to think of Cython as 'C with easy access to the Python runtime'.) However, the API certainly doesn't look scary in these examples (I've never really used it). Admittedly I didn't consider SWIG, so thanks for that. It would be nice to be rid of the Cython glue layer, so I might try that. Of course that would mean writing much of the glue (sanity checking) in C++ or Python instead of Cython.
Seeing how this uses mktemp, I assume the results of runcython build are not reused. Is that correct? If sou, wouldn't it make sense to save the results? Maybe with a switch?
Also feature requests: Windows support for run/make would be nice. Cross-compile options for make would be awesome.
The calls to itertools won't be sped up significantly. However if then you loop over the results of a call to itertools and do something to each element, then that loop might end up being much faster.
As dagw said, if your inner loops aren't inside a .pyx file, then Cython can do little to help. Even if there are, if it involves lots of fiddling with data structures like dicts and lists, or making use of dynamic or high level features of Python like generators or classes or list comprehensions, then Cython would provide only a small speed increase (say, 20%), because almost all the running time will be spent inside the Python runtime. To get the biggest speed ups out of Cython you need to write C-like code with type annotations. E.g. "for x in range(...):" loops.
I'll give you "the complexity of your algorithm" likely doesn't change, but how that algorithm is processed is usually more efficient when compiled through C, rather than running through a Python interpreter, if only because of how it's running.
I agree it is faster by running it through compiled C code.
Saying "my code is more efficient than your code" might be acceptable, but saying "my code is efficient" drives me mad when it is used as a synonym for "my code is fast". Take a look at [1] and the note that this is not about optimization. Efficient algorithms are usually algorithms with close to optimal time or space complexitiy.
Sometimes it does. For example if you write a simple O(n) for loop in Python, then convert it to Cython, then compile it, the compiler may replace the for loop by an equivalent O(1) algorithm. I use this to surprise students when benchmarking Cython code for teaching purposes in my SageMath course.
If it's just a matter of speeding up a tight loop, I find numba performs admirably and it's a lot more convenient than Cython.
When you need to wrap some c code, or use high-level data structures (that cython handles beautifully with STL integration) that's when it makes sense to drop to Cython.
> And I hate when people use efficiency when they mean speed.
Except they're usually right. "Efficiency" is not just a measure of algorithmic complexity. Something is more efficient if you are able to do more of a desired activity at the cost of fewer resources. The resource in question varies.
Yes, one often in computer science focuses on algorithmic complexity as the resource to be economized on. But time and power are also perfectly reasonable resources to have in mind. In fact, they are the resource that most audiences will naturally assume when they see the word "efficient" standing along, without further explanation.
I think GP is probably wrong to criticize as well -- making code more efficient [or, if you insist, fast] makes it easier to support more users that it could support otherwise. Is the ease with which a system can support an additional user not the very definition of the word "scalability"?
Of course, there are more powerful, and more specialized techniques for improving scalability, but that doesn't mean the author is wrong to suggest that speed improves scalability.
Its an aside, but recently Python 3 has got annotations. It would be nice if cython supported annotation syntax, so that type-annotated cython was as much as possible valid python than can also be run through an interpreter.
I understand a lot of the nuances, particularly the lack of in-function annotations etc. My own foray into Python type annotations is obiwan https://pypi.python.org/pypi/obiwan/
This is great to see! I don't know if it was inspired by runhaskell, but it does appear to be similar. I have to say, this kind of tool is something I have come to feel is incredibly helpful and important for my dev workflow.
I spend most of my time developing in statically-typed, compiled-to-machine-code languages; obviously, one of the major advantages of more dynamic or interpreted languages is the ability to rapidly prototype. This type of tool allows me to quickly prototype something without losing all the power, control and other benefits of my languages of choice (in particular, Haskell these days).
Things like this make me more amenable to python by the day! Keep up the good work!
50 comments
[ 3.0 ms ] story [ 114 ms ] threadThe same for web applications. I never liked the idea of reintroducing a separate build (concat/minify/compress) process whenever the sources change. Instead, the production mode should be as simple as the debug mode: When the main page is requested per HTTP, first rebuild the client JS app as needed, then deliver it to the browser.
Caching the build result should be no different from any other caching of automatically-generated resources.
I use this approach for almost all new software that I write and I'm quite satisfied with that. The main program (e.g. "bin/mytool") is a wrapper shell script or Python script that runs the build as needed, then starts the program. In the simplest case, this is a wrapper around "make && ./run_tool".
Of course, the program should indicate the build with a small message to stderr, especially if the build takes some seconds or more. Moreover, it should be possible to trigger the build individually such that distros can build packages from that. On the other hand, running the program with "-h" or "--help" is a kind of build-only process, too.
I personally prefer to precompile stuff, even running "python -m compileall" to create .pyc files, to avoid having to keep that path writeable by the user running the program.
More generall, I find W^X hard to enforce in typical web applications. Sure you can mount directory read-only, disable access to /tmp and observe what happens. But then they might use the database as template cache, or whatever.
In most (web) application I'd be happy if they had a central directory where they write stuff into (bonus points for making that directory configurable), instead of scattering generated files all across the source tree.
That is, I like to set it up like you say, but then not rely on it in production. Mainly because I want as few moving parts as possible on prod (especially if it means I don't need dev tools/libs on the production server).
The only real threats to Python to me come would from a properly parallelized language that would use the GPU/multicore natively in vector form. I don't see that anywhere yet.
Also, a quick question: which one is preferred nowadays, Cython or Numba?
That being said, when Numba works it's great and is much easier to work with than cython.
In Numba 0.21.0, on-disk caching of jit'd code was also introduced, which was one of the major sticking points for us to put Numba into production in areas where we needed faster start-up times. Before we could really only use Cython because we required the start-up times available only from AOT compilation.
That all said, Numba has been a bit buggy for me at times, although these get squashed pretty quickly. I've only found a single bug in Cython in all of the years I've been using it, and it's amazing how quickly Robert Bradshaw or Stefan Behnel respond and fix things considering Cython is not their full time job.
Numba is specific to numeric/array-oriented code. Cython is general-purpose and can also be used to implement e.g. datastructures.
[1] http://cliki.net/mgl-mat
[2] I http://quotenil.com/On-the-Design-of-Matrix-Libraries.html
I don't think that Lua is that much of an improvement, and I do not have big hopes for the JVM. The growing number of projects trying to improve the situation with value types, off-heap allocations and C compatibility are a sign that the VM is not suitable for scientific computing and will not be much better for a while.
I instead hope that Nim will take a leading role for scientific computation. It has an easy syntax and is flexible enough to write libraries such as numpy. At the same time it is fast and C compatibility is trivial. Of course, there is a library ecosystem to build, but I think that this is more doable than trying to work against the language itself. I am trying to start this effort with https://github.com/unicredit/linear-algebra but the road is very long and I hope it takes off.
I cannot really comment on Julia as I do not know it welll enough
I actually disagree completely with your specific examples. These projects have mostly orthogonal goals and are actually quite compatible with each other -- they all speak NumPy arrays. In fact, three of them (Blaze, Numba and Dask) are sponsored by the same company (Continuum Analytics).
Yes, the SciPy ecosystem is a complex beast. There are lots of complementary projects and its not always clear what the best tool for the job is. There are certainly improvements we could make for easier cross-compatibility between libraries (and there are likely projects that could be consolidated), but the number of options you have for scientific computing in Python is an indication of a very robust ecosystem.
Dask requires you to encode computations as a graph explicitly, essentially forcing you to write an AST manually. This means that you are sidestepping the normal language mechanisms to encode program flow, and doing so is essentially incompatible with anything else. Moreover, it does not use numpy arrays - it uses its own internal format that you can convert to a numpy array when needed as explained the overview: http://dask.pydata.org/en/latest/array-overview.html
Numba is another cool project, but - again - it deviates from the mainstream Python. Not every Python function is compilable by Numba, and arithmetic is fixed-size. This means that existing Python code may or may not be compilable by Numba, and even if it works one has to carefully check that arbitrary precision arithmetic is not used. So, it is nice when it works, but is little more than a way to write C-like code with a Pythonish syntax (in which case I greatly prefer Nim, that has actual dispatching on types, generics and so on).
Blaze is in a strange position which I do not fully understand. Apparently it uses Numpy arrays, but at the same time the Numpy author states it should be a Numpy replacement: http://technicaldiscovery.blogspot.it/2012/12/passing-torch-...
Numexpr requires you to write your code as strings, so it is essentially a separate interpreter. It is nice that it is fast, but it does not play with normal Python code.
Not a single project of these works on PyPy, which is the only fast interpreter for non-numeric Python code, nor on Jython, which is the only multithreaded Python interpreter.
Do not misunderstand me: I enjoy Python for many things, and internally we use it a lot. But I am frustrated that a lot of effort seems to be dedicated to fix things at the language level starting from a powerful library ecosystem, where I would like to see something solid at the language level that evolves libraries instead
The entire point of dask is to enable parallel and bigger than memory computations that are not be possible with NumPy arrays. It uses an internal graph representation for deferred computation because deferred computation is basically the only sensible way to do these computations. But in fact, a major point of dask is that users should not need to write these task graphs explicitly. The dask.array API is actually designed to be almost a perfect match for NumPy. The docs do a nice job of explaining the internal abstraction, but understanding it is by no means necessary to use it. (Dask is not my project, but I have made quite a few contributions to it.)
Numba certainly does deviate from mainstream Python, but in my experience it works very much in line with the needs and expectations of scientific python users.
As for Blaze, it's changed a long way since Travis wrote that blog post in 2012. This website provides a better overview of what Blaze is today: http://blaze.github.io/. Confusingly, the name is used for both an ecosystem and one of its major components more specifically. Dask is a component of the larger Blaze project.
> Jython, which is the only multithreaded Python interpreter.
This is not quite true. Python supports multithreading, and CPython's GIL is less of a obstacle to numerical computing than you might think: http://matthewrocklin.com/blog/work/2015/03/10/PyData-GIL/
Firstly numpy/scipy don't help if you can't write your algorithms in terms of operations on multidimensional arrays, and numpy overheads (creating and accessing ndarrays) are actually prohibitively large if the arrays aren't large, easy to be slower than straight Python. First I tried writing code (scientific stuff) in type-annotated Cython. But it turns out that data structures are the bottleneck if your algorithms need to read/write something other than a bunch of numpy ndarrays. If you try to use lists, dicts, etc, you still go through the Python runtime so get little speed benefit over Python. (Cython optimises ndarray accesses.)
So I ended up writing C++ code and interfacing to it using Cython. But now I have to write a huge amount of code to translate between the Python/Numpy and C++ datastructures. And it's bug prone due to memory allocation and ownership. Using multiple poorly compatible languages is a miserable experience. Julia sounds fantastic. Don't get me wrong though, I love Python, but it wasn't designed for scientific computing. And most of the time, numpy, scipy & friends are all that you need or want.
If you look in the numpy source you can begin to understand why the overheads are so large: for every operation it's first necessary to apply rules for broadcasting between different sizes and types, plus often being callable with a number of different function signatures (that includes __getitem__). And in many cases all of that is implemented in Python.
Also feature requests: Windows support for run/make would be nice. Cross-compile options for make would be awesome.
My question is if I would still get a considerable speed-up if I rewrite some of my code in Cython, given my reliance on itertools?
http://matthewrocklin.com/blog/work/2014/05/01/Introducing-C...
I know this is being a semantic weenie, but I hate when people use scalability when they mean efficiency.
Python -> Cython speeds up you program but does not change the complexity of your algorithm.
I'll give you "the complexity of your algorithm" likely doesn't change, but how that algorithm is processed is usually more efficient when compiled through C, rather than running through a Python interpreter, if only because of how it's running.
Saying "my code is more efficient than your code" might be acceptable, but saying "my code is efficient" drives me mad when it is used as a synonym for "my code is fast". Take a look at [1] and the note that this is not about optimization. Efficient algorithms are usually algorithms with close to optimal time or space complexitiy.
https://en.wikipedia.org/wiki/Algorithmic_efficiency
When you need to wrap some c code, or use high-level data structures (that cython handles beautifully with STL integration) that's when it makes sense to drop to Cython.
Except they're usually right. "Efficiency" is not just a measure of algorithmic complexity. Something is more efficient if you are able to do more of a desired activity at the cost of fewer resources. The resource in question varies.
Yes, one often in computer science focuses on algorithmic complexity as the resource to be economized on. But time and power are also perfectly reasonable resources to have in mind. In fact, they are the resource that most audiences will naturally assume when they see the word "efficient" standing along, without further explanation.
I think GP is probably wrong to criticize as well -- making code more efficient [or, if you insist, fast] makes it easier to support more users that it could support otherwise. Is the ease with which a system can support an additional user not the very definition of the word "scalability"?
Of course, there are more powerful, and more specialized techniques for improving scalability, but that doesn't mean the author is wrong to suggest that speed improves scalability.
I understand a lot of the nuances, particularly the lack of in-function annotations etc. My own foray into Python type annotations is obiwan https://pypi.python.org/pypi/obiwan/
I spend most of my time developing in statically-typed, compiled-to-machine-code languages; obviously, one of the major advantages of more dynamic or interpreted languages is the ability to rapidly prototype. This type of tool allows me to quickly prototype something without losing all the power, control and other benefits of my languages of choice (in particular, Haskell these days).
Things like this make me more amenable to python by the day! Keep up the good work!