That's a good point. I'd wonder if it had any production quality seeing that number. Yet, most reviews I read talk like it's beta quality or better for their uses. Hmm.
I'm the guy who started the Cython project and made up the name, so here are some random remarks that don't answer your question. Not only is Cython about 8 years old, it started as a fork of Pyrex (http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/), which is itself over 11 years old, since I remember using it in 2004 when evaluating languages in which to implement http://sagemath.org.
The people who did the most work on Cython on probably Stefan Behnel and Robert Bradshaw (who was my Ph.D. student). I had to sort of twist their arms to get them to be involved in the project, and they both still only do it in their spare time basically for fun. Despite Cython being an incredibly useful and powerful tool for certain specific problem domains (e.g., for SageMath it is critical), and being very heavily used in some scientific python applications, there's no company behind Cython and few people involved in the project are interested in publicity. Cython is an open source labor of love and an extremely useful tool. People that know Cython very well, know how amazingly powerful it is, and have watched many other attempts to solve similar problems come and go over the years.
I agree, that it is extremely useful, since before I knew Cython, I implemented some Python C extension modules by hand. That can become rather ugly and costs much sweat.
Today, I can use Cython. Or when I need even more speed, than I use Cython as wrapper-generator for a pure C module, that is still much, much easier to implement, as when I would have to do the wrapping work myself.
Cython is really enormous useful and precious to me!
The problem is that there is no pattern of version numbers that is meaningfully informative. It's pretty much necessary to figure out how reliable the releases are for each project that you are interested in.
For people that aren't following along, 0.23 might have slightly worse optics than 1.2, but there is plenty of software called 1.2 that is crap.
Yes, the 0.x number also put me down at first. But I am using this tool now several years, since 0.13 or maybe earlier and did never find a bug or problem.
Cython is great, in-spite of its version number!
I also inspected the generated code several times and it looks rock-solid. Also one big advantage (for me): It can produce code that runs equally on 2.x and 3.x versions (there are some limitations, though, like several really outdated versions are not supported any more).
Cython is amazingly productive for me. I am so happy to write typed pythonic code.
If there's one thing missing it's more people showing off their Cython style and tricks, like how they do macros (since it's not supported in the language), when to turn off safeties, and balancing pure python with typed.
For instance, I like C macros in header files, but I've played around with using a jinja2 pipeline in my setup.py to write cleaner looking pythonic code, and have done generics this way.
more posts like the one using cymem would be welcome.
Currently I don't do any macros at all, although the temptation is definitely strong. I have some code that could otherwise be more generic than it is.
I guess a quick tip/pattern would be to use structs as data containers, with a cdef class defined as a "shell", to provide access, but not actually hold the data.
Advantages of this over standard cdef class: Data is contiguous in memory; Data can be stack allocated; Data can be stored in C arrays or C++ containers; Data can be created without GIL.
Advantages of this over numpy array: More Pythonic syntax (i.e. data.attr, instead of data[column]; No fussing with memorybuffers, compiler declarations etc; Data can be created without GIL
Disadvantages: More code
(I wrote that cymem blog post -- I'll write up this pattern as a post when I get a chance.)
If you use Python for anything remotely performance sensitive, I highly recommend giving Cython a shot. The trick is to find your computation-intensive inner loop stuff, move it off to its own module, add types, and go. It makes your build a little more complicated (i.e. most Python projects have no build process at all) but if you're in the niche where it's useful, it's damned useful.
Here what it does. Take a look at this Python code:
import math
def distance(x1, y1, x2, y2):
return math.sqrt((y2-y1)**2+(x2-x1)**2)
It looks simple, but it's doing a lot behind the scenes. It respects __sub__/__add__ metamethods, respects and promotes between floats/longs/ints and even Bignums, it has to do a __getattr__ on the `math` object to find the sqrt function in the first place, etc (and can't cache that because this is Python so you may have changed it in between calls). After every one of these calls, a Python exception is checked for, and many reference counts are incremented and decremented. Every now and then, the interpreter checks to see if it's time to release/reacquire the GIL and if any OS signals have come in. And it does all of this by compiling Python to bytecode and running that through its interpreter.
So now instead of running through dozens of intermediate internal Python functions, you're just loading the numbers into your FPU and doing the computation there.
Of course, you've given up all of your Python dynamicness. Now this won't work with Bignums, it can't be monkey patched, and now you have a compiling step to running your Python code. You can do this to your existing code one function at a time, so you don't have to give those things up for your entire project, just where it matters. And where performance matters, it's a godsend
I ported a small number of pieces of reddit to Cython. Here are some simple examples:
As the underscore naming implies, these are only selected performance or compute sensitive functions factored out of their normal non-underscore Python modules which generally wrap them. These functions were coming up disproportionately in profiling because they're called very often and/or do a lot of arithmetic, and afterwards they are entirely nonissues. Often something like a _sorts.pyx:_confidence call got over 25 times faster.
I've also talked about the downsides in a thread here[1]. Go in...
It's also provides great benefits for quick scripts.
I had to write a one-off parser for a big csv file, that needed to be parsed lots of times, and fast. Pure python solution was taking a few minutes, so I decided to give Cython a try. Installed, read the docs for a few minutes, converted my parser's inner loop to cython, and the duration went down to a few seconds.
The "%%cython" magic for the IPython notebook is great for quickly prototyping cython code in the context of a larger Python workflow. You can also view a nice annotated HTML performance report with "%%cython -a".
15 comments
[ 2.7 ms ] story [ 40.1 ms ] threadToday, I can use Cython. Or when I need even more speed, than I use Cython as wrapper-generator for a pure C module, that is still much, much easier to implement, as when I would have to do the wrapping work myself.
Cython is really enormous useful and precious to me!
Thank you for it!
For people that aren't following along, 0.23 might have slightly worse optics than 1.2, but there is plenty of software called 1.2 that is crap.
Yes, the 0.x number also put me down at first. But I am using this tool now several years, since 0.13 or maybe earlier and did never find a bug or problem.
Cython is great, in-spite of its version number!
I also inspected the generated code several times and it looks rock-solid. Also one big advantage (for me): It can produce code that runs equally on 2.x and 3.x versions (there are some limitations, though, like several really outdated versions are not supported any more).
If there's one thing missing it's more people showing off their Cython style and tricks, like how they do macros (since it's not supported in the language), when to turn off safeties, and balancing pure python with typed.
For instance, I like C macros in header files, but I've played around with using a jinja2 pipeline in my setup.py to write cleaner looking pythonic code, and have done generics this way.
more posts like the one using cymem would be welcome.
Currently I don't do any macros at all, although the temptation is definitely strong. I have some code that could otherwise be more generic than it is.
I guess a quick tip/pattern would be to use structs as data containers, with a cdef class defined as a "shell", to provide access, but not actually hold the data.
Advantages of this over standard cdef class: Data is contiguous in memory; Data can be stack allocated; Data can be stored in C arrays or C++ containers; Data can be created without GIL.
Advantages of this over numpy array: More Pythonic syntax (i.e. data.attr, instead of data[column]; No fussing with memorybuffers, compiler declarations etc; Data can be created without GIL
Disadvantages: More code
(I wrote that cymem blog post -- I'll write up this pattern as a post when I get a chance.)
Here what it does. Take a look at this Python code:
It looks simple, but it's doing a lot behind the scenes. It respects __sub__/__add__ metamethods, respects and promotes between floats/longs/ints and even Bignums, it has to do a __getattr__ on the `math` object to find the sqrt function in the first place, etc (and can't cache that because this is Python so you may have changed it in between calls). After every one of these calls, a Python exception is checked for, and many reference counts are incremented and decremented. Every now and then, the interpreter checks to see if it's time to release/reacquire the GIL and if any OS signals have come in. And it does all of this by compiling Python to bytecode and running that through its interpreter.Here's the same code in Cython:
Cython takes that code and compiles it to this C code (a bit simplified): So now instead of running through dozens of intermediate internal Python functions, you're just loading the numbers into your FPU and doing the computation there.Of course, you've given up all of your Python dynamicness. Now this won't work with Bignums, it can't be monkey patched, and now you have a compiling step to running your Python code. You can do this to your existing code one function at a time, so you don't have to give those things up for your entire project, just where it matters. And where performance matters, it's a godsend
I ported a small number of pieces of reddit to Cython. Here are some simple examples:
* https://github.com/reddit/reddit/blob/master/r2/r2/lib/db/_s...
* https://github.com/reddit/reddit/blob/master/r2/r2/lib/utils...
* https://github.com/reddit/reddit/blob/master/r2/r2/lib/mr_to...
* https://github.com/reddit/reddit/blob/master/r2/r2/lib/wrapp...
As the underscore naming implies, these are only selected performance or compute sensitive functions factored out of their normal non-underscore Python modules which generally wrap them. These functions were coming up disproportionately in profiling because they're called very often and/or do a lot of arithmetic, and afterwards they are entirely nonissues. Often something like a _sorts.pyx:_confidence call got over 25 times faster.
I've also talked about the downsides in a thread here[1]. Go in...
I had to write a one-off parser for a big csv file, that needed to be parsed lots of times, and fast. Pure python solution was taking a few minutes, so I decided to give Cython a try. Installed, read the docs for a few minutes, converted my parser's inner loop to cython, and the duration went down to a few seconds.