I’m all for not prematurely optimising and the point of the authors post is still valid, but it is hard to argue that the resulting code is more readable or understandable than the rust code.
The rust code is pretty simple, and plain, and doesn’t need any of these tricks
There are benefits to standardizing on a single language and codebase, so it may be worth having some code that is less readable or performant in order to optimize for the global maximum and not prematurely introduce too many languages.
My least favorite thing about performance optimizations is how hard they are to guard against regressions with standard CI/CD. Unit tests, integration tests, etc. are rarely the right tools. And in some systems (e.g. distributed data processing), local performance often does not indicate true performance at scale.
If you want truly accurate results, you have to have a dedicated machine – not a VM (unless you’re controlling the hypervisor, and can guarantee no overcommit), not a container, but an entire machine – and tightly control every aspect of it. Updated packages? All results are invalid until you can prove the baseline hasn’t changed.
I don’t think most people need this level of accuracy, but it is something to consider.
Database queries are even worse, because not only do you need representative data at a similar scale (or you’ve scaled both hardware and working set accurately), but you have to hope that the table/column stats in prod and test are kept at a similar level of accuracy, which is more art than science.
Did the query regress because of the change, or because writes to an unrelated table have increased load and the analysis daemon is now struggling? Did it regress in test because someone else pushed a change to a rarely-used query, causing buffer churn when CI ran your job? Etc.
I agree with all of this. Because of this, even at massive scale it’s hard to justify spending the time and effort needed to get this level of testing.
Conversely, freeing data science/numerical analysis tooling from the tire-fire of the Python ecosystem might actually be a net benefit for the community at large. Get a bit more variety into the options, explore some alternative approaches. Who knows, maybe some ideas from other programming languages will yield some useful advantages.
Python’s ecosystem is great IFF you use 3rd party tooling carefully, and understand how and when to use it. The biggest one is to use venvs liberally, and to let as few things as possible interact with system Python.
If there’s a CLI tool written in Python, ideally install it with pipx.
Use Poetry for every project.
Unfortunately these aren’t inherently obvious until you have a raging tire fire, and even then, you have to know the source of it and that better options exist.
For those playing at home wondering why you can’t just do this on all Python loops:
The numba jit is for numerical code. You won’t magically get better performance iterating over a bunch of things you get back from your db for example.
Why? Node's packaging story is about a million times better than Python's. You literally just listed some of the issues with Python, none of which apply to Node.
> use venvs liberally
Not necessary with Node (you could argue it always uses venvs but that's transparent from a user point of view).
> to let as few things as possible interact with system Python.
Not an issue.
> If there’s a CLI tool written in Python, ideally install it with pipx.
Node has one way to install things and it isn't the way that nobody has heard of (I honestly had not heard of pipx until now and I'm forced to write Python some of the time for my job).
> Use Poetry for every project.
You don't have to magically know which project management tool out of dozens to use with Node. Just use the overwhelmingly popular one (NPM).
You missed a few more by the way. How about knowing to use `--config-settings editable_mode=compat` in order to use editable installs with static tooling? Oh and remember that that is deprecated with no alternative! And it silently doesn't work in `requirements.txt`! Yeay.
Then there's `pip`'s fun "wheel error! this probably isn't a wheel error" message. Very user friendly.
Oh and don't forget the whole `pip3`/`python3 -m pip` debacle. Did I say `python3`? I meant `python3` except on Windows.
Node is a million miles ahead of Python in terms of packaging usability.
> Why? Node's packaging story is about a million times better than Python's. You literally just listed some of the issues with Python, none of which apply to Node.
No, instead you get to deal with absurdly slow dependency solving / installation (I gather Bun is fast, so good on that guy), the existence of leftpad, and a `node_modules` directory that is stupidly large.
> Not an issue. [to let as few things as possible interact with system Python.]
Obviously, because systems aren't issuing tooling in Node.
> Node has one way to install things and it isn't the way that nobody has heard of (I honestly had not heard of pipx until now and I'm forced to write Python some of the time for my job).
This is literally untrue – I know for a fact that both npm and yarn are extremely popular, and as mentioned, the newcomer bun exists.
> How about knowing to use `--config-settings editable_mode=compat` in order to use editable installs with static tooling?
Or, you know, switch to modern solutions for this: `poetry add --path /foo/package` or `pipenv install -e /foo/package`.
> Oh and don't forget the whole `pip3`/`python3 -m pip` debacle. Did I say `python3`? I meant `python3` except on Windows.
Don't use Windows, obvs. Seriously though, I give absolutely zero fucks about development experience on Windows with Python.
> No, instead you get to deal with absurdly slow dependency solving / installation
…have you installed anything with Python recently? It’s not exactly speedy itself. Oh and who knows whether said application will do the right thing and use a venv or just spew its packages into the global path. At least node has the dignity to put all its project-scoped dependencies into a single place, separate from everything else.
> Obviously, because systems aren't issuing tooling in Node.
Given I can’t remember the last time I used an application that relied on the system providing the runtime (let alone the correct version) instead of shipping their required version, I’m not sure this is really much of a point.
> This is literally untrue – I know for a fact that both npm and yarn are extremely popular, and as mentioned, the newcomer bun exists.
And all of them are more streamlined than your average Python packaging experience, I promise. I could probably get my parents to NPM install a working with greater success and ease than trying to get them to do the same with whatever is the latest Python packaging mechanism de jour.
I have to ask, to clarify some points: are you referring to a library for use in a project, or a standalone tool?
> have you installed anything with Python recently? It’s not exactly speedy itself
Yes? I’ve never thought, “man this is slow,” which is very unlike my few experiences with npm. IIRC yarn was quite a bit faster, but I haven’t used either in some time. If you have a list of packages as a benchmark, I’m happy to try to come up with an equivalently-sized Python list and compare the two.
> Given I can’t remember the last time I used an application that relied on the system providing the runtime (let alone the correct version) instead of shipping their required version, I’m not sure this is really much of a point.
I think you’re missing the point – systems often ship with tooling written in Python because it’s a reasonable language, with a relatively stable ecosystem. Python versions don’t EOL for at least 5 years. Node is about half that.
> No, instead you get to deal with absurdly slow dependency solving / installation
NPM is definitely faster than pip to install things. Not sure what you're talking about here.
> I know for a fact that both npm and yarn are extremely popular
The point is they both work. Nobody is going to say "you picked the wrong package manager out of these 8 options". There's an obvious option and a slightly less obvious option and they both work properly.
> switch to modern solutions for this
What was I just saying about having to magically know the "right" way to do it? In any case I don't think either of those will solve the issue.
> Don't use Windows, obvs. Seriously though, I give absolutely zero fucks about development experience on Windows with Python.
Sounds like you give zero fucks about the Python development experience on Linux too, otherwise you wouldn't defend its obvious deficiencies.
I have literally never had a project for which Poetry, given an exact list of dependencies and flawless running on the "bad system", installed correctly on first try.
Most recently it defaults to asking for a keyring, even when just downloading? I believe that's marked WONTFIX. Even with that hacked around, and I thought the point was to reduce the number of hacks required, tensorflow seems to almost universally fail in absolutely any configuration I've tried.
Don't even begin to try integrating your own packages, c++ template errors are tame by comparison. I've also never seen a poetry error fit in less than 500 lines of nested exceptions.
I want to like this tool but... boy, Docker doesn't give me this crap.
Granted I haven’t tried Tensorflow with it, but I’ve literally never had any issues with it that weren’t caused by system Python changing (like Brew updating the Python minor version). And that’s fixed by installing it with pipx, which is one of their recommended ways to do so.
Poetry is tightly optimised for what they consider the 'correct' way of developing 'normal' python apps. Wander off that path and poetry becomes worse and worse.
It also fits with the Zen of Python (which tbf rarely seems to happen) of, “There should be one-- and preferably only one --obvious way to do it.”
Node / JS in general seems to take the path of, “this existing solution doesn’t precisely fit my needs, so instead of rethinking why that might be, I’m going to reinvent the wheel.”
Given that you where arguing that using python on the most popular OS in the world should fall outside of a 'normal' use case python should focus on, perhaps you have a too narrow view of what "correct and normal" python usage should be.
Language or platform standardization for data science is over-rated. Most often, analysis is performed remotely, and resulting models and other work product are deployed, "frozen" on production platforms. There is seldom need to match the data scientist's tools to match those of the organization.
As someone that doesn’t spend a lot of time in Python, but is familiar with the ecosystem and numpy. I would say this is actually a lot more approachable than I thought it would be.
The only thing that really surprised me was that rewriting corrcoef led to a speedup. I would imagine that’s a fairly optimized function in numpy.
Cool. Now, look at the "rust code" when you're tasked with changing mean to median... in the average non-programming specialist shop, where what you have access to is the production server. Which may or may not be the dev server, or a laptop.
No they're not complying to your CI/CD best standards - tell me with a straight face they're learnable in less than a month with a budget of $0 above your existing capital. Yes they're adding real value to the economy. That's "computing for everyone", and one of those tools works for that scenario.
This seems like a meaningless claim because the core parts of both itertools and pandas libraries used here are written in C, unless I'm missing some context here.
Interesting for sure, but that's definitely a more painful optimisation journey than the Rust one, which only involved Rust. No custom JITing of a subset of Rust, or calling out to C.
I think the real takeaway from my experience of Rust vs Python performance is that in practice people don't optimise 99% of Python so the real number of significance is the 8x speedup he got from just switching to Rust with no profiling whatsoever.
That's actually quite a low difference compared to my experience - I'd say 50x is more typical.
It would be a different matter if something like PyPy would be the reference implementation, but alas we are quite far from it.
As it stands, any AOT/JIT compiled language has an edge over Python, considering using only the reference implementation, which is quite relevant on systems were one cannot expect to allowed to install additional development tools.
As for the bindings to native libraries, yeah Python might be favoured for bindings, however those native libraries can be called by any language that has FFI capabilities anyway, alongside their JIT/AOT capabilities.
34 comments
[ 2.8 ms ] story [ 77.3 ms ] thread[0] https://willcrichton.net/notes/k-corrset/
[1] https://news.ycombinator.com/item?id=37964161
The rust code is pretty simple, and plain, and doesn’t need any of these tricks
It’s very readable to me (not a data scientist, but I’m decent at Python), YMMV.
I don’t think most people need this level of accuracy, but it is something to consider.
Database queries are even worse, because not only do you need representative data at a similar scale (or you’ve scaled both hardware and working set accurately), but you have to hope that the table/column stats in prod and test are kept at a similar level of accuracy, which is more art than science.
Did the query regress because of the change, or because writes to an unrelated table have increased load and the analysis daemon is now struggling? Did it regress in test because someone else pushed a change to a rarely-used query, causing buffer churn when CI ran your job? Etc.
If there’s a CLI tool written in Python, ideally install it with pipx.
Use Poetry for every project.
Unfortunately these aren’t inherently obvious until you have a raging tire fire, and even then, you have to know the source of it and that better options exist.
But hey, at least it isn’t Node.
I still have hopes that Julia and Ruby competition makes it change.
In practice, I have seen Node apps blocking or crashing far more often than Python apps, which drags the performance down just a bit.
In the meantime, @njit decorator on Python functions, and boom, done.
@njit doesn't work in regular Python, requires installing Numba and only works in very specific use cases.
As to specificity, I’d argue that it works on the things which stand to benefit the most from it. Parallelizing loops, vectorizing, etc.
The numba jit is for numerical code. You won’t magically get better performance iterating over a bunch of things you get back from your db for example.
Why? Node's packaging story is about a million times better than Python's. You literally just listed some of the issues with Python, none of which apply to Node.
> use venvs liberally
Not necessary with Node (you could argue it always uses venvs but that's transparent from a user point of view).
> to let as few things as possible interact with system Python.
Not an issue.
> If there’s a CLI tool written in Python, ideally install it with pipx.
Node has one way to install things and it isn't the way that nobody has heard of (I honestly had not heard of pipx until now and I'm forced to write Python some of the time for my job).
> Use Poetry for every project.
You don't have to magically know which project management tool out of dozens to use with Node. Just use the overwhelmingly popular one (NPM).
You missed a few more by the way. How about knowing to use `--config-settings editable_mode=compat` in order to use editable installs with static tooling? Oh and remember that that is deprecated with no alternative! And it silently doesn't work in `requirements.txt`! Yeay.
Then there's `pip`'s fun "wheel error! this probably isn't a wheel error" message. Very user friendly.
Oh and don't forget the whole `pip3`/`python3 -m pip` debacle. Did I say `python3`? I meant `python3` except on Windows.
Node is a million miles ahead of Python in terms of packaging usability.
No, instead you get to deal with absurdly slow dependency solving / installation (I gather Bun is fast, so good on that guy), the existence of leftpad, and a `node_modules` directory that is stupidly large.
> Not an issue. [to let as few things as possible interact with system Python.]
Obviously, because systems aren't issuing tooling in Node.
> Node has one way to install things and it isn't the way that nobody has heard of (I honestly had not heard of pipx until now and I'm forced to write Python some of the time for my job).
This is literally untrue – I know for a fact that both npm and yarn are extremely popular, and as mentioned, the newcomer bun exists.
> How about knowing to use `--config-settings editable_mode=compat` in order to use editable installs with static tooling?
Or, you know, switch to modern solutions for this: `poetry add --path /foo/package` or `pipenv install -e /foo/package`.
> Oh and don't forget the whole `pip3`/`python3 -m pip` debacle. Did I say `python3`? I meant `python3` except on Windows.
Don't use Windows, obvs. Seriously though, I give absolutely zero fucks about development experience on Windows with Python.
…have you installed anything with Python recently? It’s not exactly speedy itself. Oh and who knows whether said application will do the right thing and use a venv or just spew its packages into the global path. At least node has the dignity to put all its project-scoped dependencies into a single place, separate from everything else.
> Obviously, because systems aren't issuing tooling in Node.
Given I can’t remember the last time I used an application that relied on the system providing the runtime (let alone the correct version) instead of shipping their required version, I’m not sure this is really much of a point.
> This is literally untrue – I know for a fact that both npm and yarn are extremely popular, and as mentioned, the newcomer bun exists.
And all of them are more streamlined than your average Python packaging experience, I promise. I could probably get my parents to NPM install a working with greater success and ease than trying to get them to do the same with whatever is the latest Python packaging mechanism de jour.
> have you installed anything with Python recently? It’s not exactly speedy itself
Yes? I’ve never thought, “man this is slow,” which is very unlike my few experiences with npm. IIRC yarn was quite a bit faster, but I haven’t used either in some time. If you have a list of packages as a benchmark, I’m happy to try to come up with an equivalently-sized Python list and compare the two.
> Given I can’t remember the last time I used an application that relied on the system providing the runtime (let alone the correct version) instead of shipping their required version, I’m not sure this is really much of a point.
I think you’re missing the point – systems often ship with tooling written in Python because it’s a reasonable language, with a relatively stable ecosystem. Python versions don’t EOL for at least 5 years. Node is about half that.
NPM is definitely faster than pip to install things. Not sure what you're talking about here.
> I know for a fact that both npm and yarn are extremely popular
The point is they both work. Nobody is going to say "you picked the wrong package manager out of these 8 options". There's an obvious option and a slightly less obvious option and they both work properly.
> switch to modern solutions for this
What was I just saying about having to magically know the "right" way to do it? In any case I don't think either of those will solve the issue.
> Don't use Windows, obvs. Seriously though, I give absolutely zero fucks about development experience on Windows with Python.
Sounds like you give zero fucks about the Python development experience on Linux too, otherwise you wouldn't defend its obvious deficiencies.
Most recently it defaults to asking for a keyring, even when just downloading? I believe that's marked WONTFIX. Even with that hacked around, and I thought the point was to reduce the number of hacks required, tensorflow seems to almost universally fail in absolutely any configuration I've tried.
Don't even begin to try integrating your own packages, c++ template errors are tame by comparison. I've also never seen a poetry error fit in less than 500 lines of nested exceptions.
I want to like this tool but... boy, Docker doesn't give me this crap.
Poetry is tightly optimised for what they consider the 'correct' way of developing 'normal' python apps. Wander off that path and poetry becomes worse and worse.
It also fits with the Zen of Python (which tbf rarely seems to happen) of, “There should be one-- and preferably only one --obvious way to do it.”
Node / JS in general seems to take the path of, “this existing solution doesn’t precisely fit my needs, so instead of rethinking why that might be, I’m going to reinvent the wheel.”
Given that you where arguing that using python on the most popular OS in the world should fall outside of a 'normal' use case python should focus on, perhaps you have a too narrow view of what "correct and normal" python usage should be.
No they're not complying to your CI/CD best standards - tell me with a straight face they're learnable in less than a month with a budget of $0 above your existing capital. Yes they're adding real value to the economy. That's "computing for everyone", and one of those tools works for that scenario.
I think the real takeaway from my experience of Rust vs Python performance is that in practice people don't optimise 99% of Python so the real number of significance is the 8x speedup he got from just switching to Rust with no profiling whatsoever.
That's actually quite a low difference compared to my experience - I'd say 50x is more typical.
As it stands, any AOT/JIT compiled language has an edge over Python, considering using only the reference implementation, which is quite relevant on systems were one cannot expect to allowed to install additional development tools.
As for the bindings to native libraries, yeah Python might be favoured for bindings, however those native libraries can be called by any language that has FFI capabilities anyway, alongside their JIT/AOT capabilities.