Ask HN: Is anyone using PyPy for real work?

573 points by mattip ↗ HN
I have been the release manager for PyPy, an alternative Python interpreter with a JIT [0] since 2015, and have done a lot of work to make it available via conda-forge [1] or by direct download [2]. This includes not only packaging PyPy, but improving on an entire C-API emulation layer so that today we can run (albeit more slowly) almost the entire scientific python data stack. We get very limited feedback about real people using PyPy in production or research, which is frustrating. Just keeping up with the yearly CPython release cycle is significant work. Efforts to improve the underlying technology needs to be guided by user experience, but we hear too little to direct our very limited energy. If you are using PyPy, please let us know, either here or via any of the methods listed in [3].

[0] https://www.pypy.org/contact.html [1] https://www.pypy.org/posts/2022/11/pypy-and-conda-forge.html [2] https://www.pypy.org/download.html [3] https://www.pypy.org/contact.html

184 comments

[ 3.4 ms ] story [ 64.1 ms ] thread
You should probably put "Ask HN:" in your title.

Personally I don't use PyPy for anything, though I have followed it with interest. Most of the things I need to go faster are numerical, so Numba and Cython seem more appropriate.

Cut him some slack, he's only been registered for 10 years
I don’t think it’s about being strict or condescending. In some HN readers the post will show up in a different catalogue and generally be easier for people to find, thus giving the post more visibility :)

Edit; typo

I read this as humor and I imagine mattip may have done also.
I use PyPy quite often as a 'free' way to make some non-numpy CPU-bound Python script faster. This is also the context for when I bring up PyPy to others.

The biggest blocker for me for 'defaulting' to PyPy is a) issues when dealing with CPython extensions and how quite often it ends up being a significant effort to 'port' more complex applications to PyPy b) the muscle memory for typing 'python3' instead of 'pypy3'.

For the b) part, you should consider creating alias for that command, if it really might lead for you to not use it otherwise.
I had the same thought. For years I have aliased ‘p’ for ‘python’ and after reading this thread I will alias ‘pp’ for ‘pypy’.
I use CPython most of the time but PyPy was a real lifesaver when I was doing a project that bridged EMOF and RDF, particularly I was working with moderately sized RDF models (say 10 million triples) with rdflib.

With CPython, I was frustrated with how slow it was, and complained about it to the people I was working with, PyPy was a simple upgrade that sped up my code to the point where it was comfortable to work with.

Is your group still using it?
That particular code has been retired because after a quite a bit of trying things that weren’t quite right we understood the problem and found a better way to do it. I’m doing the next round of related work (logically modeling XSLT schemas and associated messages in OWL) in Java because there is already a library that almost does was I want.

I am still using this library that I wrote

https://paulhoule.github.io/gastrodon/

to visualize RDF data so even if I make my RDF model in Java I am likely to load it up in Python to explore it. I don’t know if they are using PyPy but there is at least one big bank that has people using Gastrodon for the same purpose.

What do you use RDF models for?
So I wrote this library

https://paulhoule.github.io/gastrodon/

which makes it very easy to visualize RDF data with Jupyter by turning SPARQL results into data frames.

Here are two essays I wrote using it

https://ontology2.com/essays/LookingForMetadataInAllTheWrong...

https://ontology2.com/essays/PropertiesColorsAndThumbnails.h...

People often think RDF never caught on but actually there are many standards that are RDF-based such as RSS, XMP, ActivityPub and such that you can work on quite directly with RDF tools.

Beyond that I’ve been on a standards committee for ISO 20022 where we’ve figured out, after quite a few years of looking at the problem, how to use RDF and OWL as a master standard for representing messages and schemas in financial messaging. In the project that needed PyPy we were converting a standard represented in EMOF into RDF. Towards the end of last year I figured out the right way to logically model the parts of those messages and the associated schema with OWL. That is on its way of becoming one of those ISO standard documents that unfortunately costs 133 swiss franc. I also figured out that it is possible to do the same for many messages defined with XSLT and I’m expecting to get some work applying this to a major financial standard and I think there will be some source code and a public report on that.

Notably the techniques I use address quite a few problems with the way most people use RDF, most notably many RDF users don’t use the tools available to represented ordered collections, a notable example with this makes trouble is in Dublin Core for document (say book) metadata where you can’t represent the order of the authors of a paper which is something the authors usually care about a great deal. XMP adapts the Dublin Core standard enough to solve this problem, but with the techniques I use you can use RDF to do anything any document database can, though some SPARQL extensions would make it easier.

That is a great idea! I use rdflib frequently and never thought to try it with PyPy. Now I will.
I don't use PyPy because when I'm stuck with performance issues, I go to numpy and if it really doesn't work I go to cython/numba (because it means that 99% of my python code continue to work the same, only the 1% that gets optimized is different; if I'd go PyPy, I'd have to check my whole code again). I do mostly computational fluid dynamics.

(nevertheless, PyPy is impressive :-) )

I’ve been aware of it for a long time.

I don’t use it.

Why would I use it, what’s the compelling benefit?

Errm, nothing too serious. It's way faster for CPU bound code, and allows micro-threads.

This two weird tricks tend to create wonders, tho.

From the project's homepage:

> A fast, compliant alternative implementation of Python

Performance without compromising too much on compatibility seems to be the main benefit. There is a talk on the YouTube channel «Pycon Sweden» from 5 years ago where the host showed some impressive speed gains for his workload (parsing black box dumps from planes).

It's a python runtime that contains a JIT, as a result it can be phenomenally faster. Like with any JITted runtime, it depends a bit on what your code is doing, and how long you're running it for as there is a little (but honestly very little) bit of up front overhead.
This post is a funny coincidence as I tried today to speed-up a CI pipeline running ~10k tests with pytest by switching to pypy.

I am still working on it but the main issue is psycopg support for now, as I had to install psycopg2cffi in my test environment, but it will probably prevent me from using pypy for running our test suite, because psycopg2cffi does not have the same features and versions as psycopg2. This means either we switch our prod to pypy, which won't be possible because I am very new in this team and that would be seen as a big, risky change by the others, or we keep in mind the tests do not run using the exact same runtime as production servers (which might cause bugs to go unnoticed and reach production, or failing tests that would otherwise work on a live environment).

I think if I ever started a python project right now, I'd probably try and use pypy from the start, since (at least for web development) there does not seem to be any downsides to using it.

Anyways, thank you very much for your hard work !

Second this - no psycopg2 support and to a lesser extent lxml is a nonstarter and makes it pretty difficult to experiment with on production code bases. I could see a lot of adoption from Django deployments otherwise.
Yeah we don't use pypy for those exact reasons on our small django projects.
I work on pg8000 https://pypi.org/project/pg8000/ which is a pure-Python PostgreSQL driver that works well with pypy. Not sure if it would meet all your requirements, but just thought I'd mention it.
One compromise could be to run pypy on draft PRs and CPython on approved PRs and master?
I used PyPy with SymPy when I was helping out a mathematician-friend. SymPy is not exactly fast, a free performance boost was very welcome.
Interesting. I was under the impression PyPy did not do so well with SymPy because the dynamic code paths are difficult to JIT. What kind tasks waw a speed up?
It's been a while and the code is long lost. We only touched the surface of SymPy. Functions, Substitutions, some `ingegrate` and `simplify` is what I remember. The maths was already done. My job was to verify some equations.
I'm using pypy to analyse 350m DNS events a day, through python cached dicts to avoid dns lookup stalls. I am getting 95% dict cache hit rate, and use threads with queue locks.

Moving to pypy definitely speeded me up a bit. Not as much as I'd hoped, it's probably all about string index into dict and dict management. I may recode into a radix tree. Hard to work out in advance how different it would be: People optimised core datastructs pretty well.

Uplift from normal python was trivial. Most dev time spent fixing pip3 for pypy in debian not knowing what apts to load, with a lot of "stop using pip" messaging.

Debian is its own worst enemy with things like this. It’s why we eventually moved off it at a previous job, because deploying Python server applications on it was dreadful.

I’m sure it’s better if you’re deploying an appliance that you hand off and never touch again, but for evolving modern Python servers it’s not well suited.

(comment deleted)
Yes 1000x What is it with them which makes them feel entitled to have special "dist-packages" vs "site-packages" as is the default? This drives me nuts, when I have a bunch of native packages I want to bundle in our in-house python deployment. CentOS and Ubuntu are vanilla, and only Debian (mind-boggingly) deviates from the well-trodden path.

I still haven't figured out how to beat this dragon. All suggestions welcome!

I usually make a venv in ~/.venv and then activate it at the top of any python project. Makes it much easier to deal with dependencies when they're all in one place.
second this and it's what I do on all Linux distros, just run it inside .venv as the site-installation.

if you need extra dependencies that pip can not do well in the .venv case, Conda can help with its own and similar site-based installation.

I don't know how it is different in the python installation case between ubuntu and debian, they seem the same to me.

i am a big fan of .venv/ -- except when it takes ~45 mins to compile the native extension code in question -- then I want it all pre-packaged.
At this stage [0], uncompiled native extensions are not yet a bug, but a definite oversight of the maintainer. They should come as precompiled wheels

[0]: https://pythonwheels.com

Honestly I don't think I've ever used a precompiled package in Python. Every single C stuff seems to take ages and requires all that fun stuff of installing native system dependencies.

Edit: skimming through this page, precompiling seems like an afterthought, and the linked packages don't even seem to mention how to integrate third-party libraries. So I guess I can see why it doesn't deliver on its promises.

Probably a function of the specific set of packages you use, or the pip options you specify. Pretty much all the major C packages come as wheels these days.
They all come as wheels, they just aren't precompiled.
I honestly can't remember the last time I had to compile anything, and I am on Windows.
Can you link one that comes as a wheel but is really a source distribution?
You can try pip install pillow for a good example of how it works. I suspect there's a strong survivorship bias here, as you'd only notice the packages that don't ship with wheels.
Yeah, perhaps. One I remember from last year is the cryptography and numpy package, for instance. Now they do seem to ship with binary wheels, at least for my current Python and Linux version.

Kerberos and Hadoop stuff obviously still doesn't, though. I guess the joke's on me for being stuck in this stack...

In order for a wheel to be used instead of a source distribution there needs to be one that matches your environment. For numpy you can take a look at the wheels for their latest release[1]. The filename of a wheel specifies where it can be used. Let's take an example:

numpy-1.25.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

This specifies cpython 3.9, linux, glibc 2.17 or higher, and x86_64 cpu. Looking through the list you will see that the oldest cpython supported is 3.9. So if you are running with an older version of python you will have to build from source.

I just learned a bit more about this recently because I could not figure out why PyQt6 would not install on my computer. It turned out my glibc was too old. Finally upgraded from Ubuntu 18.04.

[1] https://pypi.org/project/numpy/1.25.2/#files

Try `--only-binary :all:` to force pip to ignore sdist packages, might help avoid those slow compilations.
It's a good idea to be caching sdists and wheels — for resilience against PyPI downtime, for left-pad scenarios, and even just good netiquette — and for packages that don't have a wheel for your environment, you can fairly easily build that wheel yourself and stick it into the cache.
dist packages are a must for software written in Python that is part of the distribution itself.
You're not really answering why they are important?

Is it because .deb packages will install inside dist-packages and when you run pip install as root without a virtual env, it installs inside site-packages?

I don't really see how this helps though? Sure you won't get paths to clash between the two but you still have duplicate packages which is probably not what you want..

Debian ships packages with a coherent dependency structure that crosses language boundaries. You don't need to care what language something is written in to be able to "apt install" it. The expectation is that if it "apt installed" then it should Just Work because all the required dependencies were also pulled in from Debian at the same time.

Debian also tries to ship just one version of everything in a single distribution release to reduce the burden on its maintainers.

This is fundamentally at odds with pip. If you've pip installed something, then that'll likely be the latest version of that package, and in the general case won't be the version of the same thing that shipped in the Debian release. If there exist debs that depend on that package and they are shared between pip and debs, now the deb could be using a different version of the dependency than the deb metadata says is acceptable, leading to breakage.

Another way of putting this: it shouldn't be possible for you to pip upgrade a dependency that a deb shipped by Debian itself relies upon. Because then you'd creating a Frankenstein system where Debian cannot rely on its own dependencies providing what it expects.

This is fixed by having two places where things are installed. One for what the system package manager ships, and one for your own use with pip and whatever you want to do. In this sense, having duplicate packages is actually exactly what you want.

Yep, I screw things up all the time with packages in homebrew that are written in python, when I forget to switch into a virtual env before doing stuff with pip. Debian's solution seems very sensible. And it is the same solution as homebrew, I suppose, as long as you don't interact with any of the homebrew-installed packages via pip. But I find it quite easy to accidentally do that.

  export PIP_REQUIRE_VIRTUALENV=1
has been quite helpful in the past as pip then refuses to just install things directly.
There is https://peps.python.org/pep-0668/ which suggests that in the future this kind of behaviour will be default. I'm not sure of the specifics but I have seen lots of conversation about it in Debian circles.
OK, but... I get the same problem when I compile python from source. I'm not talking about the distribution's base files. In fact, I am in the business of creating an /opt/gjvc-corp-tools/ prefix with all my packages under there. When I compile python from source on Debian, the resulting installation (from make install) does not have a site-packages directory in-place already. That is what is mind-boggling.
> This is fundamentally at odds with pip

It's at odds with everything. I leave the system versions of any language alone and use language manager tools or docker to be able to run the exact version that any project of my customers require. Asdf is my favorite because it handles nearly everything, even PostgreSQL.

Imagine you installed python3-requests (version x.y.z). Some of your distribution's packages depend on that specific package/version.

If you pip install requests globally, you just broke a few of your distrib's packages.

Dist packages is the right way to handle Python libs. You'd prefer to have the distro package manager clashing with Pip? Never knowing who installed what. Breaking things when updates are made.
> What is it with them which makes them feel entitled to have special "dist-packages" vs "site-packages" as is the default? This drives me nuts, when I have a bunch of native packages I want to bundle in our in-house python deployment. CentOS and Ubuntu are vanilla, and only Debian (mind-boggingly) deviates from the well-trodden path.

Hi, I'm one of the people that look after this bit of Debian (and it's exactly the same in Ubuntu, FWIW).

It's like that to solve a problem (of course, everything has a reason). The idea is that Debian provides a Python that's deeply integrated into Debian packages. But if you want to build your own Python from source, you can. What you build will use site-packages, so it won't have any overlap with Debian's Python.

Unfortunately, while this approach was designed to be something all package-managed distributions could do, nobody else has adopted it, and consequently the code to make it work has never been pushed upstream. So, it's left as a Debian/Ubuntu oddity that confuses people. Sorry about that.

My recommendations are: 1. If you want more control over your Python than you get from Debian's package-managed python, build your own from source (or use a docker image that does that). 2. Deploy your apps with virtualenvs or system-level containers per app.

IMO bespoke containers using whatever python package manager makes sense for each project. Or make the leap to Nix(OS) and then still have to force every python project into compliance which can be very easy if the PyPy packages you need are already in the main Nix repo (nixpkgs) or very difficult if depends on a lot of uncommon packages, uses poetry, etc.

Since PEP 665 was rejected the Python ecosystem continues to lack a reasonable package manager and the lack of hashed based lock files prevents building on top of the current python project/package managers.

> I still haven't figured out how to beat this dragon. All suggestions welcome!

Docker

(comment deleted)
It works completely fine in my experience.
Lucky you. Having gone through multiple Debian upgrades, a Python 2->3 migration on Debian, and Debian python packaging to Pip/PyPI, it was a whole world of pain that cost us months of development time over years, as well as a substantial amount of downtime.
What distro did you move to? IME debian as a base image for python app containers is also kind of a pain.
We moved to stripped down Debian images in containers and made sure to not use any of the Debian packaging ecosystem.
> it's probably all about string index into dict and dict management

Cool. Is the performance here something you would like to pursue? If so could you open an issue [0] with some kind of reproducer?

[0] https://foss.heptapod.net/pypy/pypy/-/issues

I'm thinking about how to demonstrate the problem. I have a large pickle but pickle load/dump times across gc.disable()/gc.enable() really doesn't say much.

I need to find out how to instrument the seek/add cost of threads against the shared dict under a lock.

My gut feel is that probably if I inlined things instead of calling out to functions I'd shave a bit more too. So saying "slower than expected" may be unfair because there's limits to how much you can speed this kind of thing up. Thats why I wondered if alternate datastructures were a better fit.

its variable length string indexes into lists/dicts of integer counts. The advantage of a radix trie would be finding the record in semi constant time to the length in bits of the strings, and they do form prefix sets.

If you have very large dicts, you might find this hash table I wrote for spaCy helpful: https://github.com/explosion/preshed . You need to key the data with 64-bit keys. We use this wrapper around murmurhash for it: https://github.com/explosion/murmurhash

There's no docs so obviously this might not be for you. But the software does work, and is efficient. It's been executed many many millions of times now.

I'm in strings, not 64 bit keys. But thanks, nice to share ideas.
The idea is to hash the string into a 64-bit key. You can store the string in a value, or you can have a separate vector and make the value a struct that has the key and the value.

The chance of colliding on the 64-bit space is low if the hash distributes evenly, so you just yolo it.

One should really consider using containers in this situation.
Can you describe what in this situation warrants it?

I'm very curious about where the line is/should be.

In my experience leaving the system python interpreter the way it was shipped will save you enormous headaches down the road. Anytime I find myself needing additional python packages installed I will almost always at minimum create a virtual env, or ideally a container.
Uplift from normal python was trivial.

By definition if you lift something it is going to go up, but what does this mean?

If you replace your python engine you have to replace your imports.

Some engines can't build and deploy all imports.

Some engines demand syntactic sugar to do their work. Pypy doesn't

I've never ended up using PyPy other than to play with it. Numba has worked very well for me for real code.
I liked Psyco a lot, it was totally awesome and with very few bugs (CPython differences) but that was looong ago. PyPy looks and feels like a monstrosity, it builds longer than most software for once, which is off-putting. I would be more interested in a Python JIT which is more like LuaJIT to Lua.
I can't remember exactly what the use case was but we used at my old work (Start up providing a Web CDN/WAF type service, think the kind of stuff CloudFlare does nowadays) in ~2013 for some sort of batch processing analytics/billing type job, using MRJob and AWS Elastic Map Reduce over a seriously large data set.

The performance of PyPy over CPython saved us loads and loads time and thus $$$s, from what I can recall.

Thanks, that is hopeful, although quite a while ago.
I wonder if programs like Rye, that distribute python in a way similar to Rust's rustup, can help. Rye already supports pypy, you can just pull down pypy3.9 at will into any particular python project managed by rye.
I use it at work for a script that parses and analyzes some log files in an unusual format. Wrote a naive parser with a parsing combinator library. It was too slow to be usable with CPython. Tried PyPy and got a 50x speed increase (yes, 50 times faster). Very happy with the results, actually =)
Thanks for the feedback. It does seem like parsing logs and simulations is a sweet spot for PyPy
what cpython version and OS was that? I'd be very surprised if modern Python 3.11 has anything an order of magnitude slower like that. things have gotten much faster over the years in cpython
We don't. To be honest, I didn't realize PyPy supported Python 3. I thought it was eternally stuck on Python 2.7.

So the good: It apparently now supports Python 3.9? Might want to update your front page, it only mentions Python 3.7.

The bad: It only supports Python 3.9, we use newer features throughout our code, so it'd be painful to even try it out.

I think it supports up to 3.10, as there are official docker images for this version, I saw them this morning.

Maybe the site is not up to date ?

It supports Python3.10 now too. Thanks, I updated the site.
Their docs seem perpetually out of date, but they recently released support for 3.10. I haven't been able to try it recently because our projects use 3.10 features but in the past it was easily a 10-100x speedup as long as all the project's libraries worked.

https://downloads.python.org/pypy/

I've never used it because the (unknown) effort of switching and the chance of compatibility issues have always made it unappealing compared to just switching to a faster language.

If I could just `pip3 install pypy` and then set an environment variable to use it or something like that then I'd give it a try. It does feel a bit like adding a jet pack to a rowing boat though. I know some people use Python in situations where the performance requirement isn't "I literally don't care" but surely not very many?

Obviously if it was the default that would be fantastic.

If you use a version manager like rtx or asdf then it’s basically that simple. I just had to run a single command:

    rtx use python@pypy3.10
This downloaded and installed PyPy v3.10 in a few seconds and created an .rtx.toml file in the current directory that ensures when I run python in that directory I get that version of PyPy.
I use it for data transformation, cleanup and enrichment. (TXT, CSV, Json, XML, database) to (TXT, CSV, JSON, XML, database).

Speed up of 30x - 40x. The highest speedup on those that require logic in the transformation. (lot of function calls, numerical operations and dictionary lookups).

Similar. I was working on some ETL work with SQLite, and now PyPy is my regular tool for getting better performance at similar jobs.
Same. I have used it for many ETL jobs, usually with about a 10x speed up. It also pulled in the latency on some Flask rest apis.
I put PyPy in production at a previous job, running a pretty high traffic Flask web app. It was quick and pretty straightforward to integrate, and sped up our request timings significantly. Wound up saving us money because server load went down to process the same volume of requests, so we were able to spin down some instances.

Haven’t used it in a bit mostly because I’ve been working on projects that haven’t had the same bottleneck, or that rely on incompatible extensions.

Thank you for your work on the project!

You're welcome.

> that rely on incompatible extensions.

Which ones? Is using conda an option, we have more luck getting binary packages into their build pipelines than getting projects to build wheels for PyPI

I can't actually remember off of the top of my head, I tried it out a year or two ago but didn't get too far because during profiling it became clear the biggest opportunities for performance improvement in this app were primarily algorithmic/query/io optimizations outside of Python itself, so business-wise it didn't make too much sense, though if it had I think using Conda would have been on the table. We make heavy use of Pandas/Numpy et al, though I know those are largely supported now so I'd guess it was not one of them but something adjacent.
While the community is here, anyone has embedded pypy as scriptable language for some larger program? Like Inkscape or scripting as part of a rule engine. Or for that, CPython is more suitable?
If it was relatively up to date with Python3 I'd use it, but as it lags behind considerably I avoid it, even for personal work.
Python 3.10 is too old for your work?
In fact no, Python 3.10 is OK new enough.

There is still the lag though, Python 3.10 was out for quite a while before PyPy supported 3.10.

Yes. We have a legacy Python-based geospatial data processing pipeline. Switching from CPython to PyPy sped it up by a factor of 30x or so, which was extremely helpful.

Thank you for your amazing work!

Would love to hear more. Is it still being used?
I didn't hear about PyPy before, but I think you're doing great work.

I would be interested in seeing benchmarks where PyPy is compared with more recent versions of CPython. https://www.pypy.org/ currently shows a comparison with CPython 3.7, but recent releases of CPython (3.11+) put a lot of effort into performance which is important to take into account.

I don’t use it, but I’d like to.

The big obstacle is that for while we would have multiple execution environments. It’s not like we could flip a switch and all Dockerfiles are using PyPy.

Plus I don’t think AWS Lambda supports it.

If I could go back in time, we would use it from the beginning.

My biggest issue is that DataDog doesn’t support PyPy. Out of curiosity, I made a new branch of our app and took out DataDog and observed a significant improvement in performance when using PyPy vs CPython on the same branch (but can’t remember how much).
Do you mean the Python tracing library does not work out-of-the-box?

disclaimer: I work there but not on the APM team

Correct. I think DD only supports CPython.
I don’t actually use PyPY, but I’m very aware of it. My understanding is that the only reason to use PyPy instead of the default Python is for performance gains. For the vast majority of projects I work on, the performance of our code on the CPU is almost never the bottleneck. The slowness is always in IO, databases, networks, etc.

That said, if I do ever run into a situation where I need my code to perform better, PyPy is high on my list of things to try. It’s nice to know it’s an option.