Yeah, rattler are our low-level crates to handle conda packages (similar to how rip is a low-level crate to handle PyPI packages).
Both build on top of resolvo (our SAT solver, based on MiniSAT and libsolv) and both are used (or going to be used) in pixi, which is our package manager.
If I understand it correctly, rip::pip as to what rattler::conda? Finally pixi brings them altogether as the pkg manager to rule them all. Lofty goal but I remain optimistic that something better (esp in python land) is just around the corner.
Blog post author here – happy to take any questions. We're looking to unify the conda + PyPI ecosystems and build a really nice cross-platform, language independent package manager (which is called pixi).
Any chance you guys, Charlie Marsh/Astral, and Armin Ronacher will ever team up? I’m really excited about next-gen tooling but would love a cargo-like “one true path”. At the very least keep the tools compatible?
Yeah, I think it would be awesome to work on top of the same foundations (rip & resolvo being two of those, and konstin's, who works with Charlie at astral, pep440_rs and pep508_rs being two others).
We'll have some more conversations around this in the upcoming weeks I hope.
Perhaps packages that are borderline irrelevant should not squat on prime name real estate. Plus cant rip ship as rip-py and still install as rip on the bet that nobody is actually going to be trying to use these tools simultaneously?
No plans but we would be happy to explore this further if the poetry-devs want to. Maybe they'll join us at PackagingCon next week: https://packaging-con.org/
It doesn't work well for those yet. rip only deals with wheel files for now. The way (afaik) pip works for source dist packages is that it locally builds the wheel, and then extracts the metadata from that locally built wheel.
This is also the approach that we want to take with rip. As you can imagine, the performance will be quite bad (but nothing we can do about that). Ideally, almost all packages should ship wheels.
> Ideally, almost all packages should ship wheels.
It would be ideal but it's not always possible. I maintain a GSSAPI/KRB5 library that wraps the C libs but due to PyPI wheel policies I cannot upload a wheel without embedding those C libs which then opens up a whole bunch of problems around deps and lib conflicts :(
Ideally - however there are points where it's not possible yet.
Bindings like pycairo are a good example.
On windows a WHL can be provided.
On Linux it can't - Cairo (which pycairo binds to) is shipped by the distro, and distros are free to enable or disable different features.
Since the C bindings are linked to the cairo shared library, and that includes all the backends you can't build that and know if would work in a distro.
There are two use cases for pycairo - in one you might not care what the system provides and if the pycairo who provided its own Cairo shared object, that would be fine.
In the second use case you want to use the system provides Cairo, e.g. to work with Gtk.
It's impossible to resolve this workout changing how Cairo itself works and it's API.
The result is no pycairo whl on Linux, and users who have to install all the dependencies (which once go to pango you pull in Gtk, X, freetype etc).
Cairo isn't the only example but it's one I know.
A lot of these are libraries that bind system libraries and predate virtualenv / venv - when they were created compiling things was fine.
There's thousands more packages than listed on that website.
Also now you are moving the goalposts. Python wheels exist but not all Python packages are distributed as wheels.
If I am going through the effort to build wheels myself, then I don't need a resolver since I can just make sure to pick all the versions I need. Then I don't need rip to install anything.
Are there any gains to be had in just using this for pypi packages (no conda)? Solving for a monorepo with hundreds of packages is the cause of much time lost.
Our solver implements "state-of-the-art" CDCL (conflict driven clause learning). So there is a chance that the performance is improved over "resolvelib" which is used by pip.
Once we are feature complete enough we will work on some more benchmarking.
> The above invocation will automatically download the latest version of pixi, extract it, and move the pixi binary to ~/.pixi/bin. If this directory does not already exist, the script will create it.
Please don't pollute my home directory. Follow the XDG base directory specification [1].
37 comments
[ 4.9 ms ] story [ 102 ms ] thread> Rattler: Rust crates for fast handling of conda packages
https://github.com/prefix-dev/pixi :
> https://pixi.sh/
We'll have some more conversations around this in the upcoming weeks I hope.
Perhaps packages that are borderline irrelevant should not squat on prime name real estate. Plus cant rip ship as rip-py and still install as rip on the bet that nobody is actually going to be trying to use these tools simultaneously?
Who cares if every build script in the world breaks, progress waits for no one.
But we have a strong and true commitment to open source. A non-open-source package manager would definitely not work for our community!
Conda-forge also produces packages for much more (nodejs, Qt, HDF5, C/C++ compilers, LLVM ...).
So we have access to a pretty big repository and a large community already :)
This is also the approach that we want to take with rip. As you can imagine, the performance will be quite bad (but nothing we can do about that). Ideally, almost all packages should ship wheels.
And then a new python version is released and virtually no packages with native modules have wheels for that new python version. (too few use abi3)
Also, good luck finding wheels for e.g. s390x linux.
It would be ideal but it's not always possible. I maintain a GSSAPI/KRB5 library that wraps the C libs but due to PyPI wheel policies I cannot upload a wheel without embedding those C libs which then opens up a whole bunch of problems around deps and lib conflicts :(
Bindings like pycairo are a good example.
On windows a WHL can be provided.
On Linux it can't - Cairo (which pycairo binds to) is shipped by the distro, and distros are free to enable or disable different features.
Since the C bindings are linked to the cairo shared library, and that includes all the backends you can't build that and know if would work in a distro.
There are two use cases for pycairo - in one you might not care what the system provides and if the pycairo who provided its own Cairo shared object, that would be fine.
In the second use case you want to use the system provides Cairo, e.g. to work with Gtk.
It's impossible to resolve this workout changing how Cairo itself works and it's API.
The result is no pycairo whl on Linux, and users who have to install all the dependencies (which once go to pango you pull in Gtk, X, freetype etc).
Cairo isn't the only example but it's one I know.
A lot of these are libraries that bind system libraries and predate virtualenv / venv - when they were created compiling things was fine.
> Example setup: To build manylinux, musllinux, macOS, and Windows wheels on GitHub Actions, you could use this .github/workflows/wheels.yml
But do you or others build them with SLSA 3? https://slsa.dev/get-started#slsa-3
Also now you are moving the goalposts. Python wheels exist but not all Python packages are distributed as wheels.
If I am going through the effort to build wheels myself, then I don't need a resolver since I can just make sure to pick all the versions I need. Then I don't need rip to install anything.
Once we are feature complete enough we will work on some more benchmarking.
In fact, for a long time conda & mamba have created "hard-links" to a central package cache for the mentioned space savings.
Please don't pollute my home directory. Follow the XDG base directory specification [1].
[1]: https://specifications.freedesktop.org/basedir-spec/basedir-...
https://github.com/dirs-dev/directories-rs