Coupling with Rye is very interesting, as that or the PyBi idea seem inevitable to take over. Managing different interpreters is just too much effort which each tool is handling differently.
Of course, the PSF will remain silent on the issue and let yet another tool exist in the ecosystem. I really do not care who wins, but we need a standard to be decided.
I hadn't heard of Rye before I don't think, but sounds like a strange arrangement to me - basically the same project & idea, so the new one to start switched the old one to use it 'under the hood', 'took it over', and will replace it 'when the time is right'? Why.. any of that, why not just buy it and let it be uv instead of starting uv, or just leave it alone?
I don’t want the PSF to declare tools cannot be made. I want the PSF to pick a winner so we can finally consolidate on a workflow.
Just today, I was helping a beginner setup Python on their laptop. A nightmare collection of tooling complexity (so you have Python, but you need to worry about virtual environments, to do that, I use this combination of pipx+virtualenvwrapper, but to do actual projects, you need Poetry, but…) all the while noting that this is an opinionated workflow that I happen to utilize because of the tradeoffs that matter to me. Ask someone else, and they will definitely have a different configuration. Entire first lesson was just to get Python installed with tons of elided complexity.
I should be able to point to a document on Python.org that says, “This is the way.” Yet for decades, they have refused to take a stand.
I've been dying to see what they came up with next given how in love I am with Ruff. In my wildest dreams I didn't expect it to be a package resolver. I'm very excited about this.
edit: As a side note, if anyone from Astral happens by, any chance of an RSS feed on your blog?
I was kind of hoping it was a better type checker, something like mypy + pyright. To be fair that's an incredible amount of work, so maybe we'll see that later. Still very excited.
That was actually my first guess, and while I certainly would love to see that, packaging is such a headache for so many people that this could be amazing. Even with poetry I still occasionally get mysterious errors and wildly wrong resolver claims about versions not existing.
Pixi is even more ambitious but with a different philosophy (I think? Specifically thinking about how pixi went with their own lock and manifest format as well as first class conda support). I'd definitely prefer if they worked together instead or instead dedicated their time to type checking python which imo there still isn't a great solution for.
Yeah, pixi has decent mixed support for conda/pypi, it currently solves and installs conda (then locks it) and then solves and installs pypi. I think it's on their roadmap to solve them together, which would be a killer feature.
For type checking it is very good, however, the error messages are sometimes not very human-readable. For example:
def _deep_merge(updates: dict[str, str]) -> None:
for key, update_info in updates.items():
if isinstance(update_info, dict) and "values" in update_info:
value = update_info["values"]
Errors out with Pyright:
- error: Argument of type "Literal['values']" cannot be assigned to parameter "__key" of type "SupportsIndex | slice" in function "__getitem__"
Type "Literal['values']" cannot be assigned to type "SupportsIndex | slice"
"Literal['values']" is incompatible with protocol "SupportsIndex"
"__index__" is not present
"Literal['values']" is incompatible with "slice" (reportGeneralTypeIssues)
1 error, 0 warnings, 0 informations
It took me a great amount of starring to figure out that changing the signature of updates to dict[str, dict] was what it was complaining about.
> Similarly, uv does not yet generate a platform-agnostic lockfile. This matches pip-tools, but differs from Poetry and PDM, making uv a better fit for projects built around the pip and pip-tools workflows.
Do you expect to make the higher level workflow independent of requirements.txt / support a platform-agnostic lockfile? Being attached to Rye makes me think "no".
Without being platform agnostic, to me this is dead-on-arrival and unable to meet the "Cargo for Python" aim.
> uv supports alternate resolution strategies. By default, uv follows the standard Python dependency resolution strategy of preferring the latest compatible version of each package. But by passing --resolution=lowest, library authors can test their packages against the lowest-compatible version of their dependencies. (This is similar to Go's Minimal version selection.)
> uv allows for resolutions against arbitrary target Python versions. While pip and pip-tools always resolve against the currently-installed Python version (generating, e.g., a Python 3.12-compatible resolution when running under Python 3.12), uv accepts a --python-version parameter, enabling you to generate, e.g., Python 3.7-compatible resolutions even when running under newer versions.
This is great to see though!
I can understand it being a flag on these lower level, directly invoked dependency resolution operations.
While you aren't onto the higher level operations yet, I think it'd be useful to see if there is any cross-ecosystem learning we can do for my MSRV RFC: https://github.com/rust-lang/rfcs/pull/3537
How are you handling pre-releases in you resolution? Unsure how much of that is specified in PEPs. Its something that Cargo is weak in today but we're slowly improving.
Thanks Ed! Your work as always is a big source of inspiration.
> Do you expect to make the higher level workflow independent of requirements.txt / support a platform-agnostic lockfile? Being attached to Rye makes me think "no".
Yes, we absolutely do. We don't do this today, the initial scope is intentionally limited. But in the next phase of the project, we want to extend to multi-platform and multi-version resolution.
> How are you handling pre-releases in you resolution? Unsure how much of that is specified in PEPs. Its something that Cargo is weak in today but we're slowly improving.
This is something we talked with Jacob about quite a bit. Turns out (as you know) it's a very hard problem. For the initial release, we added a constraint: our default behavior is that if you want to use a pre-release, you _have_ to specify the package as a first-party dependency and use a pre-release marker in the version specifier. (We also support globally enabling and disabling pre-releases.) So, we basically don't support "transitive" pre-releases right now -- but we give you a dedicated error message if your resolution fails for that reason.
Any plans to tackle the Python version installation side of things and make it as seamless as rustup has? I've previously used `pyenv install` for this, but it would be nice to fold it into one tool.
Note that rye already handles it better then pyenv (it downloads pre built pythons instead of building from source). I assume they'll eventually they'll copy/move the functionality over.
> How are you handling pre-releases in you resolution? Unsure how much of that is specified in PEPs.
The living version of PEP 440 has a bit on how pre-releases are handled[1]. The basic version is that the installer shouldn't select them at all, unless the user explicitly indicates that they want a pre-release. Once opted into, they're ordered by their phase (alpha, beta, rc) and pre-release increment (e.g. `.beta.1 > `.alpha.2`).
There are a couple of promising tools written in Rust looking to replace Pip for most users.
Rip (https://github.com/prefix-dev/rip/issues) which is more of a library for other rust tools to be on top of like Pixi (which is looking to replace both Pip and Conda).
And now uv, which seems to be looking to replace Pip, Pip-Tools, and eventually Poetry and PDM.
A lot of the explosion in tools in the Python world is coming from the desire for better workflows. But it has been enabled by the fact that build configuration and calling has been standardized and tool makers are able to follow standards instead of reverse engineering easy install or setup tools.
I know a lot of people are put off by there being so many tools, but I think in a few years the dust will settle and there will emerge a best practice work flow that most users can follow.
As a primarily Python developer and someone who occasionally contributes to Pip to solve complex dependency resolution issues it does make me wonder if I should hang my hat on that and learn enough rust to contribute to one of these projects eventually.
My experience with Rust developers who dwell in Python land is that they fundamentally disagree with most of the language that is Python and think they know better than incumbents what belongs and what doesn't.
The Rust ecosystem gets so much right that honestly, even as a career-long Python developer myself (and Rust for many years, but that's less of my point), they honestly probably do know how to build a good devxp better than much of the Python ecosystem.
Put other ways: the Python ecosystem has had 30+ years to figure out how to make packaging not suck. It has continually failed - failed less and less over time, sure, but the story is still generally speaking a nightmare ("throw it all in an OCI container" is an extremely reasonable solution to Python packaging, still, in 2024). I welcome advances, especially those inspired by tooling from languages that focused heavily on developer experience.
There's a lot of important work that happens in python. Most of it isn't being done by software engineers. I think the idea of improving things for that group is plenty meaningful.
Looks awesome. I find that pip is usually pretty fast for me, and when it's not, it is mostly because it has to download so much data or wait for native libraries to compile in the background (or anything involving cuda which always seems to take forever). What really needs some help with speed is conda, which is just so absurdly slow for literally anything, even on ridiculously powerful machines.
From my testing in rye it’s significantly faster in day to day. There are numbers in the blog post obviously but it’s the feeling you get using it locally that makes it much more fun to use.
I did some basic testing today and uv was around 2x faster than pip for a clean venv and cold cache on a real, decent sized project. With warm uv cache it was incredibly fast, under 10 sec.
What are some of the reasons that teams use conda (and related tools) today? As a machine learning scientist, I used conda exclusively in the mid-2010s because it was the only framework that could reliably manage Python libraries like NumPy, PyTorch, and so on, that have complex binary dependencies. Today, though, pip install works fine for those packages. What am I missing?
Another reason I used to use conda was for easy native Windows installation. GPU accelerated packages like OpenCV were especially difficult when I used use it 6 years ago. Now there’s Linux subsystem.. has pip support dramatically improved on Windows?
The biggest advantage for poetry I found, working with a lot of non-traditional software people, is that it does a lot of things by default like pin versions and manage virtual envs. Unfortunately, it does complicate some things.
Those are interesting pointers; appreciate it! My own experience over the past three years has been similar. I tried using Pipenv, and then Poetry, for internal projects at my company; in both cases the tool seemed overly complicated for the problem, slow, and I had a hard time getting co-workers on board. About a year and a half ago, I saw [Boring Python: dependency management](https://www.b-list.org/weblog/2022/may/13/boring-python-depe...), which recommends using the third-party `pip-tools` library alongside the standard library’s `pip` and `venv`, and switched to that for the next project. It’s been working great. The project has involved a small team of scientists (four or five, depending) who use a mix of macOS and Windows. We do analysis and development locally and write production-facing algorithms in Python packages tracked in our repository, and publish releases to Gitlab’s PyPI. For our team, the “get up and running” instructions are “clone, create a venv, and pip install -r requirements.txt” and for the software team that manages the production systems, deploying an update just means pip installing a new version of the package. Every team’s got different constraints, of course, but this has been working very smoothly for us for over a year now, and it’s been easy, no pushback, with everyone understanding what’s going on. Really impressed with the progress of the core Python packaging infrastructure over the past several years.
For me it's the easiest and fastest cross-platform way to consistently install a Python version.
pip and venv work fine, but you have to get them first; and that can be a struggle for unseasoned python devs, especially if you need a version that's not what your distro ships, and even more so on Windows and macOS.
I use micromamba [1] specifically, which is a single binary.
> Today, though, pip install works fine for those packages.
pip install works, but pip's dependency management doesn't seem to (for Pytorch, specifically) which is why projects that have pip + requirements.txt as one of their installation methods will often have separate pytorch installation instructions when using that method, though if the same project supports conda installation it will be a one-stop-shop installation that way.
> pip's dependency management doesn't seem to (for Pytorch, specifically)
That’s interesting — I’ve also had difficulties with PyTorch and dependency resolution, but only on the most recent versions of Python, for some period of time after they’re released. Picking Python 3.9 as a baseline for a project, for example, has been very reliable for PyTorch and all the related tooling.
For me personally, I prefer conda because it is dependency resolution (mamba), virtual environments, and a package repository (conda-forge) all from one base miniconda installation. And for all of my use cases, all of those just work. Dependency solving used to be painfully slow, mamba solved that. Packages used to be way behind the latest, setting conda-forge as my default solved that.
After fiddling with different solutions for years and having to start fresh with a new Python install, I've been using nothing by miniconda for years and it just works
One reason to choose one over the other is the dependencies they’re bundled with. Take numpy. With PyPI, it’s bundled with OpenBLAS, and with conda, it’s bundled with Intel MKL, which can be faster. See https://numpy.org/install/#
I'd be curious to see how "packaging for production" is addressed. As described, this is a tool for development purposes:
> Think: a single binary that bootstraps your Python installation and gives you everything you need to be productive with Python, bundling not only pip, pip-tools, and virtualenv, but also pipx, tox, poetry, pyenv, ruff, and more.
A lot of that dev tooling is not needed in prod, in fact it is a liability (features available for misuse, code/image size, higher resource requirements). Would there be a "uv-prod" to only deal with the minimum subset of environment setup / container building that enables production use? Would uv build packages that are runnable in prod but themselves don't contain uv? It'd be interesting to hear the plans.
My approach for this is to use multi-stage dockerfiles. Build tools live and operate in a build stage, and you copy the /venv into your runtime stage. I think this same approach should work with uv so long as it doesn't bundle itself into the venv.
Do you plan to support platform-agnostic lockfiles? From the wording it sounds like "yes" but also "this is a feature not a bug." Which is it?
Checksum verification and portable lockfiles are kind of table stakes for me, which is why I use poetry everywhere possible. I can't give up correctness for speed.
sidenote: this would be super useful as a library so I could plug it into build systems. Managing python dependencies in standards-compliant ways is a pain, so shelling out to tools like poetry and pip are used even though it would be nicer to write that code eg in Rust and just use the parts you care about.
Yes, we absolutely want to support platform-agnostic lockfiles. Very important thing.
The "feature not a bug" tone is probably because we intentionally limited the scope of this release to _not_ support platform-agnostic resolution, since it adds a lot of complexity (and we were still able to ship something that's immediately useful for those that rely on pip and pip-tools today).
So what's the plan for monetization? NPM getting acquired by GitHub was an anomaly, and I'm wary of letting VC-backed companies becoming ecosystem lynchpins.
I hope they answer this ASAP. People in the Python community are already concerned that Astral is following the Embrace, extend, and extinguish playbook.
If they stop development on Ruff today, Ruff won't be useful anymore after a few new Python releases. If the maintainers of the tools that Ruff is replacing stop maintaining them, the whole Python community will be in a really bad place. So, there is reason to be cautious. Astral being transparent on how they plan to make money would be very helpful.
People could just fork it if that happens. Since the foundation of the tooling is solid, I 'm sure the community would rally around a fork that keeps it going.
It's MIT and Apache dual licensed. If the company fails, you fork and move on. If the company never makes a dime, that's OK for me: some folks got paid to make open source tooling.
What they are doing is already discouraging people from contributing to the projects their tools are replacing[0]. If they go out of business and stop supporting their tools, it might leave the Python community in a bad place.
I'm not sure this is a reasonable framing: LLVM stole much of GCC's thunder by being significantly easier to contribute to (and extend externally), but I don't think it would be accurate to say that LLVM is "discouraging" people from contributing to GCC.
I'm not sure if you can compare a project that came out of a university and got adopted by Apple with a project developed by a VC backed company with no revenue. I'm sure Charlie has the best intentions with both ruff and uv, but we have no idea how this is going to play out.
My understanding of Ruff's history is that it predates Astral, and was originally a side project by Charlie. I have similar reservations about VC funding and sustainability, but I don't think Ruff was created as a trojan horse or with a secret takeover plan in mind.
I agree that we have no idea how it'll play out. But I've seen nothing but good faith from Charlie and others, and I see no reason to preemptively condemn their attempt to build a business.
By this token, we should be concerned that CPython is not written in Python. But that historically has not posed a significant risk to CPython's longevity, nor to the ability of motivated contributors to learn the C necessary to contribute to it.
(Or another framing: only the tiniest fraction of the Python community was contributing to CQA tooling in the first place. It's not clear that prioritizing the interests of the 99.9% of Python programmers who -- rightfully! -- will never modify these tools makes sense.)
That video is.. weird. It claims that astral isn't contributing back despite the entirety of their code being permissively licensed. It's also sort of baffling that making tooling more accessible doesn't seem to be considered contributing back.
I'm not sure what the maker of that video wants, does he want money to be poured back into the community, or for no one else to make money?
While we don't particularly care about pip lockfiles, etc. as we're moving to the conda/mamba ecosystem a faster pip is most welcome. We invoke it a lot programmatically and it's very, very slow.
Very nice! Not long ago we got rid of a bunch of tools at work in favor of ruff. I'm looking forward to this. Specially when the time comes to be able to replace poetry with it, hoping it is a good implementation.
Iirc, some pip packages require compilation which depends on entire toolchains with e.g. gcc, g++, and with dependencies like gtk, Qt, etc. How do they intend to make that less error prone and thus more user-friendly?
That’s the really pointy end of packaging. Pillow (python +c+ External dlls) produces something like 50 wheels per release, and we’ve still got platforms where you’ve got to compile your own. Since they tend to be lower volume, they also tend to have more interesting compiler issues.
Yes, I'm using Python on an nVidia Jetson system with PyTorch, which depends on CUDA, and all my other dependencies are quite challenging to get working in concert. Every time pip3 starts compiling under the hood, I'm just praying that it will work.
Exciting stuff! I view Hatch [1] as becoming the Cargo for Python because it's already close and has an existing (and growing) user base but I can definitely see depending on this for resolution and potentially not even using pip after it becomes more stable.
This is very exciting! Congratulations to the Astral team on their work here.
I have historically expressed trepidation towards external attempts to "fix" Python packaging, largely because each attempt has historically accreted another layer of incompatibilities and hacks onto the giant pile of hacks that are already needed to keep the ecosystem running. As such, it makes me happy to see that compatibility is a priority here: the Astral team has gone out of their way to emphasize both formal (PEP) and informal (pip CLI) compatibility with existing tooling and standards.
(Separately, I share concerns about VC funding and sustainability. But everything I've seen so far indicates that Charlie and the other folks at Astral are not in it to screw or otherwise leverage the Python community. So I choose to be optimistic here.)
I don't think that's true, at least in the case of Python packaging. The pile has historically grown because of informal or non-existent standards, along with multiple independent implementations attempting to align their observed (but not specified) behaviors.
Well, they do explicitly list certain historical features that they don't intend to support, like installation from eggs or editable installs. So they're doing some work to trim the pile while they're there.
> I had to guess, that’s the path that the Astral team would take as well - expand ruff’s capabilities so it can do everything a Python developer needs. So the vision that Armin is describing here might be achieved by ruff eventually. They’d have an advantage that they’re not a single person maintenance team, but the disadvantage of needing to show a return to their investors.
This article says they don't support "platform-agnostic lockfiles" YET. And platform-agnostic lockfiles have been mentioned in several comments here, times too. I haven't encountered these before (with python, anyway) - how do they work platform-specific dependencies, e.g., binary wheels?
For instance, there are >30 dists for numpy 1.26.4 [1], with different artifacts for different platforms and python interpreters. Which of those hashes ends up in a platform-agnostic lockfile? Do you have to stick with sdists only?
PDM and Poetry would include hashes for all possible wheels, and would also follow transitive paths for platforms other than the current host. E.g. if my host is Linux and I depend on foo, and foo depends on bar only on windows, and bar depends on baz, my lock file will still contain baz.
Something I've been waiting to see from language package managers is more attention paid to security. I believe both cargo and pip packages can run arbitrary code as the local user the instant they are installed, and malicious packages have existed in the wild for years. I also recall a blog post where someone was scanning PyPI for malicious Python packages, only to realize that `pip download` also executed code.
Just downloading a library to a local source code directory should not cause arbitrary code to run on your system, and there should be safeguards in place so developers are not one typo away from a malicious package pwning their entire system. Supply chain attacks remain a huge area of concern.
Instead, when I "ctrl+f security" the homepages of any of these new packaging systems, I get 0 results. Not good.
Very cool! Of note, I made something along these lines a few years ago, although with a slightly broader scope to also include managing and installing python versions. I abandoned it due to lack of free time, and edge cases breaking things. The major challenge is that Python packages that aren't wheels can do surprising things due to setup.py running arbitrary code. (https://github.com/David-OConnor/pyflow)
For some more context, at the time, poetry and pipenv were both buggy, and had some limitations their teams listed as "wont-fix". The sort of thing where you hit a breaking bug immediately on certain systems. These have since been fixed.
Something that is super super important to me is editable installs, and local references to those editable installs. Often I have several editable packages set up in a virtualenv which I think of as like my "base" virtualenv.
One reason I struggled with Rye earlier is that it kind of felt like this wasn't entirely supported, possibly because I couldn't parse the information on the workspaces or virtual projects page. Maybe someone else figured this out? It does seem common enough that I would be surprised if it wasn't supported.
I think having a good story for local workspaces is gonna be a key piece of this next phase of the project, where we grow uv from a "pip replacement" to a "Poetry replacement" -- or, phrased differently, to something that looks more like Cargo. (Cargo does a bunch of nice things here that I want to learn from.)
I think Rye actually does handle this mostly correctly (as the sibling comment said). I got through some of it here: https://github.com/mitsuhiko/rye/issues/671. I think it's very close to what I actually want (maybe not what Armin wants with multiversion).
219 comments
[ 2.5 ms ] story [ 286 ms ] threadOf course, the PSF will remain silent on the issue and let yet another tool exist in the ecosystem. I really do not care who wins, but we need a standard to be decided.
Just today, I was helping a beginner setup Python on their laptop. A nightmare collection of tooling complexity (so you have Python, but you need to worry about virtual environments, to do that, I use this combination of pipx+virtualenvwrapper, but to do actual projects, you need Poetry, but…) all the while noting that this is an opinionated workflow that I happen to utilize because of the tradeoffs that matter to me. Ask someone else, and they will definitely have a different configuration. Entire first lesson was just to get Python installed with tons of elided complexity.
I should be able to point to a document on Python.org that says, “This is the way.” Yet for decades, they have refused to take a stand.
Please keep doing what you're doing!
edit: As a side note, if anyone from Astral happens by, any chance of an RSS feed on your blog?
https://github.com/prefix-dev/pixi/
As conda/pip interop is just not great. And even micromamba (C++ implementation of conda) is relatively slow to resolve compared to pip.
But agreed either work together or create a type checker. I use mypy currently but it definitely slows down my editor.
It took me a great amount of starring to figure out that changing the signature of updates to dict[str, dict] was what it was complaining about.
Hence they will always trigger user errors, and their image will be stained by it.
> Similarly, uv does not yet generate a platform-agnostic lockfile. This matches pip-tools, but differs from Poetry and PDM, making uv a better fit for projects built around the pip and pip-tools workflows.
Do you expect to make the higher level workflow independent of requirements.txt / support a platform-agnostic lockfile? Being attached to Rye makes me think "no".
Without being platform agnostic, to me this is dead-on-arrival and unable to meet the "Cargo for Python" aim.
> uv supports alternate resolution strategies. By default, uv follows the standard Python dependency resolution strategy of preferring the latest compatible version of each package. But by passing --resolution=lowest, library authors can test their packages against the lowest-compatible version of their dependencies. (This is similar to Go's Minimal version selection.)
> uv allows for resolutions against arbitrary target Python versions. While pip and pip-tools always resolve against the currently-installed Python version (generating, e.g., a Python 3.12-compatible resolution when running under Python 3.12), uv accepts a --python-version parameter, enabling you to generate, e.g., Python 3.7-compatible resolutions even when running under newer versions.
This is great to see though!
I can understand it being a flag on these lower level, directly invoked dependency resolution operations.
While you aren't onto the higher level operations yet, I think it'd be useful to see if there is any cross-ecosystem learning we can do for my MSRV RFC: https://github.com/rust-lang/rfcs/pull/3537
How are you handling pre-releases in you resolution? Unsure how much of that is specified in PEPs. Its something that Cargo is weak in today but we're slowly improving.
> Do you expect to make the higher level workflow independent of requirements.txt / support a platform-agnostic lockfile? Being attached to Rye makes me think "no".
Yes, we absolutely do. We don't do this today, the initial scope is intentionally limited. But in the next phase of the project, we want to extend to multi-platform and multi-version resolution.
> How are you handling pre-releases in you resolution? Unsure how much of that is specified in PEPs. Its something that Cargo is weak in today but we're slowly improving.
This is something we talked with Jacob about quite a bit. Turns out (as you know) it's a very hard problem. For the initial release, we added a constraint: our default behavior is that if you want to use a pre-release, you _have_ to specify the package as a first-party dependency and use a pre-release marker in the version specifier. (We also support globally enabling and disabling pre-releases.) So, we basically don't support "transitive" pre-releases right now -- but we give you a dedicated error message if your resolution fails for that reason.
The living version of PEP 440 has a bit on how pre-releases are handled[1]. The basic version is that the installer shouldn't select them at all, unless the user explicitly indicates that they want a pre-release. Once opted into, they're ordered by their phase (alpha, beta, rc) and pre-release increment (e.g. `.beta.1 > `.alpha.2`).
[1]: https://packaging.python.org/en/latest/specifications/versio...
Rip (https://github.com/prefix-dev/rip/issues) which is more of a library for other rust tools to be on top of like Pixi (which is looking to replace both Pip and Conda).
And now uv, which seems to be looking to replace Pip, Pip-Tools, and eventually Poetry and PDM.
A lot of the explosion in tools in the Python world is coming from the desire for better workflows. But it has been enabled by the fact that build configuration and calling has been standardized and tool makers are able to follow standards instead of reverse engineering easy install or setup tools.
I know a lot of people are put off by there being so many tools, but I think in a few years the dust will settle and there will emerge a best practice work flow that most users can follow.
As a primarily Python developer and someone who occasionally contributes to Pip to solve complex dependency resolution issues it does make me wonder if I should hang my hat on that and learn enough rust to contribute to one of these projects eventually.
Put other ways: the Python ecosystem has had 30+ years to figure out how to make packaging not suck. It has continually failed - failed less and less over time, sure, but the story is still generally speaking a nightmare ("throw it all in an OCI container" is an extremely reasonable solution to Python packaging, still, in 2024). I welcome advances, especially those inspired by tooling from languages that focused heavily on developer experience.
Being compatible with the mess that exists is where the difficulty comes from.
The last 5 or so years I've mostly done work in rust... so I guess I'm a rust developer now.
Do the last 5 years invalidate my opinions, thoughts, or skills w.r.t. python somehow?
npm -> Yarn was life changing performance-wise.
I wonder what pip -> uv is.
I’m loving the pace of release with this crop of speed obsessed projects and I cannot wait for astral to tackle typing.
Npm has improved greatly.
Npm features follow a two-step process: (1) Get implemented in Yarn (2) Get implemented in npm.
By the way, the creator of mamba started his own company at https://prefix.dev/
They want to essentially leverage the conda(-forge) infrastructure to build a new cross-platform, cross-language, cargo-like package manager: pixi
[0] https://github.com/mamba-org/mamba
https://conda.org/blog/2023-11-06-conda-23-10-0-release/
Using only „Pythons native tools“ like pip and venv simply works nowadays so good that I wonder about the purpose of many tools like poetry etc. etc.
https://www.bitecode.dev/p/back-to-basics-with-pip-and-venv
https://www.bitecode.dev/p/relieving-your-python-packaging-p...
https://github.com/python-poetry/poetry/issues/6409
pip and venv work fine, but you have to get them first; and that can be a struggle for unseasoned python devs, especially if you need a version that's not what your distro ships, and even more so on Windows and macOS.
I use micromamba [1] specifically, which is a single binary.
[1] https://mamba.readthedocs.io/en/latest/user_guide/micromamba...
pip install works, but pip's dependency management doesn't seem to (for Pytorch, specifically) which is why projects that have pip + requirements.txt as one of their installation methods will often have separate pytorch installation instructions when using that method, though if the same project supports conda installation it will be a one-stop-shop installation that way.
That’s interesting — I’ve also had difficulties with PyTorch and dependency resolution, but only on the most recent versions of Python, for some period of time after they’re released. Picking Python 3.9 as a baseline for a project, for example, has been very reliable for PyTorch and all the related tooling.
After fiddling with different solutions for years and having to start fresh with a new Python install, I've been using nothing by miniconda for years and it just works
> Think: a single binary that bootstraps your Python installation and gives you everything you need to be productive with Python, bundling not only pip, pip-tools, and virtualenv, but also pipx, tox, poetry, pyenv, ruff, and more.
A lot of that dev tooling is not needed in prod, in fact it is a liability (features available for misuse, code/image size, higher resource requirements). Would there be a "uv-prod" to only deal with the minimum subset of environment setup / container building that enables production use? Would uv build packages that are runnable in prod but themselves don't contain uv? It'd be interesting to hear the plans.
Checksum verification and portable lockfiles are kind of table stakes for me, which is why I use poetry everywhere possible. I can't give up correctness for speed.
sidenote: this would be super useful as a library so I could plug it into build systems. Managing python dependencies in standards-compliant ways is a pain, so shelling out to tools like poetry and pip are used even though it would be nicer to write that code eg in Rust and just use the parts you care about.
The "feature not a bug" tone is probably because we intentionally limited the scope of this release to _not_ support platform-agnostic resolution, since it adds a lot of complexity (and we were still able to ship something that's immediately useful for those that rely on pip and pip-tools today).
Sure, it’s rendered a bunch of projects obsolete, but it’s because it’s so much better than the predecessors.
Obviously it would be pretty crummy to end up in a situation like pyright where MS really have leant in to their embrace, extend, extinguish strategy.
That is true for every single development tool in the Python space.
Yeah. Probably precisely the same people that call everything EEE.
You’ll always find people acting alarmist about anything.
[0] https://www.youtube.com/watch?v=XzW4-KEB664
I agree that we have no idea how it'll play out. But I've seen nothing but good faith from Charlie and others, and I see no reason to preemptively condemn their attempt to build a business.
In this case they are replacing Python code with Rust which might exclude large part of Python community from being able to contribute.
(Or another framing: only the tiniest fraction of the Python community was contributing to CQA tooling in the first place. It's not clear that prioritizing the interests of the 99.9% of Python programmers who -- rightfully! -- will never modify these tools makes sense.)
I'm not sure what the maker of that video wants, does he want money to be poured back into the community, or for no one else to make money?
[1]: https://hatch.pypa.io/latest/
I have historically expressed trepidation towards external attempts to "fix" Python packaging, largely because each attempt has historically accreted another layer of incompatibilities and hacks onto the giant pile of hacks that are already needed to keep the ecosystem running. As such, it makes me happy to see that compatibility is a priority here: the Astral team has gone out of their way to emphasize both formal (PEP) and informal (pip CLI) compatibility with existing tooling and standards.
(Separately, I share concerns about VC funding and sustainability. But everything I've seen so far indicates that Charlie and the other folks at Astral are not in it to screw or otherwise leverage the Python community. So I choose to be optimistic here.)
``` black @ git+https://github.com/psf/black ```
We also support editable installs for local directories, like you mentioned:
``` black @ ../black ```
The thing we don't support is using _editable_ installs for Git and direct URL dependencies, like:
``` -e git+https://github.com/psf/black ```
In my experience, these are really rare, and we didn't see a reason to support them.
https://xkcd.com/1987/ https://xkcd.com/927/
> I had to guess, that’s the path that the Astral team would take as well - expand ruff’s capabilities so it can do everything a Python developer needs. So the vision that Armin is describing here might be achieved by ruff eventually. They’d have an advantage that they’re not a single person maintenance team, but the disadvantage of needing to show a return to their investors.
For instance, there are >30 dists for numpy 1.26.4 [1], with different artifacts for different platforms and python interpreters. Which of those hashes ends up in a platform-agnostic lockfile? Do you have to stick with sdists only?
[1] https://pypi.org/project/numpy/#files
Just downloading a library to a local source code directory should not cause arbitrary code to run on your system, and there should be safeguards in place so developers are not one typo away from a malicious package pwning their entire system. Supply chain attacks remain a huge area of concern.
Instead, when I "ctrl+f security" the homepages of any of these new packaging systems, I get 0 results. Not good.
I think you're thinking of this post[1]. The code being searched for wasn't malicious, just buggy :-)
[1]: https://moyix.blogspot.com/2022/09/someones-been-messing-wit...
Heh, I forgot about the anime catgirl part.
For some more context, at the time, poetry and pipenv were both buggy, and had some limitations their teams listed as "wont-fix". The sort of thing where you hit a breaking bug immediately on certain systems. These have since been fixed.
One reason I struggled with Rye earlier is that it kind of felt like this wasn't entirely supported, possibly because I couldn't parse the information on the workspaces or virtual projects page. Maybe someone else figured this out? It does seem common enough that I would be surprised if it wasn't supported.