The nice thing about Python packaging is that it’s standardized: if uv’s build backend doesn’t fit into the hole you want, then you can use any number of other backends that do an admirable job of conforming to the packaging standards.
With that said, I don’t understand why Gentoo needs to rebuild every single release of uv-build. It doesn’t change as much as uv itself, and it’s intentionally compatible across uv versions. Why not set a policy of only updating once weekly, or even less frequently, with exceptions for important breaking changes or bug fixes?
(For context: the only reason uv-build releases as often as it does is because it’s easier to keep its internals in lockstep with the rest of uv. But you can do a normal version bound in your Python build backend metadata and uv will happily use whatever version of uv-build is already present.)
The problem is packages that use the build backends, and Gentoo is generally not in the business of maintaining that kind of patch.
Admittedly rustification like that is a problem not only for gentoo, I have commercially deployed systems where one of the most annoyings elements of updating is presence of rust in python deps...
You pretty much opened with a... I'm sorry for a strong word, but it is what it is... a lie.
No. Python packaging isn't standardized. Tools aren't interchangeable. And they are all bad, uv including. I wish it didn't exist, so that there were fewer problems for people like me, whose job is, essentially, working around the problems created by people who build tools like uv and people who mindlessly adopt them.
To answer to your specific arguments: nobody is going to believe that your proposed solution is going to hold water longer than it takes for milk to go sour. Too many people, too often had burned on Python packaging tools to trust some tool not to break, to be consistent etc. Without even realizing it, you suggested that someone else should use a crutch to work around the problem you created, and you expected them to just go along with the flow and live by the bottom of the barrel lowest possible "standards" that exist in Python packaging world. No wonder they don't want uv-build... why would any sensible person want it?
> The nice thing about Python packaging is that it’s standardized
But it isn't. Once upon a time, it was: you used distutils. But of course distutils had a lot of problems. Instead of adopting a single standardized solution for those problems, the Python developers punted and let third parties do it. So now we do not have a single standardized solution for building Python packages; we have a bunch of third party solutions, none of which is in the Python standard library.
Yes, all those third party solutions have to be PEP 517 compatible, but that's not standardization on a tool; it's just standardization on an interface between the backend and the frontend. Which is nice, but doesn't solve the problem under discussion here.
> if uv’s build backend doesn’t fit into the hole you want, then you can use any number of other backends that do an admirable job of conforming to the packaging standards.
As the project developer, sure. But as a Linux distro building your python package, Gentoo has to use the build backend you specify. It can't say, well, this particular package is just pure python, so really all we need is a minimal tool to build a pure python none-any wheel. It has to use the backend you specified.
> I don’t understand why Gentoo needs to rebuild every single release of uv-build.
Because if you don't pin a version of uv-build in your project, that means "use the latest version available". Gentoo can't second guess you--even if it were able to verify somehow that the new version of uv-build that just got released will end up producing an identical wheel for your project as the previous version, without actually using the new version of uv-build.
> you can do a normal version bound in your Python build backend metadata
Sure, but how many users of uv-build actually do that? Seems like the answer to that is "not enough to matter".
At the end of the day, it’s hard for me to even pinpoint what is problematic for Gentoo here: I can understand that they need to support multiple build backends as a reality of Python packaging and that’s frustrating, but per below it seems extremely unlikely that they actually need to build every single patch version of uv. Nothing about uv’s defaults or the packaging template would require that, and users would have to go out of their way to configure their build backend in that way.
(And note: this is not unique to uv! You can grief anybody who does an sdist build by pinning to a random ancient version of setuptools or any other build backend, including other ones that use native code. This is why uv, like everybody else, encourages semver ranges for build backends.)
> it seems extremely unlikely that they actually need to build every single patch version of uv.
If every new patch version is still within the semver range specified in a project that uses uv-build, then yes, every Gentoo user will have to rebuild that new patch version of uv-build in order to rebuild that project. And since the build backend is a dependency of the project, pushing a new patch version will trigger a rebuild of the project.
distros patch the build process when they package stuff for the distro all the time. why would they "have to" do anything? i'm not super familiar with gentoo but it seems to me like any other 90s era distro where the distro mediates the relationship between the user and the upstream developer for all packages, by applying patches and potentially doing whatever build they want to.
> Yes, all those third party solutions have to be PEP 517 compatible, but that's not standardization on a tool; it's just standardization on an interface between the backend and the frontend. Which is nice, but doesn't solve the problem under discussion here.
Sure it does. TFA claims
> all Gentoo users need to build it from source, and given that it’s released quite frequently (12 releases in the 30 days prior to writing this post), that’s a lot of building. Like, for every 100 ms spent running uv-build to install one release of your package, Gentoo users will spend many hours building new uv-build versions over and over again.
And that is false, exactly because of that standard. If you want reproducible builds then you can stick with a previously built version of uv-build for a long time. If you want working builds on the fly using your own processes, then the standard empowers you to substitute something else. If you want to use the Python packaging ecosystem in the normal way, then it doesn't even matter what your distro is.
uv-build is for pure Python projects, which entails that the "build" process is trivial. It doesn't even bytecode-compile the .pyc files generally; wheels aren't expected to contain them (your installer may produce them at install time, or you can let Python produce them on demand). Binary reproducibility for pure-Python wheels is just not, from anything I've seen, a high priority for Python packaging.
Linux distros have these ideas about packaging that are just not suitable for Python and not in tune with what the Python ecosystem is doing. The vast majority of the Python code you install should not be integrated into your "system". (Python devs rarely find themselves wanting to "bootstrap" everything from source, either.) Gentoo's issue here appears to be their own policy, which is a problem of their own making. It's unrealistic for them to expect random Python devs to care about this.
> As the project developer, sure. But as a Linux distro building your python package, Gentoo has to use the build backend you specify. It can't say, well, this particular package is just pure python, so really all we need is a minimal tool to build a pure python none-any wheel. It has to use the backend you specified.
If you mean because of Gentoo policy, that's their fault. I can guarantee you that approximately nobody using uv-build is concerned with whether their package makes it into Portage. Python build systems aren't designed to produce packages native to Portage (or any other Linux packaging system). Probably the overwhelming majority of these users aren't expecting any Linux distro repositories to host their package in any form, and probably most of them couldn't even tell you what Gentoo is.
If you mean technically, then no; once you have the source code and are using ebuild, nothing compels you to choose the backend specified in pyproject.toml. If everything is up to standard then it's trivial to switch it over to, say, Flit. Most projects following a standard src layout will need no tool-specific config, and there's quite little of it possible regardless (and all well documented). The author even highlights:
> It’s a backend and the whole point of PEP 517 is that you can use any backend with any tool. You can use flit-core, you can use hatchling, and either of them will work just fine with uv (or any other frontend).
The edits do take time but it's a lot better than the proposed bogeyman of waiting for a new build of uv-build from Rust source.
> nothing compels you to choose the backend specified in pyproject.toml
This comes as a complete surprise to me. I thought the whole point of specifying the build backend in pyproject.toml was so the project owner can choose which build backend their project gets built with, and have that choice automatically enforced by tools.
Are you saying this kind of thing is allowed by the Python spec? If so, where? Can you give a specific document?
> The author even highlights
What he means is that the project owner doesn't have to, for example, use uv-build as their build backend in pyproject.toml just because they are using uv as their overall project management tool. I do not think he means that he, as a distro packager, is free to just ignore what's in pyproject.toml.
> the project owner can choose which build backend their project gets built with
When following an automated workflow implemented by an installer or other tooling, yes.
But you own the sdist file, and the archive contents, after downloading it. Nothing prevents you from editing pyproject.toml, using a build frontend/system that offers a flag to disregard PEP 517, running your own custom scripts on the data, manually copying the files into the target environment, or whatever else you can think of.
And since Gentoo is doing its own packaging with its own package format, nothing prevents them from doing the same kind of patching.
> you own the sdist file, and the archive contents, after downloading it.
If you're forking the project, sure. But I don't think that's what Gentoo is doing.
If you're trying to build what the project owner intended for you to build, I don't see how you can just ignore what the project owner specified. Nor do I see how that is allowed by the Python specs. That's why I asked you for a reference if you think it is in fact allowed by the Python specs.
If your position is that a distro can just ignore what's in the Python specs, I don't agree with that either, but it's a different position, and I'm not sure which one you're taking.
> uv-build is for pure Python projects, which entails that the "build" process is trivial.
The build process is more than just the Python code. There is also project metadata, which might be dynamically generated by the build backend. Indeed that's one of the main features offered by the different build backends, simplifying the project owner's life by removing the need to write a lot of boilerplate metadata that the tool can automatically discover on its own.
Yes, but these things are largely trivial in most cases.
In a wheel, there are basically two kinds of metadata: the kind that follow directly from the circumstances of the build (RECORD, which lists sizes and hashes for included files, and WHEEL, which says what environments the wheel is for and what tool built it), and then the kind that has to be specified by the developer. The latter is in METADATA. But the thing is, for most projects this data can be completely static, in which case the wheel METADATA is just a copy of the corresponding sdist PKG-INFO.
Build backends support a variety of dynamic metadata, but there are two places where that could be calculated: in the sdist (using information from pyproject.toml plus the developer's source tree to produce PKG-INFO) and when building a wheel from that sdist. The latter cases mean that the developer is asserting that code needs to run on the end user's machine in order to get correct metadata. This is… usually questionable at best, and IMO impolite without a damned good reason.
The point is that that's not your call to make. It's the project owner's call. That's what the build backend specification in pyproject.toml is for. If you're going to just ignore that, what's the point of having it at all?
> in the sdist (using information from pyproject.toml plus the developer's source tree to produce PKG-INFO)
Which is the only place that's relevant for this discussion, since we're talking about Gentoo, which builds everything from source.
> The latter cases mean that the developer is asserting that code needs to run on the end user's machine in order to get correct metadata. This is… usually questionable at best, and IMO impolite without a damned good reason.
I agree with all this, but as above, it's irrelevant to this discussion.
You can demonstrate it somewhat empirically: pip, uv, etc. all understand the build backend interface and packaging standards and can install any package that conforms with the standards.
(I think so much of the confusion around Python packaging comes down the layer of abstraction/standardization not being the one that users intuitively expect. This is unfortunate and is made worse by lackluster user-facing documentation. But there are standards that guarantee interoperability, and they’re pretty close to Pareto-optimal in terms of making all involved parties happy.)
Yeah, I think conformance is pretty good these days. PyPI itself enforces metadata well-formedness, and projects that don’t conform tend to get noticed quickly (because they don’t install correctly).
> The nice thing about Python packaging is that it’s standardized
And a problem with Python packaging is that it's far too flexible.
For a pure Python package, IMO every single file in the wheel should be bit-for-bit identical to a file in the source tree (with the possible exception of a manifest listing the files). If this were actually a guarantee, even just for a category of package that opted in, then:
a) The build "tool" could be replaced with a build file selector, which would often be completely trivial and could have default behavior that worked for almost every pure python package right in pyproject.toml and
b) Instead of the giant mess of sigstore and attestation and weird trusted builds we have today, we would have a straightforward proof that a wheel actually matched a git commit irrespective of who built it. In fact, the proof could be embedded in the wheel! And the proof could be verified without executing any package-controlled code.
Python is exactly as flexible as the people who use it want it to be. I suspect that most trivial solutions do not generalize to the interests of the broader Python community, which is how we end up with standards that are universally agreed to be not exactly what anybody wants.
Maybe, but I do think “pure Python sdists could just be a file selector” misses lots of common cases of simple code generation. Documentation boilerplate generators, version-at-runtime generators, things that generate e.g. static HTML or protobuf or whatnot results…it’s not most pure-Python packages, sure, but it’s a lot of them.
This response downplays the portability concern which is the main point of the article. The ask for Gentoo to change its release & update cadence doesn't really address his plea.
As I see it, uv-build is Rust and will stay Rust, so on SPARC or address-space-constrained 32-bit targets may never work right. Astral prioritizes its users who run on tier-1 targets, and the cost of their uv-init default lands on the downstream packages that Gentoo has to build. Gentoo wants projects to choose pure python backends to reduce that ecosystem load, Astral would prefer if more projects default to uv-build (IIRC the default went from hatchling to uv-build in mid-2025).
Long-tail vs evolution of ecosystem would be a far more interesting discussion, but the broader community has a "we are the future/adapt to us" sentiment that forecloses trade-offs.
I mean, SPARC and anything else can work right. But what uv can’t do is conjure up support for platforms that the larger ecosystem won’t even claim to support.
To that extent, I think uv (and CPython, and Rust, and so many other projects) are essentially trying to be honest in saying “we have no way to build and test that to our own standards.” I think that kind of honesty in validation is its own kind of load reduction on the ecosystem, because it means less tacit assumption that the maintainers will support anything that happens to build.
The basic question the plea raises is "how should the long tail of ecosystems get cut?".
"Upstream has no way to build and test [the long tail] to our own standards" is a certain kind of honesty but not a real answer or a trade-off based in the status quo ante.
The status quo, as I glean it, is that C/autotools ecosystem let downstream packagers get from “it compiles” to “it runs” on their own; the new Rust-based one inhibits their freedom to do so. I respect Astral's decision to prioritize their product by setting a default to the ecosystem convenient to their interests; I just don't see a need to moralize it.
Why? You can bootstrap Rust if you want, just like you can bootstrap LLVM or GCC. There’s no law or rule against it, there are just support tiers (and GNU projects have those support tiers too.)
(I don’t think it’s moralizing to do or say this.)
Maybe it's time to grow up and download binary builds like a normal person. I had my 'let's compile the gentoo kernel from source for 3.5% hypothetical performance gains' phase but you can't expect professional devs to cater to source-obsessed hobbyists
I doubt this oft-repeated stereotype was every particularly accurate. I'm not aware of any current Gentoo users on the distro specifically for performance reasons. Gentoo users are in it for control over their system and customization.
"In the early days, there was much promotion of Gentoo as being “faster”, because you could compile everything based on your particular processor, etc. And you could increase the optimization level for a boost. Their web site still touts this as a reason to use Gentoo.
In reality, the performance is more or less the same as on any other distribution. The folks who stick to Gentoo tend not to care about performance as much. Unfortunately, this perception of Gentoo remains, and I wish they would remove the verbiage from their site."
A friend of mine got me on Gentoo in 2004. He spent all of his time tweaking compiler flags like -O3 -ffast-math -fomit-frame-pointer -funroll-loops and running into so many issues in the process. I think he only lasted about 9 months on Gentoo. To this day, he's the only person I've ever met that was there for performance.
Good timing! If memory serves, you missed the Big Libpng Upgrade that either got the concept of Package Slots into Portage or convinced everyone that they were a _really_ good feature to have and use where appropriate.
> Unfortunately, this perception of Gentoo remains, and I wish they would remove the verbiage from their site.
Based on what I can see after putting a minute or two into searching, that verbiage being referred to only appears on the About page. [0]
The distro's home page says
Welcome to Gentoo, a highly flexible, source-based Linux distribution.
and has used those words since at least January 2023 [1]. Its home page has spoken about flexibility and has not spoken about performance since the big redesign in April 2016... seven years prior to the publication of that guy's blog post. [2][3]
And -speaking as a Gentoo user since 2002- neither the user community nor the maintainer community hold the opinion that Ultimate Maxxximum Performance is the distro's goal. Trouble reports that come from "ricers" pretty much always get told "Set your compiler flags to something sane (try '-O2 -pipe' or '-O2 -pipe -march=native') and rebuild. If you can still repro the problem, file a new report because this one is getting closed INVALID.".
And -while the old Gentoo Wiki burned down so I can't prove it- [4] the advice about GCC optimization in the wiki has always cautioned against using aggressive compiler flags and warned that doing so will -among other things- not substantially improve performance. The advice about reasonable compiler flags has pretty consistently been "Use -O2, -pipe, -march=$THE_CORRECT_SETTING_FOR_YOUR_CPU, and maybe -fomit-frame-pointer"... though the "-fomit-frame-pointer" advice has always come with a warning that doing that may make debugging either extremely difficult or impossible, and the "-march" advice the warning that this may make it impossible to run those binaries on another computer.
Anyway. There's only so much you can do to erase an -allegedly- pervasive myth. The fact that some folks have the belief that Gentoo Linux is all about Maxxximum Performance doesn't change the fact that nearly all of the people who use it and -almost certainly- all of the people who maintain it strongly disagree with that claim. The distro is and always has been about flexibility, user choice, and having fun. And it's a nice bonus if building for -say- -march=athlon-xp instead of -march=i586 happens to get you a bit of extra performance here and there.
[3] That year's April 1 redesign was a bit cool, IMO.
[4] The oldest copy that I can easily find is from mid 2013 [5], so you'll just have to trust me that this has been the long-standing opinion of the overwhelming majority of users and maintainers. While you're there, take a minute or two to skim the prose and notice how it consistently warns against using aggressive compiler flags.
> Extreme configurability, performance, and a top-notch user and developer community are all hallmarks of the Gentoo experience.
It's the performance part I was referring to.
Going back to when I started using it:
> Gentoo Linux is a high-performance ports-based Linux metadistribution
Regarding -O2 vs -O3, in the early days, -O3 was considered quite safe (I don't recall if it was recommended, but it wasn't discouraged). I think only after a certain GCC upgrade did they start discouraging -O3. I vaguely recall them announcing that quite a few of the -O3 benefits were now part of -O2, and that some (new?) -O3 stuff was unstable.
At the time, I knew little about compiler flags. I went with -O3 because of guidance from the Gentoo folks.
Yep, when I started using it in _2002_ that was on the home page. Ten years ago, it stopped being on the home page because folks finally got the motivation to do a bunch of work to it. As I said:
Its home page has spoken about flexibility and has not spoken about performance since the big redesign in *April 2016*... seven years prior to the publication of that guy's blog post.
There's only so much you can do to erase an -allegedly- pervasive myth. The fact that some folks have the belief that Gentoo Linux is all about Maxxximum Performance doesn't change the fact that nearly all of the people who use it and -almost certainly- all of the people who maintain it strongly disagree with that claim.
> Regarding -O2 vs -O3, in the early days, -O3 was considered quite safe (I don't recall if it was recommended, but it wasn't discouraged). ... I went with -O3 because of guidance from the Gentoo folks.
It has been too long for me to remember what -O level my first Gentoo box ended up using (and I can't be bothered to pull it out of the closet and power it on to see), but the box I installed in ~2007 is using -O2. A random sampling of the few Gentoo forum posts from the early days that are in the Wayback Machine [0], as well as a handful of Bugzilla reports from the 2002->2005 era _suggests_ that most folks in the early days were using -O2. I'm certain that I played with all of the optimization levels on my first box, didn't notice any performance difference, and then settled on one that I can no longer remember.
And like, you say that -O3 wasn't discouraged. Well, -O3 isn't really discouraged today. The advice about -O3 gives you a summary of the facts and trusts you to make the choice that's right for you [1][2]
WARNING: Using `-O3` may cause some packages to break either during the compilation or misbehave at runtime, although `-O3` retains standard conformance, hence any breakage is either undefined behaviour in the application, or (very rarely) a compiler bug.
<irrelevant stuff elided>
`-O3`: enables `-O2` as well as optimizations that are expensive in terms of compile time and memory usage. Compiling with `-O3` is likely to improve performance (although not guaranteed). While historically `-O3` could expose compiler bugs, nowadays issues with `-O3` are almost always instances of undefined behavior (UB) and the code is at fault and should be fixed. Some packages are still known-broken with `-O3`. Using `-O3` is not recommended unless running test suites for packages.
(I'm pretty sure that "unless running test suites" thing means "you shouldn't use this optimization level unless you're also using FEATURES="test" to run the package's tests".)
FWIW, it's my maybe-incorrect understanding that many (most? all?) packages that are known-broken with specific flags get those flags filtered out and replaced with known-good ones... unless they're nutball "ricer" flags, in which case the user gets told "Don't do that!".
[2] Even the advice from a decade ago [3] is substantially similar: «This makes compilation take longer, and it not only might not make things faster, it may make things slower and is known to break things sometimes.»
-O3 is still quite safe and the rare cases where it causes issues have always been due to bugs in the source code being compiled from my experience. It isn't guaranteed to produce codue faster than -O2 though.
> In reality, the performance is more or less the same as on any other distribution.
It is now but in the 32-bit days there was a nontrivial boost form compiling for e.g. a Pentium 4 when most binary distros were still stuck targetting generic i586 or something old like that.
Gentoo made a lot of sense WAY back in a pre-docker world, where you had to run the whole world on the host OS. Being able to compile it all meant you could link it all together with whatever weird combination of optional libraries you wanted. Things like getting sendmail to auth over LDAP, that nobody packaged in the base distro, become possible. And with Gentoo's tooling, you could build a coherent distro to your own unique demands.
Now though? It's way easier to have a herd of unicorn docker containers running that each handle one weird thing in isolation.
Gentoo’s decision to support source builds seems eminently reasonable to me, and I don’t think we should reward it with condescension.
The bigger problem, to me, is that it’s not really clear why this is actually a problem for Gentoo. I am glad they perform user-facing source builds, however, because they see things that other distributors don’t.
The vast majority of desktop Linux users run Ubuntu (initial motto: Linux for human beings). So one may argue the literal opposite of what you wrote, in that if Canonical didn't invest into making a Linux distro that's accessible to normal people, it wouldn't have the relevance it now does. On top of that, Linux has (or had) a number of clear advantages over Windows that dwarf the presumed upsides of compiling everything from source vs. downloading builds.
There's not much of an argument for adopting obscure niche practices for their own sake
Linux was obscure and niche for a very long time before Ubuntu existed and Ubuntu only existed because someone bothered to use and develop Linux when it wasn't normal.
The Ubuntu users are only there because after they had told the rest of us how silly we were for using Linux they eventually had to use it themselves for work.
Now we're going to listen to that sort of stuff again without rolling our eyes? ....
SLS for me. haha. :-) But now at least you have the choice! It's great really. I can use Artix because I like being nearer the bleeding edge but even that is really a lot easier to use than Linux was in the days when I had to compile my kernel.
Nevertheless I remember lots of my friends who started off saying Linux was hopeless and ended up using it and Ubuntu was their gateway. The Linux world now is full of people who are fairly conformist and don't care about how it all began and who don't value the idea that open source has margins from which who knows what world beating things might eventually come. The fact that margins can exist is what makes open source great.
The rather large giant collosal planet sized argument that you somehow missed despite that, is that every single thing that you consider good and normal today doesn't exist except for the fact that at one point it was the obscure niche thing that annoyed intolerant "normal" people.
You, like the other commenter, disregarded the core argument of the matter in favor of contrarianness for its own sake: are you trying to argue that building literally everything from source is cool and good because it's niche and weird, whereas Linux was niche and weird once but it's cool and good now? As I said, Linux had and still has clear advantages over Windows to the point that Microsoft felt threatened by it fairly soon (cue Ballmer's chair throwing, cancer remark, etc.) and tried to undermine it before giving up with Nadella. What, exactly, are the obvious and objective upsides of compiling everything from source? Many posters in the thread dispelled performance-maxxing as a meme so it can't be that. You are comparing apples to oranges here.
I feel like I'm trying to explain a shade of colour to a blind person.
building from source has uses which you believe are worthless and other people don't think are worthless.
Anyone who uses cargo or installs a module for golang is building from source. But that, probably, is considered useful. It's only people who build C code from source who are niche and wierd. ....
Less and less with the ongoing ads, taskbar issues, and RAM eaating bloatware.
By just adding
preempt=full
to most GNU/Linux distros to the grub kernel boot options (trivial even for Fedora Silverblue, just run
sudo rpm-ostree kargs --editor
) and append at the end of these boot flags you are seeing the mentioned string. Done, multimedia ready desktop for tons of tasks outperforming Windows 11.
Firefox built from source is sufficiently sped up that I can notice it starting up at least 1-2 seconds faster. Not that I use the distro for speed, but I'm not complaining about a faster browser.
If you insist on acting like a normal person, no one stops you from using something normal.
i got a debian maintainer for old IBM mainframes asking me to fix bugs and run tests for that architecture. ¯\_(ツ)_/¯ these people exist, but i'm not going to support them. feel free to fork me!
Yes, optionally apparently but I think we'll end up with forks of Python if the Rust enthusiasts push it any further. I think the option will be off for me.
I presume that those who don't want to have to build rust just to get python will stick the rust code into an LLM and get out an acceptable analog in C.
I don't think I agree with the author's assessment here:
> The other reason is performance and that argument makes even less sense. Even if it mattered, we’re realistically talking about a difference in the ballpark of 100 ms, for an action executed rarely; it is equally possible that any gain will be outweighed by the time needed to fetch the 1.5 MiB wheel for uv-build.
Modern CI architectures are building Docker stacks from scratch and using a readthrough cache of wheels. Downloading may still be the dominant factor, but the second-dominant factor does start to add up, especially when the downloading gets cached at various layers.
By a search, Gentoo accounts for less than 1% of all developers. If uv-build is more conformant to people, probably volunteers (and if they're using it, it probably is), I don't think the Gentoo people have any right to want them not to use it because it makes their lives harder. They can request, and they do request nicely, but they also present the drawbacks for them as if they are absolute, and even claim that "its benefits don’t outweigh the drawbacks".
> they do request nicely, but they also present the drawbacks for them as if they are absolute, and even claim that "its benefits don’t outweigh the drawbacks".
for some people the benefits absolutely do outweigh the drawbacks. Maybe not for you....but on the whole I don't know - if the whole issue had been solved without being given attention would you have ended up caring which way it went?
I'm sure it accounts for a much larger proportion of people using exotic or legacy systems. In any case, I think most people are using uv-build either because it's a default for uv now or just because it's hip and new. If a project absolutely needed it then so be it, but if you knew it was going to make life difficult even just for one person and no benefit to you then you'd probably choose something else.
UV build is distributed as a binary/executable, e.g. manylinux wheels on pypi: https://pypi.org/project/uv-build/#files. If a user or distro's conventions are to build things from source, this is a consequence they have implicitly accepted. For the vast majority of computer users and uses, installing binaries makes more sense.
It is still a problem where there are no binaries for platforms otherwise supported by the distro, and in fact supported by the package being built - because uv-build is the only part that does not run on given platform and can't be reliably built for it
Looks like it's being built with fat LTO by default (and gentoo patched to thin LTO). Which is fine if you want to spend CI cycles once to ship executables to thousands of users.
If you compile from source and run the tool maybe once then they should consider going further and run with maybe opt-level=1, lto=off; halfway between release and debug. This is how rustc itself builds some internal buildtools that get recompiled frequently relative to how often they run.
> for every 100 ms spent running uv-build to install one release of your package, Gentoo users will spend many hours building new uv-build versions over and over again.
maybe it's time to embrace the enemy and go "clanker, this is uv-build source, rewrite it in portable C". rewrite it in C FTW!
btw, I know of a guy who using extensive amounts of tokens to make his fork of mrustc usable via "claude, make this rustc compile 99% of crates.io", I think this also can yield some results.
You don’t even need uv_build as an explicit dependency if you already use uv. uv itself recognizes if your project specifies uv_build and does the same job.
(If you’re not using uv, you’ve made the wrong choice, but that’s a separate concern.)
78 comments
[ 0.28 ms ] story [ 14.9 ms ] thread(I am one of uv’s maintainers.)
The nice thing about Python packaging is that it’s standardized: if uv’s build backend doesn’t fit into the hole you want, then you can use any number of other backends that do an admirable job of conforming to the packaging standards.
With that said, I don’t understand why Gentoo needs to rebuild every single release of uv-build. It doesn’t change as much as uv itself, and it’s intentionally compatible across uv versions. Why not set a policy of only updating once weekly, or even less frequently, with exceptions for important breaking changes or bug fixes?
(For context: the only reason uv-build releases as often as it does is because it’s easier to keep its internals in lockstep with the rest of uv. But you can do a normal version bound in your Python build backend metadata and uv will happily use whatever version of uv-build is already present.)
Admittedly rustification like that is a problem not only for gentoo, I have commercially deployed systems where one of the most annoyings elements of updating is presence of rust in python deps...
No. Python packaging isn't standardized. Tools aren't interchangeable. And they are all bad, uv including. I wish it didn't exist, so that there were fewer problems for people like me, whose job is, essentially, working around the problems created by people who build tools like uv and people who mindlessly adopt them.
To answer to your specific arguments: nobody is going to believe that your proposed solution is going to hold water longer than it takes for milk to go sour. Too many people, too often had burned on Python packaging tools to trust some tool not to break, to be consistent etc. Without even realizing it, you suggested that someone else should use a crutch to work around the problem you created, and you expected them to just go along with the flow and live by the bottom of the barrel lowest possible "standards" that exist in Python packaging world. No wonder they don't want uv-build... why would any sensible person want it?
But it isn't. Once upon a time, it was: you used distutils. But of course distutils had a lot of problems. Instead of adopting a single standardized solution for those problems, the Python developers punted and let third parties do it. So now we do not have a single standardized solution for building Python packages; we have a bunch of third party solutions, none of which is in the Python standard library.
Yes, all those third party solutions have to be PEP 517 compatible, but that's not standardization on a tool; it's just standardization on an interface between the backend and the frontend. Which is nice, but doesn't solve the problem under discussion here.
> if uv’s build backend doesn’t fit into the hole you want, then you can use any number of other backends that do an admirable job of conforming to the packaging standards.
As the project developer, sure. But as a Linux distro building your python package, Gentoo has to use the build backend you specify. It can't say, well, this particular package is just pure python, so really all we need is a minimal tool to build a pure python none-any wheel. It has to use the backend you specified.
> I don’t understand why Gentoo needs to rebuild every single release of uv-build.
Because if you don't pin a version of uv-build in your project, that means "use the latest version available". Gentoo can't second guess you--even if it were able to verify somehow that the new version of uv-build that just got released will end up producing an identical wheel for your project as the previous version, without actually using the new version of uv-build.
> you can do a normal version bound in your Python build backend metadata
Sure, but how many users of uv-build actually do that? Seems like the answer to that is "not enough to matter".
At the end of the day, it’s hard for me to even pinpoint what is problematic for Gentoo here: I can understand that they need to support multiple build backends as a reality of Python packaging and that’s frustrating, but per below it seems extremely unlikely that they actually need to build every single patch version of uv. Nothing about uv’s defaults or the packaging template would require that, and users would have to go out of their way to configure their build backend in that way.
(And note: this is not unique to uv! You can grief anybody who does an sdist build by pinning to a random ancient version of setuptools or any other build backend, including other ones that use native code. This is why uv, like everybody else, encourages semver ranges for build backends.)
If every new patch version is still within the semver range specified in a project that uses uv-build, then yes, every Gentoo user will have to rebuild that new patch version of uv-build in order to rebuild that project. And since the build backend is a dependency of the project, pushing a new patch version will trigger a rebuild of the project.
Sure it does. TFA claims
> all Gentoo users need to build it from source, and given that it’s released quite frequently (12 releases in the 30 days prior to writing this post), that’s a lot of building. Like, for every 100 ms spent running uv-build to install one release of your package, Gentoo users will spend many hours building new uv-build versions over and over again.
And that is false, exactly because of that standard. If you want reproducible builds then you can stick with a previously built version of uv-build for a long time. If you want working builds on the fly using your own processes, then the standard empowers you to substitute something else. If you want to use the Python packaging ecosystem in the normal way, then it doesn't even matter what your distro is.
uv-build is for pure Python projects, which entails that the "build" process is trivial. It doesn't even bytecode-compile the .pyc files generally; wheels aren't expected to contain them (your installer may produce them at install time, or you can let Python produce them on demand). Binary reproducibility for pure-Python wheels is just not, from anything I've seen, a high priority for Python packaging.
Linux distros have these ideas about packaging that are just not suitable for Python and not in tune with what the Python ecosystem is doing. The vast majority of the Python code you install should not be integrated into your "system". (Python devs rarely find themselves wanting to "bootstrap" everything from source, either.) Gentoo's issue here appears to be their own policy, which is a problem of their own making. It's unrealistic for them to expect random Python devs to care about this.
> As the project developer, sure. But as a Linux distro building your python package, Gentoo has to use the build backend you specify. It can't say, well, this particular package is just pure python, so really all we need is a minimal tool to build a pure python none-any wheel. It has to use the backend you specified.
If you mean because of Gentoo policy, that's their fault. I can guarantee you that approximately nobody using uv-build is concerned with whether their package makes it into Portage. Python build systems aren't designed to produce packages native to Portage (or any other Linux packaging system). Probably the overwhelming majority of these users aren't expecting any Linux distro repositories to host their package in any form, and probably most of them couldn't even tell you what Gentoo is.
If you mean technically, then no; once you have the source code and are using ebuild, nothing compels you to choose the backend specified in pyproject.toml. If everything is up to standard then it's trivial to switch it over to, say, Flit. Most projects following a standard src layout will need no tool-specific config, and there's quite little of it possible regardless (and all well documented). The author even highlights:
> It’s a backend and the whole point of PEP 517 is that you can use any backend with any tool. You can use flit-core, you can use hatchling, and either of them will work just fine with uv (or any other frontend).
The edits do take time but it's a lot better than the proposed bogeyman of waiting for a new build of uv-build from Rust source.
This comes as a complete surprise to me. I thought the whole point of specifying the build backend in pyproject.toml was so the project owner can choose which build backend their project gets built with, and have that choice automatically enforced by tools.
Are you saying this kind of thing is allowed by the Python spec? If so, where? Can you give a specific document?
> The author even highlights
What he means is that the project owner doesn't have to, for example, use uv-build as their build backend in pyproject.toml just because they are using uv as their overall project management tool. I do not think he means that he, as a distro packager, is free to just ignore what's in pyproject.toml.
When following an automated workflow implemented by an installer or other tooling, yes.
But you own the sdist file, and the archive contents, after downloading it. Nothing prevents you from editing pyproject.toml, using a build frontend/system that offers a flag to disregard PEP 517, running your own custom scripts on the data, manually copying the files into the target environment, or whatever else you can think of.
And since Gentoo is doing its own packaging with its own package format, nothing prevents them from doing the same kind of patching.
If you're forking the project, sure. But I don't think that's what Gentoo is doing.
If you're trying to build what the project owner intended for you to build, I don't see how you can just ignore what the project owner specified. Nor do I see how that is allowed by the Python specs. That's why I asked you for a reference if you think it is in fact allowed by the Python specs.
If your position is that a distro can just ignore what's in the Python specs, I don't agree with that either, but it's a different position, and I'm not sure which one you're taking.
The build process is more than just the Python code. There is also project metadata, which might be dynamically generated by the build backend. Indeed that's one of the main features offered by the different build backends, simplifying the project owner's life by removing the need to write a lot of boilerplate metadata that the tool can automatically discover on its own.
In a wheel, there are basically two kinds of metadata: the kind that follow directly from the circumstances of the build (RECORD, which lists sizes and hashes for included files, and WHEEL, which says what environments the wheel is for and what tool built it), and then the kind that has to be specified by the developer. The latter is in METADATA. But the thing is, for most projects this data can be completely static, in which case the wheel METADATA is just a copy of the corresponding sdist PKG-INFO.
Build backends support a variety of dynamic metadata, but there are two places where that could be calculated: in the sdist (using information from pyproject.toml plus the developer's source tree to produce PKG-INFO) and when building a wheel from that sdist. The latter cases mean that the developer is asserting that code needs to run on the end user's machine in order to get correct metadata. This is… usually questionable at best, and IMO impolite without a damned good reason.
The point is that that's not your call to make. It's the project owner's call. That's what the build backend specification in pyproject.toml is for. If you're going to just ignore that, what's the point of having it at all?
> in the sdist (using information from pyproject.toml plus the developer's source tree to produce PKG-INFO)
Which is the only place that's relevant for this discussion, since we're talking about Gentoo, which builds everything from source.
> The latter cases mean that the developer is asserting that code needs to run on the end user's machine in order to get correct metadata. This is… usually questionable at best, and IMO impolite without a damned good reason.
I agree with all this, but as above, it's irrelevant to this discussion.
Has anybody done the analysis to determine if this is something that can, in practice, be relied on?
(I think so much of the confusion around Python packaging comes down the layer of abstraction/standardization not being the one that users intuitively expect. This is unfortunate and is made worse by lackluster user-facing documentation. But there are standards that guarantee interoperability, and they’re pretty close to Pareto-optimal in terms of making all involved parties happy.)
Right, but what I'm asking is: by and large, can you expect any random package on pypi to conform to the standard?
And a problem with Python packaging is that it's far too flexible.
For a pure Python package, IMO every single file in the wheel should be bit-for-bit identical to a file in the source tree (with the possible exception of a manifest listing the files). If this were actually a guarantee, even just for a category of package that opted in, then:
a) The build "tool" could be replaced with a build file selector, which would often be completely trivial and could have default behavior that worked for almost every pure python package right in pyproject.toml and
b) Instead of the giant mess of sigstore and attestation and weird trusted builds we have today, we would have a straightforward proof that a wheel actually matched a git commit irrespective of who built it. In fact, the proof could be embedded in the wheel! And the proof could be verified without executing any package-controlled code.
As far as I know, packages that use or could use uv-build or flit-core are in the just-copy-files category.
As I see it, uv-build is Rust and will stay Rust, so on SPARC or address-space-constrained 32-bit targets may never work right. Astral prioritizes its users who run on tier-1 targets, and the cost of their uv-init default lands on the downstream packages that Gentoo has to build. Gentoo wants projects to choose pure python backends to reduce that ecosystem load, Astral would prefer if more projects default to uv-build (IIRC the default went from hatchling to uv-build in mid-2025).
Long-tail vs evolution of ecosystem would be a far more interesting discussion, but the broader community has a "we are the future/adapt to us" sentiment that forecloses trade-offs.
To that extent, I think uv (and CPython, and Rust, and so many other projects) are essentially trying to be honest in saying “we have no way to build and test that to our own standards.” I think that kind of honesty in validation is its own kind of load reduction on the ecosystem, because it means less tacit assumption that the maintainers will support anything that happens to build.
"Upstream has no way to build and test [the long tail] to our own standards" is a certain kind of honesty but not a real answer or a trade-off based in the status quo ante.
The status quo, as I glean it, is that C/autotools ecosystem let downstream packagers get from “it compiles” to “it runs” on their own; the new Rust-based one inhibits their freedom to do so. I respect Astral's decision to prioritize their product by setting a default to the ecosystem convenient to their interests; I just don't see a need to moralize it.
(I don’t think it’s moralizing to do or say this.)
"In the early days, there was much promotion of Gentoo as being “faster”, because you could compile everything based on your particular processor, etc. And you could increase the optimization level for a boost. Their web site still touts this as a reason to use Gentoo.
In reality, the performance is more or less the same as on any other distribution. The folks who stick to Gentoo tend not to care about performance as much. Unfortunately, this perception of Gentoo remains, and I wish they would remove the verbiage from their site."
I'm still on Gentoo 22 years later.
Good timing! If memory serves, you missed the Big Libpng Upgrade that either got the concept of Package Slots into Portage or convinced everyone that they were a _really_ good feature to have and use where appropriate.
Based on what I can see after putting a minute or two into searching, that verbiage being referred to only appears on the About page. [0]
The distro's home page says
and has used those words since at least January 2023 [1]. Its home page has spoken about flexibility and has not spoken about performance since the big redesign in April 2016... seven years prior to the publication of that guy's blog post. [2][3]And -speaking as a Gentoo user since 2002- neither the user community nor the maintainer community hold the opinion that Ultimate Maxxximum Performance is the distro's goal. Trouble reports that come from "ricers" pretty much always get told "Set your compiler flags to something sane (try '-O2 -pipe' or '-O2 -pipe -march=native') and rebuild. If you can still repro the problem, file a new report because this one is getting closed INVALID.".
And -while the old Gentoo Wiki burned down so I can't prove it- [4] the advice about GCC optimization in the wiki has always cautioned against using aggressive compiler flags and warned that doing so will -among other things- not substantially improve performance. The advice about reasonable compiler flags has pretty consistently been "Use -O2, -pipe, -march=$THE_CORRECT_SETTING_FOR_YOUR_CPU, and maybe -fomit-frame-pointer"... though the "-fomit-frame-pointer" advice has always come with a warning that doing that may make debugging either extremely difficult or impossible, and the "-march" advice the warning that this may make it impossible to run those binaries on another computer.
Anyway. There's only so much you can do to erase an -allegedly- pervasive myth. The fact that some folks have the belief that Gentoo Linux is all about Maxxximum Performance doesn't change the fact that nearly all of the people who use it and -almost certainly- all of the people who maintain it strongly disagree with that claim. The distro is and always has been about flexibility, user choice, and having fun. And it's a nice bonus if building for -say- -march=athlon-xp instead of -march=i586 happens to get you a bit of extra performance here and there.
[0] <https://www.gentoo.org/get-started/about/>
[1] <https://web.archive.org/web/20230101024843/https://www.gento...>
[2] <https://web.archive.org/web/20150406115846/https://www.gento...>
[3] That year's April 1 redesign was a bit cool, IMO.
[4] The oldest copy that I can easily find is from mid 2013 [5], so you'll just have to trust me that this has been the long-standing opinion of the overwhelming majority of users and maintainers. While you're there, take a minute or two to skim the prose and notice how it consistently warns against using aggressive compiler flags.
[5] <https://web.archive.org/web/20130829083155/https://wiki.gent...>
> Extreme configurability, performance, and a top-notch user and developer community are all hallmarks of the Gentoo experience.
It's the performance part I was referring to.
Going back to when I started using it:
> Gentoo Linux is a high-performance ports-based Linux metadistribution
Regarding -O2 vs -O3, in the early days, -O3 was considered quite safe (I don't recall if it was recommended, but it wasn't discouraged). I think only after a certain GCC upgrade did they start discouraging -O3. I vaguely recall them announcing that quite a few of the -O3 benefits were now part of -O2, and that some (new?) -O3 stuff was unstable.
At the time, I knew little about compiler flags. I went with -O3 because of guidance from the Gentoo folks.
Yep, when I started using it in _2002_ that was on the home page. Ten years ago, it stopped being on the home page because folks finally got the motivation to do a bunch of work to it. As I said:
> Regarding -O2 vs -O3, in the early days, -O3 was considered quite safe (I don't recall if it was recommended, but it wasn't discouraged). ... I went with -O3 because of guidance from the Gentoo folks.It has been too long for me to remember what -O level my first Gentoo box ended up using (and I can't be bothered to pull it out of the closet and power it on to see), but the box I installed in ~2007 is using -O2. A random sampling of the few Gentoo forum posts from the early days that are in the Wayback Machine [0], as well as a handful of Bugzilla reports from the 2002->2005 era _suggests_ that most folks in the early days were using -O2. I'm certain that I played with all of the optimization levels on my first box, didn't notice any performance difference, and then settled on one that I can no longer remember.
And like, you say that -O3 wasn't discouraged. Well, -O3 isn't really discouraged today. The advice about -O3 gives you a summary of the facts and trusts you to make the choice that's right for you [1][2]
(I'm pretty sure that "unless running test suites" thing means "you shouldn't use this optimization level unless you're also using FEATURES="test" to run the package's tests".)FWIW, it's my maybe-incorrect understanding that many (most? all?) packages that are known-broken with specific flags get those flags filtered out and replaced with known-good ones... unless they're nutball "ricer" flags, in which case the user gets told "Don't do that!".
[0] <https://web.archive.org/web/20020922020122/https://forums.ge...>
[1] <https://wiki.gentoo.org/wiki/GCC_optimization#-O>
[2] Even the advice from a decade ago [3] is substantially similar: «This makes compilation take longer, and it not only might not make things faster, it may make things slower and is known to break things sometimes.»
[3] <https://wiki.gentoo.org&#x...
It is now but in the 32-bit days there was a nontrivial boost form compiling for e.g. a Pentium 4 when most binary distros were still stuck targetting generic i586 or something old like that.
Now though? It's way easier to have a herd of unicorn docker containers running that each handle one weird thing in isolation.
The bigger problem, to me, is that it’s not really clear why this is actually a problem for Gentoo. I am glad they perform user-facing source builds, however, because they see things that other distributors don’t.
Linux wouldn't exist if people submitted to that argument.
There's not much of an argument for adopting obscure niche practices for their own sake
The Ubuntu users are only there because after they had told the rest of us how silly we were for using Linux they eventually had to use it themselves for work.
Now we're going to listen to that sort of stuff again without rolling our eyes? ....
I started using linux in 1994 with slackware. Nowadays I use ubuntu, I've spent too many hours of my life tweaking my linux installs.
Nevertheless I remember lots of my friends who started off saying Linux was hopeless and ended up using it and Ubuntu was their gateway. The Linux world now is full of people who are fairly conformist and don't care about how it all began and who don't value the idea that open source has margins from which who knows what world beating things might eventually come. The fact that margins can exist is what makes open source great.
building from source has uses which you believe are worthless and other people don't think are worthless.
Anyone who uses cargo or installs a module for golang is building from source. But that, probably, is considered useful. It's only people who build C code from source who are niche and wierd. ....
By just adding
to most GNU/Linux distros to the grub kernel boot options (trivial even for Fedora Silverblue, just run ) and append at the end of these boot flags you are seeing the mentioned string. Done, multimedia ready desktop for tons of tasks outperforming Windows 11.If you insist on acting like a normal person, no one stops you from using something normal.
> The other reason is performance and that argument makes even less sense. Even if it mattered, we’re realistically talking about a difference in the ballpark of 100 ms, for an action executed rarely; it is equally possible that any gain will be outweighed by the time needed to fetch the 1.5 MiB wheel for uv-build.
Modern CI architectures are building Docker stacks from scratch and using a readthrough cache of wheels. Downloading may still be the dominant factor, but the second-dominant factor does start to add up, especially when the downloading gets cached at various layers.
https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-python/uv...
Looks like it's being built with fat LTO by default (and gentoo patched to thin LTO). Which is fine if you want to spend CI cycles once to ship executables to thousands of users. If you compile from source and run the tool maybe once then they should consider going further and run with maybe opt-level=1, lto=off; halfway between release and debug. This is how rustc itself builds some internal buildtools that get recompiled frequently relative to how often they run.
If only there were ways to skip this waste...
btw, I know of a guy who using extensive amounts of tokens to make his fork of mrustc usable via "claude, make this rustc compile 99% of crates.io", I think this also can yield some results.
(If you’re not using uv, you’ve made the wrong choice, but that’s a separate concern.)