Unfortunately it's not perfectly supported. If you set up a private registry on gitlab and try to push packages with epoch versions, gitlab will normalize the ! to _ so everything breaks. (We found this out the hard way recently.)
Another Python versioning gotcha I've never see discussed is use of instructions like AVX2 in C extensions. There seems to be no way in Python versioning to say "this version is for CPUs with AVX2 instructions" so I guess public C extensions have to be compiled for the least common denominator instruction set. Does anyone know otherwise? How do other packaging systems like cargo deal with this?
> If you set up a private registry on gitlab and try to push packages with epoch versions, gitlab will normalize the ! to _ so everything breaks. (We found this out the hard way recently.)
That's unfortunate; I know Poetry's support for epochs has also been spotty (at least historically, I'm not sure about currently).
> There seems to be no way in Python versioning to say "this version is for CPUs with AVX2 instructions" so I guess public C extensions have to be compiled for the least common denominator instruction set. Does anyone know otherwise?
This is my understanding as well: x86_64 in wheel tags implies the lowest common denominator for AMD64. I think the best you can generally do here is either (1) accept the pain that comes with sdists, or (2) do runtime feature detection for things like AVX2.
> How do other packaging systems like cargo deal with this?
Cargo always builds from source, so they effectively punt on this problem.
Or simply require an AVX2-capable CPU. According to Steam survey, 90% have AVX2 support. That's not as nice as the 99.18% SSE4.2 support or the 100% SSE3 support, but for some use cases such as scientific libraries, requiring a CPU that is less than 8 years old makes sense to me. Some more helpful error message than "Illegal Instruction" would still be a good idea though. This does require runtime AVX2 detection but at least it avoids the pain of trying to compile "pristine" architecture-specific code, which seems only possible for C/C++ code that does not use any standard library function or by writing assembler directly.
> Instead use the official packaging.version.Version class to parse, compare, and reason about Python versions.
I would have agreed with this previously, but a year or so ago they changed their backend and now it just errors out on certain (maybe not common, but extant) version strings, such as cloudera’s spark release version.
If I recall, it used to parse fine on their old backend parser. Sure, the maintainers can cast aspersions on uneducated devs from their ivory tower, amazed anyone would think this is a valid version. But things like this can and do show up in the wild, and the parsing package playing dumb (especially when it didn’t use to) doesn’t inspire a lot of confidence that this should be the tool you use.
Just parse the damn thing. It’s painfully obvious how to do so. I should have to write my own custom parser to wrap theirs. Sure, it’s nonstandard, but it should be handled as a best guess + a warning, not an exception. I didn’t choose this version string. Don’t punish me for trying to parse it.
"Just parse the damn thing" without a grammar leads to exploitable parser differentials[1]. Your "obvious" parse isn't everyone else's.
(Nobody anywhere has called developers "dumb" for using legacy, non-PEP 440 versions. There's nothing else they even could have done prior to version standardization across the Python ecosystem.)
I understand that the lack of standardization makes things difficult, but there are a lot of knowables.
Further, there would be nothing wrong with returning something like a NonconformingVersion object, where we can best guess at things like major, minor, patch. Parse what you think you know, and when you run into something that is nonconforming, then you fail over to return an object that has what you think you know and what you can’t sort out. E.g: in the above example, major, minor, patch versions are apparent. I would even say the vast majority of nonconforming versions have at least one or more of major/minor/patch readily parseable, the problem is we aren’t sure exactly how of handle the rest of it. And that’s fine.
This approach is super pythonic in the BTAFTP philosophy, where I will let users decide what to do with the object returned. “You asked me to parse this mess. I can’t fully do that, but here’s a best guess.” This is worlds better than just failing completely.
I understand the frustration here (I maintain a project that was directly broken by strict version checking), but I also think that reducing the problem to "best guess at major, minor, patch" understates the problem here. For example, what do you do with version strings like these?
Those last two are actual legacy Python versions[1] -- how should we compare between those two for the purposes of dependency resolution?
Whether or not it's preferable to attempt to parse unspecified inputs boils down to questions of predictability, reliability, and security and, IMO, security is an overriding factor when it comes to domains as critical as packaging.
That being said: you can attempt to parse legacy versions using this package[2].
When it comes to dependency resolution, that’s where it’s up to the application how to handle NonconformingVersion objects. For vanilla pip, it makes sense for it to treat these as errors.
I actually think it makes sense for the PyPI itself to be the one to dictate how to handle version policy via some callable or handler. Many companies have internal versions and mirrors of OSS code, and allowing them to preferentially shim these in (and reasonably version) these for their employees. Let’s say you fork the requests library. So now your versioning is requests=2.31.0.corporate-31. Sure, it makes no sense externally, but as the internal PyPI authority, you can enforce this versioning. Then your users just point to your index and get what they need (even if they just ask for requests=2.31.0).
In general, parsing and dependency resolution are two separate issues. The concerns of dependency resolution should not bleed into parsing. So it should be a valid use case to simply want to parse and compare two versions, and if it seems we have complete enough information, then make a truth estimation. E.g. in the above example, this should realistically not fail, but instead throw a NonconformingVersion warning:
> assert parse(‘2.4.0.coudera2’) >= parse(‘2.4’)
While we can’t parse the full string, we can make a really good, consistent guess at what the user is after. And if needed, parse could reasonably take an optional callable to support handling of nonconforming versions to convert them to conforming ones.
Other libraries are available: distlib is even part of PyPA, but apparently its approach to trying out new things put some people off, so it's not "blessed". It supports legacy, standard and semver versions in its version module [0].
>>> from distlib.version import LegacyVersion, NormalizedVersion
>>> NormalizedVersion('2.4.0cloudera2')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/disk2/vinay/projects/scratch/distlib/distlib/version.py", line 33, in __init__
self._parts = parts = self.parse(s)
^^^^^^^^^^^^^
File "/disk2/vinay/projects/scratch/distlib/distlib/version.py", line 276, in parse
result = _normalized_key(s)
^^^^^^^^^^^^^^^^^^
File "/disk2/vinay/projects/scratch/distlib/distlib/version.py", line 188, in _pep_440_key
raise UnsupportedVersionError('Not a valid version: %s' % s)
distlib.version.UnsupportedVersionError: Not a valid version: 2.4.0cloudera2
>>> LegacyVersion('2.4.0cloudera2')
LegacyVersion('2.4.0cloudera2')
> to parse, compare, and reason about Python versions.
If in addition to the above you want to mutate Python versions, you may consider my parver package[1]. It also doesn't normalise the input unless you ask it to, so you can preserve your preferred format after mutating.
I wrote a Clojure library that implemented version comparison for all of the package versioning algorithms I could find[1], and by faaar the Python version comparison algorithm was the weirdest, most annoying one to implement (PEP 440[2], the current standard at the time). So many exceptions to the rules that attempt to be helpful but didn't really help.
In my opinion, the best version comparison algorithm is that of the Debian package manager. Super simple, and very flexible. The description of the algorithm fits in just three paragraphs[3]. Most other algorithms implemented in the aforementioned library were actually implemented in terms of it. Prereleases, fourth version numbers, etc. anything you want basically fits nicely in that box.
PEP 440 was the only version comparison algorithm that couldn't be neatly implemented in terms of that of Debian's, beating out semver, maven, and rubygems for complexity.
PEP 440 is complicated, but it's also complicated in ways that make sense for a language packaging ecosystem: Debian is tasked primarily with distributing stable builds of software; PyPI is facilitating a source and build distribution relationship between different maintainers and users.
In other words: Debian prioritizes unified versioning and has a "fan-out" topology of users; PyPI prioritizes facilitating dependency resolution for a "graph" topology of users.
OT, but it took me an embarrassing amount of time before I realized that '5.5.1' != '5.5.10'. Maybe it's the fact that there isn't a leading zero, like .01, or something, but it was a moment of "wait...did I know that? Is that right?"
Trivia: the package with the longest version number is uselesscapitalquiz (https://pypi.org/project/uselesscapitalquiz), version 3.14159265358979323846264338327950288419716939937510582097494459230781640628620899862803482534211706798214808651328230664709384460955058223172535940812848111745028410270193852110555964462294895493038196442881097566593
This is mentioned in the post, but the actual package was not identified.
16 comments
[ 1.9 ms ] story [ 55.3 ms ] threadAnother Python versioning gotcha I've never see discussed is use of instructions like AVX2 in C extensions. There seems to be no way in Python versioning to say "this version is for CPUs with AVX2 instructions" so I guess public C extensions have to be compiled for the least common denominator instruction set. Does anyone know otherwise? How do other packaging systems like cargo deal with this?
That's unfortunate; I know Poetry's support for epochs has also been spotty (at least historically, I'm not sure about currently).
> There seems to be no way in Python versioning to say "this version is for CPUs with AVX2 instructions" so I guess public C extensions have to be compiled for the least common denominator instruction set. Does anyone know otherwise?
This is my understanding as well: x86_64 in wheel tags implies the lowest common denominator for AMD64. I think the best you can generally do here is either (1) accept the pain that comes with sdists, or (2) do runtime feature detection for things like AVX2.
> How do other packaging systems like cargo deal with this?
Cargo always builds from source, so they effectively punt on this problem.
I would have agreed with this previously, but a year or so ago they changed their backend and now it just errors out on certain (maybe not common, but extant) version strings, such as cloudera’s spark release version.
> packaging.version.InvalidVersion: Invalid version: '2.4.0.cloudera2'
If I recall, it used to parse fine on their old backend parser. Sure, the maintainers can cast aspersions on uneducated devs from their ivory tower, amazed anyone would think this is a valid version. But things like this can and do show up in the wild, and the parsing package playing dumb (especially when it didn’t use to) doesn’t inspire a lot of confidence that this should be the tool you use.
Just parse the damn thing. It’s painfully obvious how to do so. I should have to write my own custom parser to wrap theirs. Sure, it’s nonstandard, but it should be handled as a best guess + a warning, not an exception. I didn’t choose this version string. Don’t punish me for trying to parse it.
(Nobody anywhere has called developers "dumb" for using legacy, non-PEP 440 versions. There's nothing else they even could have done prior to version standardization across the Python ecosystem.)
[1]: https://blog.yossarian.net/2022/05/09/A-most-vexing-parse-bu...
Further, there would be nothing wrong with returning something like a NonconformingVersion object, where we can best guess at things like major, minor, patch. Parse what you think you know, and when you run into something that is nonconforming, then you fail over to return an object that has what you think you know and what you can’t sort out. E.g: in the above example, major, minor, patch versions are apparent. I would even say the vast majority of nonconforming versions have at least one or more of major/minor/patch readily parseable, the problem is we aren’t sure exactly how of handle the rest of it. And that’s fine.
This approach is super pythonic in the BTAFTP philosophy, where I will let users decide what to do with the object returned. “You asked me to parse this mess. I can’t fully do that, but here’s a best guess.” This is worlds better than just failing completely.
Whether or not it's preferable to attempt to parse unspecified inputs boils down to questions of predictability, reliability, and security and, IMO, security is an overriding factor when it comes to domains as critical as packaging.
That being said: you can attempt to parse legacy versions using this package[2].
[1]: https://pypi.org/project/paramiko/#history
[2]: https://pypi.org/project/packaging-legacy/
I actually think it makes sense for the PyPI itself to be the one to dictate how to handle version policy via some callable or handler. Many companies have internal versions and mirrors of OSS code, and allowing them to preferentially shim these in (and reasonably version) these for their employees. Let’s say you fork the requests library. So now your versioning is requests=2.31.0.corporate-31. Sure, it makes no sense externally, but as the internal PyPI authority, you can enforce this versioning. Then your users just point to your index and get what they need (even if they just ask for requests=2.31.0).
In general, parsing and dependency resolution are two separate issues. The concerns of dependency resolution should not bleed into parsing. So it should be a valid use case to simply want to parse and compare two versions, and if it seems we have complete enough information, then make a truth estimation. E.g. in the above example, this should realistically not fail, but instead throw a NonconformingVersion warning:
> assert parse(‘2.4.0.coudera2’) >= parse(‘2.4’)
While we can’t parse the full string, we can make a really good, consistent guess at what the user is after. And if needed, parse could reasonably take an optional callable to support handling of nonconforming versions to convert them to conforming ones.
If in addition to the above you want to mutate Python versions, you may consider my parver package[1]. It also doesn't normalise the input unless you ask it to, so you can preserve your preferred format after mutating.
[1]: https://github.com/RazerM/parver
In my opinion, the best version comparison algorithm is that of the Debian package manager. Super simple, and very flexible. The description of the algorithm fits in just three paragraphs[3]. Most other algorithms implemented in the aforementioned library were actually implemented in terms of it. Prereleases, fourth version numbers, etc. anything you want basically fits nicely in that box.
PEP 440 was the only version comparison algorithm that couldn't be neatly implemented in terms of that of Debian's, beating out semver, maven, and rubygems for complexity.
1: http://djhaskin987.gitlab.io/serovers/index.html
2: https://peps.python.org/pep-0440/
3: https://www.debian.org/doc/debian-policy/ch-controlfields.ht...
In other words: Debian prioritizes unified versioning and has a "fan-out" topology of users; PyPI prioritizes facilitating dependency resolution for a "graph" topology of users.
It may have been designed for that use case, but so were these others. They also had simpler designs.
This is mentioned in the post, but the actual package was not identified.