> This has been done in response to the discovery that the popular installer uv has a different extraction behavior to many Python-based installers that use the ZIP parser implementation provided by the zipfile standard library module.
> For maintainers of installer projects: Ensure that your ZIP implementation follows the ZIP standard and checks the Central Directory before proceeding with decompression. See the CPython zipfile module for a ZIP implementation that implements this logic. Begin checking the RECORD file against ZIP contents and erroring or warning the user that the wheel is incorrectly formatted.
Good to know that I won't need to work around any issues with `zipfile` — and it would be rather absurd for any Python-based installer to use anything else to do the decompression. (Checking RECORD for consistency is straightforward, although of course it takes time.)
... but surely uv got its zip-decompression logic from a crate rather than hand-rolling it? How many other Rust projects out there might have questionable handling of zip files?
> PyPI already implements ZIP and tarball compression-bomb detection as a part of upload processing.
... The implication is that `zipfile` doesn't handle this. But perhaps it can't really? Are there valid uses for zips that work that way? (Or maybe there isn't a clear rule for what counts as a "bomb", and PyPI has to choose a threshold value?)
Related to multiple .zip formats: I've found macOS Archive Utility sometimes refuses to extract early pkzip .zips created on MS-DOS, but yet Info-ZIP handles them just fine.
And, the macOS Archive Utility will complain that a proper .tar.bz2 is "corrupt" created using bzip2.
In general, be liberal in input and be conservative in output. Sometimes, this means using less features or certain older formats so that all/most things work without issues.
> In general, be liberal in input and be conservative in output.
That is a dangerous maxim in a world with malicious players. In fact this PyPI problem is precisely because zip files are being too readily accepted, even if they have ambiguous meaning. Their fix is (very sensibly) to be less liberal with their input.
5 comments
[ 0.26 ms ] story [ 29.8 ms ] thread> For maintainers of installer projects: Ensure that your ZIP implementation follows the ZIP standard and checks the Central Directory before proceeding with decompression. See the CPython zipfile module for a ZIP implementation that implements this logic. Begin checking the RECORD file against ZIP contents and erroring or warning the user that the wheel is incorrectly formatted.
Good to know that I won't need to work around any issues with `zipfile` — and it would be rather absurd for any Python-based installer to use anything else to do the decompression. (Checking RECORD for consistency is straightforward, although of course it takes time.)
... but surely uv got its zip-decompression logic from a crate rather than hand-rolling it? How many other Rust projects out there might have questionable handling of zip files?
> PyPI already implements ZIP and tarball compression-bomb detection as a part of upload processing.
... The implication is that `zipfile` doesn't handle this. But perhaps it can't really? Are there valid uses for zips that work that way? (Or maybe there isn't a clear rule for what counts as a "bomb", and PyPI has to choose a threshold value?)
And, the macOS Archive Utility will complain that a proper .tar.bz2 is "corrupt" created using bzip2.
In general, be liberal in input and be conservative in output. Sometimes, this means using less features or certain older formats so that all/most things work without issues.
That is a dangerous maxim in a world with malicious players. In fact this PyPI problem is precisely because zip files are being too readily accepted, even if they have ambiguous meaning. Their fix is (very sensibly) to be less liberal with their input.