would have been useful for something we ran into at a previous job. For legacy reasons large csv files had been zipped instead of gzipped (or some other streamable format), and were causing some process in our data pipeline to consume too much memory.
I wound up coming up with a quick solution that appeared to work. Taking advantage of being able to get s3 objects at arbitrary ranges, I implemented seek and read on top.
Last I was dealing with zip files in Python 3 years ago, I was unpleasantly surprised that there is no way to open Zip64 archives.
It is a semi-proprietary format not supported by zlib. Some tools, most notably Windows will sometimes use it to create large zip archives that go beyond limits of original zlib. An unsuspecting user would compress some files and there is no warning which format has been used.
3 comments
[ 12.0 ms ] story [ 112 ms ] threadI wound up coming up with a quick solution that appeared to work. Taking advantage of being able to get s3 objects at arbitrary ranges, I implemented seek and read on top.
It is a semi-proprietary format not supported by zlib. Some tools, most notably Windows will sometimes use it to create large zip archives that go beyond limits of original zlib. An unsuspecting user would compress some files and there is no warning which format has been used.
I think Zip64 _is_ supported by Python, and doesn't really involve anything at the zlib level(?)
(The reason I'm asking: a recent answer at https://stackoverflow.com/a/67579486/1319998 mentions Deflate64 and Windows)