Sounds good. If it works well, I'd love to see it become the new default on all *nix distros. It's about time some of these basic tools get updated for multi-core hardware.
The next parallelized compressor I'd like to see is LZMA. It's basically Lempel-Zip, like gzip, but with a much wider window (tens of megs, vs a few dozen k), and because it's dictionary-based compression it decompresses much faster than bzip.
What's the benefit of that? I see downsides: have to deal with the FSF, have to work with the existing codebase, have to fight to get the changes accepted, etc. It can always be ported into gzip if that becomes desirable.
We used this in SUSE Studio to decrease build time. Worked well.
There's also MPIBZIP2 which uses (IIRC) libmpi to do bzip2 compression/decompression across several machines. We messed with that too, but it wasn't worth the added complexity.
Just out of interest, does anyone have any numbers on whether or not the problems with gzip are more i/o bound or cpu bound? I would've thought with gzip i/o would've been a bigger factor, but am happy to be proven wrong.
23 comments
[ 2.8 ms ] story [ 57.0 ms ] threadWhen we changed NASA Nebula from XEN to KVM we had to tweak a few files in every image, this sped up the conversion by minutes per image.
It's definitely needed since XZ is fairly slow.
Australians would pronounce it as "pig-zed". Just one of those novelties I guess... :)
http://en.wikipedia.org/wiki/Monkey_(TV_series)
(I quite dislike the name, myself. And I think everyone's actually going to pronounce it "pigs".)
One of the main reasons I stay the hell away from OSS unless I can fork it.
There's also MPIBZIP2 which uses (IIRC) libmpi to do bzip2 compression/decompression across several machines. We messed with that too, but it wasn't worth the added complexity.