My take (I compare about ZSTD only, as that what we need to care about):
1. Lizard's decompression speeds are much higher than ZSTD for any level of compression.
2. Lizards's compression speed is significantly (several times) lower than ZSTD for similar compression ratios.
So it makes sense if we mostly don't care about CPU usage for compression.
if it's faster for both compression and decompression it just seems better period? why do you say "if we mostly don't care about CPU usage"? being faster means less CPU usage...
Faster/slower aside, it doesn't matter if it is slower to compress, because you only do that once for most use cases (thinking of downloadable repos or document stores). Decompression happens much more frequently. So, one compression algo that takes a lot of work (and time) to compress but uncompresses with less work than anything else is desirable because of the fact you only need compress once. So, less compute is required.
There could be some use cases where this doesn't work out, such as a compressed network connection, where everything has to be compressed/decompressed in a 1:1 ratio.
The thing I don't get about zstd is that their own github page shows lz4 is faster at both compression and decompression at the cost of some compression ratio: https://github.com/facebook/zstd
But most people I work with will pick zstd every time even in cases where decompression speed matters the most.
> If you are bottle-necked by reading data (eg over networks or slow storage), zstd will be faster to decompress since you are transferring less data.
This should be stressed. Benchmarks show zstd's compression ratio being 75% of lz4's. Pushing 25% less data over a network is enough to win any real-world test, not to mention the impact it has on operational costs.
I have this pathological example, I have a 30GB journal dump from bcachefs, and compressed it down with lz4 and zstd to a sata SSD, the compression ratios are 6.8% and 3% respectively (and this with default level 3!), it takes 14s to ripgrep through the lz4 version, and 11s with the zstd version. So even at 500MB/s zstd is faster for decompression.
> But most people I work with will pick zstd every time even in cases where decompression speed matters the most.
Zstd is really good. If you're a math/compression theory nerd it makes a lot of good choices. Also, if you're willing to tune zstd instead of using the defaults, so far anyway, I haven't found any application where the right answer wasn't zstd.
At this point I don't think it's unreasonable to expect new compression developments to to explicitly say "What is improved over ZStandard".
You're entirely right! And points out the blind spot that I've used it much more as a library in tools I build/use than integrated with other things. Never used brtfs, I'm assuming it uses zstd internally for something? Cool.
Also very cool that qcow contains zstd support (didn't know that either!).
> Btrfs supports transparent file compression. There are three algorithms available: ZLIB, LZO and ZSTD(since v4.14)
I've been using zstd compression for qcow2 images since last year, but I admit it's kind of hidden. The man page for qemu-img [1] doesn't say a word about zstd, but it's there alright [2].
That's the thing, even at its fastest setting, zstd is still slower than lz4. So clearly, it's not the right answer if you're serializing from a fast medium.
And this is exactly the kind of cognitive laziness I'm talking about.
> That's the thing, even at its fastest setting, zstd is still slower than lz4. So clearly, it's not the right answer if you're serializing from a fast medium.
It really depends on what you want to do, as decompression speeds can be meaningless and irrelevant in some applications, if not practically all of them.
Can you provide a specific example where you feel lz4 is hands down the best option?
> I shouldn’t have to keep explaining this. zstd isn’t some magic answer to everything.
That logic doesn't really work. It's true that zstd doesn't let you tune it to be as fast as lz4. But it's possible that a slightly different design could make that possible, so that the new version is better than lz4 at all speeds and at all ratios. Making one compression algorithm strictly better than another requires no magic.
Note that a benchmark in the README refers to zstd 1.1.1 and brotli 0.5.2, which are very old (the current versions are zstd 1.5.2 and brotli 1.0.9). The same author maintains lzbench [1] which is more current than the README. It shows that Lizard heavily trades off compression speed and ratio for decompression speed; zstd at the default setting (-3) is expected to be on par with Lizard -45 in terms of ratio, but Lizard would be >10x slower to compress while only ~3x faster to decompress.
as in you run the bitmap through QOI and then compress that output with LZ4? Or am i misremembering how QOI works and it uses LZ4 internally? PNG uses LZ, doesn't it?
27 comments
[ 3.0 ms ] story [ 71.8 ms ] threadSo it makes sense if we mostly don't care about CPU usage for compression.
"compression speed is lower"
To me this reads as though it's faster. Maybe you meant "compression speed is Slower"?
Disclaimer: I generally dislike being pedantic, especially on HN. I hope this message doesn't come across that way.
If compression time is lower, then it's faster.
There could be some use cases where this doesn't work out, such as a compressed network connection, where everything has to be compressed/decompressed in a 1:1 ratio.
But most people I work with will pick zstd every time even in cases where decompression speed matters the most.
This should be stressed. Benchmarks show zstd's compression ratio being 75% of lz4's. Pushing 25% less data over a network is enough to win any real-world test, not to mention the impact it has on operational costs.
Zstd is really good. If you're a math/compression theory nerd it makes a lot of good choices. Also, if you're willing to tune zstd instead of using the defaults, so far anyway, I haven't found any application where the right answer wasn't zstd.
At this point I don't think it's unreasonable to expect new compression developments to to explicitly say "What is improved over ZStandard".
Also very cool that qcow contains zstd support (didn't know that either!).
> Btrfs supports transparent file compression. There are three algorithms available: ZLIB, LZO and ZSTD(since v4.14)
I've been using zstd compression for qcow2 images since last year, but I admit it's kind of hidden. The man page for qemu-img [1] doesn't say a word about zstd, but it's there alright [2].
[1]: https://www.qemu.org/docs/master/tools/qemu-img.html
[2]: https://gitlab.com/qemu-project/qemu/-/blob/v7.0.0/block/qco...
And this is exactly the kind of cognitive laziness I'm talking about.
It really depends on what you want to do, as decompression speeds can be meaningless and irrelevant in some applications, if not practically all of them.
Can you provide a specific example where you feel lz4 is hands down the best option?
And again, I shouldn’t have to keep explaining this. zstd isn’t some magic answer to everything.
That logic doesn't really work. It's true that zstd doesn't let you tune it to be as fast as lz4. But it's possible that a slightly different design could make that possible, so that the new version is better than lz4 at all speeds and at all ratios. Making one compression algorithm strictly better than another requires no magic.
[1] https://github.com/inikep/lzbench
Yes, it can be very beneficial.