Today we are glad to introduce support for a new compression algorithm in Percona XtraBackup 8.0.30 – Zstandard (ZSTD).
Results shows that ZSTD not only overcame LZ4 results on all tests, but it also brought backup size to half of its original size.
When streaming is added to the mix is when we see the biggest difference between both algorithms, with ZSTD overcoming LZ4 with an even bigger margin.
This can bring users and organizations a huge amount of savings in backup storage, either on-premises or especially in the cloud – where we are charged for each GB of storage we use.
“All right, but apart from the sanitation, the medicine, education, wine, public order, irrigation, roads, a fresh water system, and public health, what have the Romans ever done for us?”
So far, the societal benefits from Facebook, WhatsApp, and Instagram have been net positive, because the massive amounts of data they're collecting have not yet been bulk misused for mass torture, society-wide oppression, or a large-scale war.
So we've got a system that backs up to an XML file nightly. I noticed a couple weeks ago the bucket was using a not-insignificant amount of disk space.
These XML files are full backups rather than deltas - so each one contains the full previous file plus the new additional data.
My assumption was if I grabbed just a swath of the old ones, say a years worth and compressed them together I would get really decent savings.
I was pretty disappointed - gzip knocked 10 gb off of about 100 gb of data. I started doing some research and found people saying 7zip and it's sliding dictionary size options were the answer. After multiple tries, each run of 7zip taking multiple days I was able to get it down to about 70 gigabytes from 100. Better then gzip but frankly nowhere near what I would expect.
Does there exist a compression that could better handle this sort of expanding documents?
You can’t embed binary data in xml raw, so a common pattern is embedding it as base64, or similar type of wrapping.
Technically, it’s possible to escape it in other ways, but it’s error prone.
Either way, XML which has text strings or whatever typical XML document data that ISN’T something like base64 encoded random data should compress dramatically better than what the poster was talking about.
You need what's called "solid compression", meaning that the files are concatenated together before compression. Because otherwise compression restarts from zero for each new file - https://en.wikipedia.org/wiki/Solid_compression
Some compressors have options for solid compression, or you can use tar to first concatenate the files. For both, you need to sort the files first, so that they are compressed in chronological order, otherwise the common information will drop out of the dictionary.
And how big is one file and how much RAM you have? Because you might need increasing the "dictionary size" of the compression algorithm, both 7zip and zstd support multi-gigabyte sizes.
Yeah. We used to store PowerDesigner files in git a while back, which are XMLs usually smaller than 100 MB, and that was pretty much a disaster of hour taking clones already. I can't imagine 700 MB working.
To expand on the dedup a bit, Borg uses content aware chunking, so any large enough repeats will be fully deduplicated without relying on any specific block size/alignment. This is amazing for "slightly changing files".
"A compression utility that excels at compressing large files (usually > 10-50 MB). Larger files and/or more free RAM means that the utility will be able to more effectively compress your files (ie: faster / smaller size), especially if the filesize(s) exceed 100 MB. You can either choose to optimise for speed (fast compression / decompression) or size, but not both."
A while back I switched from lz4 to zstd compression with borg for backups. I've always appreciated zstd but the deduplication of borg + zstd is such a dramatic difference in storage space. My home "server" is a tiny NUC-like AMD Ryzen 7 4800U and it flies with this configuration.
What's surprising to me is that zstd beat out lz4 on compress/decompress speed. LZ4 is supposed to be purpose built to optimize on those metrics. Great job zstd developers to get perf that good!
The lz4 developers are the zstd developers :) In this case I suspect the IO reductions on the output are what matter -- while lz4 is faster, it also produces more output (of course) which you then have to write to disk. This can make the wall time take longer... basically you become IO bound on output, be it network or disk.
zstd also can tune itself and adjust ratios to saturate output bandwidth, which is pretty cool (--adapt).
That is exactly it. The difference comes from compression ratio. ZSTD is compressing the data more, so you need to write less back to disk.
Also when talking about streaming, the difference is even more as it has to go over wan to S3 (provider used on blog test).
My main pc is a MacBook 2013 and brtfs with zstd has changed the way I use it drastically. It basically doubles its storage for common day things. Sure movies and audio and such don’t compress but everything else does. Makes it feel a tad snappier as well.
What else do you store besides movies and audio so that it becomes noticeble for you? I also use btrfs with zstd, but mainly see the difference for OS/apps files. Movies and audio don't compress as you noted, and code and scripts take tiny space anyway.
We did a similar implementation of Zstd on ScyllaDB some years back. You can see how it compared to LZ4, Snappy and Deflate.
The interesting thing in comparing compression algorithms is — "what are you optimizing for?" Compression speed? Decompression speed? Storage savings? There are Tradeoffs. Getting the most storage savings might be suboptimal with a database if it adds too much latency or hampers throughput.
You may get a quite different answer in 2022 as there has been significant work on the raw performance of the zstd code. On x86 with BMI2 it is noticeably faster.
The biggest compression you can typically get is if you use structural sharing between backup versions, e.g. using hardlinks or specific filesystem support.
...compared to LZ4, not exactly honest here, LZ4 on average have worse compression than GZIP, it's just faster.
We actually went back (SQL backups, not xtrabackup) because weirdly enough bzip2 made smaller files when we aimed for compression ratio with similar compress time as bzip2. THink zstd got a bit faster since then tho...
Only for MySQL, PostgreSQL were noticeably smaller with zstd
Sorry: if there is absolute maximum bit entropy that cannot be reduced to a generative function, then there can be no (lossless) compression.
This is like the claims of tape drive manufacturers who claimed double the capacity of the media.
Pure marketing wank.
If you're not already compressing and encrypting your backups, then you're not doing it right to begin with. zstd may not be the most efficient algorithm for a given dataset. There are hundreds of compressors out there. It's worth finding the best one on average for a given job and occasionally revalidating it against other candidates.
53 comments
[ 3.3 ms ] story [ 118 ms ] threadResults shows that ZSTD not only overcame LZ4 results on all tests, but it also brought backup size to half of its original size.
When streaming is added to the mix is when we see the biggest difference between both algorithms, with ZSTD overcoming LZ4 with an even bigger margin.
This can bring users and organizations a huge amount of savings in backup storage, either on-premises or especially in the cloud – where we are charged for each GB of storage we use.
https://facebook.github.io/zstd/
Maybe React too, I'm not a front-end person.
Cons: created a post-truth world which destroyed democracy and killed millions of people by spreading antivaxx propaganda
Eh.
I can't think of any real pros to Twitter. Did they create any great new open-source technologies?
Now Airbnb, those people caused untold suffering by both their impact on housing market, and by creating Airflow ...
You might as well be blaming highways and radio - Facebook made it more accessible, but far from created it.
The game isn't over yet, however.
These XML files are full backups rather than deltas - so each one contains the full previous file plus the new additional data.
My assumption was if I grabbed just a swath of the old ones, say a years worth and compressed them together I would get really decent savings.
I was pretty disappointed - gzip knocked 10 gb off of about 100 gb of data. I started doing some research and found people saying 7zip and it's sliding dictionary size options were the answer. After multiple tries, each run of 7zip taking multiple days I was able to get it down to about 70 gigabytes from 100. Better then gzip but frankly nowhere near what I would expect.
Does there exist a compression that could better handle this sort of expanding documents?
Is the XML wrapping a bunch of other random data or something?
As an example, I have zstd enabled on some zfs pool. The client-side encrypted time machine backups does not even compress 1 %, as expected.
You can’t embed binary data in xml raw, so a common pattern is embedding it as base64, or similar type of wrapping.
Technically, it’s possible to escape it in other ways, but it’s error prone.
Either way, XML which has text strings or whatever typical XML document data that ISN’T something like base64 encoded random data should compress dramatically better than what the poster was talking about.
So, what the hell is in your XML anyway?
Some compressors have options for solid compression, or you can use tar to first concatenate the files. For both, you need to sort the files first, so that they are compressed in chronological order, otherwise the common information will drop out of the dictionary.
And how big is one file and how much RAM you have? Because you might need increasing the "dictionary size" of the compression algorithm, both 7zip and zstd support multi-gigabyte sizes.
I've got 64gb of ram and 10 cores to work with, but it seems single core compression might be in order?
Yes: storing deltas. Git might work well. A backup system that does subfile deduplication may work too (e.g. Restic, Borg).
I was under the impression that git didn't do well with large files.
I've used it to store 20-80GB of files in a repo and it works quite well.
Long Range ZIP or LZMA RZIP
https://github.com/ckolivas/lrzip
"A compression utility that excels at compressing large files (usually > 10-50 MB). Larger files and/or more free RAM means that the utility will be able to more effectively compress your files (ie: faster / smaller size), especially if the filesize(s) exceed 100 MB. You can either choose to optimise for speed (fast compression / decompression) or size, but not both."
Impressive!
zstd also can tune itself and adjust ratios to saturate output bandwidth, which is pretty cool (--adapt).
The interesting thing in comparing compression algorithms is — "what are you optimizing for?" Compression speed? Decompression speed? Storage savings? There are Tradeoffs. Getting the most storage savings might be suboptimal with a database if it adds too much latency or hampers throughput.
Nicely done, Percona!
https://www.scylladb.com/2019/10/07/compression-in-scylla-pa...
We actually went back (SQL backups, not xtrabackup) because weirdly enough bzip2 made smaller files when we aimed for compression ratio with similar compress time as bzip2. THink zstd got a bit faster since then tho...
Only for MySQL, PostgreSQL were noticeably smaller with zstd
This is like the claims of tape drive manufacturers who claimed double the capacity of the media.
Pure marketing wank.
If you're not already compressing and encrypting your backups, then you're not doing it right to begin with. zstd may not be the most efficient algorithm for a given dataset. There are hundreds of compressors out there. It's worth finding the best one on average for a given job and occasionally revalidating it against other candidates.
PSA: Untested backups aren't backups.