Most hackers don't know about XZ yet, but they should. I still prefer not to see Wikipedia articles on HN since they lack context. A "7 reasons you should ditch bzip2 and use XZ" blog post would probably be better.
The HN Guidelines explicitly frown upon "X reasons..." titles because they're almost always linkbait.
If the original title begins with a number or number + gratuitous adjective, we'd appreciate it if you'd crop it. E.g. translate "10 Ways To Do X" to "How To Do X," and "14 Amazing Ys" to "Ys." Exception: when the number is meaningful, e.g. "The 5 Platonic Solids."
Agree on the wikipedia articles with no context point. Especially since the article in question is short with not very much real information on why it matters.
The main reason to continue to use gzip is speed: it's an order of magnitude faster than bzip2 for compression / decompression, for a relatively small compression ratio hit. It's fast enough to us inline in web servers, SSH sessions, etc, and if you're just moving data around a single time, often the amount of time for compress / transfer / decompress is much faster than with bzip2.
I've found that on many types of data, xz -2 is twice as fast and compresses better than gzip -9. Of course gzip -1 kicks the pants off xz but there's absolutely no reason to use gzip at its higher levels, nor to use bzip2 at all. And if you want fast, LZO does it better -- the lzop tool provides a typical unix compress interface.
Using 50MiB from a tarball I had lying around, best of 3:
Ugh. Can you try with a lighter compression level? There is a command line switch for that, and it should also show the memory requirements for each compression level.
I'm insisting because PAQ is a really incredible collection of compressors. Far more intelligence in it than in the LZ* bunch.
LZMA is much, much slower. It is simply a matter of the rate at which you generate data and the compression ratio and the rate at which you add more storage/move the files off to tape. Can you "keep up" essentially.
Right now, CPUs are fast enough that LZMA is a realistic prospect. It boils down to what the extra CPUs will cost vs the cost of the storage saved (incl. the "cost" of space in the datacentre, the administrative overhead of more hardware, etc).
I wonder if it would be possible to have page templates delivered separately from their dynamic content? The templating of the dynamic data would happen on the browser-side machine. This way, we could use really expensive compression algorithms with high CPU cost for compression, but then cache the compression result.
This could be achieved by using Javascript and fetching data as JSON, but it seems to be something that would be very useful and beneficial if it were standardized.
There is client-side XSLT already. The problem is that transformed template is usually same size or smaller than your source data.
Really expensive algorithms aren't that much better to offset cost of sending template logic to the client.
You could probably save some processing time by integrating compression directly into server-side templating - take advantage of the fact that some parts never change and keep them pre-compressed or at least cache some statistics about them to aid compression.
There is the SDCH spec: Shared Dictionary Compression over HTTP. Google Chrome uses it, and the Google Toolbar adds support to IE.
The server supplies a pre-created dictionary (template) which gets cached, and further requests get a diff to that dictionary as a response.
I've only found Google search using it on the server side, and the browser penetration is fairly low, but it seems promising. It makes requests very fast.
I've also noticed that Google search serves image thumbnails as data:// URLs directly in the original response. They are going a long way to ensuring each page loads entirely in one request.
More importantly, xz decompresses quickly. Essentially, you get the compression ratio of bzip2 at the decompression speed of gzip (and, unfortunately, the compression speed of bzip2).
EDIT: also, like gzip but unlike bzip2, you can stream data through xz (at an insignificant penalty in compression ratio).
In retrospect, I probably should have submitted the XZ Utils project page: http://tukaani.org/xz/ .
However, that page does not give the same picture of widespread adoption that the Wikipedia article does. That use was what interested me: I'm surprised that it only came to my attention when I was trying to recall the name of the 7z command line utility p7zip.
Given the increasing CPU/network gap, it's only a matter of time before the bandwidth (and thus time) saved by XZ more than compensates for the slower compression.
mileage may vary; If the time it takes to compress is costs much less than the time it takes to transfer over the network, you might want not want to use gzip. For example, you're transferring a large file (1GB? 1TB?) to a remote person to deal with, is it cheaper to gzip (lower compression rate), take longer for the network transfer (most likely slowest step), and have the other person unzip, or to use a better compressor, and have the file transferred over quicker?
So bzip2 and 7-zip are way, way slower than gzip, then?
Bandwidth is cheap. Stick to gzip.
It's not as simple as that. Which one is better depends on the use case. If you're sending a one-off file to somebody, sure, gzip is better. But if you want to distribute a file to a large number of people (like Linux distributions do with their packages), the extra CPU time is insignificant compared to the bandwidth saved over the course of thousands of downloads.
If you look at some of the stats people are posting, it's the compression that takes the most time, not the decompression. gzip has fast compression and decompression, which is why it's used for things like compressing network streams (http,ssh,etc). But when you want to package up large files for distribution to a large audience, then it makes more sense throw some extra CPU time at the compression to get a smaller package (so long as the decompression time on the other end is reasonable).
If you have to wait minutes to download the files it doesn't matter, but if you already have the file locally it is very annoying.
Also if this is used extensively on projects with a large server deployment this matters even more related to latency and energy consumption. That's why Google has their own compression algorithms derived from BMDiff and LZW (Zippy.) Think about it. Speed matters.
Decompression is more often limited by disk I/O, in my experience, particularly when the source and destination are the same disk. I can often get large improvements in decompression and installation speed by putting the source file and / or temporary installation files on a different disk.
It's not always I/O speed. You can notice when installing CPU usage goes to 100% (or fans kicking in) for BWT/LZM* and not for the DEFLATE (unless you use -9 or something like that.) While you install something at least one of your cores is unavailable for anything else.
This affects energy consumption, too.
And think about both mobile and servers. Those systems are usually more sensible to high CPU load.
I have a draft blog post with analysis of different protocols with valgrind and other tools. But it is so much data to present and graph I never get around to finish it :(
Are you willing to donate money to your favorite open source software so they can afford the bandwidth? If not, don't complain about having to spend a few more seconds decompressing the latest release (which you're getting for free).
As a programmer, I would rather work on a patent-unencumbered and open source compression algorithm solving exactly this problem. Perhaps investing months of my own unpaid time on it. HINT
The question in my mind is if 7-zip has a good multicore implementation for compression/decompression. I recall that the multicore implementation by the original gzip author increased the speed on a 4-core machine by something like a x3.75 boost.
The only reason I stick with gzip over bz2 or 7z (lzma) is that gzip is everywhere. In reality, nobody has a file compression utility that can handle LZMA (even though I do, I never use it).
Interestingly, file(1) can't identify lzma(1)-compressed files (they show up as data), but unlzma(1) reassuringly knows the difference between LZMA and random data. This came up once or twice in DEFCON CTF quals (http://www.vnsecurity.net/2010/05/defcon-18-quals-writeups-c...).
The specification ( http://tukaani.org/xz/xz-file-format.txt ) reads like this format allows the application of a chain of compression "filters". Perhaps the most interesting, and least portable is the filter that modifies code to make compression easier. (Section 5.2.3) Is this mainly for intel architecures?
I wrote a tool, pixz, that does xz compression in parallel to take advantage of multiple cores. It also indexes xz-compressed tarballs, so you can extract an individual file very quickly instead of needing to decompress the entire tarball. The parallel-compressed, self-contained tarball+index is fully compatible with regular tar/xz, you don't need any special tools to extract it.
It really depends on the data. For highly redundant data like web pages with lots of boilerplate header/footer, it can compress better because the bm_pack (first pass of bmz) looks for large common patterns over all the input. For typical text, it should be a little worse than gzip but faster.
BMZ = bmpack + lzo by default and can be combined with lzma if necessary. It's not really a BMDiff and Zippy clone, as I've never had a chance to see Google's implementation. It's based on the original Bentley & McIlroy paper: "Data Compression Using Long Common Strings", 1999. Even the two pass idea is from that paper. It was really a wacky experimental implementation (with a lot of room for improvement) to satisfy my curiosity. I'm a little surprised that the 0.1 version has been stable for quite a few people compressing TBs of data through it.
68 comments
[ 2.6 ms ] story [ 127 ms ] thread1. They're overdone
2. They're uncreative
3. They're unoriginal
4. They're a crutch
5. They're assembly-line writing
6. They're used by tabloids to appeal to supermarket zombies
7. They've helped turn respectable mags into tabloids (or are a symptom of it, not sure which came first)
8. They're psychologically manipulative (I don't know how, I just know it)
9. They work
:)
I do not mind lists, but I cannot stand lists that are seperated to pages for inflated page counts!
If the original title begins with a number or number + gratuitous adjective, we'd appreciate it if you'd crop it. E.g. translate "10 Ways To Do X" to "How To Do X," and "14 Amazing Ys" to "Ys." Exception: when the number is meaningful, e.g. "The 5 Platonic Solids."
http://ycombinator.com/newsguidelines.html
This guy did a not so rigorous analysis but it is mostly OK: http://changelog.complete.org/archives/931-how-to-think-abou...
A little bird told me there's a better algorithm in speed/compression rate coming soon ;)
Using 50MiB from a tarball I had lying around, best of 3:
http://en.wikipedia.org/wiki/PAQ
I'm insisting because PAQ is a really incredible collection of compressors. Far more intelligence in it than in the LZ* bunch.
http://partiallystapled.com/~gxti/misc/2010/06/25-shootout.p...
http://partiallystapled.com/~gxti/misc/2010/06/25-linux-2.6....
xz format seems to have this feature, but: $ xz --list xz: --list is not implemented yet.
Right now, CPUs are fast enough that LZMA is a realistic prospect. It boils down to what the extra CPUs will cost vs the cost of the storage saved (incl. the "cost" of space in the datacentre, the administrative overhead of more hardware, etc).
This could be achieved by using Javascript and fetching data as JSON, but it seems to be something that would be very useful and beneficial if it were standardized.
Really expensive algorithms aren't that much better to offset cost of sending template logic to the client.
You could probably save some processing time by integrating compression directly into server-side templating - take advantage of the fact that some parts never change and keep them pre-compressed or at least cache some statistics about them to aid compression.
The server supplies a pre-created dictionary (template) which gets cached, and further requests get a diff to that dictionary as a response.
I've only found Google search using it on the server side, and the browser penetration is fairly low, but it seems promising. It makes requests very fast.
I've also noticed that Google search serves image thumbnails as data:// URLs directly in the original response. They are going a long way to ensuring each page loads entirely in one request.
EDIT: also, like gzip but unlike bzip2, you can stream data through xz (at an insignificant penalty in compression ratio).
Second, I am reading Of Mice or Men for the first time this summer. It is new to me!
However, that page does not give the same picture of widespread adoption that the Wikipedia article does. That use was what interested me: I'm surprised that it only came to my attention when I was trying to recall the name of the 7z command line utility p7zip.
Here are some comparisons between the big 3 compression algorithms (taken from http://blogs.reucon.com/srt/tags/compression/ -- he used a 163 MB Mysql dump file for the tests):
Bandwidth is cheap. Stick to gzip.
Bandwidth is cheap. Stick to gzip.
It's not as simple as that. Which one is better depends on the use case. If you're sending a one-off file to somebody, sure, gzip is better. But if you want to distribute a file to a large number of people (like Linux distributions do with their packages), the extra CPU time is insignificant compared to the bandwidth saved over the course of thousands of downloads.
http://news.ycombinator.com/item?id=1458697
If you have to wait minutes to download the files it doesn't matter, but if you already have the file locally it is very annoying.
Also if this is used extensively on projects with a large server deployment this matters even more related to latency and energy consumption. That's why Google has their own compression algorithms derived from BMDiff and LZW (Zippy.) Think about it. Speed matters.
This affects energy consumption, too.
And think about both mobile and servers. Those systems are usually more sensible to high CPU load.
I have a draft blog post with analysis of different protocols with valgrind and other tools. But it is so much data to present and graph I never get around to finish it :(
Looking good on ubuntu 10.4
Same folder compressed in 1/2 the size as with tar cvfz, sick.
lzma is depreciated by its author in favor of xz.
http://github.com/vasi/pixz
The interface is still very rough, but it works. The xz utility comes with a very nice library and API, which made this a lot easier--thanks, Lasse!
http://bitbucket.org/mattsta/bmz/src
It's is a fast compression scheme implemented using BMDiff and a Google Zippy clone (based off LZO).
The results of my unscientific test (compression only):
BMZ = bmpack + lzo by default and can be combined with lzma if necessary. It's not really a BMDiff and Zippy clone, as I've never had a chance to see Google's implementation. It's based on the original Bentley & McIlroy paper: "Data Compression Using Long Common Strings", 1999. Even the two pass idea is from that paper. It was really a wacky experimental implementation (with a lot of room for improvement) to satisfy my curiosity. I'm a little surprised that the 0.1 version has been stable for quite a few people compressing TBs of data through it.
http://www.7-zip.org/