HN Explorer

Show HN: New compression algorithm beats ZSTD by 14% (github.com)

1 points by seccode ↗ HN
Hi all, I made a data compression algorithm that works as follows: Replace common bigrams with a fixed value: "of the" => "of e" "and she" => "and e" "when there" => "when e"

To decompress, look up the bigram dictionary for the right value of e.

I've had some weird bugs in the decompression process, so I'm looking forward to others looking at this.

4 comments

[ 3.8 ms ] story [ 23.0 ms ] thread
metadat
What kind of data is it 14% better on?

The algorithm code looks like it's actually using Zstd, is that right?

Please show some benchmarks. I'd like to repeat your test!

seccode
I tested on the dickens dataset (Charles Dickens novels). This algorithm works well on English text. I do not expect it to be any better than zstd at binary data. I haven't tried code.

It does used zstd. My algorithm is a preprocessing step for zstd.

metadat
Commit that data + test code :)

Add the results to the readme ::))

[deleted]
(comment deleted)