> Run-length encoding (RLE) is a very simple form of lossless data compression in which runs of data (that is, sequences in which the same data value occurs in many consecutive data elements) are stored as a single data value and count, rather than as the original run. This is most useful on data that contains many such runs. Consider, for example, simple graphic images such as icons, line drawings, and animations. It is not useful with files that don't have many runs as it could greatly increase the file size.
> RLE may also be used to refer to an early graphics file format supported by CompuServe for compressing black and white images, but was widely supplanted by their later Graphics Interchange Format. RLE also refers to a little-used image format in Windows 3.x, with the extension rle, which is a Run Length Encoded Bitmap, used to compress the Windows 3.x startup screen.
> Typical applications of this encoding are when the source information comprises long substrings of the same character or binary digit.
That is a certain type of encoding. There is always overhead with this type. Columnar databases often provide lots of different encoding options (mixed in with compression) for these type of things. Products like Vertica can build the associativity for you ahead of time so you don't even have to maintain it.
Also, I believe you misunderstood how a columnar storage would work as your example would be for encoding values on a single row where I was referring to a column type.
Savestates for Rollercoaster Tycoon 1 and 2 are RLE. Makes sense as a savestate is one fixed sized/positioned datastructure with many runs of null bytes.
Doesn't fit the "a lot" usecase, but I wanted to at least give one real world example ;)
Hi powturbo! Seeing as these benchmarks are for Sandy Bridge, I thought it would be interested to see how the numbers changed for Haswell and Skylake. I couldn't immediately figure out how to replicate the benchmark, though. Could you give (or point to) some explicit directions on the steps to follow? Thanks.
hi Nathan,
glad you want to benchmark TurboRLE on Haswell and Skylake.
Well, it is very simple:
1- download the lkatest TurboRLE from github
2- cd TurboRLE
3- make
4- ./trle file
The enwik9bwt must be generated.
The two other files can be directly downloaded.
You must also download rle64 if you want to use test it.
11 comments
[ 4.0 ms ] story [ 13.7 ms ] thread> Run-length encoding (RLE) is a very simple form of lossless data compression in which runs of data (that is, sequences in which the same data value occurs in many consecutive data elements) are stored as a single data value and count, rather than as the original run. This is most useful on data that contains many such runs. Consider, for example, simple graphic images such as icons, line drawings, and animations. It is not useful with files that don't have many runs as it could greatly increase the file size.
> RLE may also be used to refer to an early graphics file format supported by CompuServe for compressing black and white images, but was widely supplanted by their later Graphics Interchange Format. RLE also refers to a little-used image format in Windows 3.x, with the extension rle, which is a Run Length Encoded Bitmap, used to compress the Windows 3.x startup screen.
> Typical applications of this encoding are when the source information comprises long substrings of the same character or binary digit.
https://en.wikipedia.org/wiki/Run-length_encoding
Also, I believe you misunderstood how a columnar storage would work as your example would be for encoding values on a single row where I was referring to a column type.
Doesn't fit the "a lot" usecase, but I wanted to at least give one real world example ;)
The enwik9bwt must be generated. The two other files can be directly downloaded. You must also download rle64 if you want to use test it.
The better option is to use TurboBench: https://github.com/powturbo/TurboBench