Fixed-width binary data (e.g., a sequence of double floats) often benefit from a simpler transform: just transpose the bits/bytes so that, e.g., the least significant bytes form a contiguous region, followed by all the second least significant bytes, etc.
> Meant for binary data: can take advantage of the type size meta-information for improved compression ratio (using the integrated shuffle pre-conditioner).
16 comments
[ 3.3 ms ] story [ 51.0 ms ] threadhttp://nbviewer.ipython.org/github/Blosc/movielens-bench/blo...
The results are explained here:
http://www.blosc.org/docs/bcolz-EuroPython-2014.pdf
and, more in-depth here:
https://python.g-node.org/wiki/starving_cpu
This sounds like the Burrows-Wheeler transform, which bzip2 uses:
https://en.wikipedia.org/wiki/Burrows%E2%80%93Wheeler_transf...
> Meant for binary data: can take advantage of the type size meta-information for improved compression ratio (using the integrated shuffle pre-conditioner).
makes it sound like that's what Blosc is doing.
http://www.blosc.org/images/shuffle.png
I usually don't nitpick terminology but memcpy() is a C language runtime library function and not a Linux/Win32 os call.