HN Explorer

Show HN: Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB

112 points by spidy__ ↗ HN
I built an experiment that uses an overfitted transformer and arithmetic coding to compress individual files.

Instead of training the model to generalize, I train a 900KB transformer to memorize a single file and predict the next byte. Those predictions are fed into an arithmetic coder to produce the compressed output.

On a 100MB NYC taxi CSV, it compresses to about 7MB (~0.5 bits/byte). On a 100MB slice of enwik9, it compresses to about 21MB (~1.68 bits/byte).

It's pretty slow right now (roughly 20–30 minutes of training and 45 minutes each for compression and decompression on my AMD 7800XT).

Checkout the repo - https://github.com/samyak112/pym-particles

33 comments

[ 0.18 ms ] story [ 69.8 ms ] thread
7373737373
What does it compress the full 1GB file to? http://prize.hutter1.net/
cellular
Maybe everyone should compress the 1st 100MB worth of digits of pi, for an apples-to-apples comparison?

Edit: oh wait that's too easy. Need to generate /publish random digits so everyone can use it.

purple-leafy
That’s so awesome! I want to try something similar. I’ve been going crazy with compression work. I reckon I can beat that prize link
spidy__
Reallly?? So have you published something so far? Can i read something? Sounds like you got some interesting ideas.
purple-leafy
I will be showcasing something on hackernews soon! Basically I found a way to “compress” a multiplayer game state from ~100KB+ to ~1KB

But it’s only for the game I’m building and it’s not pure compression work, I had to do some tricky things

tae0086
Neat approach. Since the 900KB model ships with the compressed file, is there a file size below which the model overhead just eats the gains? Curious where the crossover is.
SubiculumCode
What do those compress to with conventional approaches? For comparison.

I am curious. A classic machine learning ensemble approach is to overfit a collection of small models then bag them (e.g. voting) allowing the models to generalize.

I'm sure someone's tried to overfit a bunch of transformers for compression like this, then bag them to see how well it does?

wildstrawberry
Three questions:

1. How much was AI used to generate documentation for this project?

2. The 100MB CSV data sources are not provided in the repo so it doesn't seem possible to reproduce your results. The enwik9 dataset says it is a "slice" of the larger data set, and there are many NYC taxi trip record datasets that exist. Can you provide the datasets used to generate your results?

3. I am surprised to see performance comparisons only between your transformer and WinZIP. What were your results when comparing your transformer to more modern approaches like LZMA2 (level 9), BZIP2 and ZPAQ (max effort)?

rtpg
I've had this idea of building a codec that would similarly overfit to specific images. But the codec itself would not be a fixed size transformer... instead you could just mess around with the sizing to get better quality/smaller size.

So the codec would be something like: <header describing image size + transformer layer shape> <transformer data itself>

I've seen experiments where people have a "fixed" pipeline but I think having something more dynamic would work quite well.

jmspring
The model is the important part, a huffman code or adaptive huffman or other sorts of encoders would be much better on a dataset based on the model. You need the model to also decode. And on a dataset of sufficient size, embedding the model and the benefit of it's memorization of the file can be offset.

A non-general compression algorithm (model - I don't mean a distinct llm, but "modeling data") targeted at a specific dataset will always do better than a general algorithm.

The reason I mentioned the "encoder" doesn't matter - arithmetic coding, for the data it is presented, will beat huffman/adaptive huffman every day, but it's the model that is where the real "compression" comes into play.

I've implemented enough "coders" over the years, including arithmetic for both commercial and research purposes (was a student of Glen Langdon).

purple-leafy
Dumb question: can you train a model to predict the next byte of ANOTHER MODEL

So apply this same logic to compressing a bigger model within a smaller model

I know this is absolutely regarded, but humour me please

test1072
So has anyone tried to you know for example keep constant weights base model and just transmit the data, might be better compression
totetsu
if first bit is 1 then decompress to a picture of my cat, else its just Huffman
whacked_new
Somewhat related is stavros's method to compress 500KB to something like 50 bytes https://www.stavros.io/posts/compressing-images-with-stable-...

main drawback is that it's not lossless ;-)

but this is great. I hope this actually becomes a format that wraps the weights and transformer module (maybe this can also be NAS-optimized too?). Maybe it would even work for video?

It's like calling gzip but instead of compression level you choose kolmogorov complexity level

VorticonCmdr
Great work. Just Yesterday I thought about LLMzip and asked myself if this is something which could vastly improve HTML compression when done at Google scale and shipped with browsers. I haven't done any research though.
fennecbutt
Given it's an overfitted transformer isn't is still terribly inaccurate/is liable to add random bytes here and there?
freakynit
Recently read about some upcoming GPU chip vendor that speeds up multiply-and-add (the core of [pretty much any neural network, including LLM's) oepration by transforming them to log scale where the multiplication changes to addition.

There issue was the size of such log tables that would be needed, and hence, they settled for some trade-off.

My question is: can the expensive log-domain addition / correction function be implemented as fixed hardware lookup tables or approximate units?

Update: found the vendor: https://www.tensordyne.ai/silicon-and-math