LLM-Deflate: Extracting LLMs into Datasets (scalarlm.com)

What open source code do you use to pull synthetic data from LLMs?

Remember those programming books that were like "1000+1 tips for C++", making those with llms would be trivial now.

I wonder how many cycles of train->extract->train->extract->... you can do before most of your output will be hallucinations.

> This compression is lossy

Is compression really lossy? What is an example of lost knowledge?

My gripe with an approach like this is the lack of any grounding to these generated topics. Hallucination accumulates like error in this case so every generation that is conditioned by a previous one (the recursive "hierarchical topic exploration" in TFA).

I suspect most of the "leafs" are unusable.

The claims in this paper don't make sense. There is no proof that anything has been decompressed

Wouldn’t this method be good if applied on humans in job interviews?

My understanding was that the Alpaca data was a distillation from text-davinci-003

Learning == Compression of information.

It can be a description by a shorter bit length. Think Shannon Entropy and the measure of information content. The information is still in the weights but it is reorganized and the reconstructed sentences (or lists of tokens) will not provide the same exact bits but the information is still there.