Deploying large language models (LLMs) is challenging because they are memory inefficient and compute-intensive for practical applications. In reaction, researchers train smaller task-specific models by either finetuning with human labels or distilling using LLM-generated labels. However, finetuning and distillation require large amounts of training data to achieve comparable performance to LLMs. We introduce Distilling step-by-step, a new mechanism that (a) trains smaller models that outperform LLMs, and (b) achieves so by leveraging less training data needed by finetuning or distillation. Our method extracts LLM rationales as additional supervision for small models within a multi-task training framework. We present three findings across 4 NLP benchmarks: First, compared to both finetuning and distillation, our mechanism achieves better performance with much fewer labeled/unlabeled training examples. Second, compared to LLMs, we achieve better performance using substantially smaller model sizes. Third, we reduce both the model size and the amount of data required to outperform LLMs; our 770M T5 model outperforms the 540B PaLM model using only 80% of available data on a benchmark task.
It does, they are largely using a full sized LLM to guide a smaller one, but it does mean we can pool resources for the large LLM and then build a company smaller LLM for wide distribution.
This is true (and sort of the point of PaLM - it was an experiment on truly huge model sizes). But also Llama 65B is very efficient for its size: it generally outperforms models twice as large on most benchmarks.
Large model trains small model, I suspect you end up with a better training session than purely unsupervised. You can think of it like transfer learning
> Knowledge distillation has been successfully used
to transfer knowledge from larger, more competent
teacher models into smaller student models affordable for practical applications
It sounds like a bit of Chain of Thought going on too
> We propose a new paradigm, Distilling step-bystep, that leverages the ability of LLMs to reason
about their predictions to train smaller models in
a data-efficient way.
If we anthropomorphize a bit, you can think of this as a person reading all the things and learning on their own vs learning with a mentor, which results in much better results for humans. So maybe this is an interesting, expected result. It will also be interesting to see how this can be combined with DeepMind's RETRO ideas.
Googler here, but not in the research team that authored this paper.
It seems that using a "teacher" LLM to train a smaller model in this step-by-step fashion you get much more out of your parameters. Specifically, if you look at section 3 in the paper, they mention that they use LLMs generated rationales as additional guidance for the smaller model. This approach has already been tried, but it had some limitations (section 3.2) which they circumvent by doing things differently:
"In this work, instead of using rationales as additional model inputs, we frame learning with rationales as a multi-task problem. ... (this) enables the model to learn to generate the intermediate reasoning steps for the prediction, and could therefore guide the model in better predicting the resultant label."
It seems that this is the key for the success they are showing. I am still digesting the paper and I am not 100% sure that is the key factor.
The wording makes it seem that the small one outperforms the large one, all else equal. But PaLM is tested in the few-shot setting, specifically chain-of-thought prompting, which means:
Part of what is going on here is that this paradigm is to train a smaller, task-specific model from a large, generalized model. When comparing to other methods for doing this, it outperforms. So one of these smaller, specialized models can outperform PaLM on some tasks, but not all.
The other benefit comes at inference time, where you can get much faster and cheaper outputs.
from abstract, it looks like it was one task. It is usually much easier to build much more performant specialized model for single task, while it will be weak in many other tasks compared to larger but generalized model.
(1) This is like a new prolog in which the rules are linguistic rules (rationales), and the distilling step-by-step is a way of learning the rules. Call the new prolog lprorat (learning programming with rationales).
(2) Hypothesis: The success of distilling chain of thought learning is that it provides a pseudo large context window. The input pair (label,rationale) allows the student LLM to mimic having a large context window (the context window of a human agent that informs the system by providing the rationale data). So it reduces the embedding distance between words for which entailment is difficult to obtain with short context windows.
(3) The above hypothesis suggests to perform the training process in two phases (a) and (b):
(a) A small model is used as a teacher, this allows the learning model to connect d-distant words. (b) A larger LLM is used as a teacher, the training process continues with new data using the rationale provided by a stronger larger LLM.
In phase (a) the learning system learns to infer d-distant words relations, in phase (b) the system is already prepared to relate (2*d)-distant words because it have developed the necessary skills phase (a).
(4) The two-phase approach can be directly generalized to a multi-phase approach using three meta-parameters: the number of teachers, the relative power of each LLM teacher, and the percentage of data in the training set for each teacher.
(5) Need of pruning: From a geometrical point of view, each phase allows the system to strengthen relations between distant words, increasing latent spaces density promotes part of the latent space getting disconnected, that is creating outliers. Pruning should reduce the risk of outliers that promote hallucinations. Outliers are the germ for establishing relation between very distant words but the rationale data perform that function well, so removing outliers don't pause the learning pace.
(6) It is clear that the rationale data enhances the attention mechanism. This suggests modifying the attention mechanism by giving appropriate weights to the attention with respect to the rationale and self attention for the tag.
(7) It would be desirable for the authors to release those small and powerful LLMs. Therefore giving individuals the power to enter into the SOTA realm and allowing us to create new low latency and cost applications with enhanced privacy.
I was referring to the prolog programming language. What would happen if we replace the rationale with a concrete program in prolog: Example
tag = daughter(X), rationale = father(_,X),female(X); mother(_,X),female(X).
unlikely, the paper was uploaded yesterday and they make no mention of such an intention.
More generally, I think we've reached the point where the big players are investing a lot of resources and trade secrets into their giga-models. Having them effectively distilled and shared may expose these secrets or erode the competitive advantage. You can fine-tune an OpenAI model, but I don't think they are going to let you download the weights. I expect the same with Google and Microsoft.
I dunno, if it's all a valuable trade secret why bother publishing the paper then?
It's not like these models are so big it's prohibitive for anyone else to reproduce the training, we're already seeing a profusion of open 1-11B param GPT-style models... the high-performing 770M T5 described here is smaller than that
you don't need access to the weights though, unless I missed something? this is akin to the various groups using GPT-3.5/4 to generate data for fine-tuning smaller models
Would using llama as the llm teacher for a commercial model mitigate licensing concerns since it's not being used directly but only to create a separate artifacts?
What could impact the ability to discern whether it's llama or open llama? Are there artifacts of a given Llm that could be exposed via distilling process?
Not a lawyer, but I think that depends on who's training this model for what purpose? If you're training the second model specifically to be able to use a llama-like model commercially, that's arguably a for-profit use of llama. On the other hand, if somebody else distills knowledge with llama in order to make a better model, and happens to release it for free without having a commercial use for it themselves it should be fine.
llama 7b in full precision requires 28GB of GPU ram. I know little about AI. Can I estimate that a 770 model takes ~2.8GB? It means vast devices can run the model locally.
Something like that, plus some overhead but likely < 3GB.
That being said though, half-precision would almost certainly work without a huge performance hit (so 2GB), 8bit/4bit quantitization could be done but usually don't work super well on the small parameter count models.
Also you can still batch it through but it'll be slower. In theory ~2GB including the overhead though...
It's worth noting that it's current performance though is only for a specific category of tests, and isn't the most useful model either so it's not chatGPT-on-your-phone just yet.
I think this and somewhat related approaches (eg, the Alpaca approach of generating RLHF data from chatGPT and training a smaller model with it) are really interesting.
If we think of intelligence as the ability to reverse entropy (as in being able to compress information into the smallest representation) there's this weird conceptual idea where a strong enough model is able to transfer its knowledge like this without wandering off track and teaching the wrong things.
It'd be interesting to have formal benchmarks of this ability because I think it is closely related to things people care about in the generative space. For example H2O recently introduced a nice new LLM with a good license. But informally in my testing is seems to hallucinate much worse than Llama derived models - for example it my testing it hallucinated that "Crown Prince Harry" (!) married Phillip Mountbatten (!!)[1].
I suspect but can't prove that a model like this would be much worse at distilling than a Llama based model of the same size.
Testing this knowledge transfer would be really instructive I think.
37 comments
[ 4.2 ms ] story [ 91.0 ms ] threadDeploying large language models (LLMs) is challenging because they are memory inefficient and compute-intensive for practical applications. In reaction, researchers train smaller task-specific models by either finetuning with human labels or distilling using LLM-generated labels. However, finetuning and distillation require large amounts of training data to achieve comparable performance to LLMs. We introduce Distilling step-by-step, a new mechanism that (a) trains smaller models that outperform LLMs, and (b) achieves so by leveraging less training data needed by finetuning or distillation. Our method extracts LLM rationales as additional supervision for small models within a multi-task training framework. We present three findings across 4 NLP benchmarks: First, compared to both finetuning and distillation, our mechanism achieves better performance with much fewer labeled/unlabeled training examples. Second, compared to LLMs, we achieve better performance using substantially smaller model sizes. Third, we reduce both the model size and the amount of data required to outperform LLMs; our 770M T5 model outperforms the 540B PaLM model using only 80% of available data on a benchmark task.
That sounds too good, can someone more knowledgeable comment?
> Knowledge distillation has been successfully used to transfer knowledge from larger, more competent teacher models into smaller student models affordable for practical applications
It sounds like a bit of Chain of Thought going on too
> We propose a new paradigm, Distilling step-bystep, that leverages the ability of LLMs to reason about their predictions to train smaller models in a data-efficient way.
If we anthropomorphize a bit, you can think of this as a person reading all the things and learning on their own vs learning with a mentor, which results in much better results for humans. So maybe this is an interesting, expected result. It will also be interesting to see how this can be combined with DeepMind's RETRO ideas.
https://web.mit.edu/5.95/readings/bloom-two-sigma.pdf
It seems that using a "teacher" LLM to train a smaller model in this step-by-step fashion you get much more out of your parameters. Specifically, if you look at section 3 in the paper, they mention that they use LLMs generated rationales as additional guidance for the smaller model. This approach has already been tried, but it had some limitations (section 3.2) which they circumvent by doing things differently: "In this work, instead of using rationales as additional model inputs, we frame learning with rationales as a multi-task problem. ... (this) enables the model to learn to generate the intermediate reasoning steps for the prediction, and could therefore guide the model in better predicting the resultant label."
It seems that this is the key for the success they are showing. I am still digesting the paper and I am not 100% sure that is the key factor.
770M model uses 80% of dataset
540B model uses 0.01% of dataset
The other benefit comes at inference time, where you can get much faster and cheaper outputs.
(2) Hypothesis: The success of distilling chain of thought learning is that it provides a pseudo large context window. The input pair (label,rationale) allows the student LLM to mimic having a large context window (the context window of a human agent that informs the system by providing the rationale data). So it reduces the embedding distance between words for which entailment is difficult to obtain with short context windows.
(3) The above hypothesis suggests to perform the training process in two phases (a) and (b): (a) A small model is used as a teacher, this allows the learning model to connect d-distant words. (b) A larger LLM is used as a teacher, the training process continues with new data using the rationale provided by a stronger larger LLM.
In phase (a) the learning system learns to infer d-distant words relations, in phase (b) the system is already prepared to relate (2*d)-distant words because it have developed the necessary skills phase (a).
(4) The two-phase approach can be directly generalized to a multi-phase approach using three meta-parameters: the number of teachers, the relative power of each LLM teacher, and the percentage of data in the training set for each teacher.
(5) Need of pruning: From a geometrical point of view, each phase allows the system to strengthen relations between distant words, increasing latent spaces density promotes part of the latent space getting disconnected, that is creating outliers. Pruning should reduce the risk of outliers that promote hallucinations. Outliers are the germ for establishing relation between very distant words but the rationale data perform that function well, so removing outliers don't pause the learning pace.
(6) It is clear that the rationale data enhances the attention mechanism. This suggests modifying the attention mechanism by giving appropriate weights to the attention with respect to the rationale and self attention for the tag.
(7) It would be desirable for the authors to release those small and powerful LLMs. Therefore giving individuals the power to enter into the SOTA realm and allowing us to create new low latency and cost applications with enhanced privacy.
Edited several times for grammar and new ideas.
More generally, I think we've reached the point where the big players are investing a lot of resources and trade secrets into their giga-models. Having them effectively distilled and shared may expose these secrets or erode the competitive advantage. You can fine-tune an OpenAI model, but I don't think they are going to let you download the weights. I expect the same with Google and Microsoft.
It's not like these models are so big it's prohibitive for anyone else to reproduce the training, we're already seeing a profusion of open 1-11B param GPT-style models... the high-performing 770M T5 described here is smaller than that
> First, given an LLM and an unlabeled dataset, we prompt the LLM to generate output labels along with rationales to justify the labels
> Second, we leverage these rationales in addition to the task labels to train smaller downstream models.
This doesn't sound like online use of LLMs to me, but rather using them to pre-generate augmented examples.
At the end of the paper is this table:
Let's do a back-of-envelope calculation:OpenAI costs "gpt-3.5-turbo: $0.002 / 1K tokens"
let's say each rationale consumes 250 tokens ...that's $275 to augment 550M examples in e-SNLI dataset
Even if that estimate is quite far off in terms of token count, it seems like it'll only cost a few thousand dollars at most? GPT4 costs 10x as much.
It's more than I personally would ever want to pay as a curious bystander/hobbyist, but it doesn't seem like a serious "moat" to competitors.
Edit: just saw I wrote 550M in my original comment, even though I was thinking of it as the correct figure (550k)
What could impact the ability to discern whether it's llama or open llama? Are there artifacts of a given Llm that could be exposed via distilling process?
That being said though, half-precision would almost certainly work without a huge performance hit (so 2GB), 8bit/4bit quantitization could be done but usually don't work super well on the small parameter count models.
Also you can still batch it through but it'll be slower. In theory ~2GB including the overhead though...
It's worth noting that it's current performance though is only for a specific category of tests, and isn't the most useful model either so it's not chatGPT-on-your-phone just yet.
> Also you can still batch it through but it'll be slower.
Thanks!
If we think of intelligence as the ability to reverse entropy (as in being able to compress information into the smallest representation) there's this weird conceptual idea where a strong enough model is able to transfer its knowledge like this without wandering off track and teaching the wrong things.
It'd be interesting to have formal benchmarks of this ability because I think it is closely related to things people care about in the generative space. For example H2O recently introduced a nice new LLM with a good license. But informally in my testing is seems to hallucinate much worse than Llama derived models - for example it my testing it hallucinated that "Crown Prince Harry" (!) married Phillip Mountbatten (!!)[1].
I suspect but can't prove that a model like this would be much worse at distilling than a Llama based model of the same size.
Testing this knowledge transfer would be really instructive I think.
[1] https://twitter.com/nlothian/status/1653267700288987138