The distillation is done on-policy like RLHF -- the student model is generating the sequences and teacher is providing feedback in terms of logits.
His page mentions that he's not working currently due to personal reasons, so you might not see his signature. http://www.cs.toronto.edu/~hinton
The distillation is done on-policy like RLHF -- the student model is generating the sequences and teacher is providing feedback in terms of logits.
His page mentions that he's not working currently due to personal reasons, so you might not see his signature. http://www.cs.toronto.edu/~hinton