It's fascinating how this works with such a small model. Especially given that the training is a kind of meta learning of "how to do in-context learning". I wonder, is there a good intuition of the role of the MLP in…
It's fascinating how this works with such a small model. Especially given that the training is a kind of meta learning of "how to do in-context learning". I wonder, is there a good intuition of the role of the MLP in…