It makes sense to me that distributing across more parameters results in models that can be quant more heavily (information theory - more bits available)
That sounds very similar to what we known in self-supervised learning to representation collapse. Wonder if we could copy some of the anti collapse mechanisms from SSL into GPT... after all, they are ways to increment the differential entropy. However, I'm not sure if it could be useful after all: any pure function cannot produce more entropy than the entropy it receives... and natural language as text has much less entropy than other domains...
The cone-shaped anisotropy is known since at least Gao et al. 2019. That lineage explained it fairly intuitively as an artifact of word frequency and softmax geometry (so a training dynamic).
A variety of papers followed up by adding post-hoc ‘whitening’ steps, then adding regularizers to the loss to penalize the anisotropy, eventually penalizing the covariance matrix (a la VICReg), and then the SIGReg method as a computationally much cheaper way to approximate the full covariance.
As another commenter pointed out it’s also similar to the InfoNCE/contrastive learning objectives. Where terms to increase uniformity (spread out evenly) on the hyper sphere were added. Like the SimCSE (Gao 2021) paper or the excellent alignment/uniformity breakdown from Wang & Isola 2020.
8 comments
[ 3.4 ms ] story [ 32.7 ms ] threadYou might want to look at Physics of Language Models[1]. IIRC, the authors estimate it to be ~2 bits of factual knowledge per parameter.
[1]: https://physics.allen-zhu.com/
The cone-shaped anisotropy is known since at least Gao et al. 2019. That lineage explained it fairly intuitively as an artifact of word frequency and softmax geometry (so a training dynamic).
A variety of papers followed up by adding post-hoc ‘whitening’ steps, then adding regularizers to the loss to penalize the anisotropy, eventually penalizing the covariance matrix (a la VICReg), and then the SIGReg method as a computationally much cheaper way to approximate the full covariance.
As another commenter pointed out it’s also similar to the InfoNCE/contrastive learning objectives. Where terms to increase uniformity (spread out evenly) on the hyper sphere were added. Like the SimCSE (Gao 2021) paper or the excellent alignment/uniformity breakdown from Wang & Isola 2020.
Big topic early 2020s