Contextual Position Encoding: Learning to Count What's Important (arxiv.org)

This sounds like a big deal, we'll have to see if it replicates at bigger scales.

I've always felt that the position encodings being used in practice (learned position encoding or RoPE) are pretty hacky since the 'number of tokens' is not a good metric for distance between two parts of the context.