Unlock Massive Context Windows: Positional embeddings are just training wheels (xcancel.com)

ref: https://x.com/hardmaru/status/2010565943269999091

Would this generate the same completion for 'the cat sat on the' as 'on the cat sat the'?