Break the quadratic wall of Transformer attention: WERSA, paper+code open source (huggingface.co)

I'm not deep enough in the ecosystem to ingest that paper but it's title reminded me of an interesting video I saw recently: "AI can't cross this line and we don't know why" https://youtube.com/watch?v=5eqRuVp65eY