Transformer vs. Post-Transformer Debate: Kaiser, Kosowski, Jones, Lechner [video] (youtube.com)

I found the disagreement striking. Kaiser argues Transformers still win unless someone shows a better scaling curve while the other researchers argue the field is overfitting to current hardware and missing better architectures.

There was a back-and-forth on scaling, hardware constraints, continual learning and latent reasoning.

Very Interesting panel, Kaiser is one of the only scientists in the field with no social accounts, no public presence and who appears in the media maybe twice a year. That’s exactly why the full talk matters.