1 comment

[ 2.7 ms ] story [ 13.1 ms ] thread
Notable it's a hybrid model, 7 linear attention layers for every traditional O(n^2) attention layer. They claim this allows it to outperform other LLMs (including Gemini) on long context benchmarks.