Look up what some Chinese LLMs are doing. Mamba2 layers, Gated Delta Net (2), Kimi Delta Attention and recent Raon Delta Attention (Gated Delta Net 2 + improvement from Kimi Delta Attention) You are likely to find…
Very little in terms of the layers they use. Calling it now, they are using global layers everywhere, making the model basically unusable due to high KV Cache use.
How would it compare to current state of the art tested state space layers like Kimi Delta Attention?
When it released, it cost $595 (back then, not in today's money), considering the price dropped to just $229 the following year, that was not a good buy when it came out.
Replace all of the engineers with open weight models. And add GLM 5.3 Flash as a builder. I am interested how will they fare.
CH32V003 has RV32EC core, for RV64, yes, there is no point.
Look up what some Chinese LLMs are doing. Mamba2 layers, Gated Delta Net (2), Kimi Delta Attention and recent Raon Delta Attention (Gated Delta Net 2 + improvement from Kimi Delta Attention) You are likely to find…
Very little in terms of the layers they use. Calling it now, they are using global layers everywhere, making the model basically unusable due to high KV Cache use.
How would it compare to current state of the art tested state space layers like Kimi Delta Attention?
When it released, it cost $595 (back then, not in today's money), considering the price dropped to just $229 the following year, that was not a good buy when it came out.
Replace all of the engineers with open weight models. And add GLM 5.3 Flash as a builder. I am interested how will they fare.
CH32V003 has RV32EC core, for RV64, yes, there is no point.