This direction is clearly not unique to DeepSeek. Similar ideas have shown up before, like Gemma’s per-layer embeddings, RWKV’s deepembed, and ByteDance’s UltraMem. From an algorithm perspective, using hash-based n-gram…
This is impressive work on quantization. It really validates the hypothesis that software optimization (and proper memory management) matters more than just raw FLOPs.
This direction is clearly not unique to DeepSeek. Similar ideas have shown up before, like Gemma’s per-layer embeddings, RWKV’s deepembed, and ByteDance’s UltraMem. From an algorithm perspective, using hash-based n-gram…
This is impressive work on quantization. It really validates the hypothesis that software optimization (and proper memory management) matters more than just raw FLOPs.