I’ve been using local models on a Framework desktop. I get > 50 tokens/s on Qwen 3.6 35B A3B, and I find that speed ok for coding (though the model not quite). If machine could do a SOTA model with say 0.5-2T parameters like GLM 5.2 at 100 tok/s, it would be very useable.
In 2 years presumably the SOTA models will be much bigger, but I won’t be surprised if there’s a fable level sub 50B parameter model by that point either, in which case 1000 tok/s may be reasonable.
It's just assumed that we need to make models bigger to get more out of them. That is from observation, no one has actually distilled different kinds of intelligences from an LLM to say this is the minimal model size needed for this kind of problem.
Examples: Databases scale with data stored in them. It's a linear relationship. We have databases that load pages on demand as needed and keep hot ones in cache.
There is no 'database page' alternative for LLM's yet. Someday there will be and the paradigm shift will happen.
3 comments
[ 3.5 ms ] story [ 15.7 ms ] threadIn 2 years presumably the SOTA models will be much bigger, but I won’t be surprised if there’s a fable level sub 50B parameter model by that point either, in which case 1000 tok/s may be reasonable.
Examples: Databases scale with data stored in them. It's a linear relationship. We have databases that load pages on demand as needed and keep hot ones in cache.
There is no 'database page' alternative for LLM's yet. Someday there will be and the paradigm shift will happen.
Local 100 tok/s+ is a gamechanger for sure.