That's an interesting approach, but... what's the performance impact? It seems the check was done only against really trivial tasks. Why not run some known coding benchmark against the original and pruned one?
Also Gemma is Moe not modem (https://arxiv.org/html/2410.07490v1) so will try to balance export choice for each token - I'm suspicious about expectations for this model.
A 6x improvement from 50tps to 300tps for example would be pretty amazing. There’s apparently a threshold where virtually all problems can be brute forced by having a lot of fast tokens rather than a larger model.
10 comments
[ 0.24 ms ] story [ 28.5 ms ] threadAnother step forward for local AI / local LLM inferencing on (very!) resource-constrained consumer GPU hardware!
Also Gemma is Moe not modem (https://arxiv.org/html/2410.07490v1) so will try to balance export choice for each token - I'm suspicious about expectations for this model.
A 6x improvement from 50tps to 300tps for example would be pretty amazing. There’s apparently a threshold where virtually all problems can be brute forced by having a lot of fast tokens rather than a larger model.