Exactly. I also feel like being able to choose a model for the use case could be worth an idea. So instead of trying to squeeze all kinds of knowledge into a single model, even if it's moe, just focus models on use…
Quite some years ago there was some data backed evidence of this I would say. https://www.dkriesel.com/blog/2019/1229_video_und_folien_mei... Here is the link to his talk directly…
Looks interesting. The 12GB minimum RAM requirements seem quite steep though. Why so much?
Exactly. I also feel like being able to choose a model for the use case could be worth an idea. So instead of trying to squeeze all kinds of knowledge into a single model, even if it's moe, just focus models on use…
Quite some years ago there was some data backed evidence of this I would say. https://www.dkriesel.com/blog/2019/1229_video_und_folien_mei... Here is the link to his talk directly…
Looks interesting. The 12GB minimum RAM requirements seem quite steep though. Why so much?