[–] syntaxing 2y ago ↗ The thing I’m confused about is if MoE is supported by Metal? More specifically, can I run the 8x7B on a MacBook (with enough RAM, which I believe is 48GB required?)? [–] leetharris 2y ago ↗ There's nothing about MoE that is incompatible with Metal.There is a prototype GGUF implementation (https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GGUF) that is out this morning, but it requires a branch that hasn't been merged into the main llamacpp release.In fact, the #1 challenge with MoE is increased RAM usage, which is something that the unified memory of Macs specializes in.
[–] leetharris 2y ago ↗ There's nothing about MoE that is incompatible with Metal.There is a prototype GGUF implementation (https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GGUF) that is out this morning, but it requires a branch that hasn't been merged into the main llamacpp release.In fact, the #1 challenge with MoE is increased RAM usage, which is something that the unified memory of Macs specializes in.
2 comments
[ 2.0 ms ] story [ 20.9 ms ] threadThere is a prototype GGUF implementation (https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GGUF) that is out this morning, but it requires a branch that hasn't been merged into the main llamacpp release.
In fact, the #1 challenge with MoE is increased RAM usage, which is something that the unified memory of Macs specializes in.