Not really, no. You want to use the GPU not the CPU. Macs are neat here since they can use the shared memory with a rather high bandwidth. So even if the GPU is much slower, the ram is much worse than proper vram, and ridiculously overpriced for that,... often the bottlenecks are ram amount and bandwidth.
I only have some light use cases, so I use a cheap laptop (<$250) with a ryzen APU 8gb soldered/shared ram. Then added a 16gb ram stick, booted off a usb bios from github, increased uma buffer to 8gb. I had stable diffusion working, it was slow, but I'm pretty sure its faster than cpu/ram (2 min for 512x512 20-25 step).
Depending on your internet speed, with llamafile you can do it even faster. Go to https://github.com/Mozilla-Ocho/llamafile , find Quickstart section and have fun. Scroll down a bit further for more models.
It could be even easier, we implemented a two click install open-source local AI manager (+RAG and other cool stuff) for Windows / Mac / Linux. You can check it out in shinkai com or check out the code in https://github.com/dcspark/shinkai-apps
ollama. download a few models. bobs your uncle. simple as.
better still you can the use python to call it with langchain chatollama and build anything you want with a little help from claude and chatgpt or codeqwen if you want to do it all locally.
absolute AI don. impress the ladies with that one, you'll be beating them off with a stick when they see it in action.
19 comments
[ 2.7 ms ] story [ 48.0 ms ] threadAnd yes, but there's no lower limit on the memory; it's entirely dependent on the model or kernel size.
Even my M1/16GB gets decent speeds. 7+ tokens/second with llama3
better still you can the use python to call it with langchain chatollama and build anything you want with a little help from claude and chatgpt or codeqwen if you want to do it all locally.
absolute AI don. impress the ladies with that one, you'll be beating them off with a stick when they see it in action.
just need plenty of VRAM after that.