This seems pretty useful for AI inference if it can pass Apple approval. I've wanted to use my Nvidia GPUs with a Mac Mini, this would enable it to run CUDA directly. Very cool!
> As much as I hate to admit it, step one in most of my projects now is to ask AI about it. Maybe it’ll tell me something I don’t know.
Or, more likely, it will tell you something it doesn't know.
Reminds me of yesterday, when I was arguing with ChatGPT that the 5070TI was an actual video card. It kept trying to correct me by saying I must have meant a 4070ti, since no such 5070ti card exists.
I have been bothering the VM team for years for VM GPU pass through. I worked on the Apple Silicon Mac Pro and it would have made way more sense if you could run a linux VM and pass through the GPU that goes inside the case!
Sadly, as you can tell, they have not taken me up on my requests. Awesome that other people got it working!
This is pretty impressive. My impression was that eGPUs simply do not work with Apple Silicon.
(EDIT: Apple agrees with my impression. “To use an eGPU, a Mac with an Intel processor is required.” And, on top of that, the officially supported eGPUs were all AMD not NVIDIA. https://support.apple.com/en-us/102363)
This is not using an eGPU with macOS, ie you can't run your chrome on macOS with its GPU acceleration coming from this eGPU. This is tunneling that eGPU to a Linux VM.
The game benchmarks are fun but the LLM improvements are where this gets really interesting for practical use. I love Apple platforms as an approachable way to run local models with a lot of RAM, but their relatively slow prompt processing speed is often overlooked.
> Here you can see the big issue with Macs: the prompt processing (aka “prefill”) speed. It just gets worse and worse, the longer the prompt gets. At a 4K-token prompt, which doesn’t seem very long, it takes 17 seconds for the M4 MacBook Air to parse before we even start generating a response. Meanwhile, if you strap the eGPU to it, it’ll only take 150ms. It’s 120x faster.
The prefill problem goes unnoticed when you’re playing around with the LLM with small chats. When you start trying to use it for bigger work pieces the compute limit becomes a bottleneck.
The time to first token (TTFT) charts don’t look bad until you notice that they had to be shown on a logarithmic scale because the Mac platforms were so much slower than full GPU compute.
Been hearing this for over a decade, except back then it was eGPU in Intel Macs which were closer to other PCs if anything. Even if this didn't require so much DIY and if Thunderbolt could do PCIe speeds, most people don't want to add drama when they can just use a PC with regular PCIe slots and native compatibility with Nvidia. The native way already has enough edge cases without adding an unusual setup.
What would be native enough here? What if they got Asahi working with NV gpus for rendering and running cuda kernels? Would eGPU on asahi be sufficient or do you really only see pcie worthwhile?
Some of us mainly want more gpu options on a high performance consumer arm machine (for Linux).
Man, Apple fans are still proving the stereotype to be accurate after 20 years.
Ignoring the fact that the Mac OS gets in your way every time you try to do something that Apple doesn't like, with no guarantee that an update won't break anything existing, ignoring the fact that Macs are non repairable, non upgradable, ignoring the fact that they don't support multiple displays flawlessly, I hope you realize that egpu support natively is NEVER coming to Macs, because why the fuck would they enable it when they can just charge you full price for a desktop computer? Apple is built on the sole image that Apple users have money, so buying another Mac Mini or Mac Pro in addition to your laptop is what you are supposed to do.
Android is way ahead of Mac with Android Desktop mode and Samsung Dex, to the point where you don't even need to own a laptop anymore. Ive been using my S24/S25 with lapdock for over 3 years now as a laptop, and it works flawlessly. Apple can easily do this with iPhone, but they won't because that means one less macbook purchase.
> Man, Apple fans are still proving the stereotype to be accurate after 20 years.
Who is this straw man you're flogging?
> Ignoring the fact that the Mac OS gets in your way every time you try to do something that Apple doesn't like, with no guarantee that an update won't break anything existing, ignoring the fact that Macs are non repairable, non upgradable, ignoring the fact that they don't support multiple displays flawlessly,
Lot to unpack there, most of it does not matter to most normies. When I bought my current mini-pc to drive my egpu I didn't focus on any of this stuff. Just about all I looked for was something that can drive a gpu over TB4/5 and has good perf/watt in a small form factor.
> I hope you realize that egpu support natively is NEVER coming to Macs, because why the fuck would they enable it when they can just charge you full price for a desktop computer?
Sounds like you are more hopeful they wont than I am that they will. They've already enabled RDMA over TB5 for ML applications, and they've left their boot loader open enough for the asahi community to reverse engineer tons of functionality.
I do think eventually there will be some form of GPGPU programing popularized on the mac that isn't Metal (gross).
> Apple is built on the sole image that Apple users have money, so buying another Mac Mini or Mac Pro in addition to your laptop is what you are supposed to do.
I think you have a very specific use case in mind, chiefly gaming. There's a lot more eGPUs offer, and it has nothing to do with turning your normie laptop into a sick gaming rig.
> Android is way ahead of Mac with Android Desktop mode and Samsung Dex, to the point where you don't even need to own a laptop anymore. Ive been using my S24/S25 with lapdock for over 3 years now as a laptop, and it works flawlessly. Apple can easily do this with iPhone, but they won't because that means one less macbook purchase.
I fail to see how Android is relevant in this context at all? For one, the arm64 hardware would have to exceed the single thread and perf per watt of an M5 and secondly you'd actually need tools and applications worth using for desktop use.
I am seeing some of the newer AMD 370/395 and Intel Ultra 7/9 socs as being much more of a serious alternative to the M4/5 here. In fact my current eGPU setup is an Ultra 9 mini-pc with an egpu, its just a shame im still on x86.
The only thing Apple silicon has going for it is power use and that gap is getting closed. I can't really see any reason why I would switch to Mac, it just seems like you pay a lot more for a closed expensive environment that fights you at every step.
I'll never pay anyone for a developer licence or fee either. They can sponsor me to port my software to their platform.
Is it? I recently paid $999 for a pre-build intel mini-pc system thats best case in line in perf with a M2 from four years ago. That seems roughly the same as what I'd paid for an equivalent mac mini in the past, and I thought prices for custom builds were going up quite a bit too?
> Because OpenGL is not well-supported anymore on macOS, the game is completely unplayable there, even with CrossOver. Ironically, it plays totally fine on a Windows PC, but this is a game you literally can’t play on Mac without this eGPU setup.
I understand that this is true it seems that Doom does support Vulkan but you would need to add VK_NV_glsl_shader to MoltenVK. Probably much less work than what went into hanging an RTX 5090 off of a M4. Still, kudos to the scott and the local AI Inference speeds are pretty cool. What a crazy project! <applause>
interesting. that might be a fun intro project to MoltenVK. I hadn't dug into what was missing for Doom. I thought maybe the issue was that the intro/menu always ran in opengl mode or something. If it's just one missing op, that's way easier.
Very nice effort. This has incredible technical depth, particularly in the DMA and QEMU sections. I also like that you didn't oversell it as the ideal Mac gaming solution. I found the AI inference results to be the most fascinating. Overall, it was a great read.
Wait, this is incredible. I have a spare 5090 lying around and run a claw-like on my M4 Mini. Just plugging it into some sort of 3D print frame for stability and plugging it into the TB port might get me a pretty viable tool for local inference. Would need something neat to ensure the power etc. is well fed.
The problem is `max-num-seqs` and `max-model-len` fight each other, and unless you're in the pure single-client mode you'll need multiple slots so to speak.
> step one in most of my projects now is to ask AI about it. Maybe it’ll tell me something I don’t know.
Bingo. This is exactly how I use LLM. I like getting a gut check, seeing what the first recommendations are or if there is some deep flaw in what I think the approach is, and I almost never copy/paste whatever it spits back or just follow its instructions.
I came into the post thinking it would be running a VM through the slow tinygrad driver... but this is much, much better.
It'd be amazing if Apple would provide better support, and allow more than that 1.5 GB window to make this easier. Arm overall has some quirks with PCIe devices, but at least in Linux, it's gotten so much easier since most modern drivers treat arm64 as a first class citizen.
i don't know for sure, but i suspect what makes the tinygrad stuff slow isn't the macos host driver itself. i think they're doing something very similar to what i'm doing, which is just mapping the PCI BARs to userspace, then they have a bunch of python code that drives the GPU.
this is only speculation, but i think the big thing that makes tinygrad slow is that the tinygrad inference engine has not really been optimized much for all these open LLM models. probably most of the work has gone towards optimizing the stack for george's self-driving hardware company. since you can't just run the existing CUDA kernels on their engine, that makes things a lot tougher, engineering-wise.
i am actually curious if my project could share a macos host driver with them. i think it would need some changes, but it seems like there's a lot of overlap
The lack of native games on Apple Silicon is one of the greatest crimes ever committed against computing.
I got Fallout 3 working on my M2 MBP as well as it did on Windows back in the day. Temps were cool, battery was decent. If they sold my college years gaming collection (15-ish years ago) in a way that ran natively through GoG or Steam, I'd buy every single title.
50 comments
[ 2.5 ms ] story [ 71.3 ms ] thread"no - not in any practical sense today, and "maybe" only in a very deep, borderline-impractical research sense."
This is why humans will always rule over crappy LLMs.
Or, more likely, it will tell you something it doesn't know.
Reminds me of yesterday, when I was arguing with ChatGPT that the 5070TI was an actual video card. It kept trying to correct me by saying I must have meant a 4070ti, since no such 5070ti card exists.
Sadly, as you can tell, they have not taken me up on my requests. Awesome that other people got it working!
It’s these people, not the ones who refuse to use LLMs, who are as they say, “cooked”.
(EDIT: Apple agrees with my impression. “To use an eGPU, a Mac with an Intel processor is required.” And, on top of that, the officially supported eGPUs were all AMD not NVIDIA. https://support.apple.com/en-us/102363)
The game benchmarks are fun but the LLM improvements are where this gets really interesting for practical use. I love Apple platforms as an approachable way to run local models with a lot of RAM, but their relatively slow prompt processing speed is often overlooked.
> Here you can see the big issue with Macs: the prompt processing (aka “prefill”) speed. It just gets worse and worse, the longer the prompt gets. At a 4K-token prompt, which doesn’t seem very long, it takes 17 seconds for the M4 MacBook Air to parse before we even start generating a response. Meanwhile, if you strap the eGPU to it, it’ll only take 150ms. It’s 120x faster.
The prefill problem goes unnoticed when you’re playing around with the LLM with small chats. When you start trying to use it for bigger work pieces the compute limit becomes a bottleneck.
The time to first token (TTFT) charts don’t look bad until you notice that they had to be shown on a logarithmic scale because the Mac platforms were so much slower than full GPU compute.
Some of us mainly want more gpu options on a high performance consumer arm machine (for Linux).
Ignoring the fact that the Mac OS gets in your way every time you try to do something that Apple doesn't like, with no guarantee that an update won't break anything existing, ignoring the fact that Macs are non repairable, non upgradable, ignoring the fact that they don't support multiple displays flawlessly, I hope you realize that egpu support natively is NEVER coming to Macs, because why the fuck would they enable it when they can just charge you full price for a desktop computer? Apple is built on the sole image that Apple users have money, so buying another Mac Mini or Mac Pro in addition to your laptop is what you are supposed to do.
Android is way ahead of Mac with Android Desktop mode and Samsung Dex, to the point where you don't even need to own a laptop anymore. Ive been using my S24/S25 with lapdock for over 3 years now as a laptop, and it works flawlessly. Apple can easily do this with iPhone, but they won't because that means one less macbook purchase.
Who is this straw man you're flogging?
> Ignoring the fact that the Mac OS gets in your way every time you try to do something that Apple doesn't like, with no guarantee that an update won't break anything existing, ignoring the fact that Macs are non repairable, non upgradable, ignoring the fact that they don't support multiple displays flawlessly,
Lot to unpack there, most of it does not matter to most normies. When I bought my current mini-pc to drive my egpu I didn't focus on any of this stuff. Just about all I looked for was something that can drive a gpu over TB4/5 and has good perf/watt in a small form factor.
> I hope you realize that egpu support natively is NEVER coming to Macs, because why the fuck would they enable it when they can just charge you full price for a desktop computer?
Sounds like you are more hopeful they wont than I am that they will. They've already enabled RDMA over TB5 for ML applications, and they've left their boot loader open enough for the asahi community to reverse engineer tons of functionality.
I do think eventually there will be some form of GPGPU programing popularized on the mac that isn't Metal (gross).
> Apple is built on the sole image that Apple users have money, so buying another Mac Mini or Mac Pro in addition to your laptop is what you are supposed to do.
I think you have a very specific use case in mind, chiefly gaming. There's a lot more eGPUs offer, and it has nothing to do with turning your normie laptop into a sick gaming rig.
> Android is way ahead of Mac with Android Desktop mode and Samsung Dex, to the point where you don't even need to own a laptop anymore. Ive been using my S24/S25 with lapdock for over 3 years now as a laptop, and it works flawlessly. Apple can easily do this with iPhone, but they won't because that means one less macbook purchase.
I fail to see how Android is relevant in this context at all? For one, the arm64 hardware would have to exceed the single thread and perf per watt of an M5 and secondly you'd actually need tools and applications worth using for desktop use.
I am seeing some of the newer AMD 370/395 and Intel Ultra 7/9 socs as being much more of a serious alternative to the M4/5 here. In fact my current eGPU setup is an Ultra 9 mini-pc with an egpu, its just a shame im still on x86.
I'll never pay anyone for a developer licence or fee either. They can sponsor me to port my software to their platform.
I understand that this is true it seems that Doom does support Vulkan but you would need to add VK_NV_glsl_shader to MoltenVK. Probably much less work than what went into hanging an RTX 5090 off of a M4. Still, kudos to the scott and the local AI Inference speeds are pretty cool. What a crazy project! <applause>
The problem is `max-num-seqs` and `max-model-len` fight each other, and unless you're in the pure single-client mode you'll need multiple slots so to speak.
Bingo. This is exactly how I use LLM. I like getting a gut check, seeing what the first recommendations are or if there is some deep flaw in what I think the approach is, and I almost never copy/paste whatever it spits back or just follow its instructions.
It'd be amazing if Apple would provide better support, and allow more than that 1.5 GB window to make this easier. Arm overall has some quirks with PCIe devices, but at least in Linux, it's gotten so much easier since most modern drivers treat arm64 as a first class citizen.
this is only speculation, but i think the big thing that makes tinygrad slow is that the tinygrad inference engine has not really been optimized much for all these open LLM models. probably most of the work has gone towards optimizing the stack for george's self-driving hardware company. since you can't just run the existing CUDA kernels on their engine, that makes things a lot tougher, engineering-wise.
i am actually curious if my project could share a macos host driver with them. i think it would need some changes, but it seems like there's a lot of overlap
I got Fallout 3 working on my M2 MBP as well as it did on Windows back in the day. Temps were cool, battery was decent. If they sold my college years gaming collection (15-ish years ago) in a way that ran natively through GoG or Steam, I'd buy every single title.