I don’t really get why anyone would be buying ai compute unless A) to your goal is to rent out the compute B) no vendor can rent you enough compute when you need it C) you have an exotic funding arrangement that makes compute capex cheap and opex expensive.
Unless you can keep your compute at 70% average utilization for 5 years - you will never save money purchasing your hardware compared to renting it.
Long story short, performance considerations with parallelism go way beyond Amdahl's Law, because supporting scale-out also introduces a bunch of additional work that simply doesn't exist in a single node implementation. (And, for that matter, multithreading also introduces work that doesn't exist for a sequential implementation.) And the real deep down black art secret to computing performance is that the fastest operations are the ones you don't perform.
I thought the conclusion should have been obvious: A cluster of Raspberry Pi units is an expensive nerd indulgence for fun, not an actual pathway to high performance compute. I don’t know if anyone building a Pi cluster actually goes into it thinking it’s going to be a cost effective endeavor, do they? Maybe this is just YouTube-style headline writing spilling over to the blog for the clicks.
If your goal is to play with or learn on a cluster of Linux machines, the cost effective way to do it is to buy a desktop consumer CPU, install a hypervisor, and create a lot of VMs. It’s not as satisfying as plugging cables into different Raspberry Pi units and connecting them all together if that’s your thing, but once you’re in the terminal the desktop CPU, RAM, and flexibility of the system will be appreciated.
Love Jeff's ansible roles/playbooks and his cluster building ! Quite interesting, I should reserve some time to play with a Pi cluster and ansible, sounds fun
I read through it and it’s amusing but along with the title being something I’d receive in email from a newsletter mailing list I’ve never subscribed to (hoping it has an unsubscribe link at the bottom), there’s nothing really of hacker curiosity here to keep me hooked. It’s shallow and appeals to some LCD “I did the thing with the stuff and the results will shock you because of how obvious they are now click here” mentality. Vainposting at its most average. The Mac restoration video was somewhat easier to sit through if only because the picture quality beats out a handful of other YT videos doing the exactly same thing as I’m holding back a jaw grating wince of watching someone butchering a board with poor knowledge of soldering iron practice, so YMMV? Back to hackaday for me I think. I’m not here to read submarine resumes of people applying to work at Linus Tech Tips.
Oh come on Jeff, you forgot to buy GPUs for your AI cluster. Such a beginner mistake.
All you needed to do is buy 4x xtx 7900 used on ebay and build a four node raspberry pi cluster using the external GPU setup you've come up with in one of your previous blog posts [0].
If he was building compute device for LLM inference specifically it would help to check in advance what that would entail. Like GPU requirement. Which putting bunch of RPis in the cluster doesn't help one bit.
I really don’t understand the hype over raspberry Pi.
It’s an overrated, overhyped little computer. Like ok it’s small I guess but why is it the default that everyone wants to build something new on? Because it’s cheap? Whatever happened to buy once, cry once? Why not just build an actual powerful rig? For your NAS? For your firewalls? For security cameras? For your local AI agents?
There is a reason all the big supercomputers have started using GPUs in the last decade. They are much more efficient. If you want 32bit parallel performance just buy some consumer GPUs and hook them up. If you need 64bit buy some prosumer GPUs like the RTX 6000 Pro and you are done.
Nobody is really building CPU clusters these days.
as someone who has built various raspberry pi clusters over the years (I even got an academic paper out of one) the big shame is that as far as I know it's still virtually impossible to use the fairly powerful GPUs they have for GPGPU work
Am I the only one who looks at both the Pi Cluster and the Framework PC and wonders how they are both slower and less cost effective than a MacBook Pro M4 Max? 88 token/s on a 2.3b model is not exactly great, most likely you will want a 32 or 70b model.
58 comments
[ 2.7 ms ] story [ 52.6 ms ] threadWas it a learning experience?
More importantly, did you have some fun? Just a little? (=
I would be pretty regretful of just the first sentence in the article, though:
> I ordered a set of 10 Compute Blades in April 2023 (two years ago), and they just arrived a few weeks ago.
That's rough.
'Worth it any more'? At this size, never. A Pi is a Pi is a Pi!
A few are fine for toying around; beyond that, hah. Price:perf is rough, does not improve with multiplication [of units, cost, or complexity].
I assumed this was a novelty, like building a RAID array out of floppy drives.
Unless you can keep your compute at 70% average utilization for 5 years - you will never save money purchasing your hardware compared to renting it.
Or the oldie-but-goodie paper "Scalability! But at what COST?": https://www.usenix.org/system/files/conference/hotos15/hotos...
Long story short, performance considerations with parallelism go way beyond Amdahl's Law, because supporting scale-out also introduces a bunch of additional work that simply doesn't exist in a single node implementation. (And, for that matter, multithreading also introduces work that doesn't exist for a sequential implementation.) And the real deep down black art secret to computing performance is that the fastest operations are the ones you don't perform.
If your goal is to play with or learn on a cluster of Linux machines, the cost effective way to do it is to buy a desktop consumer CPU, install a hypervisor, and create a lot of VMs. It’s not as satisfying as plugging cables into different Raspberry Pi units and connecting them all together if that’s your thing, but once you’re in the terminal the desktop CPU, RAM, and flexibility of the system will be appreciated.
Thank you!
Not at all the best, but they were cheap. If i WANTED the best or reliable, i'd actually buy real products.
All you needed to do is buy 4x xtx 7900 used on ebay and build a four node raspberry pi cluster using the external GPU setup you've come up with in one of your previous blog posts [0].
[0] https://www.jeffgeerling.com/blog/2024/use-external-gpu-on-r...
Slower. 4 times slower.
Maybe I'm missing something.
It’s an overrated, overhyped little computer. Like ok it’s small I guess but why is it the default that everyone wants to build something new on? Because it’s cheap? Whatever happened to buy once, cry once? Why not just build an actual powerful rig? For your NAS? For your firewalls? For security cameras? For your local AI agents?
Nobody is really building CPU clusters these days.