Ask HN: How would you build a budget CPU compute cluster in 2023?

I would investigate buying used servers, workstations, desktops. Often you can find them for very cheap and in large quantities.

How long is this gonna be up and running? Cases may not be needed. If you're optimizing for cheap; then perhaps motherbaords and CPUs of a couple generations back are a better bet. "Refurb" deals can save a lot of money there.

Of course "cheap" can cost too much: if you need reliability and want it to run first time after assembly, then it might pay to spend more.

What are you trying to optimize for? Solely the initial cost of hardware? Power consumption? Is there some performance target, task that needs to complete in X minutes, etc?

For example, I might suggest buying used Lenovo Tiny M75q's on eBay. The Ryzen 3400GE is significantly slower than your Ryzen 4500, but also lower TDP and very cheap procurement cost. Also fits your "smaller the better" wish. No ECC, though.

Cloud vs. on prem is going to come down basically to your duration of use and average/peak utilization ratio. For instance, if you want to run large parallel experiments 1 hour a day then 23 hours a day your compute is sitting idle then that will favor cloud more. Fundamentally on-prem you incur capital costs proportional to peak use whereas cloud opex is proportional to average use.

Once you have your intended compute lifecycle figured out you can compute the cloud cost and hardware cost and compare. Given you’re mentioning k8s I’m assuming this might be a continuous load in which case you’d amortize your hardware capital costs much faster.

Mobos these days don't come with iGPUs. You'll need CPUs with graphics or buy cheap graphic adapters on eBay.

You can buy a couple of 16 to 22 core Xeons on AliExpress and a dual CPU motherboard for them. Plenty of reviews on YouTube.

This will heavily depend on power costs in your area, but if your tasks are not just CPU heavy but also require a lot of RAM it's totally worth to consider old Xeons from China with used Samsung RAM.

This used hardware can be easily be 2-4 times cheaper than building using modern CPUs, but power usage is also much higher.

or hetzner

beefy dedicated servers for 50/100 eur per month

you can use it for a few months and return it any time monthly contract)

> Does it make sense to go cloud (even spot) when I would aim at high utilization?

Hybrid cloud ?

"combines and unifies public cloud, private cloud and on-premises infrastructure to create a single, flexible, cost-optimal IT infrastructure."

Hetzner has a dedicated cheap server: ( monthly pricing )

https://www.hetzner.com/de/dedicated-rootserver/matrix-ax

- AMD Ryzen™ 5 3600 ( € 37.30 + VAT) / month

- AMD Ryzen™ 7 7700 ( € 59.00 + VAT ) / month + setup

- AMD Ryzen™ 9 5950X ( € 103.30 + VAT ) / month

- AMD EPYC™ 7502P ( € 119.80 + VAT ) / month + setup

You can buy complete Dell Workstations with e5 v4/v3 CPUs or even Xeon Gold CPUs for a pittance on ebay. They already have power management, etc built in and tend to run whisper quiet.

I just upgraded compute hardware at home and went with: ASRock Deskmeet X300 (itx case, motherboard, and 500w power supply), Ryzen 5600g, 32 gb ddr4, 512 gb m.2 for around $400. Pretty good bang for the buck.

Graphics. You may not need to have a graphics card in nodes once they are installed. So you may find you can get a single low end card and use that to install each host.

Power and heat. Will you have enough power for the nodes? What is the power trade-off if you get low-end chips vs higher-end chips? Have a look at the Ryzen page on wikipedia to get a feel for power use of each chip. How will you understand how much cooling you need? (more cooling takes more power)

RAM. How much does accuracy matter? Should you use ECC RAM? You can get UDIMMs to work in Ryzen kit, but not with the chips with integrated graphics card (i.e. avoid APU chips if you want ECC). Get Asrock or Asus AM4 motherboards, then get RAM like this - Samsung M393A4K40DB3-CWE. If you go cloud, you may find the hardware has ECC.

IO. Once the grid-of-nodes is in place will you be moving data to functions, or functions to data? How much data are you moving over the network per-job? Might there be IO bottlenecks when you scale up? How detailed a model of IO can you build before you settle on hardware?

I'd go with a rack and rack cases, if you can. There's a reason why the industry uses it.

I find that once you have a bunch of equipment piled up it makes a huge hard to manage mess, and that happens a lot faster than you'd expect. Before finally getting a rack I ended up with with a bunch of hardware caked in dust because it was all lying in such a precarious way that I was afraid to touch anything in there.

How much data do you need to move in/out? How much variation is there in demand?

Get as much cores as you can, use dual socket boards. You can find 12 core Xeon E5-2670V3 for about $12 on Aliexpress and dual socket X79 boards for about $75. Used ECC RAM is cheap.

Build your own boxes.

You can use Kerrighed or OpenSSI for the software side.

It may not be the best use case for you. But you can use the Oracle Cloud forever free tier to run a basic Kubernetes setup. If you don't want the limitations on the networking then you can always pay extra for another public IP.

If you want high throughput, you may want to consider 10gbe and nvme.

> As for the potential node spec:

> - Ryzen 4500/5500 (seems best perf/$)

Price/performance is great for the CPU, but you have to spend hundreds of dollars on the motherboard, RAM, power supply, and case for each one.

You need to look at the overall system cost. If you’re building new, it could be cheaper overall to put 12-core or 16-core CPUs into a smaller number of machines than it would be to put a lot of $100 budget CPUs into many machines.

Unless your goal is to build a cluster for the sake of building a cluster, you might have a better price/performance ratio by building a single 16-core 7950X box than you would with three separate Ryzen 4500 or 5500 machines.

Even with perfect scaling, you would need at least 4 separate Ryzen 5500 machines to have a chance at beating a single 7950X for CPU-bound tasks. The 7950X CPU alone is barely more than 4X the cost of a Ryzen 5500, but you only need to buy one motherboard, one power supply, and so on.

As others have said -this is vague.

For my home server, I pick the smallest case that can fit a desktop CPU, so just a bit bigger than Intel Nuc. Those have laptop CPU's, you are overpaying. I am willing to pay extra for it to be small.

The two best contenders for me are Asrock DeskMini barebones system (picoITX) and IN WIN Chopin case - you gave to buy an ITX motherboard.

I use Chopin with an Intel CPU, they work for my usecase.

Also some motherboars can boot a ryzen without any GPU at all. Asrock usually will. If you are willing to deal with a totally headless system, go for it.

cpu cost is only a small fraction of overall cost: for every cpu, there is a power supply, ram, motherboard and case. You also haven't mentioned anything about data: do they need local storage or will it be only temporarily stored locally?

it sound like minipcs can be an excellent solution for you.

For how long will you be using this? AWS may be preferable in the short term while local hardware may be cheaper in the long term/a lot of cpu hours.

there is also the question of your application performance on different cpus. there are older servers available for very cheap prices but is it worth it to buy a 12/20core xeon cpu that consumers 200-300W if its performance is similar to a 5900 at 150W ?

If 2-3 Ryzen machines is all you need, you should just go for a Xeon/Epyc server. You can build a decent Epyc workstation/server with 64 cores for as little as $2k USD when using 2nd gen Epyc processors.

If it's just for fun and learning purposes you can build a RaspberryPi cluster: https://www.youtube.com/watch?v=X9fSMGkjtug

I have a couple similar machines in my homelab, running in ASRock X300 boxes w/Ryzen APUs (cpu+gpu), in a similar role (k3s).

The reason I have two is I started with a 3400g (4c/8t) due to supply limitations, then upgraded to a 4750g (8c/16t) when it became feasible. Over time I upgraded memory and storage, so eventually I had everything but the case for a second “half-power” machine.

Having multiple medium-power machines can be useful for rolling upgrades (and for learning purposes), but otherwise it’s very uneconomical.

If your goal is to maximize cores/$ then a single beefy machine will do best.

If your workload is able to be split up and run efficiently on different machines, just buy as many cheap (or free) 1-to-10 year old used computers as you can, and run HTCondor on them for scheduling the jobs.

Honestly, look on ebay for used Lenovo ThinkCentre M/P "Tiny" or Dell Optiplex "Tiny" computers. They can be purchased by the lot for most of them as businesses get rid of the old ones.

I use 4 Lenovo M910x's as a kubernetes cluster and home lab. Have them all connected with a netgear switch. The whole setup costs about the same as a single new quality work station. Each has: i7 8700 (6c - 12t), 32gb memory, 1TB SSDs, <1L case, they're practically silent. easy to find parts, they even use lenovo laptop chargers. if one dies, I can easily purchase + replace in a few days.

You can even go cheaper if you don't need the absolute fastest cpus. Some of these older tiny computers can be purchased for around 100 bucks if you look for them. It has worked like a charm for me. Not sure how much horsepower you really need, but this is a cheap way to build a home cluster. I think they hover around using 40w most the time, so power isn't really too big of a cost either.

If you run them 24/7 another idea will be to rent cheap dedicated servers. Hetzner has some: https://www.hetzner.com/dedicated-rootserver

They even have auctions: https://www.hetzner.com/sb