Posted before I saw this comment. Yes passive cooling is fine with recent firmware and a good heatsink case. I have stress tested with the following case for hours with all cores pegged at 100% and there is no CPU throttling with passive cooling. They do run hotter, but they do not throttle. Old firware and bad cases will though.
https://www.amazon.com/Geekworm-Raspberry-Computer-Aluminum-...
Its cool and fun and teaches you a skill at a low price point. There is essentially no functional reason to do this over a single high power server unless for some reason you need a bunch of ARM servers for something.
All fourteen Pis, even v4s (which are significantly hungrier), would be lower power (as in electrical draw) than even a feeble (as in compute power) traditional machine.
If you can distribute your workload (or it's not intensive anyway), i.e. having no single machine with a faster processor, then I think it's a good choice, cheaper to run.
Edit: I'm happy to be corrected if down-voters would like to reply... A Pi 4 draws 6-7W at full load. A former desktop PC turned server likely idles over 100W.
They're different beasts, so I'm not saying 'a Pi is better than a PC', I'm saying 'if your load could also be handled by a (bunch of) Pi(s), then all else being equal that might be better than the PC'. Of course all else is never equal, ymmv, you may already have the old PC, you may need very many Pis, so even if pennies cheaper to run it might take years and years to break even on the Pi acquisition cost.
Idle as in powered on, not sleeping, just 'no' workload? I stand corrected if so, I didn't realise that was the case.
I also like the idea of having an easy homogeneous setup, each Pi having one disk, vs. my alternative (old PC parts) being something like '4 SATA ports on this one, and 6 on that one, but one's broken, less one on each for the boot disk' - I quite like the idea of having a bunch of Pis, and one image that I apply to all of them, which can include details like /dev/sda being its only data disk. That's not really a functional advantage, just appealing for keeping it simple to come back to and remember/understand how I set it up when I need to tweak something (and really easy to add another one increasing available cpu/ram/disk).
This high end system draws ~35W idle: Intel Core i9-10900K @ 5,0 GHz, AMD Radeon RX 6900 XT (16 GB), Asus ROG Maximus XII Hero Wi-Fi, 2× 16 GB G Skill Trident Z Royal 3 600 MHz 16-16-16-36, Samsung 970 Evo M.2 1 TB, Samsung 860 Evo 1 TB, Seasonic Prime Ultra Titanium, 1 000 W.
Picture in Swedish, from graphics card review comparing average power draw for the entire system for various GPUs, idle and while benchmarking in Metro Exodus, numbers rounded to nearest multiple of 5: https://cdn.sweclockers.com/artikel/diagram/22452?key=2c6f21...
It's not that difficult to have a reasonably modern desktop PC idling more in the 10-20W range. For 100W you need to do something seriously wrong, or add extras that would blow up the Pi usage too (e.g. a home server full of hard disks will of course use more power, but so would Pis with the same number of drives)
Don't forget that Pis also draw quite some power while idle. The Pi 4 draws around 3.4 Watts [1], or around 2.7 Watts since they are running headless in a server setup you'd probably disable HDMI [2]. Times fourteen that's 37.8 Watts just sitting there. An idling headless Desktop PC will be below that for sure.
A good functional reason is when you need to have high redundancy for low-intensity jobs while keeping power consumption ridiculously low. As long as you don't use SD cards, Pis can run for years without stability issues.
lol? what if you don't need a heap of compute power but require a lot of redundancy? A single server is not good for that...Also not every site has internet so yes having small redundant easily replaceable nodes on-site does have plenty of use cases.
Y'all negative folks against this seem to have very restricted use cases in mind. Not all redundant rack servers live in a 36U case in a air-conditioned internet connected server farm. There are plenty of use cases outside of this.
Advantages: Cheaper than conventional servers, limited physical form factor suitable for constrained environments such as apartments, luggage, etc. Low power consumption, suitable for mobile/off-grid/renewable.
Commercial advantage: Run highly cost and power efficient data centers, call centers, etc. Eliminate redundant state. Remove manual configuration rollout. Infrastructure as code. Full CI/CD with kernel-level control, per node, for any OS. Basically becoming a lost art in this day of "outsource to cloud". With SDN knowledge, confidently build your own cloud provider.
You absolutely could. But you wouldn't learn how to build a system that recovers from yanking a power or network cable, wouldn't own it, and wouldn't be able to run it on your terms. You'd need a credit card with an American-run payments network to operate, with a declared identity and a bank account, for which a prerequisite is a few months of stable living and income in a particular jurisdiction, itself something large portions of the world do not have access to. Further you'd need to be happy with the fact none of your data was in your country, under your physical control, and never would be. But feel free to ignore the rest of the world and the fact that all this rental crap runs on open source software, just throw in the towel and be a rent-payer: a strategy that has its perks and efficiencies, for sure. But don't discourage those who seek to learn more.
> You'd need a credit card with an American-run payments network to operate, with a declared identity and a bank account, for which a prerequisite is a few months of stable living and income in a particular jurisdiction,
Try being from a random South Asian or African village. Tell me how you go buying cloud services with your nonexistant banking infrastructure and personal credit line.
The physical location of a person isn't generally the key barrier to being unbanked; instead it's poverty.
"The most common one was having too little money to use an account. Two-thirds cited this as a reason for not having a financial institution account, and roughly a fifth cited it as the sole reason. Cost and distance were each cited by about a quarter of those responding to the question, and a similar share said they do not have an account because a family member already has one. Lack of documentation and distrust in the financial system were both cited by roughly a fifth of adults without a financial institution account, and religious concerns by 6%."
This sounds like some kind of fantasy; how many people without stable living conditions or without a bank account that can be processed by the US are concerned with...cluster computing?
"What about all the homeless Iranian Ferrari owners!" doesn't seem like much of a rallying cry.
I "show an interest" in space, but I'm not entitled/deluded enough to expect someone to deliver a shuttle to me, because that would be a waste of time/materials/energy/money.
If you had spoken to people in the 1960s and told them how many transistors we would have on a chip in 2020, they probably would have called you delusional.
Also you are comparing a space shuttle to a few raspberry pis, hardly apples with apples.
RamNode has some starting at $15/year on their OpenVZ platform. Have been happy enough with that for several years now, although it's quite limited as to what it can do in terms of performance. But as another commentor mentioned, check out LowEndBox. It's where I discovered this.
One issue I have with going the VPS route is that what I'm charged is often not what I'm expecting. I will use estimation calculators and figure my bill will be $25/month (which I get for a couple months) and then suddenly get a $120 bill. This has happened more than once and has had me loose all confidence in these services for hobby projects. I'd rather just buy a Pi cluster at this point.
Your agreement was $25/month, so if they start to charge you $120 you pay them $25 and then cancel the service. This is very annoying but do happen dealing with unscrupulous businesses. Although if they have 500% margin eg. you pay $5 for something you can get for $1 elsewhere they will care more about keeping you as a customer. Compared to for example a cloud bill, renting a VPS is both cheaper and more predictive. If you are however paying $25/month you should start looking at dedicated servers, where all resources are preserved for you.
Sounds like AWS or one of the big guys. The smaller providers usually have an all in price for a VPS with a fixed amount of rosources, no surprises. Lightsail is similar but kind of sucks.
> Honestly, could be as valid with a setup that just involves running a bunch of VMs.
Slightly less valid. You don't have the same IO latency between nodes using VMs unless you start messing around with artificial attempts at inducing latency on each node individually.
It's much easier to accurately simulate real world networking by not bothering to simulate and just using real world networking.
There's a huge difference between latency on a wire vs latency within a hypervisor, and you won't know what you've papered over through lower latency via VM until you move to real hardware.
If you want to learn clustering with physical hardware, and you don't have the power and size constraints, you're better off buying some off-lease business desktops. These i3 and i5 systems with 4 to 8GB of RAM can be had for less than $100 each.
When you're done with the cluster, you can likely sell them for nearly what you paid for them.
I know there are thinkpad hoarders with like 10 or 20+ old thinkpads. Seems to be a nice hobby, but you will rarely find one who builds a thinkpad cluster.
In the case of Raspberry Pi it is pretty normal nowadays to connect a bunch of them into a cluster and have some fun. In fact, it has become so popular, that you u can find stackable cluster cases and other Pi cluster hardware on amazon.
Is the power consumption really that good? I mean comparing to just getting a single 5950x or something would this actually win in terms of performance per watt?
I have a friend that uses similar setups to tech kubernetess but previously he also used it as example for all sorts of distributed computing (when Hadoop was still a cool thing to do)
My guess would be Nextcloud put a lot of load on both CPU and disk (for thumbnail generation, file encryption, etc.) making Raspberry Pi which lack performance in both front not very suitable for the task.
I tried running Nextcloud on Raspberry Pi 4 (4GB) a while ago, and hitting performance issue on SD card very fast (initial DB import take very long). I tried to solve this by moving all Nextcloud data into iSCSI disk but RAM & CPU became a problem once I started adding more files. After a week or so, I tore everything down and went back to a VPS instance.
It might works better with Raspberry Pi cluster on Kubernetes, though. (This very Raspberry Pi is now happily running as a DVB streaming server.)
Thanks. I keep meaning to try it out but booting from an SSD instead. I was hoping anything intensive (e.g. thumbnail generation) would be a one-time thing so not noticeable in daily usage. Perhaps I will re-think, I have a ThinkCentre M75q I could use but that seems overkill.
I find you can get a much better experience with a cheap celeron NUC than a RPi because everything from single thread performance to IO speeds favor x86 right now.
This. I think the Hacker News mindset has substantially shifted in the last few years from the "Hacker" mentality of doing things because they are fun and knowledgeable, and more towards Enterprise News..
Not everything needs a business case, some things (most things) should be done just because you want to see if you can.
If someone really needs to think in items of ROI and that everything needs a business case, then this ideology of doing things for fun and testing limits is the only way to really innovate.
I bought a Turing Pi 1 . It's a 7 node mini-cluster, not very expensive (I think about £300 fully populated), that fits into an MiniITX-style case with a single small power supply and single ethernet exposed to the world. You can easily pick the thing up with one hand. It's certainly not powerful, each node is a very modest ARM processor with "only" a gigabyte of RAM and 8 GB of local storage.
The alternatives are much more powerful, but far more expensive and not very attractive in other ways. Blade server? Huge, noisy, power-hungry and I wouldn't get much change from many thousands of pounds. Rack of NUCs? Powerful of course but far more expensive, large, separate power bricks, ethernet switch and lots of wiring required. VMs running on a desktop? Still more expensive, larger, and not the same tangible experience as having a real cluster. VMs on a laptop? Laptops at the same price point would have too little RAM to run lots of VMs well.
So these Raspberry Pi clusters fit a niche which is not well served by other solutions.
You can neatly mount your Pis in your rack where you have the rest of your gear supported by network switches, power delivery, cabling, cooling, good internet connectivity connectivity (if housed in a machine room or colo) etc.
But in terms for actual cost and performance; how does this compare in terms of peak performance, power consumption and actual total cost to mass produced quality hardware like the Apple Mac Mini?
I don't the actual numbers but I have the feeling it won't fare that well.
For practical purposes 8 high performance cores will be much more performant that 56 cores with lower performance; in particular if the cores are spread over 14 physical machines.
Maybe. But you have a network bottle neck between the 14 separate computers that you have on a single computer. If you have a computational task you split in 56 pieces and run it across the setup, you can utilies the cores fully. However even in that case I think the Mac Mini will outperform this.
You can use it as a USB-hub though as I suppose you get 2 USB-ports pr. raspberry pi.
I doubt it - each cpu core only has access to the 4 or 8 GB that is on its machine - to communicate to other parts of the setup it will have to go over network or a shared drive. I think that will be much slower than a highly integrated device like the mac mini just using its hard drive.
It was apparently built for a very specific use case (doing CI for some application) so I guess it has to be ARM Linux. You can run Linux in a VM guest on Mac mini, but some limitations remain:
- The project was obviously started before the Mac mini was available
- Linux guests are still rather experimental and might not behave the same way as a real machine.
- 32-bit Arm applications do not run on the M1, it is a 64-bit-only CPU
- Nested virtualization is said to not work yet under MacOS, so you cannot run KVM guests inside of the Linux guest
- If you need to spin up lots OS instances simultaneously to test various setups and each of them need 8GB, you can have at most 2 instances on a Mac.
For most use cases, none of those apply, so you would be better off with a Mac, a Graviton instance or an off-the-shelf PC.
Obviously in terms of single-core performance your Mac (or any desktop system for that matter) will always do better. As for actual power consumption, it always depends on the load and additional devices connected to the Pis. The only easily comparable aspect is the price, but it's meaningless on its own.
I tried to setup something like this, but let's be frank. The performance of this thing makes this largely pointless. I was able to learn much more by setting up virtual machines and configuring them into a cluster. Only advantage of RPi cluster is that you can physically pull the Ethernet plug and test the failures. But that's more like a "fun" factor rather that utility.
It's definitely worth the effort so why not just give it a try ....Promotion ends soon so DONT GET LEFT OUT...Click on the link, join and start earning.
The Pi does not do poe natively. POE is available as a hat, but I have not seen one for the Pi4 yet. The hats are nearly as expensive as the Pi itself.
I'm using the existing PoE hat on several Pi4's in a production environment and it works great. Never had shutdowns or issues with power. Note that I'm not drawing extra power from any of the USB ports, so perhaps the existing PoE hat isn't sufficient when you have USB devices attached to the pi4.
Good to know. I couldn't see any reason that they wouldn't work for a Pi4, but I couldn't find a single manufacturer that said Pi4 was actually supported.
Running a 6 node cluster of Pi4s at home now. Did some benchmarks. Long story short.
USB thumb drive boot disks are better performance and cheaper than SD cards.
Fans are completely unnecessary on a Pi4 with an aluminum heat shield style "case". They do run hotter than with a fan, but they will not be heat throttled with recent firmware.
looks cool, and the breakout board is nice, but they don't show how it attaches to the pi. it's aluminum yeah, but does it actually do anything for heat dissipation?
I've been using the FLIRC case, which appears to be die-cast and looks a little bit more family friendly sitting next to the TV on your entertainment center.
They recommend Kubernetes for private cloud workloads. But I am wondering if there isn't room in the ecosystem for a new virtualization / containerization technology to orchestrate tasks on multi gpus? Have heard NVidia is developing its own OS, or rather two, one for Cloud AI and one for real time robotics, but its still research phase...
yeah, I'm excited to get my hands on some Jetsons too. the 4gb memory on the pi is really a big pro though. when one of these SBCs finally comes with 8GB and GPU acceleration for < 100$ we will see some real progress I think.
That's a slick cluster in a box. I'm not sure which modules I'd put on it, though. Unless you harvest them from the 2 GB dev kits, the total price pushes up against a single Jetson Xavier AGX, which is a better value (and has an M.2 slot for an SSD).
any usb 3.0 drive works, but I am using SanDisk Ultra Fits. price per byte is less and performance is similar or slightly better for reads and way better for sustained writes.
No Overclock.
99-100% CPU on all 4 cores for hours and no throttling. they do run at 75-78C instead of 65C, but I could not get them to throttle.
Even with the cheapo 3 piece heat sinks there was almost no throttling, but they did throttle occasionally after sustained stress.
missed the SD cards. I am benchmarking against the best SD cards I can get because that is always the bottleneck on the Pi to my experience. The bandwidth and clock rate on USB3 vs sdcard bridge is just faster, but the media is also cheaper in USB3 format in general.
For a build like the one in the article, I imagine something that puts two nvme sticks beside each other, sandwiched between every other rpi, with passive aluminum heat shields in the remainder of the spaces between the rpi’s. I bet it could be even denser (more pis and more storage) and quieter.
I wonder if space can be saved by making a heat shield heatsink that has a pi on both sides, and fins in the middle.
>Fans are completely unnecessary on a Pi4 with an aluminum heat shield style "case". They do run hotter than with a fan, but they will not be heat throttled with recent firmware.
1 x 4k@60hz with no case and alu heat shield and my RPI4 does throttle. Latest firmware. Seems you don't punish it enough.
USB isn't very good for high vibration environments (e.g. inside robots) though :-/ Worst abomination of a connector design I've seen in a couple decades. I wish they had locking industrial connectors on the USB ports on the Pi ... although a SATA/m.2 would be nicer.
Not really, because that makes it really hard to unplug/replug for servicing.
I wish USB was designed to at least snugly support housing to housing of connectors and not have the entire connector supported by a flimsy piece of sheet metal soldered to a PCB.
cloud style microservice architectures. saves a lot of power and no single node is carrying all the weight. distributed computing is scalable. 1 big xeon is not.
This is so exciting. Is this type of "movement" (trend) likely to see data centers coming back on premises for enteprise markets? (ie. back from cloud)
Right now, yeah. But if this tech trend (miniature, low power, cheap, commodity) continues and we can run our apps on them ... it might be a game changer. No?
Just for a different perspective, I grew up in a very low income household, and the Pi was the only computer I was able to convince my mum to get me as an early teenager. "Cheap" is relative, and I disagree that the Pi wasn't a big step forward in terms of price on the very low-end.
This is what an engineer would say and that's important but, behind the scenes it's all about operating costs (e.g. administration). I worked with Cisco on a contract, they had about 22 folks managing the day to day operation for a specific on premise integration. Imagine if Cisco didn't have to pay 22 folks to manage the day to day? Let's say we cut that need in half but the licensing price remained the same or you could repurpose good engineers to deliver value elsewhere?
All of the elastic and horizontal stuff is great, but that's covered under SLA.
Depending what you are doing. For use cases like Dropbox cloud was never an option, other use cases can realize 30% year over year saving just by moving to the cloud. I think where RPIs make the most sense is soho, office building use cases, onsite backups, NAS, hosting infra services (DNS server for example) where you can use many nodes without much power and with a good amount of redundancy.
My only beef with RPIs as servers is disk I/O is terrible. If that's been solved, I'd love to know! If the SOC has PCI-E lanes that were accessable externally I feel like the server side would really take off
I think recent firmwares would allow Sd-card-less pxe boot. I was wondering the other day if you could then run the whole system from RAM disk only and/or maybe mount volumes via iSCSI from SAN (keeping in mind limitations of RPi 1Gb nic).
Pi's don't last as long as traditional hardware, but if you are running a self-healing cluster of machines, why wouldn't you use this for production environments?
There isn't one. For actually delivering an application, a traditional server is better in every way, unless your application is so small that you are wasting your hardware and get killed on power costs.
It's more or less the same as the people in /r/homelab who have full server racks in their basements but never do anything with most of their eBay servers other than install CentOS on them. At least those can serve movies via Plex, though.
I mean there are probably a few super edge cases for something like this. A digital art display, or theater production for instance. Something you can custom 3d print yourself is just about perfect for that.
lol brrrrt. Actually there's heaps of cases. Traditional servers are large, power hungry and generally 10-100x the compute power most jobs need if you buy a piece of hardware for a site. Traditional server also requires expensive monkey like me to service.
RPI rack on other hand. plug and play. Cheap as chips to keep failover as you can keep backup rigs in a cupboard. It doesnt require expensive monkey to unplug bits. Most sites I serve have minimal if any internet at all. So cloud compute not an option everything has to be done on-site. RPI is perfect. It has more than enough grunt, costs bugger all, performance is rock solid.
Disks, man. Where are your disks??? Compute by itself is next to useless for anything serious. You need data. What do you need a compute node for if you don't have any data? You could just do it client-side in the browser.
making something like this with eg, amlogic socs, would be significantly faster and require no fans and less power. it would be cool to see someone do something like this with the not terrible rpi broadcom chips
That is not true at all. For compute, it's about 5x slower than the cheapest Intel processor you can buy.
It's certainly much better than the original Raspberry Pi (which felt like a 486), but it is not using recent technology and it just doesn't compare to what is available today. It is for projects where you would normally use a microcontroller, but want HDMI and Linux out of the box. People using the Pi for art projects, signage, dashboards, controlling their sprinkler system, etc. have the right idea. People using them to serve interactive traffic are going to have a bad time. They are really, really slow.
Hi! thanks for posting my project here ) I will be glad if you subscribe to my instagram https://www.instagram.com/uptime.lab/ I don't sell anything, but feedback is very important for me
The hotswap 3D printed mechanism is pretty neat (reminds me of the swappable harddrive days). As for Rpi clusters, its just a neat idea to think about. It's not entirely trivial hardware or software wise to load balance 14 Rpis for "one purpose".
I think for me there's just a hobby/cool factor. I have no intention to build one nor can I think of a good use case but it's still neat to look at someone else's handiwork.
172 comments
[ 1405 ms ] story [ 2807 ms ] threadThere's room for a much larger fan so it seems a case of bad design by Raspberry Pi
If you can distribute your workload (or it's not intensive anyway), i.e. having no single machine with a faster processor, then I think it's a good choice, cheaper to run.
Edit: I'm happy to be corrected if down-voters would like to reply... A Pi 4 draws 6-7W at full load. A former desktop PC turned server likely idles over 100W.
They're different beasts, so I'm not saying 'a Pi is better than a PC', I'm saying 'if your load could also be handled by a (bunch of) Pi(s), then all else being equal that might be better than the PC'. Of course all else is never equal, ymmv, you may already have the old PC, you may need very many Pis, so even if pennies cheaper to run it might take years and years to break even on the Pi acquisition cost.
I also like the idea of having an easy homogeneous setup, each Pi having one disk, vs. my alternative (old PC parts) being something like '4 SATA ports on this one, and 6 on that one, but one's broken, less one on each for the boot disk' - I quite like the idea of having a bunch of Pis, and one image that I apply to all of them, which can include details like /dev/sda being its only data disk. That's not really a functional advantage, just appealing for keeping it simple to come back to and remember/understand how I set it up when I need to tweak something (and really easy to add another one increasing available cpu/ram/disk).
Picture in Swedish, from graphics card review comparing average power draw for the entire system for various GPUs, idle and while benchmarking in Metro Exodus, numbers rounded to nearest multiple of 5: https://cdn.sweclockers.com/artikel/diagram/22452?key=2c6f21...
If the direct picture link doesn't work, it's under "Effektmätning" here: https://www.sweclockers.com/test/30908-amd-radeon-rx-6900-xt...
[1] https://magpi.raspberrypi.org/articles/raspberry-pi-specs-be...
[2] https://www.pidramble.com/wiki/benchmarks/power-consumption
Y'all negative folks against this seem to have very restricted use cases in mind. Not all redundant rack servers live in a 36U case in a air-conditioned internet connected server farm. There are plenty of use cases outside of this.
Advantages: Cheaper than conventional servers, limited physical form factor suitable for constrained environments such as apartments, luggage, etc. Low power consumption, suitable for mobile/off-grid/renewable.
Recommended software: https://clusterlabs.org https://www.kernel.org/doc/html/latest/admin-guide/nfs/nfsro... https://wiki.gentoo.org/wiki/Diskless_nodes
Commercial advantage: Run highly cost and power efficient data centers, call centers, etc. Eliminate redundant state. Remove manual configuration rollout. Infrastructure as code. Full CI/CD with kernel-level control, per node, for any OS. Basically becoming a lost art in this day of "outsource to cloud". With SDN knowledge, confidently build your own cloud provider.
Many other small VM hosts accept payment by Bitcoin.
None of this is true.
"The most common one was having too little money to use an account. Two-thirds cited this as a reason for not having a financial institution account, and roughly a fifth cited it as the sole reason. Cost and distance were each cited by about a quarter of those responding to the question, and a similar share said they do not have an account because a family member already has one. Lack of documentation and distrust in the financial system were both cited by roughly a fifth of adults without a financial institution account, and religious concerns by 6%."
https://www.cashmatters.org/blog/globally-17-billion-people-...
(And to state the obvious: poverty is also a barrier to owning physical infrastructure)
"What about all the homeless Iranian Ferrari owners!" doesn't seem like much of a rallying cry.
Or maybe to Don Quixote, but he was famously delusional.
It's the same here.
Also you are comparing a space shuttle to a few raspberry pis, hardly apples with apples.
What's the best way to go about clusters since I think you need 1gb RAM per node and lowest I've seen is $5.
Thanks
Honestly, could be as valid with a setup that just involves running a bunch of VMs.
> Low power consumption
Yeah, well, a typical laptop should draw around 50W, which is what, like 0.05$ a day?
> Commercial advantage: Run highly cost and power efficient data centers
Hu, no, in terms of power efficiency per unit of work, you will not beat specialized hardware.
> Infrastructure as code. Full CI/CD with kernel-level control, per node, for any OS. With SDN knowledge, confidently build your own cloud provider.
Sure, also, blockchain, 5G, serverless, microwaves, rust, wasm
Lets come back to earth for a second. There is objectively no purpose for this, except it's fun and nerdy. Do we really have to find excuses?
> Honestly, could be as valid with a setup that just involves running a bunch of VMs.
Slightly less valid. You don't have the same IO latency between nodes using VMs unless you start messing around with artificial attempts at inducing latency on each node individually.
It's much easier to accurately simulate real world networking by not bothering to simulate and just using real world networking.
There's a huge difference between latency on a wire vs latency within a hypervisor, and you won't know what you've papered over through lower latency via VM until you move to real hardware.
At least where I live that would be 1.2kWh and about 13 cents per day, around $4 per month, or around $50 per year if running 24x7.
Pacific Northwest has I believe some of the cheaper power in the U.S. at 10-11 cents per kilowatt hour.
When you're done with the cluster, you can likely sell them for nearly what you paid for them.
I host a lot out of my house on a T430S with a cracked screen.
In the case of Raspberry Pi it is pretty normal nowadays to connect a bunch of them into a cluster and have some fun. In fact, it has become so popular, that you u can find stackable cluster cases and other Pi cluster hardware on amazon.
I tried running Nextcloud on Raspberry Pi 4 (4GB) a while ago, and hitting performance issue on SD card very fast (initial DB import take very long). I tried to solve this by moving all Nextcloud data into iSCSI disk but RAM & CPU became a problem once I started adding more files. After a week or so, I tore everything down and went back to a VPS instance.
It might works better with Raspberry Pi cluster on Kubernetes, though. (This very Raspberry Pi is now happily running as a DVB streaming server.)
The RPi4 with a USB SSD should be fast enough and use little power.
Not everything needs a business case, some things (most things) should be done just because you want to see if you can.
If someone really needs to think in items of ROI and that everything needs a business case, then this ideology of doing things for fun and testing limits is the only way to really innovate.
The alternatives are much more powerful, but far more expensive and not very attractive in other ways. Blade server? Huge, noisy, power-hungry and I wouldn't get much change from many thousands of pounds. Rack of NUCs? Powerful of course but far more expensive, large, separate power bricks, ethernet switch and lots of wiring required. VMs running on a desktop? Still more expensive, larger, and not the same tangible experience as having a real cluster. VMs on a laptop? Laptops at the same price point would have too little RAM to run lots of VMs well.
So these Raspberry Pi clusters fit a niche which is not well served by other solutions.
But in terms for actual cost and performance; how does this compare in terms of peak performance, power consumption and actual total cost to mass produced quality hardware like the Apple Mac Mini?
I don't the actual numbers but I have the feeling it won't fare that well.
You can use it as a USB-hub though as I suppose you get 2 USB-ports pr. raspberry pi.
- The project was obviously started before the Mac mini was available
- Linux guests are still rather experimental and might not behave the same way as a real machine.
- 32-bit Arm applications do not run on the M1, it is a 64-bit-only CPU
- Nested virtualization is said to not work yet under MacOS, so you cannot run KVM guests inside of the Linux guest
- If you need to spin up lots OS instances simultaneously to test various setups and each of them need 8GB, you can have at most 2 instances on a Mac.
For most use cases, none of those apply, so you would be better off with a Mac, a Graviton instance or an off-the-shelf PC.
Or a kiosk with 14 HDMI displays.
Serving web pages. A single Raspberry Pi 4 can manage 943 Mbit on its Gigabit port.
You might be able to buy (expensive) 40 Gigabit network adapters for the two TB4 ports of the Mac Mini. You'd still be slower.
https://t.me/ETH_Ads_jp_bot?start=1498941506
It's definitely worth the effort so why not just give it a try ....Promotion ends soon so DONT GET LEFT OUT...Click on the link, join and start earning.
https://t.me/ETH_Ads_jp_bot?start=1498941506
[1] https://www.raspberrypi.org/products/poe-hat/
USB thumb drive boot disks are better performance and cheaper than SD cards.
Fans are completely unnecessary on a Pi4 with an aluminum heat shield style "case". They do run hotter than with a fan, but they will not be heat throttled with recent firmware.
https://www.amazon.com/Geekworm-Raspberry-Computer-Aluminum-...
https://www.waveshare.com/pi4-case-argon-one.htm
Edit to add the fan is optional, so it can run fanless with no throttling.
https://flirc.tv/more/raspberry-pi-4-case https://flirc.tv/more/raspberry-pi-case (for RPi 2 and 3)
Admittedly, that's a simple load, but my Pi never gets over 42 degrees C, in a tropical area, averaging around 30 degrees C.
https://www.seeedstudio.com/Jetson-Mate-Cooling-Kit-p-4784.h...
They recommend Kubernetes for private cloud workloads. But I am wondering if there isn't room in the ecosystem for a new virtualization / containerization technology to orchestrate tasks on multi gpus? Have heard NVidia is developing its own OS, or rather two, one for Cloud AI and one for real time robotics, but its still research phase...
It might have GPU acceleration but the RPi already comes with 8GB of RAM for about 80$.
https://www.raspberrypi.org/blog/8gb-raspberry-pi-4-on-sale-...
Do you overclock?
If you ran at 99-100% CPU on average, would you still not need fans?
No Overclock.
99-100% CPU on all 4 cores for hours and no throttling. they do run at 75-78C instead of 65C, but I could not get them to throttle.
Even with the cheapo 3 piece heat sinks there was almost no throttling, but they did throttle occasionally after sustained stress.
However, there are usb3 nvme enclosures that are incredibly cheap, and blow both out of the water:
https://www.jeffgeerling.com/blog/2020/fastest-usb-storage-o...
For a build like the one in the article, I imagine something that puts two nvme sticks beside each other, sandwiched between every other rpi, with passive aluminum heat shields in the remainder of the spaces between the rpi’s. I bet it could be even denser (more pis and more storage) and quieter.
I wonder if space can be saved by making a heat shield heatsink that has a pi on both sides, and fins in the middle.
Super cool article.
1 x 4k@60hz with no case and alu heat shield and my RPI4 does throttle. Latest firmware. Seems you don't punish it enough.
I wish USB was designed to at least snugly support housing to housing of connectors and not have the entire connector supported by a flimsy piece of sheet metal soldered to a PCB.
Hmm ... seems like you can also network boot them over DHCP/TFTP:
https://www.raspberrypi.org/documentation/hardware/raspberry...
https://www.sparkfun.com/products/14882
The HN hug is strong this morning
Why do so many people want to see a higher purpose in any fun experiment? Can't it just be that, a fun and cool hobby project?
All of the elastic and horizontal stuff is great, but that's covered under SLA.
My only beef with RPIs as servers is disk I/O is terrible. If that's been solved, I'd love to know! If the SOC has PCI-E lanes that were accessable externally I feel like the server side would really take off
The RPi 4 also has support for USB 3 disks using UAS. I've attached a couple of SSDs to mine. https://rwmj.wordpress.com/2020/09/24/raspberry-pi-4-running...
Edit: someone has done this already. Of course! :) ( https://tech.xlab.si/blog/pxe-boot-raspberry-pi-iscsi/ )
It's more or less the same as the people in /r/homelab who have full server racks in their basements but never do anything with most of their eBay servers other than install CentOS on them. At least those can serve movies via Plex, though.
RPI rack on other hand. plug and play. Cheap as chips to keep failover as you can keep backup rigs in a cupboard. It doesnt require expensive monkey to unplug bits. Most sites I serve have minimal if any internet at all. So cloud compute not an option everything has to be done on-site. RPI is perfect. It has more than enough grunt, costs bugger all, performance is rock solid.
Compared to something we all know, e.g. MBP 16" 2019, how does this rack of Raspberry Pi's stack up?
I know it can be like comparing apples to oranges but I was just curious how fast this can be.
It's certainly much better than the original Raspberry Pi (which felt like a 486), but it is not using recent technology and it just doesn't compare to what is available today. It is for projects where you would normally use a microcontroller, but want HDMI and Linux out of the box. People using the Pi for art projects, signage, dashboards, controlling their sprinkler system, etc. have the right idea. People using them to serve interactive traffic are going to have a bad time. They are really, really slow.
Every step deeper, or obscure loses you more and more mass appeal.
Therefore, projects with high max appeal make it to the top of lists like HN. It's no conspiracy.
How its going: Raspberry Pi Server Mark III