112 comments

[ 3.0 ms ] story [ 181 ms ] thread
It's incredible how powerful these small things are. I have a couple and I use them for just about anything, from access servers to access my home network, to running various applications inside containers on them. I did plan on making a cluster some time ago but for various reasons I've put that plan at the back of the line.

The real question here is did they use it to run some heavy applications on the cluster and how did it perform? Basically do they have a purpose for it or was it simply a "because we can" type of thing?

I think an objective might have been demonstrating the benefits of Oracle Autonomous Linux regarding automatic patching and tuning; lower TCO being the advertised selling point.
No benchmarks. Cool but just seems like an ad for Oracle.
Of course, you're not allowed to benchmark Oracle publicly, it's in the EULA :)
Seriously? As someone with purchasing power, I could only assume they were slower than alternatives in that case. Anything they quoted me would have to be considered marketing fluff without independent numbers.
I can almost understand why they do it. Benchmarking is incredibly hard to do correctly, and it requires a very deep understanding of software and hardware.

For example, I remember hearing about Joyent's port of KVM to Illumos. They were testing the Illumos KVM against Linux KVM, and the Illumos KVM was faster. After a lot of debugging, they realized the Illumos machine was turboboosting better than the Linux machine, since it was in a colder part of the data center.

I wonder what the cost-effectiveness is in terms of compute power. I'd assume not great, but who knows? Raspberry PIs pack a lot of punch for their size and price.
Definitely not great but thats one of the tradeoffs.
If they can leverage the GPU on the Pis it could be quite a bit of compute power spread across that array. Even if they're purely CPU bound that is 4240 cores which is not nothing.

Power wise each Pi going flat out burns around 5W, so all told you're burning ~6kW, which is quite modest for a big computing setup.

It looks like you can get 400 single precision MFLOPS for those 5W. So 80 MFLOPS/W +/- 20%. It isn’t the worst and it isn’t the best, but it is respectable. It would be a big hassle to make and maintain a cluster of thousands of these.

https://www.arnabkumardas.com/platforms/nvidia/nvidia-jetson...

https://raspi.tv/2019/how-much-power-does-the-pi4b-use-power...

https://en.wikipedia.org/wiki/Performance_per_watt

...but you could experiment with "disposable computing" (ie: similar to original google datacenter racks: never replace a node, only replace a rack at a time if 51% is failing).
That and interesting configurations (systolic arrays, message passing architectures, Erlang on a large number of nodes and so on). Lots of fun with stuff like this.
Oh jeez. Out of curiosity I did a comparison against a gaming GPU. A GTX 1080 uses 200W of power for 9 TFLOPS, which is roughly 500x better performance per watt than a raspberry pi. Outside of the fun of it, and maybe a cheap way to prototype larger systems, I can't see a serious reason to make a raspi cluster.
When it comes to cluster-scale computing, size is not the main issue. The main issues are: compute/$ and compute/watt.

There is also the matter of how much RAM and disk can be attached to each board, and the speed of the NIC. Gigabit ethernet is a little slow these days.

GbE is slow, yes, but GbE for the amount of compute a raspi 4 can do is massive. Raspis don’t even have the oomph to max out GbE links. You hook 48 raspi 4 into a switch and using a 25 GbE link to a storage cluster.

Edit: It is conceivable that a raspi has the oomph to max out a GbE link but is limited to 300 Mbps due to the use of a USB 2.0 bridge in between the ethernet interface and the SoC. My mistake.

The RPi4 is capable of gigabit. I benchmarked mine at 900-950mbit via iperf yesterday. That was to a usb3 gigabit ethernet adapter on a laptop.
Ah thanks. My previous post is dated knowledge, applicable only to the raspi 3.
I've been curious about RPis for a while.

I have a file server running AMD Phenom II 955 that I'm thinking of replacing with a RPi. Is there a practical way to have RPis manage a software RAID? Performance is not a consideration. Is it even worth looking at RPi in this scenario? The advantages I can think of is small footprint, lower electricity cost.

> Is there a practical way to have RPis manage a software RAID?

I can't think of any good reason to do use software RAID, let alone doing so on a Raspberry Pi. I would recommend hardware RAID, ZFS, or a Synology appliance.

If performance is truly not a concern, yes it is possible to do this with a Pi but I don't think you are going to be super impressed.

ZFS is 100% software raid.
Sorta. At the end of the day, yes, the disks are being managed by your CPU and not a dedicated raid controller... but it's a completely different ballgame than something like mdadm.
Not "sorta", it's software raid and thus, like software raid, it's portable to any system that'll run the software and JBOD the drives.

Most would argue RAIDZ is better, but many of these reasons are functionality based over and about just redundancy, i.e. Copy on Write, et all.

I say sorta because in most circles when one is speaking of 'software RAID' they are referring to a very specific configuration and are explicitly not referring to ZFS.
Friends don't let friends create RaidZ pools.

Bite the bullet of the cost of N*2 mirrors.

ZFS doesn’t support hardware RAID controllers?
It supports it, but ZFS itself is operating on the host CPU, not the RAID controller. Additionally, ZFS prefers that the RAID controller is disabled/defeated and operating in JBOD mode where disks are passed-thru to the OS without any caching.
Not just caching. The problem is that non-JBOD HBAs also swallow error codes and mask retries, meaning ZFS is potentially unaware of some read errors and is not in control of the latency as the controller firmware has its own timeouts and retry counters. Moreover, most will not pass through SMART or else fake it.
(comment deleted)
Software RAID = simple. At least when I built this machine nearly 10 years ago. Since then it's been rock solid, if not blazing fast. There were also horror stories of hardware raids failing due to controllers failing; or subtle differences between different revisions of RAID controllers. That said I also had budgetary restrictiions at the time that made software raid attractive.

I looked at NAS appliances, but I think it's still cheaper to create one from scratch (assuming existing knowhow, and "free" time :)

Software RAID is easier to manage, easier to do disaster recovery, does not require specific driver support, and easier to move to other hardware.
Not a Pi but the rockpro64 has a pcie port in which you can plug what you want (a sata card for example).

I made a nas with one 3tb drive and zfs. Zfs was a bit difficult to install but now it works really well and the power consumption is low.

You have to use the Linux images mades by ayufan (a third party) though. The manufacturer just list a bunch of distro that run on the cards and most of them seem to be made by some volunteer. I'm not a big fan of that.

There is nothing wrong with software raid. Synology uses SW raid in all their NAS products as far as I know, right up to the flagship models with > 10 drives and 100T+ of storage.
Hardware raid controllers are a single point of failure and when they fail the vendor sells a different product so you will always have to stock up and buy two at once.
It's a bit difficult since the RPi doesn't have any form of storage controller, just a USB hub. So your system would have to be built out of externally powered USB drives, which would be slow even by your lax standards and likely finicky.

It's hard to see the advantage over a dedicated NAS device, since the savings on the controller gets eaten up by needing to buy a whole bunch of drive enclosures and power supplies.

Maybe if you could use something like AoE turning the network port into the backend storage interface a la CoRaid. Otherwise like others have said you have USB.
For RAID, no. The pi has a few strikes against it for that case. If you're simply after redundancy, ZFS may work for your case.
Usually you can get a new NAS really cheap on Black Friday. Even today you can get a 119j for $103, which isn't much more than a rpi with card and case.

https://www.newegg.com/synology-ds119j/p/N82E16822108418

A one-bay NAS... didn’t even know those existed
Yeah I use as backup, and it syncs to the cloud. So I figure if it dies I'm still good as I have the original and cloud. It also means I dont have to pay for 2 disks or think about RAID. It is slow and there is a learning curve but mostly just works and is very low power - at it shuts down disk when not in use.
You could set up a couple of GlusterFS or Ceph nodes. With the RPi4 having USB3 and Gigabit Ethernet you no longer get cramped by the previous generation's internal USB 2.0 Hub.
I think the most effective approach in this case is to set the requirements and start from them.

This is not trivial advice :-) Requiring 4 drives in RAID10 has different consequences from 2 drivers in RAID1. Requiring a price < 100$ is different from < 200$. Hacked storage setup (eg. USB drives) versus proper (eg. storage controller with multiple SATA ports) is different.

For example, if you're willing to spend more, but not so much more, than say, 80$, why limiting to the RPi? The Odroid H2, for example, gives a lot of options.

Among the other things, I use it as file server with a COW-filesystem on two (cheap) SSD disks in RAID1. The board has two SATA ports, so it's a solid hardware setup. It has two ethernet ports, so it's ready to work as router. It's also x86, so it will have software support for a very long time¹. And it also runs at an incredibly cool temperature - I bought the fan, but it rarely spins. To be fair though, the H2 cases are expensive trash.

You can certainly do pretty much everything on a RPi4, as much as you can commute 50 km a day with a bike, which is... not ideal :-)

¹=RPi have an enthusiastic community, but ARM boards are second class citizens when it comes to Linux support. While Linux is formally compatible, it's still adviced to run the specific ARM distro.

There are one or two NanoPi models with an M.2 slot and full gigE, but that doesn’t get you an array unless there’s some m.2 breakout box I’m not familiar with.
I'm (still) in the process of re-purposing an old laptop for a home server. 2TB laptop hard drives are a reasonable price, and on older laptops with a CD/DVD drive bay, you can usually get an adapter caddy to put in a second drive. Pay attention to the allowable drive height though.

This system is going to be set up with software RAID.

I expect to have higher compute power, more RAM, and much better IO throughput than a RPi solution. Also, the laptop battery (which is still good) serves as a built-in UPS.

maybe the Pi 4 would be able to do this with a USB attached disk box, but you'd be much better off with something like a commercial NAS - and I say this as someone who has used ARM boards since the Sheevaplug.

It all depends - do you want a fileserver or a crash course in how to build a fileserver?

> I have a file server running AMD Phenom II 955 that I'm thinking of replacing with a RPi. Is there a practical way to have RPis manage a software RAID?

I did this using an rpi2. It is non-trivial in terms of effort.

In a practical sense the rpi is limited in terms of power to the USB ports, so any build with 2+ HDDs and some sort of wireless USB keyboard and mouse dongle will use up too much power.

In other words, I had to buy a powered USB hub.

I used mdadm to do a software RAID, which more or less worked out of the box but took a while to clone the original drive. Throughput wasn't amazing for anything except occasional pulls and puts; I wasn't going to stream 4K video off of this, though I did have some luck with Subversion and music streaming.

It was clunky enough that I got a Synology box a year and a half later and never looked back.

That's what Azure feels like sometimes, running like a cluster of Pis.
I really do wonder why Azure is so slow. You have to optimize somewhat aggressively to squeeze average performance out of Azure SQL, for instance.
Maybe because Windows updates and xbox love run through the network?
Pretty sure all of Netflix runs on AWS and I've got to think that's several orders of magnitude more traffic than Windows updates or xbl.
FWIW, the video streaming part of Netflix does not run in AWS (but, AIUI, pretty much everything else does).
Oh, TIL I guess. Thanks for the correction.
Netflix hosts video content at ISPs' sites: https://openconnect.netflix.com/en/
Edge content yes, but everything else is in AWS
The edge traffic makes up 99%+ of the traffic in question.
Do they publish how many ISPs participate? I am sure the big ones do but who knows if local Fiber is connected to someone who is behind an OpenConnect
Non-participating ISPs get it from participating ISPs. Basically no one hits AWS.
Dominican here studying Telecommunications. I was giving a tour to the Datacenter of the 3rd largest telecom company in my country. They were using the Netflix box. I was giving a tour to the 4th largest telecom company, they too had the Netflix box. I was told the 1st and 2nd largest telecoms in the country also had the box.
My understanding is that Netflix will pretty much subsidize the cost of the CDN box to nearly any ISP above a certain minimum traffic level ingress from Netflix.

Great model to attack the incumbent [transport / transit] carrier costs - end run around the middle-men and drop your CDN boxes directly at the edge (last-mile) ISPs.

ISP here, the netflix cache appliance boxes (a single 1RU server is the minimum) are free, but you have to provide rack space, power and cooling for them. For an ISP in a place like a caribbean island it's a great idea because their upstream/IP transit rates are significantly higher than in the US48 states.
Back in 2012 or 2013 I built an auto-scaling WordPress hosting off of Azure. It worked OK, better than shared hosting, but the $15 base VM at Azure (it was a cheaper time back then) couldn't compete with the base $10 droplet at Digital Ocean (it was a more expensive time back then).

On top of that Digital Ocean's API was so much better than Azure. So I spent a week and switched to Digital Ocean. Needed to scale the instances less, cost less, so more profit!

Unfortunately after 2 months of development, and half a year of marketing it, we only managed to attract a handful of customers, and had to shut it down.

Why did try to scale when you had no customers?
If you want to offer a completely automated process, auto-scaling is important. Otherwise it'll fail as soon as the first customers come in.
I guess I wasn't perfectly clear. The technology was to scale the customer's wordpress installation when certain metrics were met CPU or RAM > 75%, clone droplet and load balance.

The technology was basically an a beefy server for NFS, a beefy MySql server, and a beefy load balancer. When you started hosting with us, you paid $50+/mo and got what felt like a standard WordPress install on a dedicated digital ocean droplet, when metrics showed that you were hitting the scale threshold, we would create a new droplet run an initialization script to mount NFS, set up the nginx server, and then attach it to the loadbalancer.

When the combined CPU or RAM droped below the initial scale threshold, it would drop one VM from the load balancer at a time keeping it provisioned for an hour in case it needed to scale again.

Worked amazingly well. Sure the load balancer was a single point of failure, but in the 10 or so months that it was running, we never had a second of outage based on pinging the websites.

Now that is very cool! I've always wanted to try something like that.

What was it written in?

The main application I left hosted at Azure. It was an ASP.Net MVC app.

I used the Digital Ocean API to provision infrastructure, and SSH to remotely login and run the initialization bash script. The init script configured nginx, NFS (maybe it was SMB, i dont rmember now), a cron job to minutely heartbeat statistics back to the MVC app.

There are so many better ways to do it now, though. Off the shelf load balancers with letsencrypt and an api, hosted dns, managed & clustered databases, monitoring api. It would be a lot simpler today than it was just 2 years ago, let alone 7.

These days you can do it (relatively) easily by using wordpress docker image on kubernetes (or any other container platform). e.g. using Traefik for load balancer, a mysql container or an external mysql db instance (aws, google and do now offer managed mysql), and storage provider that can be mounted from multiple nodes (nfs, glusterfs, ceph, etc) and you're done! You can throw in an sftp container that mounts /var/www so you can manage the wordpress installation using an sftp client.
It's massively, massively overcommitted. They managed to actually run out of hardware in a whole US region recently, even though of course most customers had lots of unused quota. A typical Azure VM must be extraordinarily busy given some of the latencies and speeds I've seen on them.
Why...
Exactly, I just feel that Oracle just needed to do a huge flex to the world to show that they are relevant.

Weird flex but okay, I'll just carry on with my ARM64 Cavium kubernetes cluster that blows this toy out of the ocean.

But they probably built it as test cluster to do initial work on before going for a big shot on a big boy cluster.
I’ve seen a few of these RPi clusters (obviously none this big), I think the main reason people make them is to demo distributed systems or build a homelab cluster(sometimes they will argue its cause of the power efficiency). Also they are usually a nerd magnet so it makes for a great conversation piece as this probably was for their convention.
This seems like Oracle's way of marketing to devs that they're a cool place to work... And it might have actually just worked
I think this is cool, but engineering for engineering's sake aside I've seriously looked at building a raspi cluster a couple of times and have always concluded that it isn't worth it.

The real selling point for me is energy efficiency and price, but the reason I could never make it happen is because most of the applications I want to use are x86 only and I don't have the resources to be a trailblazer in that area. In addition sourcing becomes an issue with raspberry pis when you aren't a big company like Oracle. It's hard to buy more than a handful as an individual.

The real selling point for me would be physical isolation. If you run all of them single-core you wouldn't have to worry at all about snooping using the various side channel attacks.
Yes, but how many RPis do you need to match the workload of a single x86 core? How much does it cost to cluster and maintain these systems? IMO you would probably have a better story all around if you based your cluster on low end AMD chips, or perhaps even old refurbished systems.

That's not to say this isn't cool engineering for engineering's sake. Just that if you are considering practical applications for this, maybe think twice.

When you need that guarantee of isolation you are picking your solution for different reasons than when you need a particular workload. So there is no point comparing the workload if that wasn't the deciding factor in the first place.
My point is that you can probably get physical isolation for less money on x86.
You're going to have to explain a bit more on how you intend to achieve that economy. I'm not aware of a single SOC+peripherals based on x86 that would beat the Pi in price/performance, is there such a thing?
The Raspberry Pi Zero's are ridiculously affordable. For the cost of a regular Pi 3 you can get one thats wireless with a case with 3 covers for the case (1 for the Raspberry Pi camera). So maybe the Zero W's are more of what you might want. If you dont need them all to have wireless then you can just get the Zero's instead.
I'd want a zero for projects. I have a 3B for messing around with and wish I already had PoE out so I could just set and forget a Pihole box or something. At the moment my Pihole instance is on an x86 Raspbian VM for simplicity but I think tonight or this weekend I'll set up the 3B do be a backup and point my router to it as a secondary DNs server. At least it'd be using the Pi vs it sitting in my drawer.
By sourcing, I wasn't arguing about their price. The problem is that it's hard to find anyone who will sell you more than a handful of them at low prices.

I haven't checked recently, but at least that's what I remember.

> most of the applications I want to use are x86 only

Really? What software in this space (headless server cluster) is still x86 only?

Unfortunately some proprietary scientific software is compiled for x86 only and we just get access to the binaries. I don't like it, but that's just how some fields work.
This is purely for education and show.

I worked on two top-50 supercomputer clusters for science applications. You learn a lot by doing. You find out what happens to your electrical power, TFTP, DHCP, and syslog servers, and other infrastructure when those hundreds/thousands of nodes boot up all at once. Your Cisco switch reloads when you add a secondary IP address to all of your network interfaces because it's CAM table overflows. You learn how to automate without stuff like Ansible and Chef because they don't exist yet.

Ha, it was my experience with HPC that lead me to convince our CCIE network admin that CAM tables even could overload on a vlan of that size (we were using layer 3 switching to route tons of other vlans and this particular vlan was very active in traversing those boundaries and was also using multicast and lots of other weird things). We were seeing an issue when where computers on the same switch would lose their ability to see each other, even if they could see other machines on a different switch and everything about it smelled like the cam table was not accepting new entries. I finally convinced him to clear it out on one affected port and it worked.
That's a pretty stupid reason. The real reason ist that most software doesn't run on a cluster at all and an 8 core Ryzen costs less than $170 and $340 for a full system without a GPU but is significantly faster than a 10 node rasperry pi cluster ( doesnt even include sd card power supply network gear)
Now load any piece of Oracle software onto it and get charged for running 1060 CPUs. Whoops.
Oracle doesn't charge per CPU it charges per core. Enjoy paying for 4240 cores!
I don’t know why you’re getting downvoted, you’re correct!
I believe there's a scaling factor, so lower powered cores like this are charged as a fraction of a Xeon you'd normally use.
ARM isn't listed on their table, so it slots into "all other multicore CPUs". Scaling factor is 1. So no help there.
“The only thing that scales better than an Oracle database is their licensing cost! “

- heard years ago from a friend. Not sure of the origination of the quote.

Aha - I stubled upon this https://jira.mariadb.org/browse/MDEV-10872 yesterday and thought "why would they want to achieve compatibility with Oracle DBs?" => now (I think that) I do understand that it would be to offer an alternative that would provide the same capabilities (but maybe with a different performance/reliability/whatever) for a different price, for any software that Oracle sells which wants to run on an Oracle DB, right?

Is the process "Google vs. Oracle about Java" (I understood that it's about the functions/methods declarations and not about the code itself) similar to a theoretical similar situation where DB XXXX would provide the same functions as the ones provided by the Oracle DB?

Whenever I see any technology news about Oracle, I mentally translate it as "Larry Ellison needs a bigger yacht".
I saw an article written by an Oracle employee about "why Java is relevant in $CURRENT_YEAR". And what was the illustration accompanying the article? The coffee cup logo? A graph? A cartoon? Nay.

It was a photo of Larry Ellison's yacht.

And suddenly the intended audience of the article came into focus.

It's 4240 cores total, Oracle Enterprise DB is $47,500 per core. So ~$201M.

Java SE is priced per processor. $17.50 per month at that CPU count. And so, a relative bargain per year at 17.50 x 12 x 1060 = ~$222k / year.

> At Oracle OpenWorld 2019, tucked away in the Oracle Code One area [...]

As someone not familiar with Oracle events, I'm rather confused about Oracle Code One. I found a FAQ [1] which says:

> Oracle Code One is the ultimate developer event where we will cover the technologies and programming languages developers have come to love over the years. Expect talks on Go, Rust, Python, JavaScript, and R and SQL along with more of the latest Java technical content.

but looking at the photos, I see in the background quite a few arcade video games. Is this some sort of break area for people to hang out in between talks, or something like that?

[1] https://www.oracle.com/code-one/faq.html

(comment deleted)
Oracle Code One is the successor of JavaOne. Since Oracle acquired Sun, the JavaOne conference was co-hosted with the much larger Oracle OpenWorld conference. Recently, JavaOne was rebranded into Oracle Code One, morphing it into a cross-technology conference rather than a Java-focused one.
I remember when JavaOne was definitely worth going to. I've not heard anything like that for years and just assume it's dead as "Code One." Anyone want to confirm?
TFA says this cluster was part of "developor relations" and runs OAL and Java.

Perhaps the arcade games are also running on Java and Oracle is simply trying to "show off" a variety of uses of Java or something.

Not much info on what they can actually do with it
>an array of USB power supplies.

Surprised they didn't just bridge 5V to GPIO pins.

This explains so much about Ellison's vision for Oracle Cloud.
Talk about a thermal design nightmare. Definitely just an "look-at-me" exhibit