Today we are thrilled to announce the Turing Pi V2. The Turing Pi V2 is s compact cluster in a mini ITX form factor with 4 x cluster nodes, 2x mini PCIe (Gen 2) ports, 2x SATA (Gen 3) ports, and new Raspberry Pi compute modules 4 support.
The Turing Pi 2 cluster is architecturally similar to the AWS Graviton clusters. The CM4 processor uses the ARMv8 architecture. You can build images and applications for AWS instances of Graviton 1 and 2, host cloud native apps locally or at the edge and more.
m2 connector is something we are still experimenting with. The first two nodes are exposed to mini PCI express port, the third node is exposed to 2-ports 6 Gbps SATA controller. You can use the network file system within the cluster to access hard drives from other nodes.
I guess it's possible to adapt the mini PCI express port to M.2 pretty easily too. Having it PCI express lets you do other interesting things though too besides just storage.
Would it be possible to install a modern graphics card on this (e.g. some ati Radeon)? That would require some physical adapter on the one hand, and driver support for armv8 on the other.
> The Turing Pi 2 cluster is architecturally identical to the AWS Graviton clusters.
Graviton2 uses Armv8.2-A, as opposed to the Armv8.0-A on the CM4. Graviton2's cores are "Neoverse" vs CM4's Cortex A73, which is also a significant difference. A lot of code will be portable, but these are significantly different cores.
I'd like to know more about the PCIe Gen2. From your comment it sounds like the root port is capable of Gen2x2, or is it 2x Gen2x1 ports? If I wanted to attach a PCIe accelerator card with a standard PCIe connector, is there some kind of adapter cable I would need to buy?
On the page you mention CI/CD applications and we are considering using Raspberry Pi for our CI infra (for the lack of a more powerful option). While there are various possible setups, one fairly common arrangement (especially in a more public-facing services) is to start a throw-away VM for each CI request. This is how our CI service works and you can see it here[1] in action.
And while it would have been nice to use Turing Pi, it does not appear to fit our requirements very well because we need to have high-performance storage attached to each node (where we store/run the VMs). An ideal board for us would be a network switch plus for each module a PCIe (or better yet, NVMe) slot and USB 2.0 ports (at least 2x).
I spent a bit of time testing the v1 board, and while it isn't without fault, it's the best SBC-based cluster project I've used (haven't personally tested the SOPINE Clusterboard though).
Since the CM4 is 2x-3x faster than the CM3+ in pretty much every aspect[1], I'd be very excited to be able to upgrade my Pi Dramble[2] 4x Raspberry Pi 4 model B cluster to a Turing Pi V2!
PCIe switching would have been really interesting to see, even in a non-concurrent mode.
One also desperately wishes that multi-host network adapters had a low-end market. A 1 x 25Gbe connection shared between 6 hosts would be epic. Or if sharing the connection is too much, just a 4 x 2.5Gbe NIC chip would be acceptably interesting & useful to have. I forget, maybe the gbe is free on the Pi4's chip & this is more of a non-issue.
PCIe switching sounds nice, but let me tell you that it is a terrible nightmare on every level.
Hardware: these PCIe switches are finicky as all get out; will it link up to your device reliably? Who knows, depends on how the switch and device manufacturers interpreted the PCIe specs.
Software: there is no nice easy standard for connecting host-to-host (which I’m assuming is what one would want on a multinode system Like the one linked). There exists the idea of a PCIe non-transparent bridge, which if I recall correctly, allows CPUs on either side of it to do blind memory reads and writes to each other (hope your IOMMU is set up correctly to protect you from ugliness that could occur here).
Long story short there’s a reason PCIe is a local bus; it was never designed to do anything else. Anything else is largely a proprietary bolt on.
I was definitely thinking of NTB, for host-to-host connectivity. As you say, it doesn't provide a whole lot of capabilities. However, there's been a pretty competent ethernet driver available on Linux since Linux 3.9 released April 2013. There's out of kernel NTRDMA if you want ibverbs like communication that will make really good use of the PCIe bus too.
In addition to mapped memory, there are some other modest capabilities: support for ringing the peer's doorbell (one way), & scratchpad configuration. There's a pretty good 2016 presentation on NTB in Linux here: https://events.static.linuxfound.org/sites/events/files/slid...
I for one think more people should be investigating & harnessing the open-source Linux NTB tools. Switches are a bit expensive, but compared to ethernet switches their cost/throughput is great. Alas, yes, the whole ecosystem is woefully undersupported, under-developed. Isn't PCI part of the whole PC thing, this great revolution in computing that allowed interoperability to rise? Such a pity that all the PCI SIG's seemed to fizzle out, that not a lot of people participated. Sure we got some SR-IOV but that feels like as far as we got.
Pretty anecdotal use case, but when trying to do some type of neural architecture search for ML models that are ultimately going to be run on the edge, running some performance evaluation on a raspberry pi is a decent proxy. I have seen teams build their own type a raspberry clusters for this to be able to evaluate many models in parallel. It's basically a testing proxy for ARM architecture.
But this thing only has 4 RPI per box so I don't quite understand which problem they are targeting either.
I use mine to host some non-revenue-generating websites (they are super cheap for that purpose), and also for some ARM image builds that I integrate into my Docker/Kubernetes pipeline.
I haven't yet moved to ARM CPUs on AWS for some of my projects, but may be able to more quickly if I can have a local cluster like this for testing that isn't as slow as the CM3+-based one the Turing Pi V1 supported.
i often see people claim raspberry pi cheap but in reality it's not. a cluster like this is probably not cheap and performance wise, it's slower than, say an AMD x86 system.
You'd be hard pressed to find an old laptop with 32GB of RAM, and a laptop doesn't provide the same operational conditions experienced by a cluster of 4 computers communicating over a network connection.
Scaling horizontally presents entirely new classes of challenges that aren't experienced while scaling vertically.
Anyone can buy a brand new raspberry pi 4 with 4GB of RAM for around 50€, which fits on the palm of your hand. You can even get them delivered to your doorstep with a few mouse clicks.
The same can't really be said for 10 year old laptops.
ThinkPad W530 fits the bill quite well. Broken screen puts it at around $100-150, $300 gets you fully working one. Less tyhan 4 CM4 modules, and what about Turing Pi 2 case? power supply?
>Scaling horizontally presents entirely new classes of challenges that aren't experienced while scaling vertically.
Pay more for challenges? If you really want to fight with bad scaling just run a VM.
>>Besides academic uses to perhaps teach distributed systems on these are there any real world use cases for these clusters?
A considerable amount of people is already assembling and running clusters of raspberry pis, including hosting companies. A cursory search through eBay or Amazon can provide you with an idea of how many people are already operating these small clusters and spending money on this sort of setup. I'm sure there's a significant demand for a COTS cluster solution that allows people to avoid having to MacGyver their own cluster just to get Docker Swarm or Kubernetes running on a small cluster of raspberry pis.
I know people are building these the questions is for what, CI/CD for ARM like others mentioned is plausible but only if you are targeting the Pi or something very similar just due to how “fragmented” the ARM ecosystem is with its multiple uArchs.
If you are building for mobile then it doesn’t makes sense to run CI/CD on the Pi and if you are targeting your own bespoke edge then likely having dev boards which can represent your edge hardware more accurately is probably going to be a better way.
Running CI/CD for ARMLinux apps is pretty much the only use case I can think of and even that might be limited if you are using extensions.
> I know people are building these the questions is for what, CI/CD for ARM like others mentioned is plausible but only if you are targeting the Pi or something very similar just due to how “fragmented” the ARM ecosystem is with its multiple uArchs.
I don't understand what point you tried to make. These hosting services, and also clients, are quite intentionally targetting the raspberry pi. More specifically, they do target Debian running on Raspberry Pi. I don't understand the confusion. I mean, is it also hard to understand why AWS users pay for ARM instances running Amazon's custom linux distro? In both the Pi and Graviton you can run nginx, nodejs, Python, ssh... So, what's the problem again?
Outside of doing it for the heck of it (which is a valid enough use case , IMHO), what practical use do you see out of this ? To me, it looks like I can do all of the same things mentions (perhaps even waay faster) on a x86 processor in the same size factor.
That's one of the point too. We have industrial customers who deploy apps in the places with a limited access to the internet, so increased reliability something important to them. Another thing is that they can host apps with different purposes on different nodes in containers. Besides that all the infrastructure could be managed with a tool like Kubernetes and organically work with existing cloud infrastructure. And the last but not least in a cluster like that you can combine general purpose modules with accelerated compute modules.
That's really nice. There are high-end products out there with redundant backplanes (dual or triple) and redundant PSUs (2 or more). To match that level of redundancy it seems you would need 4x Turing Pi, to be able to survive a simultaneous failure of 1x arbitrary PSU and failure of 1x arbitrary backplane (ie. the integrated RTL8370 GigE switch).
The 2x external GigE ports on the Turing Pi helps a lot, in that each cluster can connect simultaneously to 2 external GigE switches, in case one of those things fails.
Could be an interesting exercise to set up and get it all working, in particular K3s + dynamic routing might be a fun challenge.
Theoretically yes, but there seem to be only 2 PCIe (and probably x1 slots). And the interconnect will be quite slower (I suspect) btw. the modules. All those GPIO pins are gonna be wasted and if I am going to commercially deploy containers for customers, I'd rather get a solid industrial Ryzen/Intel fanless solution and call it a day
There is nothing in the schematics about multiple power supplies or battery backups, etc. If you are hooking up all 4 rpis to the same power source , might as well replace it with one x86 processor.
Yes, but I would opt for the newly released HC4 if two drives were fine. If you are looking for four to five drives, I'm sure someone will make a PCB at some point that uses A PCIe connected SATA controller.
Price becomes a real problem though. The Pi foundation carrier board is dirt cheap, everyone else's boards seem to be a lot of money. Probably due to low volume.
Stupid question, how do these 4 SoCs share the 2x SATA ports? Do you assign them to a specific processor somehow? Are the other machines booting off of PXE boot?
Out of curiosity, does the firmware in the Compute Module allow for PXE boot?
Would be a neat way to handle corruptions of the SD / eMMC. Flick the power off through the Turing Pi's I2C, have one of the other nodes act as a PXE server, flick the power on again to the affected node, and have it pull in a tiny kernel and init-script through PXE that flashes the SD / eMMC with an image from the PXE node.
Any thoughts on keeping accurate RTC time on these things at sites with poor internet connectivity, weird latency or restrictions on outbound NTP?
I'm thinking a nice addition would be a small module to plug on top of the 40-pin GPIO connector with a GPS receiver in it and an antenna attached. The attached node could run NTP and pull accurate time from satellites. Throw in 3 of the modules, and NTP can use quorum to keep the time accurate even if one GPS module fails.
I can't think of anything better than a GPS receiver.
About 10 years ago we had a legacy system that was using a weird shaped (think UFO spaceship) FM radio receiver to get time. I never understood why our provider chose that over a simple USB GPS receiver.
73 comments
[ 2.7 ms ] story [ 132 ms ] threadThe Turing Pi 2 cluster is architecturally similar to the AWS Graviton clusters. The CM4 processor uses the ARMv8 architecture. You can build images and applications for AWS instances of Graviton 1 and 2, host cloud native apps locally or at the edge and more.
And will it be set up where each board gets one of those slots (e.g. 2 boards get single PCIe, 2 boards get single SATA)?
Graviton2 uses Armv8.2-A, as opposed to the Armv8.0-A on the CM4. Graviton2's cores are "Neoverse" vs CM4's Cortex A73, which is also a significant difference. A lot of code will be portable, but these are significantly different cores.
I’m sorry if I missed it, but have you announced pricing?
For now i'm running a few RPi devices and i was dream about such device to put them in for more processing power.
And while it would have been nice to use Turing Pi, it does not appear to fit our requirements very well because we need to have high-performance storage attached to each node (where we store/run the VMs). An ideal board for us would be a network switch plus for each module a PCIe (or better yet, NVMe) slot and USB 2.0 ports (at least 2x).
[1] https://ci.cppget.org
Since the CM4 is 2x-3x faster than the CM3+ in pretty much every aspect[1], I'd be very excited to be able to upgrade my Pi Dramble[2] 4x Raspberry Pi 4 model B cluster to a Turing Pi V2!
[1] https://www.jeffgeerling.com/blog/2020/raspberry-pi-compute-...
[2] http://www.pidramble.com
One also desperately wishes that multi-host network adapters had a low-end market. A 1 x 25Gbe connection shared between 6 hosts would be epic. Or if sharing the connection is too much, just a 4 x 2.5Gbe NIC chip would be acceptably interesting & useful to have. I forget, maybe the gbe is free on the Pi4's chip & this is more of a non-issue.
Hardware: these PCIe switches are finicky as all get out; will it link up to your device reliably? Who knows, depends on how the switch and device manufacturers interpreted the PCIe specs.
Software: there is no nice easy standard for connecting host-to-host (which I’m assuming is what one would want on a multinode system Like the one linked). There exists the idea of a PCIe non-transparent bridge, which if I recall correctly, allows CPUs on either side of it to do blind memory reads and writes to each other (hope your IOMMU is set up correctly to protect you from ugliness that could occur here).
Long story short there’s a reason PCIe is a local bus; it was never designed to do anything else. Anything else is largely a proprietary bolt on.
In addition to mapped memory, there are some other modest capabilities: support for ringing the peer's doorbell (one way), & scratchpad configuration. There's a pretty good 2016 presentation on NTB in Linux here: https://events.static.linuxfound.org/sites/events/files/slid...
Interesting to note, Nvidia has docs for settings up NTB on their Pegasus & Xavier systems: https://docs.nvidia.com/drive/drive_os_5.1.6.1L/nvvib_docs/i...
I for one think more people should be investigating & harnessing the open-source Linux NTB tools. Switches are a bit expensive, but compared to ethernet switches their cost/throughput is great. Alas, yes, the whole ecosystem is woefully undersupported, under-developed. Isn't PCI part of the whole PC thing, this great revolution in computing that allowed interoperability to rise? Such a pity that all the PCI SIG's seemed to fizzle out, that not a lot of people participated. Sure we got some SR-IOV but that feels like as far as we got.
https://www.raspberrypi.org/products/compute-module-4/?varia... also lists it as for sale.
Besides that the main direction here is edge computing.
But this thing only has 4 RPI per box so I don't quite understand which problem they are targeting either.
I haven't yet moved to ARM CPUs on AWS for some of my projects, but may be able to more quickly if I can have a local cluster like this for testing that isn't as slow as the CM3+-based one the Turing Pi V1 supported.
Scaling horizontally presents entirely new classes of challenges that aren't experienced while scaling vertically.
The same can't really be said for 10 year old laptops.
>Scaling horizontally presents entirely new classes of challenges that aren't experienced while scaling vertically.
Pay more for challenges? If you really want to fight with bad scaling just run a VM.
A considerable amount of people is already assembling and running clusters of raspberry pis, including hosting companies. A cursory search through eBay or Amazon can provide you with an idea of how many people are already operating these small clusters and spending money on this sort of setup. I'm sure there's a significant demand for a COTS cluster solution that allows people to avoid having to MacGyver their own cluster just to get Docker Swarm or Kubernetes running on a small cluster of raspberry pis.
If you are building for mobile then it doesn’t makes sense to run CI/CD on the Pi and if you are targeting your own bespoke edge then likely having dev boards which can represent your edge hardware more accurately is probably going to be a better way.
Running CI/CD for ARMLinux apps is pretty much the only use case I can think of and even that might be limited if you are using extensions.
I don't understand what point you tried to make. These hosting services, and also clients, are quite intentionally targetting the raspberry pi. More specifically, they do target Debian running on Raspberry Pi. I don't understand the confusion. I mean, is it also hard to understand why AWS users pay for ARM instances running Amazon's custom linux distro? In both the Pi and Graviton you can run nginx, nodejs, Python, ssh... So, what's the problem again?
+ ARM cards
+ X86 cards
+ RISC-V cards
Maybe you could argue that four RPis on a network are more reliable than a single x86 with VMs, but that doesn't seem to be the point of this machine.
The 2x external GigE ports on the Turing Pi helps a lot, in that each cluster can connect simultaneously to 2 external GigE switches, in case one of those things fails.
Could be an interesting exercise to set up and get it all working, in particular K3s + dynamic routing might be a fun challenge.
I have an irrational desire to create a cluster, but I would like to use it for something practical like a NAS. Bad idea?
Price becomes a real problem though. The Pi foundation carrier board is dirt cheap, everyone else's boards seem to be a lot of money. Probably due to low volume.
> The Turing Pi 2 is more functional than V1 and we also expect it to be cheaper to manufacture.
[0]https://turingpi.com/turing-pi-2-announcement/
Would be a neat way to handle corruptions of the SD / eMMC. Flick the power off through the Turing Pi's I2C, have one of the other nodes act as a PXE server, flick the power on again to the affected node, and have it pull in a tiny kernel and init-script through PXE that flashes the SD / eMMC with an image from the PXE node.
I'm thinking a nice addition would be a small module to plug on top of the 40-pin GPIO connector with a GPS receiver in it and an antenna attached. The attached node could run NTP and pull accurate time from satellites. Throw in 3 of the modules, and NTP can use quorum to keep the time accurate even if one GPS module fails.
About 10 years ago we had a legacy system that was using a weird shaped (think UFO spaceship) FM radio receiver to get time. I never understood why our provider chose that over a simple USB GPS receiver.