So we replace "expensive and complicated" storage controller with expensive consumables (ssds have limited lifespan) and move complicated part to the server. What are the upsides?
Right now the bottleneck is memory bandwidth & CPU lanes. NVMEoE lets you access storage from a VM that you can move around and, depending on what you are doing, you are probably already serving traffic over your NIC. If you only need to do a minimal amount of I/O to disk all of the lanes used for I/O when you're idle is useless. If you are doing things like cache servers, DBs that manage replication for themselves, etc you can really optimize storage using this approach.
Basically the difference between 25GB/s between your server and it's storage and 25GB/s between your server and it's clients is just where the packets are going.
2x2.5GB/s for the whole system. Everything you can do with this you can do in classic system using cheaper COTS components. I simply dont see the point other than inventing new proprietary drives to cut off third party suppliers.
I might be misreading it, but it's a 2x25GbE connection to each drive.
So 24x 25GbE in that one switch chassis.
The network-switch-chassis has 6x 100GbE (or is it actually 200GbE) for external connections.
Since the drives are dual-port, the other chassis has another 6x100/200GbE if you wanted to split traffic over the second disk port + network-switch-chassis.
Upsides seem to be:
- Can scale up storage by plugging in another switch/disk shelf
- No storage-enforced limits on the number or type of arrays
- Can define namespaces on drives so you can adjust storage to clients based on need.
- Can scale up performance for a single client by making more drives available and/or faster client NICs
- Not limited to the number of PCI lanes in the client so you could have fast access to FAR more disks than you could physically plug into one host. Performance would be limited to the client's NIC(s).
At the end of the video Patrick talks about that while they demo it being done directly on a host, they can also have DPU cards that could do all of the mounting/array management bits. There's a video on that coming soon.
"Marvell 98EX5630 Ethernet switch, 6x 100GbE on the rear of the chassis = 600Gbps. 25GbE for each drive * 24 drives = 600Gbps providing a 1:1 ratio"
Whats the appeal of proprietary drives talking to fiber switch directly versus NVME chassis with 3 2x100G PCIe NICs on the back? From logical point of view there is no difference. You can parturition, provision, bond it up just the same. You just move PCIE-fiber conversion from central point (cheaper) to bespoke solution inside proprietary drives (sounds expensive vendor lock-in).
I don't think these are proprietary drives or connectors. I can't see anything there to indicate it is proprietary.
I think Marvell just happens to be the first out with the NVMEoF drive adapter and switch chips.
Kioxia and the switch vendor seem to be first out with the drives and switches using Marvell chips/boards.
I wouldn't be surprised to hear in a year or two that you can buy a NVMEoF switch from a bunch of vendors, and then NVMEoF drives from your choice of drive vendor.
The specifics of advantages over existing solutions I can't speak to except repeating much of what is in the article and video.
I might be placing too much faith in Patrick + STH, but they seem excited, and other times there have been proprietary solutions for things they have pointed it out.
A nas like you propose is a bunch of drives, connected to a cpu, connected to a nic. Which then talks to a switch that the actual box using the drive connects to.
This solution proposes dropping a lot of hardware & getting the same outcome. There's one box with 12x 100Gbe on the back (across two switches) and a bunch of nvme (over rdma ethernet, "NVMeoF") plugged directly into that switch.
Why bother having a whole cpu & expensive nic just to connect some drives to the network? This box replaces not just that nas, but perhaps the top of switch box too.
This isnt supposed to be bespoke or proprietary. Its supposed to just the infra we already have. If your nas is really good it already spoke NVMeoF anyhow. Whats a little fancy is that you still need some PHY between switch and drive, which i think is throwing you off & confusing you a bit & making you think this is more bespoke than it is. The drive is an ethernet drive, could hardly be less bespoke.
Reminds me what ATA over Ethernet[0] promised. I bought one of Coraid's Parallel IDE to 100mbit single-device developer kits to experiment with, and then attempted to use their blade software to serve up mdadm devices. Worked okay for short periods of time (5-10 minutes) before the client machines would throw a wobbly and irrecoverably corrupt the filesystem (tried both XFS and JFS) on the device.
I used iSCSI with iPXE back in the day to boot a whole network (~120 machines) of Windows boxes. It was served on a single server by a RAID 10 array of 15k SCSI disks built on Debian. Good times; I'd forgotten about that! iSCSI was solid even with Windows, so you know it's good.
I always found this protocol fascinating due to its apparent simplicity. As a first-year student, I was fiddling with configuring the kernel, and found the ATA-over-Ethernet driver and immediately thought I'd understood it: just throw the ATA commands in an Ethernet frame. After college, I started working at Oracle who partnered with high-end storage vendors for their projects. I was amazed and figured it would be so much cheaper to use Linux :)
My WD My Book Live does basically the same thing but with the cool little piggyback card running a full Linux disto.
Unfortunately they went with 32bit PowerPC and it is no longer supported by Debian so no more upgrades unless I bust out a soldering iron and mess with u-boot to get something like netbsd on it.
Seems like a good product for the super cheap arm SoC makers, a board that attaches to the back of a hard drive with Ethernet just like WD did but without the big clunky case.
I don't entirely understand where these are meant to be used? PCIe switching as a change to the storage/networking paradigm, I can understand especially for cloudy/gaming/bigdata kind of stuff, but I don't see where this product fits in.
This isn't meant as a dig, but I'm genuinely curious as to what the use case for these things is.
For clarity pcie over Ethernet is just that, pcie over Ethernet. Pcie switching is already a thing in normal pcie.
As for the place it's really not much different from where existing large scale storage systems fit, just a different way of delivering them using newer transports which give a bit more flexibility.
The one other idea I'd try to get into people's minds here is: what would be possible if every computer in the rack had full speed access to any disk in the rack.
Right now a drive generally belongs to one computer. But disaggregating the rack & making disks independent means tbis association, of a drive or partition to a particular computer, is arbitrary. Potentially anyone could read or write to any drive.
This would require a clustered filesystem, a fs designed for multiple computers interacting with it at once. I believe there are some offerings out there, but mostly proprietary, and I've heard so little in the past decade- kt feels.like it should be cytting edge & exciting, not dormamt & specialized. I like to think we could get up to a lot of interesting computing & data systems if we had more io over fabrics.
A much more modest, pre-clustered idea might just be to heavily partition a drive, make 64 partitions, then build some control-plane of who currently gets which partitions. Letting chunks of data change ownership, without moving.
It'll be interesting to see if other fabrics than ethernet eventually start being useful with multi-root, multi-computer. PCIe is building new Scalable Virtualization, but as far as I know is still focused on being a peripheral bus, not interconnect. Most of the CXL discussion i see is similarly conventional- huge plans for build out, attaching vast memory pools and huge accelerators, but I havent seen multiple computers sharing a pool of resources shown off. I havent seen CXL as a way to interconnect standalone computers.
Reminds me of a WD or Seagate research project. They took a 3U/16 disk chassis and instead of having a linux box inside with SAS to the 16 disks, they put in a ethernet switch and ran ethernet to each of the disks, actually two interfaces per disk via SAS connectors (but not protocol obviously).
They supported a few key/value stores and claims a big win on less complexity and cool things like migration/rebalancing that could be done disk to disk. I think there was a dev kit for developing for the inside the disk platform.
Seemed like a cool idea, not sure what happened to it, I think it was for sale for interested parties, at least for awhile.
21 comments
[ 3.6 ms ] story [ 59.8 ms ] thread>23 drives we can then RAID 0
umm, is this satire?
So we replace "expensive and complicated" storage controller with expensive consumables (ssds have limited lifespan) and move complicated part to the server. What are the upsides?
Basically the difference between 25GB/s between your server and it's storage and 25GB/s between your server and it's clients is just where the packets are going.
Pretty sure that's per drive.
This chassis has 1.2Tbps switching, half for drives, half for attached computers.
It's not proprietary. The drives can plug into ethernet. They speak industry standard NVMeoF; you could not get more standard if you tried.
So 24x 25GbE in that one switch chassis. The network-switch-chassis has 6x 100GbE (or is it actually 200GbE) for external connections.
Since the drives are dual-port, the other chassis has another 6x100/200GbE if you wanted to split traffic over the second disk port + network-switch-chassis.
Upsides seem to be:
- Can scale up storage by plugging in another switch/disk shelf
- No storage-enforced limits on the number or type of arrays
- Can define namespaces on drives so you can adjust storage to clients based on need.
- Can scale up performance for a single client by making more drives available and/or faster client NICs
- Not limited to the number of PCI lanes in the client so you could have fast access to FAR more disks than you could physically plug into one host. Performance would be limited to the client's NIC(s).
At the end of the video Patrick talks about that while they demo it being done directly on a host, they can also have DPU cards that could do all of the mounting/array management bits. There's a video on that coming soon.
Whats the appeal of proprietary drives talking to fiber switch directly versus NVME chassis with 3 2x100G PCIe NICs on the back? From logical point of view there is no difference. You can parturition, provision, bond it up just the same. You just move PCIE-fiber conversion from central point (cheaper) to bespoke solution inside proprietary drives (sounds expensive vendor lock-in).
I think Marvell just happens to be the first out with the NVMEoF drive adapter and switch chips.
Kioxia and the switch vendor seem to be first out with the drives and switches using Marvell chips/boards.
I wouldn't be surprised to hear in a year or two that you can buy a NVMEoF switch from a bunch of vendors, and then NVMEoF drives from your choice of drive vendor.
The specifics of advantages over existing solutions I can't speak to except repeating much of what is in the article and video.
I might be placing too much faith in Patrick + STH, but they seem excited, and other times there have been proprietary solutions for things they have pointed it out.
This solution proposes dropping a lot of hardware & getting the same outcome. There's one box with 12x 100Gbe on the back (across two switches) and a bunch of nvme (over rdma ethernet, "NVMeoF") plugged directly into that switch.
Why bother having a whole cpu & expensive nic just to connect some drives to the network? This box replaces not just that nas, but perhaps the top of switch box too.
This isnt supposed to be bespoke or proprietary. Its supposed to just the infra we already have. If your nas is really good it already spoke NVMeoF anyhow. Whats a little fancy is that you still need some PHY between switch and drive, which i think is throwing you off & confusing you a bit & making you think this is more bespoke than it is. The drive is an ethernet drive, could hardly be less bespoke.
[0]: https://en.wikipedia.org/wiki/ATA_over_Ethernet
Performance could have been a bit better, but this was back when 128gb enterprise SSDs were nearly the largest you could buy.
Unfortunately they went with 32bit PowerPC and it is no longer supported by Debian so no more upgrades unless I bust out a soldering iron and mess with u-boot to get something like netbsd on it.
Seems like a good product for the super cheap arm SoC makers, a board that attaches to the back of a hard drive with Ethernet just like WD did but without the big clunky case.
This isn't meant as a dig, but I'm genuinely curious as to what the use case for these things is.
As for the place it's really not much different from where existing large scale storage systems fit, just a different way of delivering them using newer transports which give a bit more flexibility.
Right now a drive generally belongs to one computer. But disaggregating the rack & making disks independent means tbis association, of a drive or partition to a particular computer, is arbitrary. Potentially anyone could read or write to any drive.
This would require a clustered filesystem, a fs designed for multiple computers interacting with it at once. I believe there are some offerings out there, but mostly proprietary, and I've heard so little in the past decade- kt feels.like it should be cytting edge & exciting, not dormamt & specialized. I like to think we could get up to a lot of interesting computing & data systems if we had more io over fabrics.
A much more modest, pre-clustered idea might just be to heavily partition a drive, make 64 partitions, then build some control-plane of who currently gets which partitions. Letting chunks of data change ownership, without moving.
It'll be interesting to see if other fabrics than ethernet eventually start being useful with multi-root, multi-computer. PCIe is building new Scalable Virtualization, but as far as I know is still focused on being a peripheral bus, not interconnect. Most of the CXL discussion i see is similarly conventional- huge plans for build out, attaching vast memory pools and huge accelerators, but I havent seen multiple computers sharing a pool of resources shown off. I havent seen CXL as a way to interconnect standalone computers.
They supported a few key/value stores and claims a big win on less complexity and cool things like migration/rebalancing that could be done disk to disk. I think there was a dev kit for developing for the inside the disk platform.
Seemed like a cool idea, not sure what happened to it, I think it was for sale for interested parties, at least for awhile.