I mean at lot of what a homelab is experimenting. For example I have an absurd amount of communication 56GbE Mellanox switches but never really saturate it.
DGX Sparks have 200Gbps interlinks for clustering. Apparently switches are the only practical way to scale such clusters beyond 2 nodes even though each Spark has 2 such ports.
Well, the problem is that you want each node to be able to directly connect to each other, so you run into a mesh situation. https://www.youtube.com/watch?v=QJqKqxQR36Y shows 8 running at 100gbps and clustered.
I understand that part, what I don't quite get is why we can't divide the layers such that each node only needs to talk to at most 2 nodes. I assume I'm misunderstanding how work is distributed between them, but haven't looked into the details yet.
I think this is the distinction between pipeline parallelism (first N layers on machine 1, next N layers on machine 2, etc.) vs tensor parallelism where all machines contribute to the same matrix multiplication operation within the same layer. Pipeline parallelism doesn't need exotic high speed networking, but also doesn't deliver the kind of speed increase that tensor parallelism provides.
I purchased my SSD based on price and I have a AM4 APU so I'm limited to pci-e 3, but a quick check with crystaldiskmark says I'm hitting about 3.6 GB/s on sequential reads in seq1m q8t1 ... That would saturate 25gbps (assuming you have the right access patterns). seq1m q1t1 does 2.3 GB/s on my system, close enough to 25 gbps.
By my math, pci-e 5.0 should do 4x the throughput, so if you had 4 pci-e ssds you wanted to serve over the network and you had the pci-e lanes, you could saturate 400 gbps.
I'm pretty happy with my little baby network with a handful of 10G and 2.5G ports, and I'm cheap so if I made a big upgrade it'd probably be something used that has more 10g ports plus a 40G I could DAC to the adjacent home server ... but my home servers are barely powerful enough to push 10g anyway.
(tl;dw - guy connects first four DGX and then another 4 "DGX-alikes". Uses 400gbps breakout cables (you can split interfaces, TIL) so he has 2 ports on the first switch, and then gets this one, I believe. You can buy cables that are 1x400 to 4x100gig, etc.
I believe the 30W figure was power usage before SFPs or DACs. Looks like 400G sfps are about 10W, so you'd double it when fully loaded. I believe passive twinax doesn't use anywhere near that though.
I haven’t been directly involved in networking for a while, so it was funny to tune back in and see that now not only is 10Gbps apparently no big deal, they even put it on the management interface!
I believe it can be broken down to 200GbE and 100GbE though. It is too bad that it can't route to the management ports for a 10GbE outgoing connection (or maybe it can and I wasn't paying enough attention).
I can see a few uses for this:
1) cluster 4x GB10 machines
2) Connect a small group of video editors via 100GbE to a NVMe file server.
3) Overkill home office network, just because.
I'd really like them to take one or two of those 400G ports and split them into internal port splitters to generate a 32x10/25g switch with 400g uplinks for $2k. They do a 8x50G one (crs812_ddq)
Ultimately this is a Marvell 98dx7335 with a mikrotik management interface on the top. Most switches you tend to encounter use Broadcom chipsets rather than Marvell.
Setting aside what features routeros exposes, This chip seems to be somewhere between a Trident 3 and Trident 4 in capability (hardware support for things like evpn/vxlan, q-in-q, mpls, macsec etc)
Last mikrotik switch I tried only supported PTP Transparent clock, this chip does do boundary clock, but I don't think it's implemented through the linux layer let alone exposed via the RouterOS layer.
I think the chip is mainly aimed at service providers, which fits in well with mikrotik's main customer base (ISPs)
22 comments
[ 0.19 ms ] story [ 36.0 ms ] threadBy my math, pci-e 5.0 should do 4x the throughput, so if you had 4 pci-e ssds you wanted to serve over the network and you had the pci-e lanes, you could saturate 400 gbps.
I'm pretty happy with my little baby network with a handful of 10G and 2.5G ports, and I'm cheap so if I made a big upgrade it'd probably be something used that has more 10g ports plus a 40G I could DAC to the adjacent home server ... but my home servers are barely powerful enough to push 10g anyway.
(tl;dw - guy connects first four DGX and then another 4 "DGX-alikes". Uses 400gbps breakout cables (you can split interfaces, TIL) so he has 2 ports on the first switch, and then gets this one, I believe. You can buy cables that are 1x400 to 4x100gig, etc.
It really is when you're talking 400G.
OK there's very few ports on this, but in general if you're looking at 400g interfaces. $1100 won't even get you a genuine Arista 400G passive DAC.
https://www.fs.com/eu-en/products/82464.html
Claims "Near-Zero Power", but I have zero experience with >100g
Most people probably would use them for something else.
That I can't break down to 10gbit or 1gbit without another switch...
> The list price is $1295, but these often sell at a street price of $1100 or so, making this under $0.70/Gbps
It doesn't work that way...
I like mikrotik, but who is this switch really made for?
I can see a few uses for this: 1) cluster 4x GB10 machines 2) Connect a small group of video editors via 100GbE to a NVMe file server. 3) Overkill home office network, just because.
Ultimately this is a Marvell 98dx7335 with a mikrotik management interface on the top. Most switches you tend to encounter use Broadcom chipsets rather than Marvell.
Setting aside what features routeros exposes, This chip seems to be somewhere between a Trident 3 and Trident 4 in capability (hardware support for things like evpn/vxlan, q-in-q, mpls, macsec etc)
Last mikrotik switch I tried only supported PTP Transparent clock, this chip does do boundary clock, but I don't think it's implemented through the linux layer let alone exposed via the RouterOS layer.
I think the chip is mainly aimed at service providers, which fits in well with mikrotik's main customer base (ISPs)