Gah that hurts. Not that it concerns HN, but I immediately thought 'oh, that's neat I have to mail that to friend x', and then while composing the mail in my head I halfway through realized he's been dead for a year. :(
You can hardly really advertise it. Nobody invests in 100 GbE NICs taking advertisements seriously rather than knowing some reasons to choose/avoid particular brands/models.
Yes, if it's a one way traffic flow, but a 100GbE link as used in a router is full duplex. If looking at a traffic chart, you could have 75-80Gbps of traffic inbound and also 70-80Gbps of traffic outbound simultaneously.
Using a smaller traffic example, on a much older Juniper router that has linecards with 20Gbps full duplex to the backplane, each line card can hold two 10GbE PICs without oversubscription.
If you want to have a line-rate-capable 100GbE x86-64 whitebox router using FRR or similar, you want to budget PCI-E bus bandwidth for a maximum of approximately 24GB/s per dual port 100GbE NIC.
More importantly, 10GbE cards are cheap as free on eBay. That + a Mikrotik CRS305-1G-4S+IN and some direct connect cables you can have a 10GbE network in your house for well under $300.
The cheap used 10Gb NICs and the switch mentioned use SFP+ ports rather than RJ45. 10GBASE-T SFP+ modules are expensive and power-hungry compared to direct-attach twinax cables or fiber + transceivers.
I picked up the 8 port version from Microtik and picked up a Mellanox ConnectX-2 of eBay which came with a pair of 10G SFP+ fiber modules. A 10 meter single mode LC fiber off Amazon was like $20.
As for the NAS, I snagged a Supermicro mini-ITX 8-core Xeon D (Broadwell era) board from eBay as well for like $400. It brings out the dual integrated 10G NICs on the Xeon D, a PCIe 3.0 x16 port, a 4 lane M.2 port, and 6x SATA ports, 4 slots for DDR4 ECC RDIMMs, and an ASPEED based IPMI. Adding 64 GiB of RDIMMs, 6x 10 TB WD drives, and 2x M.2 drives on a bifurcating riser made for a killer NAS and VM host (crammed inside a Fractal Node 304)
There are no Internet providers in Canada that do more than 1.5Gbps though (Ball Canada claims to have 1.5Gbps down) and even that is mostly just claimed and rarely delivered. These days my compute workloads are mostly in the cloud and all my files are there as well. Thus local speeds do not matter much if the Internet can not also feed them.
I guess this matters for data centres, but primarily my data centres are run by AWS and GCP, so I do not buy this hardware myself.
Not being the bottleneck doesn’t make increased throughput completely superfluous. The faster you can communicate across a channel, the quicker you can shut up and get to idle, saving power on both transmission and policing of resource contention. You also have more time to handle locally scoped retries and error correction.
Speaking of Canada, Linus "Tech Tips" Sebastian mentioned this very argument in a recent video where he was installing 100G and 25G in his office for workgroup video editing. Not everything is in the cloud yet.
I hope all the patents expire soon and that good products at good prices follow quickly. The standard was released in 2006, but I'm not sure how close to that release date the last relevant patent involved was issued.
"10GBASE-T, or IEEE 802.3an-2006, is a standard released in 2006 to provide 10 Gbit/s connections over unshielded or shielded twisted pair cables, over distances up to 100 metres (330 ft)."
What is pantented about implementations? I’m surprised that the nature of the implementation of a public specification is patentable.
I spent a few years implementing 802.3 physical layer testing. At the conferences it was clear that things were written carefully to not cover implementation, but they went so far in the necessary details that there were only a handful of reasonably performant and economical solutions.
> I’m surprised that the nature of the implementation of a public specification is patentable.
Prepare to be very surprised, I guess? It’s pretty routinely done, eg) look at any of the MPEG standards.
I don’t know what patents, if any, cover 10GigE, but in general IEEE, which governs that standard, will accept Essential Patent Claims in a standard provided the Patent Holder will provide a Letter of Assurance indicating that they will license said patents under FRAND (fair, reasonable, and non-discriminatory) terms – which does not mean free.
Yeah I don't think patents will play a major role in the price. WiFi would be a lot more patent-encumbered with technologies like MIMO. Yet it doesn't seem to preclude cheap commodity hardware. My Unifi WiFi 6 AP was only a bit more expensive than the AC one.
10GBASE-T is unreasonably power hungry. SFP+ passive cables are far superior for short lengths and fiber optics are superior for anything longer. I've got two SFP+ 10GBASE-T transceivers on my home network and they seem to sit around 80°C all the time.
But indeed as others have said, even broadband connections are exceeding this speed now. USB exceeds it by a factor 40 and is on cheap laptops. Yet Ethernet remains so expensive.
People will compare browser benchmarks over milliseconds so why not file transfers over the home network?
In general terms, cynicism about the "need" may be right. Most home users in most use cases likely won't notice much by going to 10G.
But then there is the rest of us and the other use cases.
As for systems that can push this much data. Almost anything with a M.2 SSD will easily keep up.
So for a use case, backing up your zillions of photos to at least one other point of failure on your home network. May not be the best backup plan ever, but regular home users are not IT gurus and will do this.
Yeap.
Mellanox has one advantage , even though it pcie x8, if you put it to pcie x4 slot, it works nicely (according to specs it downgrades itself).
Intel - not that sure about it
It definitely does, in my PC right now. Why wouldn't it? Not downgrading would probably be an egregious violation of PCIe specs. PCIe is built to work no matter the restrictions. You can connect an x16 gen4 GPU via an adapter to an x1 gen2 miniPCIe slot and it will work.
When I was shopping for one a few months ago, it wasn't explicitly mentioned it X520 specs but mentioned in Mellanox specs.
I know that in the theory it supposed to downgrade itself, but implementations may vary depends on fantasy of designers.
At the end, I just got motherboard with few 10GbE and SPF+ on it.
You can pick up used HP NC523SFP or NC552SFP (better!) NICs for about $15-$25 each off eBay. The drivers fell out of support on RHEL8/CentOS8 but I think Fedora Server, RHEL7/CentOS7 and Ubuntu Server are fine with them. TrueNAS can make use of them as well. Then all you need is a 10GbE switch that can do bonding. Or, just make yourself a router with a bunch of dual port NICs.
56 comments
[ 2.4 ms ] story [ 65.1 ms ] threadhttps://www.mouser.com/ProductDetail/Broadcom-Avago/BCM95750...
Good to take some time to remember them sometimes.
https://iperf.fr/
https://community.mellanox.com/s/article/howto-install-iperf...
https://serversupply.com/NETWORKING/NETWORK%20ADAPTER/100%20...
Using a smaller traffic example, on a much older Juniper router that has linecards with 20Gbps full duplex to the backplane, each line card can hold two 10GbE PICs without oversubscription.
If you want to have a line-rate-capable 100GbE x86-64 whitebox router using FRR or similar, you want to budget PCI-E bus bandwidth for a maximum of approximately 24GB/s per dual port 100GbE NIC.
As for the NAS, I snagged a Supermicro mini-ITX 8-core Xeon D (Broadwell era) board from eBay as well for like $400. It brings out the dual integrated 10G NICs on the Xeon D, a PCIe 3.0 x16 port, a 4 lane M.2 port, and 6x SATA ports, 4 slots for DDR4 ECC RDIMMs, and an ASPEED based IPMI. Adding 64 GiB of RDIMMs, 6x 10 TB WD drives, and 2x M.2 drives on a bifurcating riser made for a killer NAS and VM host (crammed inside a Fractal Node 304)
I guess this matters for data centres, but primarily my data centres are run by AWS and GCP, so I do not buy this hardware myself.
It really feels like we've had 1GbE like forever (it was already standard on my 2004 PowerBook!!). But 10GbE is still hardly affordable.
https://en.wikipedia.org/wiki/10_Gigabit_Ethernet#10GBASE-T
"10GBASE-T, or IEEE 802.3an-2006, is a standard released in 2006 to provide 10 Gbit/s connections over unshielded or shielded twisted pair cables, over distances up to 100 metres (330 ft)."
I spent a few years implementing 802.3 physical layer testing. At the conferences it was clear that things were written carefully to not cover implementation, but they went so far in the necessary details that there were only a handful of reasonably performant and economical solutions.
Prepare to be very surprised, I guess? It’s pretty routinely done, eg) look at any of the MPEG standards.
I don’t know what patents, if any, cover 10GigE, but in general IEEE, which governs that standard, will accept Essential Patent Claims in a standard provided the Patent Holder will provide a Letter of Assurance indicating that they will license said patents under FRAND (fair, reasonable, and non-discriminatory) terms – which does not mean free.
And can most systems push this kind of data?
(home systems)
SSDs are an order of magnitude faster than gigabit ethernet.
Google fiber is rolling out 2Gbps internet in many locations
1 Gigabit is definitely good enough for most general use cases, but once you start moving files around regularly it becomes a huge bottleneck.
So yes, it's trivial to saturate a 10 Gbit/s link, assuming that the kernel and userland can keep up with the hardware.
Wifi 6E can do almost wigig/uwb speeds, near 7Gbit hypothetical.
A single modern NVMe drive does 56Gbit.
USB4 can do 40Gbit. It can connect direct computer to computer.
But indeed as others have said, even broadband connections are exceeding this speed now. USB exceeds it by a factor 40 and is on cheap laptops. Yet Ethernet remains so expensive.
People will compare browser benchmarks over milliseconds so why not file transfers over the home network?
In general terms, cynicism about the "need" may be right. Most home users in most use cases likely won't notice much by going to 10G.
But then there is the rest of us and the other use cases.
As for systems that can push this much data. Almost anything with a M.2 SSD will easily keep up.
So for a use case, backing up your zillions of photos to at least one other point of failure on your home network. May not be the best backup plan ever, but regular home users are not IT gurus and will do this.
At the end, I just got motherboard with few 10GbE and SPF+ on it.
https://www.ebay.com/itm/LOT-OF-2-593717-B21-HPE-NC523SFP-DU...