I have 1.5/900 fibre to my house, and I bring a 2.5 line from the modem to my home office where a 2.5 switch delivers it to my workstation, laptop, and unraid NAS. But those devices are all themselves just gigE I think, and I've yet to come up against a download (even a torrent) that seems like it would have really benefitted from having the entire theoretical 1.5 pipe available.
I might have been lucky, but in the one home and one office were I've connected 10gbit switches and PCIe cards, it has just worked. Especially the office was a nice surprise, because it is at least 20 meters (probably more) of unknown cabling and at least one unknown patch panel between the utility closet where the NAS lives and the desk area. The cables were run 15 years ago, so I expected it to be cat 5, but clearly not.
It is nice moving/streaming large files across the network at 10 gbit. It really is ten times less waiting than with plain old gigabit.
Of course, most of the time I'm working with lots of small files and then the spinning disk array in the NAS has no chance to saturated the this giant pipe, or even a normal gigabit connection...
I was surprised that the old Cat5e in my home supports 10Gbps without any issue, so went ahead and upgraded the rest of the network with 10Gbps switches (expensive Ubiquiti gear, but worth it to talk at 10Gbps between all my machines, even though the internet is only 5Gbps Fiber).
I'm extremely happy after upgrading my network to 10gbit copper ethernet. It was much more expensive than I thought it should be, but worth it even if I only max it occasionally. Now I can easily fully saturate my 10gbit ethernet doing a first Time Machine backup or transferring files to my M.2 SSD NAS which saves me waiting rime and is satisfying to watch.
It's wild to me that 10gbit isn't the norm by now and tech people who should know better seem to think WiFi matches or even exceeds even 1gbit ethernet. My MBP connects to my WiFi7 setup(Ubiquiti E7) at a nominal 1.5-1.9gbit but Time Machine backups and file transfers are slower than plugging into 1gbit ethernet, probably in large part due to latency and retransmissions. Not to mention that ethernet works with near 100% reliability with dramatically less variation in speed and error rate.
> It's wild to me that 10gbit isn't the norm by now
I honestly just don't need it. Part of that is my ISP options top out around 1200Mbps, certainly. But I also just don't need that kind of speed inside my home. Streaming video needs at most 20Mbps or so, and I don't do much in the way of large file transfers. And when I do large file transfers, it's usually from or to the internet, and my home network is not the bottleneck there.
Getting Time Machine to work reliably over a network is painful, even the old Apple-made Airport with built in TM stopped working twice a year.
However, I have multiple Macs where I simply have a USB-C laptop SSD attached for Time Machine and they have worked without issue for years. These laptop SSDs come in huge sizes nowadays, and you don't need an especially performant one, so they can be pretty cheap.
You can definitely get stable 1gig throughput through WiFi. Doesn’t even need WiFi 7/6E - possible on 6. Ran a WiFi bridge like that for years and - no packet loss, consistent gig through and maybe +1ms latency
The gotcha is both ends needs to be good radios. So a router to router bridge tends to work better than router to end user device. Also had near LoS which presumably helped a ton
I've been running it like this in a closed comm box for the last 3 years without any issues. SFP+ modules actually do not use that much power, it's just that it's concentrated into a small package, resulting in high temps.
One thing I'll add that I learnt in the process of doing my own house: It's not just the cable type but also the SFP module that can limit the distance. I used MicroTik hardware and their S+R10J modules are limited to ~30m for 10Gbps speeds.
Unfortunately the blog didn't link to the SFP+ module they're using, but everyone should know there's effectively 2 different generations of 10gbit sfp+ to ethernet^H10BASE-T modules. The old gen, labeled as 30 meters, draws ~3 W, and gets extremely hot (to the point it'll usually cause link flaps), and the newer gen, usually labeled as 100m or 80m, draws ~1.5 W, and runs much, much cooler.
the AQC chips at least work with 1/10 only ports on older switches. And, it's not that hot, a single 1x1x0.5cm heatsink takes care of it. One large issue is that mikrotik's thermal design for SFP+ is a joke.
> Build in Marvell chip, Low Power Consumption (2.0W MAX @ 30m)
I have had 0 problems. One I use for my Sonic.net Ont and the other for my a 10G switch. Maybe I don't have issues because the cable lengths are so short (1-2')?
I just checked and both are VERY hot to the touch. Now I'm worried.
For me the threshold has always been "can I stream a 4K movie from the NAS downstairs or from my seed box". No real need for anything above. Still I ran 10Gb single mode fiber in all the ducts.
The only thing I'd caution anyone else looking to do the same is doing a software router/fw like the Portectli is it's usually not hard to get the raw bandwidth to look nice with big flows but the new connection latency, connections per second, jitter, and QoS handling tend to suffer vs something with hw offloads (which is what most are used to even with cheapo gigabit home AP+router+switch combos). It's also not usually the cheapest way to get the 10G class NAT/L4 FW bandwidth, but it is usually the cheapest way to get "full" FW functionality if you don't care as much about the performance.
If you want a full FW solution that can actually FW+NAT at 10G bidirectional without breaking a sweat then something like the FortiGate 90G is the cheapest thing I've found that performs really well across the board. Great QoS, great latency, amazing throughput performance (does well with even small packet sizes in a single stream), easy enough to use UI (once you get oriented), low power. If you want to enable all of the NGFW stuff (e.g. AV and IPS) then it'll dip below line rate though.
If you just want something that NATs/connection direction oriented filtering like a "normal" home router then something like the MikroTik CCR2004 can get you better than the performance they got on the VP2440 + give you 12 ports of 10G SFP+ to work with. If you were planning to do "fancy" FWing/functionality beyond a normal home NAT FW (with decent managed switching built in) then the feature set will be a bit limiting, of course.
I put cat6a in my last house and plan to in my current house "just in case". The cable isn't that much more expensive and nobody wants to do cabling again so why not.
But I'll keep using a gigabit switch because I have absolutely no idea what I'd use 10G for. It's crazy that gigabit was affordable for me as a student in the early 00s and between then and now we've gone from DVDs to 4K and it's still plenty fast enough. In fact, most people are happy with WiFi (not me, though).
Regarding the proxmox cluster, I would advise not to set up a cluster unless you really need to, it adds complexity in order of shutting down the server, HA, Quorum votes, ceph monitor, among many others, and if one server goes offline, or even the wrong order, it will impact the others and sometimes some tough time in recovery, or data loss.
> The most important question was the structured cabling in the walls; was it CAT-5E or CAT-6, or even CAT-6A? Remember from the last post, 10GBASE-T might work over short runs of -5E (even though officially it's not meant to be able to).
This is not quite correct.
The primary problem is cross-talk. Copper wire itself will carry the relevant frequencies up to 100m without issue but even with balanced pairs the balancing is not perfect and the "dirty paper precoding" is not perfect so some cross-talk will occur. How long you can go with Cat-5e depends on how well the wire is twisted, how many wires are bundled together, are there any loops or tight bends, and other factors. Cat-6A guarantees less cross-talk with more twists, better balancing, and a plastic separator inside the cable to make the cross-talk more regular and thus easier to cancel out.
Bottom line is: for almost any normal home or apartment any quality Cat-5e cable properly terminated will carry 10GBase-T without issue. In fact if you have problems I would first re-terminate the cable before assuming you need to run new cable. Cat-6 or 6A just isn't necessary.
As a PSA: beware of "CCA". I've noticed Amazon and eBay are absolutely flooded with cheap chinese electrical and networking cable that shows nice shiny copper in the pictures but is actually "copper clad aluminum". If they mention anything at all they code it as "CCA" cable without explaining what that means.
CCA cable cannot, by definition, be ethernet cable. I won't get into the full technical details but the standard was amended to clarify that only pure copper wires are acceptable for ethernet. Personally I would not dare use CCA for anything. It has lower performance, lower current-carrying capability for the same wire diameter (inherent in aluminum), and introduces the risk of oxidation and loosening of connections as people will treat them as copper connections when aluminum needs special installation procedures and connections to avoid them coming loose over time. For electrical connections especially this not only can but absolutely will lead to a fire over time if not treated with the appropriate care. All it takes is a little bit of mechanical action scraping off the thin copper layer and you now have an effectively aluminum wire - a time bomb ticking away.
> Cat-6A guarantees less cross-talk with more twists, better balancing, and a plastic separator inside the cable to make the cross-talk more regular and thus easier to cancel out.
This isn't quite right. Cat6A guarantees the frequency and therefore effective bandwidth that is available. But it doesn't talk about physical cable type. A common structured cable for Cat6A is U/FTP which has no plastic separator, no outer shield, but rather each pair is wrapped in foil and is actually less tightly twisted than even Cat5e. Anyone can buy U/FTP cable, but to actually be Cat6A it has to be installed correctly and the installation certified to support Cat6A speeds.
Yeah that's true, it doesn't require a physical implementation per-se.
It does specify cross-talk specs and frequency response. IIRC Cat-5E was 125, Cat-6 250, Cat-6A 500Mhz. But copper is copper and all else equal well-made copper of a given size is going to carry those frequencies regardless. Cat-5E can carry 500Mhz just fine. They don't mix some magic sauce into the molten copper before casting & drawing into wire that changes its frequency response. The real limiting factor (IMHO) and where the actual manufacturing really matters is cross-talk.
Using SFP+ modules for 10GbE is interesting, I deployed a full 10GbE network in my home connected by 5Gbit/s symmetric fiber from AT&T and a 1Gbit/s x 100mbit/s DOCSIS 3.1 connection from Spectrum as backup, I did this around August of last year and used UniFi/Ubiquiti for everything. Their new XG and XGS products all support 10GbE natively, without needing SFP+ modules to connect. I'm using the UDM Pro Max with an XGSPON ONU flashed with the community firmware from PON Wiki (see https://pon.wiki/category/att/ ) and the 2.5GbE port as my second WAN link to the Spectrum modem. I'm not doing any sort of bonding/load balancing, just failover so I'm almost always just using AT&T. There's a short DAC cable connecting the UDM Pro Max to a Switch Pro XG 24-POE which serves 10GbE to the rest of the house. The only SFP+ module involved is the XGSPON ONU. Most of the devices in the house connect to WiFi served by the UniFi U7 Pro XGS APs, which take 10GbE w/ POE and give off WiFi 7.
I pretty much always get at least 4600Mbit/s both directions over AT&T, and generally cap it out. Spectrum typically gives me at least 800Mbit/s down. The ISPs are definitely the bottleneck because neither provides dedicated infrastructure from each house to the CO, instead you have some sort of aggregation point which has a shared backhaul and that means you compete for resources, but having the largest plan on each at least gives you traffic priority.
Realistically, I don't need any of this, I was doing just fine with normal gigabit fiber with a 2.5GbE network before I moved last year, but it's nice knowing everything is as fast as possible in my path and that eliminates congestion and local network resource contentions as causes when a problem arises, I know it's pretty much always upstream of me somewhere.
I'm surprised they got a reliable 10Gbps USB adapter. I've tried 3 of them and they're all trash. They cannot sustain multi-gigabit speeds and end up doing like 400Mbps when you hit them hard. I think I have sabrent and ugreen and one other.
They all just sit in a drawer, if I need a USB-C ethernet adapter i just grab my trusty 1 gig one.
Yep... Did this 3 years ago in my home and came to the same conclusion. If I were to do it again, I'd run fiber through the walls instead of Cat-6a. It took forever to find SFP+ modules that would work with my Unifi setup... (not wanting to pay for more than one router or switch with native 10g RJ-45 ports, which are still very expensive.) I loose POE but, on the whole, it would've been much easier and much less costly--I think--to have just run fiber.
It's 4x faster? If you have a storage server, its easy to exceed 2.5 or even 5 gigabit with SATA spinning disks if they're in an array. 10 gig is fast enough to work with network storage like it's local: do your work, store your steam library, edit videos, etc, but store it all somewhere else and share it with multiple computers.
I went a slightly different route. My switches are linked with 10Gbps SFP+ across the apartment, but it was way too late (and too much hassle) to pull proper in-wall fiber. Instead I used one of those ultra-thin (around 0.1mm), unshielded fiber cables, and just snaked it through door frames and taped it along the walls. I'm genuinely impressed this stuff exists, it makes retrofitting fiber into a finished space so much less painful.
Most of my edge devices are still on 2.5GbE though, and I'm increasingly aware that for anything with plain SATA disks, the drives are the real bottleneck. Once I LAG'd 2×2.5GbE to get a 5Gbps pipe, it became obvious the network wasn't the slow part anymore in a lot of cases.
And yeah, the 10GbE SFP+ modules run hot, so hot that I would not lay my fingers on them for more than 2 seconds. I stuck 2 copper heatsinks on my module, not sure they do much but the module runs smoothly. Even so, I'm pretty happy with the overall setup: from my 10GbE-equipped Mac I can saturate multiple machines at once and I no longer think about the network most of the time, which was the goal.
43 comments
[ 11.2 ms ] story [ 1939 ms ] threadhigh latency, high error rates, and terrifying heat output from SFPs (which the author noted for himself)
the only cat6 left in my home network is the link to verizon's ont, because in their infinite wisdom the ONLY connectivity offered was 10g-base-t
I have 1.5/900 fibre to my house, and I bring a 2.5 line from the modem to my home office where a 2.5 switch delivers it to my workstation, laptop, and unraid NAS. But those devices are all themselves just gigE I think, and I've yet to come up against a download (even a torrent) that seems like it would have really benefitted from having the entire theoretical 1.5 pipe available.
It is nice moving/streaming large files across the network at 10 gbit. It really is ten times less waiting than with plain old gigabit.
Of course, most of the time I'm working with lots of small files and then the spinning disk array in the NAS has no chance to saturated the this giant pipe, or even a normal gigabit connection...
Man I wish I had the kind of money/hardware to get 10gbit disk io out of a nas.
Meanwhile I'm sat here wishing I could justify running any ethernet in my apartment, but improving wi-fi tech means I never can...
It's wild to me that 10gbit isn't the norm by now and tech people who should know better seem to think WiFi matches or even exceeds even 1gbit ethernet. My MBP connects to my WiFi7 setup(Ubiquiti E7) at a nominal 1.5-1.9gbit but Time Machine backups and file transfers are slower than plugging into 1gbit ethernet, probably in large part due to latency and retransmissions. Not to mention that ethernet works with near 100% reliability with dramatically less variation in speed and error rate.
I honestly just don't need it. Part of that is my ISP options top out around 1200Mbps, certainly. But I also just don't need that kind of speed inside my home. Streaming video needs at most 20Mbps or so, and I don't do much in the way of large file transfers. And when I do large file transfers, it's usually from or to the internet, and my home network is not the bottleneck there.
However, I have multiple Macs where I simply have a USB-C laptop SSD attached for Time Machine and they have worked without issue for years. These laptop SSDs come in huge sizes nowadays, and you don't need an especially performant one, so they can be pretty cheap.
The gotcha is both ends needs to be good radios. So a router to router bridge tends to work better than router to end user device. Also had near LoS which presumably helped a ton
You can fix the thermal issue either by adding a small fan (Noctua is great) or by adding more radiators: https://pics.ealex.net/share/UxeSf_AWHLIuc-qzK5zl7JIgQvQDAZh...
I've been running it like this in a closed comm box for the last 3 years without any issues. SFP+ modules actually do not use that much power, it's just that it's concentrated into a small package, resulting in high temps.
Example of the new gen: https://www.amazon.com/Wiitek-Transceiver-Compatible-UF-RJ45...
Old gen: https://www.amazon.com/10Gtek-SFP-10G-T-S-Compatible-10GBase...
Typically the old gen uses a Marvell AQR113C, and the new gen uses a Broadcom chip that I forget the number of off hand.
> Built-in Newest Marvell chip, Max. Power Consumption 1.65W
Is that wrong?
Also, I'm using this:
https://www.amazon.com/dp/B0B3F5DSXJ?ref_=ppx_hzsearch_conn_...
which says
> Build in Marvell chip, Low Power Consumption (2.0W MAX @ 30m)
I have had 0 problems. One I use for my Sonic.net Ont and the other for my a 10G switch. Maybe I don't have issues because the cable lengths are so short (1-2')?
I just checked and both are VERY hot to the touch. Now I'm worried.
If you want a full FW solution that can actually FW+NAT at 10G bidirectional without breaking a sweat then something like the FortiGate 90G is the cheapest thing I've found that performs really well across the board. Great QoS, great latency, amazing throughput performance (does well with even small packet sizes in a single stream), easy enough to use UI (once you get oriented), low power. If you want to enable all of the NGFW stuff (e.g. AV and IPS) then it'll dip below line rate though.
If you just want something that NATs/connection direction oriented filtering like a "normal" home router then something like the MikroTik CCR2004 can get you better than the performance they got on the VP2440 + give you 12 ports of 10G SFP+ to work with. If you were planning to do "fancy" FWing/functionality beyond a normal home NAT FW (with decent managed switching built in) then the feature set will be a bit limiting, of course.
But I'll keep using a gigabit switch because I have absolutely no idea what I'd use 10G for. It's crazy that gigabit was affordable for me as a student in the early 00s and between then and now we've gone from DVDs to 4K and it's still plenty fast enough. In fact, most people are happy with WiFi (not me, though).
This is not quite correct.
The primary problem is cross-talk. Copper wire itself will carry the relevant frequencies up to 100m without issue but even with balanced pairs the balancing is not perfect and the "dirty paper precoding" is not perfect so some cross-talk will occur. How long you can go with Cat-5e depends on how well the wire is twisted, how many wires are bundled together, are there any loops or tight bends, and other factors. Cat-6A guarantees less cross-talk with more twists, better balancing, and a plastic separator inside the cable to make the cross-talk more regular and thus easier to cancel out.
Bottom line is: for almost any normal home or apartment any quality Cat-5e cable properly terminated will carry 10GBase-T without issue. In fact if you have problems I would first re-terminate the cable before assuming you need to run new cable. Cat-6 or 6A just isn't necessary.
As a PSA: beware of "CCA". I've noticed Amazon and eBay are absolutely flooded with cheap chinese electrical and networking cable that shows nice shiny copper in the pictures but is actually "copper clad aluminum". If they mention anything at all they code it as "CCA" cable without explaining what that means.
CCA cable cannot, by definition, be ethernet cable. I won't get into the full technical details but the standard was amended to clarify that only pure copper wires are acceptable for ethernet. Personally I would not dare use CCA for anything. It has lower performance, lower current-carrying capability for the same wire diameter (inherent in aluminum), and introduces the risk of oxidation and loosening of connections as people will treat them as copper connections when aluminum needs special installation procedures and connections to avoid them coming loose over time. For electrical connections especially this not only can but absolutely will lead to a fire over time if not treated with the appropriate care. All it takes is a little bit of mechanical action scraping off the thin copper layer and you now have an effectively aluminum wire - a time bomb ticking away.
This isn't quite right. Cat6A guarantees the frequency and therefore effective bandwidth that is available. But it doesn't talk about physical cable type. A common structured cable for Cat6A is U/FTP which has no plastic separator, no outer shield, but rather each pair is wrapped in foil and is actually less tightly twisted than even Cat5e. Anyone can buy U/FTP cable, but to actually be Cat6A it has to be installed correctly and the installation certified to support Cat6A speeds.
It does specify cross-talk specs and frequency response. IIRC Cat-5E was 125, Cat-6 250, Cat-6A 500Mhz. But copper is copper and all else equal well-made copper of a given size is going to carry those frequencies regardless. Cat-5E can carry 500Mhz just fine. They don't mix some magic sauce into the molten copper before casting & drawing into wire that changes its frequency response. The real limiting factor (IMHO) and where the actual manufacturing really matters is cross-talk.
I pretty much always get at least 4600Mbit/s both directions over AT&T, and generally cap it out. Spectrum typically gives me at least 800Mbit/s down. The ISPs are definitely the bottleneck because neither provides dedicated infrastructure from each house to the CO, instead you have some sort of aggregation point which has a shared backhaul and that means you compete for resources, but having the largest plan on each at least gives you traffic priority.
Realistically, I don't need any of this, I was doing just fine with normal gigabit fiber with a 2.5GbE network before I moved last year, but it's nice knowing everything is as fast as possible in my path and that eliminates congestion and local network resource contentions as causes when a problem arises, I know it's pretty much always upstream of me somewhere.
They all just sit in a drawer, if I need a USB-C ethernet adapter i just grab my trusty 1 gig one.
How about this one? https://mikrotik.com/product/crs305_1g_4s_in
You can use fiber optic SFP+ in it which should not run as hot.
Run fiber everywhere for the fun of it, instead of copper Ethernet. Here is an interesting post about it:
https://sschueller.github.io/posts/wiring-a-home-with-fiber/
You can simplify it and avoid splicing which is the most complex thing about it.
Most of my edge devices are still on 2.5GbE though, and I'm increasingly aware that for anything with plain SATA disks, the drives are the real bottleneck. Once I LAG'd 2×2.5GbE to get a 5Gbps pipe, it became obvious the network wasn't the slow part anymore in a lot of cases.
And yeah, the 10GbE SFP+ modules run hot, so hot that I would not lay my fingers on them for more than 2 seconds. I stuck 2 copper heatsinks on my module, not sure they do much but the module runs smoothly. Even so, I'm pretty happy with the overall setup: from my 10GbE-equipped Mac I can saturate multiple machines at once and I no longer think about the network most of the time, which was the goal.