It would make for a good enough edge/home cluster node. Buy 5 of these, each with two SATA SSDs/HDDs, SFP+ uplink to cheap 10G switch [1], run k8s/k3s, might even be able to squeeze in Ceph.
2.5G switches are expensive and rare. If you want to make use of multiple links to an L2 ToR, you have to set up LACP which is annoying. You could do ECMP for L3 LB instead, but that requires an even more expensive switch.
EDIT: as for commitment, I'm literally looking to solve the problem of 'I want a low power k8s cluster, and I'm okay with a switch SPOF' for bgp.wtf's FMT location. If I could buy 5-6 of these, with 10G uplink, especially ready to rack, I'd probably order them today.
Yeah, but they offer what provides most utility for their intended target audience first and foremost, and SFP+ would be inconvenient there (especially as copper pluggables ain't free).
10G SFP+ would be more useful IMO too. It's becoming more and more common on switch hardware. Seems to me that anyone remotely interested in 4x2.5Gbps will be using VLANs on a decent switch, so they might as well trunk them on a single 10G SFP+ to make things simpler.
I think this targets the case where the device itself is the switch/router/main-device/what-have-you not cases the device is hanging off a larger, faster, hardware switch. I see room for both use paradigms though, a 10G adapter would be nifty in addition.
Not OP, but chiming in: I do many workflows with very large photo and video files. Having an affordable, widely-available single-board-like or single-board-compatible product with a 10Gb SFP+ cage would have a measurable impact on how quickly I can seek assets from my home NAS.
Right now the cheapest options are to get a 10Gb PCIe card from Newegg or similar, which'll run you about $200 where I am, and that's just a PCIe extension card.
> Right now the cheapest options are to get a 10Gb PCIe card from Newegg or similar, which'll run you about $200 where I am, and that's just a PCIe extension card.
If you want to go _very_ cheap and dirty you can get some FlexibleLOM/PCIe adapters [1] and some HP FlexibleLOM NICs [2]. This results in a 2x 10G NIC for around ~$15, including the adapter.
I'm not necessarily going for rock-bottom prices; I wouldn't necessarily trust a $15 2x10G NIC to have good consistency/stability or thermal management, among other concerns.
That said I think there's a market for boards between that $15 mark and the $200 I'd have to pay for something considerably higher-end.
If by 'generic PCIe slot' you mean an NVMe slot that I have to buy janky adapters for, and figure out mechanical mounting for, then yeah. But I'm not sure I'd like to depend on that.
And then I might as well do that with an Intel NUC, or just go for a normal mini-ITX motherboard.
I think everyone is missing the point of this cheap hobbyist board.
The PCIe slot on the H2 is PCIe 2.0 x4, which doesn't have enough bandwidth to saturate more than one 10G port.
This is why hobby hardware and the enthusiast market is so difficult: You can make a great product at a cheap price point and people will still complain that it doesn't satisfy their high-end niche edge case.
Right... the parent is saying that 4x10G is pointless compared to 4x2.5G, because PCIe 4 lanes will top out at forwarding around 7gbps of traffic.
You can't do line rate on all ports either (limited by PCIe alone, let alone CPU for smaller packets), but you can certainly fill an individual port, which I suspect is the goal.
> because PCIe 4 lanes will top out at forwarding around 7gbps of traffic [...] limited by PCIe alone
Are you sure about that? With 5GT/s (or 500MB/s) per lane, and with 4 lanes, that should be plenty, no? Intel adapters like the x520-DA2 are specced at 2x 10G, and use PCIe 2.0 x8.
FWIW, I was also able to iperf3 around 3.7Gbps on a X520-DA2 connected to an RPi4's single-lane PCIe 2.0.
But PCIe is full duplex! With PCIe 2.0 x4 there's 4 lanes in each direction [1], so when 'forwarding' over a single 10G link you can expect to send and receive simultaneously at the speed I mentioned earlier.
Yah, I guess dividing by 2 isn't fair. But transmitting does impact receiving and vice-versa: when you're reading DMA descriptors, you need to wait/hold for posted completions, etc. It's not fully uncontended between send and receive, but more uncontended than a naive division by 2 would imply.
You are right that 4 PCIe 2.0 lanes have an aggregate speed of 16 Gb/s (even if in marketing speak they would be wrongly called as 20 Gb/s), which is not enough even for two 10 Gb/s ports, but only for one.
However, that is not due to being a cheap hobbyist board, but due to Intel, who have not provided their Atom line of CPUs with more lanes or with faster lanes.
Only the Elkhart Lake and Jasper Lake CPUs, which have been just launched by Intel as a replacement for the Gemini Lake Refresh CPUs used in ODROID H2, are the first Intel CPUs of the Atom series that have PCIe 3.0.
> 2.5G is perfect for hobbyist use because it is significantly cheaper than anything 10G at the moment
No, not really, it's the other way around.
SFP+ based 10g networking equipment is much cheaper and and with much higher availability. And if needed, you can always adapt it to 1/2.5/5.10g networking using sfp+->rj45 transceivers. Only disadvantage was that you had to run fiber.
Maybe it will change this year, bot for now 10g was a way to go
> Too bad it's not a single SFP+ 10G cage, that would be actually useful then. :/
You can plug an M.2 to PCIe adapter and a used eBay 10GbE PCIe cards into the ODROID-H2+. A quick search showed several folks talking about this. eg: https://forum.odroid.com/viewtopic.php?t=39174 . It'd be nicer to have a straight M.2 to SFP+ but oh well.
I've currently a Unifi USG at home with my 1gbps symmetric connection.
Generally it's pretty good.
However a recent software update has completely broken IPv6 prefix delegation.
So I've been thinking of replacing it with a more open router.
I've seen pfsense is popular, but OpenWRT also seems to be quite active and Linux based which I'm more familiar with. But also for smaller less power hungry machines.
My use case is a router so I don't need cores or ram, but rather being able to meet the network throughput for a low power consumption.
Is there anything that bits the bill?
Preferably something that was released recently to ensure software updates will be easy on it.
Does the AMD Jaguar GX-412TC SOC really handle full Gigabit with non-trivial rules and/or SQM? Also the 6W is the minimum, it goes up to 10 under load.
But if you're comparing with products from Ubiquiti (especially the Unifi line) you're probably going to see similar performance anyway. I switched to a APU2 as my home router two years ago, and I've been very happy with it. Doesn't seem to break a sweat handling a gigabit connection in a typical home setting, running VyOS. I'd probably think about something more powerful in an office or data center.
I’ve been using APU2 systems for several years. It can handle gigabit if you’re using Linux and your ISP doesn’t use PPPoE. On PPPoE it tops out around 500mbps for download and 800mbps for upload.
I've applied all of the tweaks and I haven't ever seen it sustain anywhere near 1 Gigabit on a single connection. That's using both a datacenter iperf3 instance and sites like speedtest.net.
Multiple connections can easily breach 1 Gigabit though.
I've ran pfSense on older ALIX and the APU boards. On the APU I sadly got nowhere close to my symmetric gigabit speeds. I'm not that knowledgeable on network stacks but from what I heard it's probably BSDs network stack that doesn't cut it on that hardware.
I've read that MikroTik's RouterOS delivers significantly more performance on an APU Board but I've not tried that yet (but plan to).
To be frank current versions of PfSense have been nothing but disappointing since the recent parent acquisition of PfSense. I got frustrated enough with errant problems that I just bought a proper enterprise grade router. Never looked back.
Since I run a lot of 10G fiber and use proper 10G switches (none of this Ubiquiti garbage) layer 3 routing from the router itself was an absolute must. I decided on the PaloAlto Networks PA-220 https://www.paloguard.com/Firewall-PA-220.asp .
Yes, I know - it's expensive. However, their support is fantastic and my employer agreed to pay for a portion of the licensing since I mostly require this for WFH. Zero issues, passive cooling, top notch security and this thing just freaking purrs.
This is likely off-topic but I am looking hard at having home 10Gbps network and I struggle to find noise-less switches that allow copper 10Gbps links. I am okay with buying one fanless Mikrotik 10Gbps switch with SFP cages and buy adapters but I am open to other ideas.
Can you recommend quiet 10Gbps switches that support copper links?
Copper links in the form of ethernet 10g base t or copper links as direct attach sfp+ cables? 10g base t ethernet uses quite a bit more power, so those switches are usually always loud. Also, 10g base t tranceivers when used long term can actually damage sfp+ switches and cost wise they make absolutely zero sense.
To be frank, in the long run if you plan to make any changes to your setup, buying transceivers from FS.com and fiber cables (even armored fiber) for that matter is about the same price as copper direct attach cables. Also much easier to cable manage and they can be used with cable lengths from 0.2m to 400m in most cases.
It's kind of loud but I keep my hardware in a closet and to be honest, enterprise hardware like this makes heat and needs fans for a good reason haha.
If you're looking for a more consumer entry and don't care about L3 routing Ubiquiti actually makes one hell of a damn good sfp+ "switch" for the money -> https://www.ui.com/unifi-switching/unifi-switch-16-xg/ and again - cannot recommend FS.com enough for their service and quality products. Even if it was a little weird that their sales rep immediately added me on LinkedIn?
Sadly I don't have enough free space currently so I have to bet on fanless tech (but I do enjoy it).
I meant normal Cat6 Ethernet 10Gbase-T cables, yep. If not, I can indeed buy a fanless Mikrotik switch with SFP+ cages and just buy 8-10 adapters from SFP+ to 10Gbase-T, I suppose.
Bookmarking your link, definitely will look into that switch when I have the proper closet space for server tech.
No worries! Keep in mind that sfp+ transceivers are "branded" - in that you have to buy the right sfp+ transceiver to match the brand of switch or interface you're using. For instance, intel NIC's generally require intel "marked" transceivers etc. They last a damn long time though. Those small MikroTik switches are also awesome!
Is that true of the ubiquity? I didn't think they required branded SFPs, they don't on some of their other hardware.
Similarly for mikrotik.
I usually find that companies with "licensable" features on their hardware are the ones locking their SFPs to their switches/etc in an effort to enforce what is basically a port license (aka buy our $80 SFPs instead of the generic ones we are rebranding so we can make $ on your port usage vs just adding a $$$/port license fee).
I haven't started redoing my home network in 10Gbps yet but after scanning forums for a few weeks (back in the summer), I've seen many people say that Mikrotik switches work without any issue with several non-Mikrotik DACs (not sure I'm remembering the abbreviation well but it's a small device that converts from SFP+ to 10Gbase-T Ethernet cable).
There are devices out there to rebrand a transceiver. Probably not worth it for home use, but my employer has one of these, so maybe ask around if you could get wrongly branded ones for cheap.
I tried OPNsense as well, but I'm getting to a point in life where I'm okay spending money to not become my own devops guy when I don't have to. Running a Saas company as a side gig will do that to you ;)
Another popular low-cost option is to run pfSense on an old thin client. The ideal hardware is something like the HP T610+ or T620+, which even have a PCIe slot for you to add a secondary ethernet interface if you don't want to get a VLAN switch and have to do a router-on-a-stick configuration. The form factor is nice because it's quiet, a lot smaller than a full PC, it's set up for wall/VESA mounting, and you don't need to muck about with buying specialty cases as you do for MiniITX motherboards or other SBCs.
Anyway, predictably these things are being constantly cycled out and can typically be had on ebay for $100-200.
Yes, but it also super compact, draws very little power and it has multiple network interfaces. It's also fanless, which makes it ideal for long running set-up-and-forget scenarios that home routers typically are.
GX-420CA also has a GPU which GX-412TC lacks. In headless operation the difference would not be that much. For reference, I have a t620 with GX-415GA (TDP 15W) which draws 12W at the wall in a CPU load scenario.
$150 (roughly the cost of APU2+case+power brick+16GB SSD) for a t620 would be daylight robbery. I got the non-plus variant on the secondary market for $30 delivered and it works very well, uses roughly the same CPU as APU2 as well (4-core AMD Jaguar derivative, AES-NI & AVX extensions). Power usage is 6W idle (headless), 12W full-load. UEFI is basic but functional, I can boot Linux straight from the EFI boot entry.
t620 Plus has a PCIe riser so would be better suited for a network gateway.
The Wyse equivalents from that era would be Zx0Q/Dx0Q series although I don't think they had a PCIe version.
Another interesting option is something like Fujitsu Futro S9010/S940 (4-core Goldmont, you get SHA ISA extensions which is nice), has a PCIe riser as well, but there are fewer of these on the secondary market.
The H2, with a Gemini Lake Refresh CPU, is certainly much faster than the APU2 which somebody suggested and also much faster than any cheap ARM device that uses old ARM cores like Cortex-A72 or Cortex-A73.
I think that H2 is a good choice for a router/firewall, it is faster than anything in that price range, it is silent with passive cooling and the power consumption in normal use is much less than 10 W.
Being a standard PC, any Linux distribution, as well as FreeBSD, Windows or any other OS can be installed without problems.
I have used in the past a few older ODROID models and I have been content with them.
Or I don't know, SBC vendors, you can get one AMD R1606G board and actually bother to make its 2x 10Gbps on-chip NICs usable? As opposed to everyone attaching 2x 1Gbps to it for whatever reasons (really tempted to say its laziness but maybe there are legitimate reasons, although I don't think so; why would AMD make a board on which an on-chip dual NIC is unusable?).
Piling such addons on top of an otherwise single-board computer feels quite strange to me. What are the thermal implications of such a setup?
Not an expert by any stretch but I am very skeptical.
I very very very much loath the lag in the market. The SBC market is a sham, as far as I'm concerned, with these stupid ARM toys they keep pushing. 100%, the real embedded market has much better equipment.
I looked around some for R1606G offerings, and there's quite a number of boards around $350. Alas, none use the on-chip 10Gbe ports. I haven't found anything under $700 that offers 10Gbe. Even the $350 is questionable, when by compare one can get a 1L sized mini-pc with a 35W desktop chip that will rip it to shreads. Those systems will idle under 10w. Mine does.
The R1606G chip you talk about, the R1606G, is 2 Zen cores running 2600-3500 Hz & uses 12-25W. That's a bit north of what something like this here Celeron kicks out, but it does match a lot of Chromebox style computers very closely, which is a little smaller than the typical business 1L mini-pc.
There is also a R1305G and R1102G which have lower clocks & single-channel in the R1102G's case. They also have dual 10Gbe. I'd love to know what price chips like these come at. I suspect AMD plays the same game Intel does: even though you are getting 1/8th the physical chip size versus a desktop, the fact that it has competitive performance in it's price range means AMD/Intel are still going to try & charge you the same ~$300 they'd charge for a desktop chip.
Same experience here with the R1606G. I got very hyped when I have seen the performance and power usage... and then NOBODY offered it with the stock dual 10Gbe NICs.
Do you have a link to the full specs of your system that idles under 10W? Quite amazing. My OptiPlex 3060 (the micro form factor, no bigger than a book) is likely idling like that as well but haven't measured yet (hey, any recommendations for "smart" power monitors that can be aggregated in a central dashboard?).
Also, do you have links to R1606G setups with 10Gbe somewhere? Your comment made it seem like you did eventually find such setups. Don't care much if they are $700 or more, it will be a one-time investment for at least 5 years in the future.
The lower-clocked Ryzen 1000 boards I didn't have much interest in. I am looking to build a super low-power home NAS with the R1606G, plus some light VM and Docker activities.
And yeah, pretty sure somebody recognized how valuable these boards are and are charging arm and leg for them. :(
This is becoming a super sweet routing platform! $120 board + $50 add-on card is a little on the pricey side, but ok. I wondered how they did it, but ahh, it's an x86 celeron chip, so of course there's PCIe. I really hope we start seeing more System-on-Chip from ARM &c with PCIe soon. RPi3's ethernet is powered by a single lane, which is more than most competitors do! Time to raise the stakes a little all!!
But all this.. it just makes me so so so so sad how little wifi gear is available to DIY with. If you want to DIY wifi you basically have to rely on OpenWRT, which is all hacked routers. Support for modern chipsets has been woefully scary bad. We're seeing some hopes, some code drops of some semi-modern Qualcomm chipsets, but as has been the case for most of the decade, Broadcom is totally 100% no-go useless, has no support, no drivers, I don't even know if we have a way to any of these boards any more.
But relying on prepackaged consumer equipment- it's un-ideal. I'd way rather be able to, like this add-on, build something myself. The cryptocurrency craze has shown that, in fact, PCIe works fairly well over a decent length of USB cable, and Thunderbolt mirrors that idea at 10x the price point/sophistication/complexity. The idea of having a m.2 card with some wifi hardware on it should not be that out there.
Yet there's near to no marketplace for good wifi gear. Compex is one of the few/only companies making anything what-so-ever, and it's quite expensive.
I've been meaning to try to run a Debian install on my Netgear X4S (R7800) router for a long time now. It'll mean, likely, trying to compile an OpenWRT & borrowing it's kernel & probably some modules. And kind of slap-dash throwing them in-to a Debian distro, and either uboot or perhaps having to kexec from OpenWRT into Debian. I'd really like to have a less exotic system to build wifi with. For now, my increasingly aged router continues to be one of the best options available for wifi that gives me any power or control. It's scary having such radically increasingly consumerized options dominate. It's scary to me how many techs around me in the world are happy to just buy more expensive fancy gear, go get Ubiquiti, rather than invest themselves in such a core critical foundational social technology. I want very much there to be some kind of call to action, some way to rally, but hardware keeps getting further & further away. This is the actual story of Big Tech that scares me, because it restricts what is possible. What big social networks do? Just go do something else online. But wifi? That is the last & most important mile of connectivity, to us & those around us. I know it doesn't entirely make sense, but it's something I think geeks ought to feel is important & ought to invest themselves, not just their money in. I'm tired of being well served by enterprise gear in the home, but not serving ourselves.
If you want to also have a WiFi access point, besides an Ethernet switch/router, there are 2 choices.
You can use an Intel NUC or similar, which has the perfectly supported and high-speed Intel WiFi, together with several USB to Ethernet adapters, to have both a WiFi AP and multiple Ethernet ports. That is what I am currently using.
Or, for a cheaper device, you can use Odroid H2 or something similar, with an USB WiFi dongle, taking care to select a model that has WiFi drivers for whatever operating system you use.
i should try a more modern intel wifi chip, yes, but in my experience these _client_ wifi chips make really limited access points, with not good range, and most noticably have critical/severe/cripplingly bad performance with multiple clients.
many usb chipsets straight up refuse to allow more than a handful of clients. i think my Alfa wireless usb kit with realtek chipsets refuse more than 7 connections. other kit just has multi-second jitter creep in every now and then, in a relatively un-hostile wifi environment.
it absolutely 100% _should_ be possible & easy & work well to do either of these options. perhaps with current gen modern chipsets things are better. here, alas, things fail again, because it seems many modern wifi chipsets don't have available linux drivers at all, much less competent ap mode drivers. "good silicon with bad drivers is just expensive sand"
go to a reliable respected wifi vendor like rokland[1] and look around: how many of these devices are <2 years old? <3 years old? many of the best sellers are 5+ years old. good wifi is becoming frighteningly scarce.
this is very common advice you've presented, but i've found it to be woefully critically unacceptable. just incredibly bad wifi. we are doing a mis-service to people telling them these are ok options. i've tried real hard to make these work, across years, bought dozens of devices, & it's been unbelievably unsatisfactory. i would strongly recommend against taking @adrian_b's listed options, in emphatic terms. do not do this to yourself.
but please, make this a future we can do. as i begged for, please, we need some folks like Compex putting modern product on the market unbundled. we need some drivers. for modern wifi. that we can use. please. please.
Realtek and Broadcom are both also in my "Don't even consider"-List for most uses because of the extremely bad driver situation with both of them. Intel Wifi joined this list for most use-cases.
I'm quite happy with my atheros chips (various QCA9XXX) and the mediatek chips (in the modes, where the driver doesn't crash hard)
You are probably right about the problems with many clients, I would not know.
While I have a network with more than a dozen computers, most are wired and I seldom connect through WiFi more than a single laptop.
In this use case, I did not have problems, and I much prefer to use as router a computer over which I have complete control instead of some appliance that does whatever it wants, but I have not tried to see what happens with many WiFi clients in this configuration, because I have not needed that.
Does Intel really have chips in 2021 that can finally support standards like 802.11s(mesh) (finalized 10+ years ago) and RSN-IBSS with SAE/WPA3 using encryption with MTU allowing encapsulating standard 1500 ethernet, on 5Ghz, along with the basic boring modes like AP/VLAN, station?
Atheros and Mediatek do this out of the box, since years, on mainline kernels.
Unfortunatelly, distributions usually compile wpa_supplicant without MESH support, at least Debian does (or did).
So many people don't actually realize, how crippled these Intel Wifi cards are. Especially, now that kernel 5.5 removed the "disable_lar" (Used to enable AP-on-5GHz) module option.
I'm expecting a lot of disappointed customers dumping their Intel cards on the second hand market soon.
Last time I checked, Intel consumer wi-fi cards would not even let you do AP on 5GHz (the entire spectrum was marked No-IR in firmware regardless of regulatory domain with no way to override it). But this was a while ago, I'd be also curious to know if they came to their senses.
The "disable_lar" option enabled "iw reg set" to override what the eeprom stupidly forbids, until linux kernel-version 5.5, when the nice people @Intel removed the option without an alternative, that doesn't mark your kernel as tainted.
(If I'm wrong, and you/somebody knows a way to re-enable AP-on-5GHz-on-Linux-5.5+ for iwlwifi on signed distribution kernels without Out-Of-Tree Taint, please comment)
And on top it they refuse to release support for 802.11s mesh! Even for 2.4GHz!
I'm curious how many of the new commercial home wifi mesh systems out there use 802.11s.
A couple of years, I'd been under the impression that many folk in wifi mesh land were using BATMAN or Babel or one of the two dozen other routing protocols[1]. OLPC was the last big 802.11s user that I knew of, and I never heard glowing reviews, but that was a decade ago. I'm curious how things are shaping up these days.
Rather well for the Batman, using any underlying type of transport technology/protocol like 802.11s, RSN-IBSS, wired links!, etc.
These days, batman is absolutely painless to setup in Debian and OpenWRT, and works great. (Only caveat is, that this excludes Intel Wifi because they support neither 802.11s nor RSN-IBSS for total meshing.)
Hardkernel folks: Amazing product. Please consider having a warehouses in every continent. Or some resellers. $50+ for shipping on a $120 product is a bit too steep.
90 comments
[ 4.2 ms ] story [ 151 ms ] thread2.5G switches are expensive and rare. If you want to make use of multiple links to an L2 ToR, you have to set up LACP which is annoying. You could do ECMP for L3 LB instead, but that requires an even more expensive switch.
[1] - https://mikrotik.com/product/crs317_1g_16s_rm
EDIT: as for commitment, I'm literally looking to solve the problem of 'I want a low power k8s cluster, and I'm okay with a switch SPOF' for bgp.wtf's FMT location. If I could buy 5-6 of these, with 10G uplink, especially ready to rack, I'd probably order them today.
SFP+ is cheaper and better, but not exactly common in consumer networks where I imagine most low-cost ARM boards end up.
Right now the cheapest options are to get a 10Gb PCIe card from Newegg or similar, which'll run you about $200 where I am, and that's just a PCIe extension card.
If you want to go _very_ cheap and dirty you can get some FlexibleLOM/PCIe adapters [1] and some HP FlexibleLOM NICs [2]. This results in a 2x 10G NIC for around ~$15, including the adapter.
[1] - https://github.com/TobleMiner/HPE-FlexibleLOM-adapter
[2] - https://www.ebay.com/itm/HP-10GB-530FLR-649869-001-647579-00...
That said I think there's a market for boards between that $15 mark and the $200 I'd have to pay for something considerably higher-end.
[1] - https://www.cdw.com/product/hp-reman-enet-10gbe-530flr-sfp/3...
A 10 Gb/s Ethernet NIC from ASUS, with the Aquantia controller, is $93 at Amazon.
I use a few of those, bought in 2019, and they work fine.
https://www.amazon.com/ASUS-XG-C100C-Network-Adapter-Single/...
The number of people with faster than 1G internet is extremely small.
Business customers who have >1G commercial connections are not in the market for using ODROID boards for their routing needs.
2.5G is perfect for hobbyist use because it is significantly cheaper than anything 10G at the moment, while being faster than common 1G.
If you need 10G switching, get a 10G switch.
And then I might as well do that with an Intel NUC, or just go for a normal mini-ITX motherboard.
The PCIe slot on the H2 is PCIe 2.0 x4, which doesn't have enough bandwidth to saturate more than one 10G port.
This is why hobby hardware and the enthusiast market is so difficult: You can make a great product at a cheap price point and people will still complain that it doesn't satisfy their high-end niche edge case.
You can't do line rate on all ports either (limited by PCIe alone, let alone CPU for smaller packets), but you can certainly fill an individual port, which I suspect is the goal.
Are you sure about that? With 5GT/s (or 500MB/s) per lane, and with 4 lanes, that should be plenty, no? Intel adapters like the x520-DA2 are specced at 2x 10G, and use PCIe 2.0 x8.
FWIW, I was also able to iperf3 around 3.7Gbps on a X520-DA2 connected to an RPi4's single-lane PCIe 2.0.
Yah, I said forwarding-- so each packet goes over the PCIe link twice.
So, to use your test-- 3.7 x 4 / 2 = 7.4gbps. I said around 7. 7.4gbps would be a bit more than I'd expect, but I'd not be shocked.
[1] - https://en.wikipedia.org/wiki/PCI_Express#Pinout
Ethernet speeds are given in data bits per second, while USB 3.0, PCIe 1.0 and PCIe 2.0 add in their speed values the extra bits used for encoding.
On the other hand, 10 Gb/s USB and 8 Gb/s PCIe 3.0 have really those speeds. Complicated :-(
One PCIe 2.0 lane is more than enough for 2.5 Gb/s Ethernet, but not enough for 5 Gb/s Ethernet or 10 Gb/s Ethernet.
However, that is not due to being a cheap hobbyist board, but due to Intel, who have not provided their Atom line of CPUs with more lanes or with faster lanes.
Only the Elkhart Lake and Jasper Lake CPUs, which have been just launched by Intel as a replacement for the Gemini Lake Refresh CPUs used in ODROID H2, are the first Intel CPUs of the Atom series that have PCIe 3.0.
No, not really, it's the other way around. SFP+ based 10g networking equipment is much cheaper and and with much higher availability. And if needed, you can always adapt it to 1/2.5/5.10g networking using sfp+->rj45 transceivers. Only disadvantage was that you had to run fiber. Maybe it will change this year, bot for now 10g was a way to go
You can plug an M.2 to PCIe adapter and a used eBay 10GbE PCIe cards into the ODROID-H2+. A quick search showed several folks talking about this. eg: https://forum.odroid.com/viewtopic.php?t=39174 . It'd be nicer to have a straight M.2 to SFP+ but oh well.
Generally it's pretty good. However a recent software update has completely broken IPv6 prefix delegation.
So I've been thinking of replacing it with a more open router.
I've seen pfsense is popular, but OpenWRT also seems to be quite active and Linux based which I'm more familiar with. But also for smaller less power hungry machines.
My use case is a router so I don't need cores or ram, but rather being able to meet the network throughput for a low power consumption.
Is there anything that bits the bill? Preferably something that was released recently to ensure software updates will be easy on it.
Would the H2 work?
So I'm a bit worried about future software support.
Reading this:
https://teklager.se/en/knowledge-base/apu2-1-gigabit-through...
It looks like 1gbps is possible with tweaks. My hope is whether there's a more plug and play solution.
But if you're comparing with products from Ubiquiti (especially the Unifi line) you're probably going to see similar performance anyway. I switched to a APU2 as my home router two years ago, and I've been very happy with it. Doesn't seem to break a sweat handling a gigabit connection in a typical home setting, running VyOS. I'd probably think about something more powerful in an office or data center.
I've applied all of the tweaks and I haven't ever seen it sustain anywhere near 1 Gigabit on a single connection. That's using both a datacenter iperf3 instance and sites like speedtest.net.
Multiple connections can easily breach 1 Gigabit though.
I've read that MikroTik's RouterOS delivers significantly more performance on an APU Board but I've not tried that yet (but plan to).
Yes, I know - it's expensive. However, their support is fantastic and my employer agreed to pay for a portion of the licensing since I mostly require this for WFH. Zero issues, passive cooling, top notch security and this thing just freaking purrs.
Can you recommend quiet 10Gbps switches that support copper links?
To be frank, in the long run if you plan to make any changes to your setup, buying transceivers from FS.com and fiber cables (even armored fiber) for that matter is about the same price as copper direct attach cables. Also much easier to cable manage and they can be used with cable lengths from 0.2m to 400m in most cases.
I currently use a Cisco Nexus 3064 - https://www.ebay.com/p/219487215
It's kind of loud but I keep my hardware in a closet and to be honest, enterprise hardware like this makes heat and needs fans for a good reason haha.
If you're looking for a more consumer entry and don't care about L3 routing Ubiquiti actually makes one hell of a damn good sfp+ "switch" for the money -> https://www.ui.com/unifi-switching/unifi-switch-16-xg/ and again - cannot recommend FS.com enough for their service and quality products. Even if it was a little weird that their sales rep immediately added me on LinkedIn?
Sadly I don't have enough free space currently so I have to bet on fanless tech (but I do enjoy it).
I meant normal Cat6 Ethernet 10Gbase-T cables, yep. If not, I can indeed buy a fanless Mikrotik switch with SFP+ cages and just buy 8-10 adapters from SFP+ to 10Gbase-T, I suppose.
Bookmarking your link, definitely will look into that switch when I have the proper closet space for server tech.
Similarly for mikrotik.
I usually find that companies with "licensable" features on their hardware are the ones locking their SFPs to their switches/etc in an effort to enforce what is basically a port license (aka buy our $80 SFPs instead of the generic ones we are rebranding so we can make $ on your port usage vs just adding a $$$/port license fee).
Part of why I'm grateful for the unstoppable OPNsense https://opnsense.org
Anyway, predictably these things are being constantly cycled out and can typically be had on ebay for $100-200.
https://www.cpu-world.com/CPUs/Puma/AMD-G-Series%20GX-412TC....
https://www.amd.com/en/system/files?file=2017-06/g-series-so...
t620 Plus has a PCIe riser so would be better suited for a network gateway.
The Wyse equivalents from that era would be Zx0Q/Dx0Q series although I don't think they had a PCIe version.
Another interesting option is something like Fujitsu Futro S9010/S940 (4-core Goldmont, you get SHA ISA extensions which is nice), has a PCIe riser as well, but there are fewer of these on the secondary market.
I recommend this charmingly old-fashioned site for reference on old thin clients https://www.parkytowers.me.uk/thin/
https://www.reddit.com/r/PFSENSE/comments/8ytqqn/hp_t620_plu...
The CPU is more of a limiter if you're trying to run a VPN or something.
I think that H2 is a good choice for a router/firewall, it is faster than anything in that price range, it is silent with passive cooling and the power consumption in normal use is much less than 10 W.
Being a standard PC, any Linux distribution, as well as FreeBSD, Windows or any other OS can be installed without problems.
I have used in the past a few older ODROID models and I have been content with them.
Piling such addons on top of an otherwise single-board computer feels quite strange to me. What are the thermal implications of such a setup?
Not an expert by any stretch but I am very skeptical.
I looked around some for R1606G offerings, and there's quite a number of boards around $350. Alas, none use the on-chip 10Gbe ports. I haven't found anything under $700 that offers 10Gbe. Even the $350 is questionable, when by compare one can get a 1L sized mini-pc with a 35W desktop chip that will rip it to shreads. Those systems will idle under 10w. Mine does.
The R1606G chip you talk about, the R1606G, is 2 Zen cores running 2600-3500 Hz & uses 12-25W. That's a bit north of what something like this here Celeron kicks out, but it does match a lot of Chromebox style computers very closely, which is a little smaller than the typical business 1L mini-pc.
There is also a R1305G and R1102G which have lower clocks & single-channel in the R1102G's case. They also have dual 10Gbe. I'd love to know what price chips like these come at. I suspect AMD plays the same game Intel does: even though you are getting 1/8th the physical chip size versus a desktop, the fact that it has competitive performance in it's price range means AMD/Intel are still going to try & charge you the same ~$300 they'd charge for a desktop chip.
Do you have a link to the full specs of your system that idles under 10W? Quite amazing. My OptiPlex 3060 (the micro form factor, no bigger than a book) is likely idling like that as well but haven't measured yet (hey, any recommendations for "smart" power monitors that can be aggregated in a central dashboard?).
Also, do you have links to R1606G setups with 10Gbe somewhere? Your comment made it seem like you did eventually find such setups. Don't care much if they are $700 or more, it will be a one-time investment for at least 5 years in the future.
The lower-clocked Ryzen 1000 boards I didn't have much interest in. I am looking to build a super low-power home NAS with the R1606G, plus some light VM and Docker activities.
And yeah, pretty sure somebody recognized how valuable these boards are and are charging arm and leg for them. :(
But all this.. it just makes me so so so so sad how little wifi gear is available to DIY with. If you want to DIY wifi you basically have to rely on OpenWRT, which is all hacked routers. Support for modern chipsets has been woefully scary bad. We're seeing some hopes, some code drops of some semi-modern Qualcomm chipsets, but as has been the case for most of the decade, Broadcom is totally 100% no-go useless, has no support, no drivers, I don't even know if we have a way to any of these boards any more.
But relying on prepackaged consumer equipment- it's un-ideal. I'd way rather be able to, like this add-on, build something myself. The cryptocurrency craze has shown that, in fact, PCIe works fairly well over a decent length of USB cable, and Thunderbolt mirrors that idea at 10x the price point/sophistication/complexity. The idea of having a m.2 card with some wifi hardware on it should not be that out there.
Yet there's near to no marketplace for good wifi gear. Compex is one of the few/only companies making anything what-so-ever, and it's quite expensive.
I've been meaning to try to run a Debian install on my Netgear X4S (R7800) router for a long time now. It'll mean, likely, trying to compile an OpenWRT & borrowing it's kernel & probably some modules. And kind of slap-dash throwing them in-to a Debian distro, and either uboot or perhaps having to kexec from OpenWRT into Debian. I'd really like to have a less exotic system to build wifi with. For now, my increasingly aged router continues to be one of the best options available for wifi that gives me any power or control. It's scary having such radically increasingly consumerized options dominate. It's scary to me how many techs around me in the world are happy to just buy more expensive fancy gear, go get Ubiquiti, rather than invest themselves in such a core critical foundational social technology. I want very much there to be some kind of call to action, some way to rally, but hardware keeps getting further & further away. This is the actual story of Big Tech that scares me, because it restricts what is possible. What big social networks do? Just go do something else online. But wifi? That is the last & most important mile of connectivity, to us & those around us. I know it doesn't entirely make sense, but it's something I think geeks ought to feel is important & ought to invest themselves, not just their money in. I'm tired of being well served by enterprise gear in the home, but not serving ourselves.
You can use an Intel NUC or similar, which has the perfectly supported and high-speed Intel WiFi, together with several USB to Ethernet adapters, to have both a WiFi AP and multiple Ethernet ports. That is what I am currently using.
Or, for a cheaper device, you can use Odroid H2 or something similar, with an USB WiFi dongle, taking care to select a model that has WiFi drivers for whatever operating system you use.
many usb chipsets straight up refuse to allow more than a handful of clients. i think my Alfa wireless usb kit with realtek chipsets refuse more than 7 connections. other kit just has multi-second jitter creep in every now and then, in a relatively un-hostile wifi environment.
it absolutely 100% _should_ be possible & easy & work well to do either of these options. perhaps with current gen modern chipsets things are better. here, alas, things fail again, because it seems many modern wifi chipsets don't have available linux drivers at all, much less competent ap mode drivers. "good silicon with bad drivers is just expensive sand"
go to a reliable respected wifi vendor like rokland[1] and look around: how many of these devices are <2 years old? <3 years old? many of the best sellers are 5+ years old. good wifi is becoming frighteningly scarce.
this is very common advice you've presented, but i've found it to be woefully critically unacceptable. just incredibly bad wifi. we are doing a mis-service to people telling them these are ok options. i've tried real hard to make these work, across years, bought dozens of devices, & it's been unbelievably unsatisfactory. i would strongly recommend against taking @adrian_b's listed options, in emphatic terms. do not do this to yourself.
but please, make this a future we can do. as i begged for, please, we need some folks like Compex putting modern product on the market unbundled. we need some drivers. for modern wifi. that we can use. please. please.
[1] https://store.rokland.com/
I'm quite happy with my atheros chips (various QCA9XXX) and the mediatek chips (in the modes, where the driver doesn't crash hard)
You are probably right about the problems with many clients, I would not know.
While I have a network with more than a dozen computers, most are wired and I seldom connect through WiFi more than a single laptop.
In this use case, I did not have problems, and I much prefer to use as router a computer over which I have complete control instead of some appliance that does whatever it wants, but I have not tried to see what happens with many WiFi clients in this configuration, because I have not needed that.
Atheros and Mediatek do this out of the box, since years, on mainline kernels.
Unfortunatelly, distributions usually compile wpa_supplicant without MESH support, at least Debian does (or did). So many people don't actually realize, how crippled these Intel Wifi cards are. Especially, now that kernel 5.5 removed the "disable_lar" (Used to enable AP-on-5GHz) module option.
I'm expecting a lot of disappointed customers dumping their Intel cards on the second hand market soon.
https://www.intel.com/content/www/us/en/support/articles/000...
(If I'm wrong, and you/somebody knows a way to re-enable AP-on-5GHz-on-Linux-5.5+ for iwlwifi on signed distribution kernels without Out-Of-Tree Taint, please comment)
And on top it they refuse to release support for 802.11s mesh! Even for 2.4GHz!
A couple of years, I'd been under the impression that many folk in wifi mesh land were using BATMAN or Babel or one of the two dozen other routing protocols[1]. OLPC was the last big 802.11s user that I knew of, and I never heard glowing reviews, but that was a decade ago. I'm curious how things are shaping up these days.
[1] https://en.wikipedia.org/wiki/Wireless_mesh_network#Routing_...
These days, batman is absolutely painless to setup in Debian and OpenWRT, and works great. (Only caveat is, that this excludes Intel Wifi because they support neither 802.11s nor RSN-IBSS for total meshing.)
The H2+ is a quad-core 2.3 GHz intel board. Anyone know if that would be enough juice to handle a gigabit wireguard link?
https://routerchart.com/linksys/linksys-wrt3200acm-wrt3200ac...
The sell both ARM and x86.
The H2(+) is an x86, and is well-supported by the standard Linux.
ARM boards may require kernel forks, instead, so I suspect you had a board with this architecture.