Be careful when using a very small SSD or something like a CF card for a router. Enabling logging to disk can wear the flash memory out in a matter of weeks
I have a 64GB SSD and a little over 4.5GB used in the last year. I don't think that flash exhaustion is likely in the near future... but I also have a 32GB USB stick plugged in, which gets a backup copy once a week. The great thing about building your own router is that you can easily replace parts.
I have been using a non-commercial, 8GB CompactFlash card with my pfSense install for 2 years with no problems, and I run the full pfSense rather than the NanoBSD version.
I do not run extra services like Squid or Snort, so most of my "writes" are probably within my massive 1GB of RAM.
With modern SSD that isn't going to be a concern. You can write to them at full speed constantly for months and real world tests have shown them to last just fine and you are not going to max out 500MB/s writes to an SSD with log files :)
As a point of comparison, the iptables syntax as shown in the Ars article is far harder to grok at first glance than either of the pf examples. For example:
pass in on egress inet proto tcp from any to (egress) port { 80 443 } rdr-to <ipaddr>
Yeah, I've always found iptables to be a little hard to understand.
It also sounds like it's getting replaced[0]. My guess is that we will see iptables around for a long time after it's been deprecated. ifconfig, for example, is deprecated[1] yet it's still around and being used.
ifconfig is still used on BSD and, most importantly, MacOS X. Also, the Linux replacement for ifconfig is ip from iproute2, and that was made by the guy who did Linux QoS. He sort of sucks at usability.
For example, `ifconfig eth0 down` becomes `ip link set dev eth0 down` and `ifconfig eth0 192.168.1.2` becomes `ip add add 192.168.1.2/24 brd + dev eth0`; I had to look these up to make sure I had the correct syntax.
Okay, the actual command is `ip address add` and not `ip add add`, but it allows extreme abbreviation at the expense of discoverability.
i would say that openbsd is the better tool for the job in my opinion. pf is just much nicer and simpler to set up and configure than linux iptables, and openbsd is hardened which is important for a machine that is always facing the internet.
Another thing that's nice about the BSDs is that they are whole systems, not different sets of kernel+utilities packed up by third parties.
Generally speaking, that means that there's one canonical way to do something instead of a bunch of different ways like in Linux (think 'ifconfig' vs 'ip'). Of course, FreeBSD ships with three different firewalls[0], so that's not always true.
There are downsides, of course. A freshly-installed BSD has a lot less stuff than a freshly-installed OpenSUSE/Ubuntu/Fedora/etc.
i don't think that the minimalism of the default install (i am talking about openbsd mainly, since i have little experience with the other three) is too much of a problem. in the default install is practically all the tools and components one needs to set up and configure a router. any extra components can be grabbed from ports and packages as shown in the aforementioned tutorial, where the dnscrypt client is installed for encrypted dns queries.
PFSense is the same thing below the hood, but with a web front-end and plugins.
Most off the shelf wireless routers work fine as an access point, but are quite bad as a router. So you can just plug your old wireless router into this thing (with DHCP etc turned off), and your whole setup will be much better.
So you go through all that trouble and end up with a terribly slow NIC, a "grownup" OS to maintain and the shitbox MIPS architecture to boot? What a pain. These MIPS boxes have their place, and that's squarely in the OpenWRT / system builder niche.
With the electricity prices the way they are in the US, no way I'd run something like that over a proper x86 as in the article that can saturate a 1Gbps and manage decent disk IO.
>>a proper x86 as in the article that can saturate a 1Gbps
"x86" covers a lot of ground. Certainly, in the "about $100" range, your choices in X86 land can't saturate 1Gbps either. The x86 box in the article was $280 total. About the same cost as the logical upgrade to the EdgeRouter Lite...their ER-8, which can saturate 1Gbps, and has 8 ports.
I had trouble getting a hold of one of those C1037U boxes from China. The seller would 'run out of stock' frequently if I found one for a decent price.
I ended up going with the APU2B4 board (an upgrade from the APU1D mentioned in the article.) I put pfSense on it, and it's been running perfect for a few weeks now.
Even that board is probably massive overkill for most people. I have 50/50 internet, and with full bandwidth used by torrents, a VPN and ssh session open to the router, and the web interface open, I'm still only getting about 10-15% CPU.
Looks like the one listed in the article is both out of stock and going for ~$1400 (I'm guessing auto pricing?).
The APU boards are nice too; I thought about going for one when I was shopping for a better router and slapping OpenBSD on it. Ultimately I went with an Ubiquiti ERL, mostly because I didn't really want to buy an RS-232 cable, but the PC Engines boards are probably one of the best fully-DIY options you can get.
Interesting. I have been using such PCs in my Chinese office for about a year. They are made in a factory about 30 minutes from me. I was considering designing a better looking case and bundling a more reliable power supply to export these but I got busy with bigger business.
These Shenzhen factories are somehow getting these Intel CPUs for next to nothing. Factory price for the i5 model was about $100.
At least in the past, I've heard of people getting lots of low binned / questionable QC'd intel chips for dollars per chip. They basically go for auction to the highest bidder. If you're willing to deal with a high defect rate (Either extensive QC yourself or just don't give a shit), it is a pretty good deal.
Could you please recommend accompanying parts if someone wanted to use that board with pfsense to end up with a home/office wireless router? I was looking for a packaged solution but the official pfsense two port appliance with the wireless option seemed fairly expensive.
I already had a USB->Serial adapter. For storage I used a 32GB Class 10 microSD card in a SD adapter, which I found acceptably fast (and faster than my usb sticks.) Note, if you want to run Snort or Squid, you might want a small SSD, but for a router-only it is overkill.
I assembled the board+case, wrote the 4gb nano-bsd image of pfSense to the SD card, and booted right up.
Total cost was < $200 USD.
edit: I noticed you mentioned wireless...I've heard the wireless support in pfSense using PC Engines boards is better than it was, but I prefer to use (multiple) separate APs for wifi. One of the APs I'm using is: http://www.amazon.com/TP-LINK-TL-WA801ND-Wireless-300Mbps-Re..., which I've had no issues with. I prefer the ability to upgrade/replace my APs without messing with the router, and my router is in the basement, while the APs are on other floors.
I'll probably move to 5GHz wireless later this year, and this separation allows that.
Me too! As others in the thread have mentioned, OpenBSD and pf make for a (IMHO) much easier configuration. Not sure what kind of difference in performance one might expect. I suspect both Linux and OpenBSD are more than capable of keeping up with any traffic one might throw at such a router.
BSD is at a huge disadvantage in QoS capability, to the point that it really shouldn't be recommended as the OS for the gateway on a typical low-speed bufferbloated residential ADSL or DOCSIS connection.
I believe the syntax for writing QoS on FreeBSD and OpenBSD provides very good expressive capability [1]. By using tagging [2], one can assign QoS priority to anything that a firewall rule can define.
Having used FreeBSD QoS on dial-up, ISDN, DSL and cable over the years, it is this expressiveness that is one of the reasons I prefer the pf packet filter and thus BSD.
Here's an example for bandwidth limited wan. Interactive ssh sessions get a queue with a minimum bandwidth; scp and sftp bulk transfers go to a separate queue.
queue rootq on em0 bandwidth 100M max 100M
queue ssh parent rootq bandwidth 20M
queue ssh_interactive parent ssh bandwidth 10M min 5M
queue ssh_bulk parent ssh bandwidth 10M
queue std parent rootq bandwidth 20M default
block return out on em0 inet all set queue std
pass out on em0 inet proto tcp from any to any port 22 set queue(ssh_bulk, ssh_interactive)
What's the actual queuing algorithm in use? My experience with BSD QoS has been that it's easy to set up, but far inferior to the more modern QoS options in Linux (fq_codel, etc).
I just wish tc had an interface understandable by your average sysadmin. I have to dig into the source to figure out what anything does, but not many users will have the ability/desire to do that.
OpenBSD doesn't support any form of Active Queue Management or fair queuing. It used to support an old and mostly useless AQM (RED) but dropped support in 5.6. Now there isn't even a way to apply ECN marking.
A hierarchy of dumb FIFO queues does not make a real QoS system. It can produce reasonable-looking benchmark numbers when the prioritization rules and benchmark are contrived to match, but in the face of real-world traffic that uses HTTP to carry vastly different kinds of traffic over links that don't have constant bandwidth and latency, OpenBSD is hopeless. Even the rate-limiting capability that OpenBSD has is rudimentary and lacks the ability to account for per-packet overhead and framing overhead, which is necessary for accurate traffic shaping on common service types like ADSL.
If you say you don't see any disadvantages for OpenBSD on QoS, then your idea of QoS is twenty years out of date.
Yes, they've got ECN capability, if you're satisfied with RED as your AQM. pfSense apparently also has an implementation of plain CoDel. Work is also underway to bring fair queuing to the BSD world so that networks with more than one simultaneous traffic flow can get some good QoS: http://caia.swin.edu.au/freebsd/aqm/
All of the BSDs are still way behind; the best by only a few years, while others are stuck in the 1990s.
CoDel doesn't do fair queuing so its performance on a mixed stream of traffic is much worse than fq_codel and other AQMs that have a fair queuing component. Please, read some of the literature on what's been developed in recent years. You don't seem to know anything about this subject other than having seen some of these terms show up in pfSense configuration pages and thinking it means it's at parity with Linux.
I doubt it would help to point you to anything specific, since you've managed to not find for yourself any of the wealth of relevant information that's been written over the past several years, from draft RFCs to LWN articles and Wikipedia articles to blog and mailing list discussions by internet luminaries and even some of their comments in recent HN discussions. At some point you have to actually put in the minimal effort of googling a topic and skimming at least one of the results, instead of assuming that the people around you are making stupid and obvious mistakes.
I bought MikroTik for the netflow feature, which can reveal active malware via hardware packet counters by endpoint.[0] The next cheapest router with netflow is in the thousands of dollars. MikroTik is $180 at Amazon.
Which one do you have? Mine is an RB751U-2HnD and it is plagued with problems, to the point that I set-up an automatic reboot every other day and I'm thinking about switching to Ubiquiti gear...
A client of mine has a RouterBOARD 1100 X2 AH that I also decided to reboot daily for stability reasons. A RouterBOARD 951G 2HnD used in a branch office lost all IPSec configuration and hangs when viewing the settings on the command line. Certified MikroTik technician says the thing has to be reformatted and set up from scratch.
My first experience with MikroTik products. Not good.
Just to chime in with another anecdote, I've got several MikroTik RB493G's acting as PPPoE access concentrators that are stable as hell and have been in service for a couple of years with no issues.
In general, I've had better luck with the lower end of their product line.
I haven't pushed it much, as RouterOS is very powerful but has rather a steep learning curve. I've never had to reboot it in the five months I've had it.
I have been running Linux boxes for 20 years as my home router, but just recently bought a Cisco RV325. Sort of got tired of maintaining it, and it took allot more power.
How will these smaller, embedded motherboards handle 1G Ethernet? Will be getting google fiber within next year.
Been using an APU board since they came out and I have a 1Gbps fibre connection. Unfortunately my measured speed comes to about 500/700Mbit but I belive that's due to either shitty equipment in the city wide fibre grid or my own switches/cables. I'm not really a network tech.
Either way the APU handles it fine for a home network and generates no noticeable heat.
I guess most people, when they hear the word Router, they think of their DSL-box. And those commonly have a WiFi access-point built in. (commonly also called CPE (customer premises equipment) by the telephone companies).
$ ip route
default via x.x.x.1 dev eno1 proto dhcp src x.x.x.x metric 1024
x.x.x.x/nn dev eno0 proto kernel scope link src x.x.x.x
192.168.2.0/24 dev br0 proto kernel scope link src 192.168.2.114
192.168.2.1 dev br0 proto dhcp scope link src 192.168.2.114 metric 1024
$ ip link
(...)
2: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> (...)
Network guys, when hearing the word Router think of something that holds a full BGP view...
BGP router identifier 213.200.87.253, local AS number 65534
BGP table version is 7863026, main routing table version 7863026
578096 network entries using 58387696 bytes of memory
578096 path entries using 27748608 bytes of memory
344077 BGP path attribute entries using 20645160 bytes of memory
129723 BGP AS-PATH entries using 3908750 bytes of memory
1054 BGP community entries using 58126 bytes of memory
4 BGP extended community entries using 96 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 110748436 total bytes of memory
Dampening enabled. 145 history paths, 277 dampened paths
BGP activity 730922/149406 prefixes, 731552/150036 paths, scan interval 60 secs
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
213.200.64.93 4 3257 2823796 74946 7863002 0 0 3w5d 577951
Woodworkers, when hearing the word Router think of something that can effortlessly carve a groove into wood.
Well, all of them. A router in computing parlance is just any device that forwards packets between networks. So the first two examples are both routers.
Old network guys remember when networks were not all IP, and there were Novell IPX routers, and AppleTalk routers, and DECnet Phase IV routers, and CLNP lived on for a while due to phone carriers and IS-IS. And even in IP, BGP is only the latest in a long, sad story of routing protocols, and other protocols are usually used inside networks… including IS-IS.
Oops. While I know that wifi access points and routers can be separate, I'm used to people conversationally combining the two as "routers." Sorry about that!
Not only can they be separate, they should be separate.
Think of where you want to place your router ? (Usually in your server closet or close to where the fiber enters your house) Now think of where you want your access-point to be. Preferably somewhere central in your house, where it will give you the best coverage.
So the best place to put an AP is usually a shitty place to put a router and vice-versa. They have no business being in the same box.
realistically, what is the best option for adding wireless networking to this or any other setup based on a generic box? i assume one can build an access point with a typical wifi dongle but i am not aware of any of the software means required.
Realistically, WiFi driver code sucks, and there is inherent tension between routing, which benefits from wires, and WiFi, which benefits from open space. See, for example, how the Google OnHubs have the minimum of just 1 LAN port.
You can add WiFi to a generic box, but the Ars Technica staff are promoting a consensus that it’s better to use a separate access point that is designed to be good as an access point.
Yep. Just install the samba/nfs/iscsi/other networking daemon and off you go. About the only thing to keep in mind is that you are going to need to configure the daemon before you use it to ensure that it's not listening on your WAN address, because you probably don't wanna have your files visible to all and sundry.
If you like that, check out the Nexx WT3020H. Very similar specs but you can get them from China for about $13 USD.
Best of all, they're based around a MediaTek CPU, which doesn't have the same USB quirks as the Atheros AR9330 used in the GL-iNet.
I've personally upgraded my 3020H units from 8MB SPI to 16MB, but I've also heard that you can order them directly from the factory with 16MB if your order is large enough, or they're willing to customize.
ip_conntrack_tcp_be_liberal is still in kernel sources and it is enabled in distributions like openwrt. The author does not mention this, so very likely that his/her custom router will drop traffic. Recently I ran into the issue with Netflix traffic which seemed to use window scaling. In my case I did not disable scaling, but had to enable this option on arch Linux.
It is almost 4 years since World IPv6 Launch. I’m very disappointed that, other than a few randomly timed rants from Iljitsch van Beijnum, Ars Technica has made no visible movement to IPv6. No AAAA record for Arstechnica.com, no guides to installing IPv6, and now a tutorial for setting up routers spreading FUD about how difficult it is to install IPv6.
88 comments
[ 3.2 ms ] story [ 157 ms ] threadI do not run extra services like Squid or Snort, so most of my "writes" are probably within my massive 1GB of RAM.
[1]https://omnia.turris.cz/en/
I like those automatic security updates and network traffic analysis...
http://www.openbsd.org/faq/pf/example1.html
As a point of comparison, the iptables syntax as shown in the Ars article is far harder to grok at first glance than either of the pf examples. For example:
vs:It also sounds like it's getting replaced[0]. My guess is that we will see iptables around for a long time after it's been deprecated. ifconfig, for example, is deprecated[1] yet it's still around and being used.
[0] https://lwn.net/Articles/564095/ [1] https://lists.debian.org/debian-devel/2009/03/msg00780.html
See also dladm in illumos as another variation on a theme.
For example, `ifconfig eth0 down` becomes `ip link set dev eth0 down` and `ifconfig eth0 192.168.1.2` becomes `ip add add 192.168.1.2/24 brd + dev eth0`; I had to look these up to make sure I had the correct syntax.
Okay, the actual command is `ip address add` and not `ip add add`, but it allows extreme abbreviation at the expense of discoverability.
Generally speaking, that means that there's one canonical way to do something instead of a bunch of different ways like in Linux (think 'ifconfig' vs 'ip'). Of course, FreeBSD ships with three different firewalls[0], so that's not always true.
There are downsides, of course. A freshly-installed BSD has a lot less stuff than a freshly-installed OpenSUSE/Ubuntu/Fedora/etc.
[0] https://www.freebsd.org/doc/handbook/firewalls.html
PFSense is the same thing below the hood, but with a web front-end and plugins.
Most off the shelf wireless routers work fine as an access point, but are quite bad as a router. So you can just plug your old wireless router into this thing (with DHCP etc turned off), and your whole setup will be much better.
People are running FreeBSD and Linux on it:
http://www.daemonology.net/blog/2016-01-10-FreeBSD-EdgeRoute...
https://wiki.gentoo.org/wiki/MIPS/ERLite-3
With the electricity prices the way they are in the US, no way I'd run something like that over a proper x86 as in the article that can saturate a 1Gbps and manage decent disk IO.
"x86" covers a lot of ground. Certainly, in the "about $100" range, your choices in X86 land can't saturate 1Gbps either. The x86 box in the article was $280 total. About the same cost as the logical upgrade to the EdgeRouter Lite...their ER-8, which can saturate 1Gbps, and has 8 ports.
""" ...and whether you want automatic security upgrades. (Spoiler: Yes, you do.) """
I ended up going with the APU2B4 board (an upgrade from the APU1D mentioned in the article.) I put pfSense on it, and it's been running perfect for a few weeks now.
Even that board is probably massive overkill for most people. I have 50/50 internet, and with full bandwidth used by torrents, a VPN and ssh session open to the router, and the web interface open, I'm still only getting about 10-15% CPU.
http://pcengines.ch/apu2b4.htm
The APU boards are nice too; I thought about going for one when I was shopping for a better router and slapping OpenBSD on it. Ultimately I went with an Ubiquiti ERL, mostly because I didn't really want to buy an RS-232 cable, but the PC Engines boards are probably one of the best fully-DIY options you can get.
These Shenzhen factories are somehow getting these Intel CPUs for next to nothing. Factory price for the i5 model was about $100.
APU2C4 http://pcengines.ch/apu2c4.htm
case1d2blku http://pcengines.ch/case1d2blku.htm (black cases run cooler)
http://pcengines.ch/ac12vus2.htm (power plug)
db9cab1 http://pcengines.ch/db9cab1.htm (serial null modem)
I already had a USB->Serial adapter. For storage I used a 32GB Class 10 microSD card in a SD adapter, which I found acceptably fast (and faster than my usb sticks.) Note, if you want to run Snort or Squid, you might want a small SSD, but for a router-only it is overkill.
I assembled the board+case, wrote the 4gb nano-bsd image of pfSense to the SD card, and booted right up.
Total cost was < $200 USD.
edit: I noticed you mentioned wireless...I've heard the wireless support in pfSense using PC Engines boards is better than it was, but I prefer to use (multiple) separate APs for wifi. One of the APs I'm using is: http://www.amazon.com/TP-LINK-TL-WA801ND-Wireless-300Mbps-Re..., which I've had no issues with. I prefer the ability to upgrade/replace my APs without messing with the router, and my router is in the basement, while the APs are on other floors.
I'll probably move to 5GHz wireless later this year, and this separation allows that.
I believe the syntax for writing QoS on FreeBSD and OpenBSD provides very good expressive capability [1]. By using tagging [2], one can assign QoS priority to anything that a firewall rule can define.
Having used FreeBSD QoS on dial-up, ISDN, DSL and cable over the years, it is this expressiveness that is one of the reasons I prefer the pf packet filter and thus BSD.
Here's an example for bandwidth limited wan. Interactive ssh sessions get a queue with a minimum bandwidth; scp and sftp bulk transfers go to a separate queue.
[1] PF - Packet Queueing and Prioritization http://www.openbsd.org/faq/pf/queueing.html[2] PF - Packet Tagging (Policy Filtering) http://www.openbsd.org/faq/pf/tagging.html
I just wish tc had an interface understandable by your average sysadmin. I have to dig into the source to figure out what anything does, but not many users will have the ability/desire to do that.
A hierarchy of dumb FIFO queues does not make a real QoS system. It can produce reasonable-looking benchmark numbers when the prioritization rules and benchmark are contrived to match, but in the face of real-world traffic that uses HTTP to carry vastly different kinds of traffic over links that don't have constant bandwidth and latency, OpenBSD is hopeless. Even the rate-limiting capability that OpenBSD has is rudimentary and lacks the ability to account for per-packet overhead and framing overhead, which is necessary for accurate traffic shaping on common service types like ADSL.
If you say you don't see any disadvantages for OpenBSD on QoS, then your idea of QoS is twenty years out of date.
You're right, OpenBSD no longer has ECN, and my comments regarding OpenBSD were out of date.
And yes, ECN is far more effective because it provides feedback to the sender.
All of the BSDs are still way behind; the best by only a few years, while others are stuck in the 1990s.
[0] http://www.irongeek.com/i.php?page=videos/houseccon2015/t302...
My first experience with MikroTik products. Not good.
In general, I've had better luck with the lower end of their product line.
MikroTik - CRS125-24G-1S-RM
http://www.amazon.com/MikroTik-CRS125-24G-1S-RM-rackmount-en...
I haven't pushed it much, as RouterOS is very powerful but has rather a steep learning curve. I've never had to reboot it in the five months I've had it.
How will these smaller, embedded motherboards handle 1G Ethernet? Will be getting google fiber within next year.
Either way the APU handles it fine for a home network and generates no noticeable heat.
So, who's right?
Think of where you want to place your router ? (Usually in your server closet or close to where the fiber enters your house) Now think of where you want your access-point to be. Preferably somewhere central in your house, where it will give you the best coverage.
So the best place to put an AP is usually a shitty place to put a router and vice-versa. They have no business being in the same box.
You can add WiFi to a generic box, but the Ars Technica staff are promoting a consensus that it’s better to use a separate access point that is designed to be good as an access point.
http://arstechnica.com/gadgets/2015/10/review-ubiquiti-unifi...
I've got a about 50 deployed, managing them with Ansible, super nice and cheap. USB powered as well.
Best of all, they're based around a MediaTek CPU, which doesn't have the same USB quirks as the Atheros AR9330 used in the GL-iNet.
I've personally upgraded my 3020H units from 8MB SPI to 16MB, but I've also heard that you can order them directly from the factory with 16MB if your order is large enough, or they're willing to customize.