Running out of state table space is a very common issue for firewalls subjected to a DoS or other unprecedented high traffic situation. The only thing OpenBSD-specific here is probably the very conservative default table size.
For DNS servers that are set up with rate limits, keeping state on the firewall is pointless, so it's probably best to skip that (on OpenBSD: `pass no state`, I think) -- the destination server itself knows best what's abusive traffic and what's not, so on the firewall, do nothing except count the packets...
In my opinion Linux and eBPF is considerably better and more flexible when it comes to firewalling and deep packet inspection based blocking.
The mentioned use case perfectly fits the eBPF XDP module's feature set, and could easily be mitigated with the LPM trie offered by the kernel.
I wish that in those regards BSD maintainers would start to cooperate more with Linux, because everybody benefits from cross-kernel compatibility when it comes to eBPF.
I always admired the BSD maintainers, back in the days they were very innovative but I always have the feeling that at some point when their innovations take off they just stop following up with it due to either very opinionated views or lack of interest. Especially with the ongoing eBPF community.
It's not a mono culture but the idea of true cross-platform APIs that could make supporting BSD kernels much much easier in both terms of development time and interoperability.
> I would prefer that we avoid monocultures.
Where were you to say "Stop the C and POSIX monoculture", because that's a pretty opinionated view on how to build kernels, too, you know.
I'd prefer interoperability over having to implement everything myself all the time, any day.
People do regularly point out that POSIX limits exploration of interesting ideas in computing, and a segment of the community is actively attempting to deprecate C. I think it's a continuum; I personally think being unix-like is a useful baseline without being too limiting, but the cost/benefit rapidly degrades after that. As a topical example, I'm glad that everything didn't adopt iptables, because pf seems to be more friendly to work with. I don't mind ebpf, but I wouldn't be surprised to see one of the BSDs leapfrog it.
My point of view is always a little too much security focussed when it comes to kernel mod dev. I completely agree with embracing Rust and other memory safe languages. Hence why I think eBPF has huge potential due to their CORE abstractions that are very strict in terms of memory management and validation.
But when the source code of the NSA's ebpf malware was leaked, I got concerned because currently there is no failsafe way to audit a running kernel.
I think that the logical conclusion to this has to be something like a manifest file for each running kernel module, and a better feature-based sandbox for each module that is running or hooked (e.g. usb, network, filesystem, home folder etc). Similar to how Android's sandboxes are designed, but with a concept that is able to sandbox kernel mods, too.
Seccomp sandboxes are not enough, and there is gonna be a decade of new ebpf malware ahead of us.
I wish that rust got more traction, because that would at least solve all the pointer corruptions and overflows that can/are going wrong in module codebases.
I disagree on the firewalling front. I mean sure eBPF and iptables likely scales well but for home or small business users until there’s a simple declarative syntax like PF that abstracts the need for small C programs to run as eBPF requires it won’t gain adoption or be used or advocated by many.
I’ve never had to deep packet inspect so I’m not even sure there. The article just mentions that if state tables are exhausted packets start dropping. This is the case in every switch, too. It’s a common thing. Being able to increase this is a nice stop gap. It’s not a fatal flaw wrt to PF.
Also for better or worse BSD being different is what makes it interesting. BSD devs and hackers are a fraction of what the Linux kernel and greater ecosystem have in terms of the all of K8s and other Linux foundation projects. But BSDs have been innovating within their community of hackers for years and they don’t need to just be a derivative of Linux.
That being said the concepts of eBPF could be ported and the interface adopted could be interesting. If anyone pushed or funded that effort it might be the Netflix folks.
>Also for better or worse BSD being different is what makes it interesting. BSD devs and hackers are a fraction of what the Linux kernel and greater ecosystem have in terms of the all of K8s and other Linux foundation projects. But BSDs have been innovating within their community of hackers for years and they don’t need to just be a derivative of Linux.
I disagree, you can still be unique while having interoperability with the mainstream.
Just take a look at netbsd, sure it's all about ofc it runs netbsd, but it's hidden feature is trying to be interoperable.
tl;dr: I think eBPF is great and that there are many things to criticize about the BSDs, but I think the claims about innovations taking off and BSD developers not cooperating are baseless.
I'm sorry. There's much to criticize the BSDs for, but compared to Linux a lack of innovation simply isn't among them.
From containers to capabilities, from networking to storage, from event handling to audio (and I am talking about the kernel side of things here) Linux runs in circles, re-inventing half-baked solutions that becomes yet another deprecated, yet still supported API/syscall a few years in.
Also on the note of corporation. There has been more than one event where corporation was the intention and some Linux guy posted on some public mailing list about the OpenBSD developers will be sued.
And when it comes to user space the situation is even more clear.
But then again one has to admit that there are philosophical clashes, especially when it comes to OpenBSD.
If you look at FreeBSD there's a lot more interaction. See OpenZFS. Here FreeBSD decided to corporate. At the same time, even though that is user space, Wayland and other projects are defined as "for Linux".
I believe in general there's a lot more cooperation than you think. It's just that devs aren't really much into making everything some public ceremony. If you share a common fields, you might go to common events or send an email, jump into an IRC, Discord, etc. Especially given the history regarding licenses. People - on both sides (if you can even call it that) - now behave properly and talk things out directly instead of having mud fights.
And if you look at the OpenBSDs Innovation page on their website, compare the list of Programs and subsystems as well as the list of Projects maintained by OpenBSD developers with what people think is great software or with software that's on your system and you don't even think about cause it never ever made problems. And then you have things like sndio which I've seen shipped with more than one game on Steam, because it just works and everyone doing proper updates (Android, RPM, etc.) using bsdiff (granted, that one is in fact older).
The main thing where the BSDs sometimes lag behind (with exceptions such as Apple's ARM processors, etc.) is supporting each and every device of hardware. However, one has to say that your Thinkpad has a better chance of working properly out of the box than any Linux distribution I ever encountered. And while that worked decades ago, I currently don't manage to have any system properly handling audio input/output states. It either switches around in terms of muting, or it forgets that the headphones were connected, outputs to the wrong place, etc. To be fair that also does seem a problem with some other OSs, but on OpenBSD it's always correct. I mention this because it's one example of hardware actually working, and not being wonky. And I don't have to constantly be scared that the next driver version breaks stuff, or that an API is deprecated. Yes, Linux is absolutely excellent on keeping compatibility, but the sad state is that you'll end up with 10 APIs with different parts not working properly.
And don't get me started with epoll vs kqueue. It's one great example of what I mean by half-baked solutions. Or think about how hacky setuid for processes on multi-threaded processes on Linux is. Setting owners of a thread instead of of the whole process? Every other OS gets that one right, but on Linux even big projects like Go struggle:
Btw: I say that at someone who is a huge fan of eBPF, even though I have to admit that it really is a double-edged sword. It's not one of these "this is at least better than not having it in all situations" things, even though some people sadly act like it. And on...
Your main critic doesn't really hold much water since distros and the kernel is pretty much the only "designers" of Linux.
Wayland is part of freedesktop.org for instance.
Systemd would be an exception I would admit, but there doesn't seem to be much interest from the BSD to port or attempt to use the systemd fork or openrc.
OpenBSD does have interesting projects, but why would a game needs their own audio server.
Not only that but openbsd isn't exactly releasing big new innovations a cross the board, but usually small security focused or KISS versions of something.
Stability is also what you make of it, while it's true that BSD probably have a higher uptime than say fedora rawhide, it's selection of software and not breaking them is a whole other thing.
On the other hand, pf is cross-platform, running in Solaris and FreeBSD. That carries some weight.
I have seen the Linux kernel migrate from ipfwadm to ipchains to iptables to nftables, and this has involved occasional reworking of some simple firewall rules from time to time.
pf may be on all those platforms, but pf rules are not cross-platform, because different platforms have different rule syntax and different capabilities.
In general, when you get DDOSed (not neccesarilu DOSed though,you would want to route around the firewalls because they process and inspect every packet/connection their rps/pps capacity makes them a bottleneck.
The effective approach for DDOS mitigation is to anycast the attacked subnet to diffuse the attack and intercept it with firewall nodes at the edge which use ASIC/FPGA offload to filter the attack at line rate then forward the filtered stream back to its usual edge.
eBPF direct compilation to NPU instructions is already supported on Linux by some devices, which is one reason eBPF approaches have a lot of potential in this space.
Most people can't anycast, and even if they could I did not say that was even a mitigation strategy, you arguing a strawman.
"firewall nodes at the edge which use ASIC/FPGA offload" and most people have that? I can't even think of companies that aren't FAANG or CDN providers that have anycastable ASIC enabled firewalls that can handle high pps/rps.
> eBPF direct compilation to NPU instructions is already supported on Linux by some devices, which is one reason eBPF approaches have a lot of potential in this space.
Ok, I think maybe we are talking about the same thing but what you call "firewalls" are in-house built DDoS scrubbers, which is not what I am talking about. There are also commercial DDoS scrubbers people can use, with or without anycasting.
Back to the topic of this thread: firewalls, as in devices that inspect traffic using their CPU. PF based firewalls, sonicwall, fortigate, paloalto, sourcefire, those. You can build your own ddos scrubber compiling eBPF to NPUs and then afford those. Those are highly specialized purpose specific devices, they don't represent what people mean when they think of firewalls, as in ones you can buy or rent.
It comes down to capacity, they typically can handle throughout just fine but high number of active sessions and very high pps is where most start choking.
What's the point of using a stateful firewall in front of a server ?
The concept of stateful firewall was created to protect clients:
1. Clients initiate flow toward the dangerous wild Internet;
2. The stateful firewall check that returning traffic from dangerous-Internet belongs to a flow initiated by an inside-client.
=> So it consumes CPU and RAM (session tables) in exchange of protecting inside-clients from dangerous-Internet.
But using a stateful firewall in front of a server seems useless to me:
1. The inside-server behind the firewall MUST reply to all clients, so the stateful firewall must allow all Internet initiated-sessions to reach the server (no usage of a state table here).
2. When the inside-server replies to the outside-clients, the firewall will then consume CPU and RAM to check its state table. So it will consume resources to guarantee that our inside-serve is correctly replying to the outside-clients.. which is pointless.
This could be useful in rare case when you have a very big firewall able to limit the number of sessions toward your small inside-server.
But in case of a DNS server, that are natively able to generate a large number of packets-per-second UDP traffic, this seems to waste firewall CPU and RAM for this useless state table.
> 1. The inside-server behind the firewall MUST reply to all clients, so the stateful firewall must allow all Internet initiated-sessions to reach the server (no usage of a state table here).
I don't understand this requirement, (more specifically the second half of it).
Isn't a firewall literally a device which allows in some connections, rather than all connections?
(And state plays a role in letting the firewall remember which external clients are acting DOSy?)
By my reading the rate-limiting case was covered as the exception in a latter part of the comment ("This could be useful in rare case when you have a very big firewall able to limit the number of sessions toward your small inside-server.")
But in the usual case you'll block connection attempts to the wrong port statelessly, and there's no benefit in tracking the served sessions in an external firewall.
I'm not in favor of doing it, but you pick up some benefits in some cases.
For TCP, if your server doesn't deal well with bogus packets, having a stateful firewall allows dropping data packets without an associated connection. Stateful firewalls may have syncookies where your server os doesn't, in case you're running 20 year old servers or macos servers.
For UDP servers, it could be potentially useful if the rate limiting in the firewall is more flexible than in the server application. But for DNS, usually you're looking at one request -> one response, I don't know why you would want a state for that.
Of course, it is useful to mix stateful and stateless: allow all the service related traffic without state, but allow for incidental outbound traffic with state. Drop everything else. But that's really more about having a clear access control layer than protecting a host from network abuse.
Tangential, but this reminds me of a problem I've had with pf tables. But for other reasons, and with FreeBSD.
On a FreeBSD server, using pf, I wanted to open one port to only IP addresses from two specific countries[1], and disallow from everywhere else. Everything else is behind a VPN, but I'm managing this server for other people, and, for some reason, a WireGuard VPN is a no (this would have instantly solved the problem).
So then, why a whole country? Because searching for a way to auto-allow their specific /32 public IP addresses each time they changed sounded like a PITA. So, rejected.
So I went and created pf tables for allowing whole-country address ranges, this was the easy part. But then I couldn't get FreeBSD to stop complaining about the tables being too large, even after tweaking some sysctl settings and stuff.
I don't remember the actual size, but at the very least the plaintext file that contained one subnet per line, could fit in my 64GB RAM with a lot of room to spare.
I just gave up for now and told them to use SSH forwarding on a bastion server, which is an option they accept.
[1]: Yes, I know, this isn't bulletproof. I just want to reduce noise from spray-and-pray bs on my SSH and HTTPS logs.
But still, learning nftables has been on my backlog for some time already, so I appreciate it being mentioned. If anything I should try doing this on Linux with nftables and see how it goes, since it sounds like a good way to learn.
As a general rule it’s a mistake to put a stateful firewall in front of a DNS server (at least for UDP). Sadly it’s one of those nuggets of operational knowledge that keeps being rediscovered the hard way.
pf is on by default in OpenBSD, which is definitely the wrong thing to do on simple servers. I typically disable it and bring the system down to minimal services instead.
OTOH, pf can handle very large state tables with ease. I also use a cluster of pf boxes to front-end a network with 20000+ users and it rides around 500000 to 800000 states every day.
Let’s say, for example, you make the mistake of putting a stateful firewall between your users and your recursive DNS servers. As a rough guide, you can expect about 0.1 DNS query per second per user, so about 2000 qps for your site (average, peaks will be higher). Firewalls often rely on a timeout for expiring UDP state, let’s say, 5 minutes or 300 seconds. In this example, the firewall will be keeping 600 000 useless states.
i just checked my main user-facing dns servers, and indeed i disabled pf on them
on my big-boi pf NAT cluster, i keep the udp timeout relatively low to deal with state table congestion:
set limit { states 1000000, src-nodes 100000 }
set state-defaults pflow, sloppy
set state-policy floating
set timeout src.track 240
set timeout udp.multiple 120
28 comments
[ 0.25 ms ] story [ 64.3 ms ] threadFor DNS servers that are set up with rate limits, keeping state on the firewall is pointless, so it's probably best to skip that (on OpenBSD: `pass no state`, I think) -- the destination server itself knows best what's abusive traffic and what's not, so on the firewall, do nothing except count the packets...
The mentioned use case perfectly fits the eBPF XDP module's feature set, and could easily be mitigated with the LPM trie offered by the kernel.
I wish that in those regards BSD maintainers would start to cooperate more with Linux, because everybody benefits from cross-kernel compatibility when it comes to eBPF.
I always admired the BSD maintainers, back in the days they were very innovative but I always have the feeling that at some point when their innovations take off they just stop following up with it due to either very opinionated views or lack of interest. Especially with the ongoing eBPF community.
> I would prefer that we avoid monocultures.
Where were you to say "Stop the C and POSIX monoculture", because that's a pretty opinionated view on how to build kernels, too, you know.
I'd prefer interoperability over having to implement everything myself all the time, any day.
But when the source code of the NSA's ebpf malware was leaked, I got concerned because currently there is no failsafe way to audit a running kernel.
I think that the logical conclusion to this has to be something like a manifest file for each running kernel module, and a better feature-based sandbox for each module that is running or hooked (e.g. usb, network, filesystem, home folder etc). Similar to how Android's sandboxes are designed, but with a concept that is able to sandbox kernel mods, too.
Seccomp sandboxes are not enough, and there is gonna be a decade of new ebpf malware ahead of us.
I wish that rust got more traction, because that would at least solve all the pointer corruptions and overflows that can/are going wrong in module codebases.
I’ve never had to deep packet inspect so I’m not even sure there. The article just mentions that if state tables are exhausted packets start dropping. This is the case in every switch, too. It’s a common thing. Being able to increase this is a nice stop gap. It’s not a fatal flaw wrt to PF.
Also for better or worse BSD being different is what makes it interesting. BSD devs and hackers are a fraction of what the Linux kernel and greater ecosystem have in terms of the all of K8s and other Linux foundation projects. But BSDs have been innovating within their community of hackers for years and they don’t need to just be a derivative of Linux.
That being said the concepts of eBPF could be ported and the interface adopted could be interesting. If anyone pushed or funded that effort it might be the Netflix folks.
I disagree, you can still be unique while having interoperability with the mainstream.
Just take a look at netbsd, sure it's all about ofc it runs netbsd, but it's hidden feature is trying to be interoperable.
I'm sorry. There's much to criticize the BSDs for, but compared to Linux a lack of innovation simply isn't among them.
From containers to capabilities, from networking to storage, from event handling to audio (and I am talking about the kernel side of things here) Linux runs in circles, re-inventing half-baked solutions that becomes yet another deprecated, yet still supported API/syscall a few years in.
Also on the note of corporation. There has been more than one event where corporation was the intention and some Linux guy posted on some public mailing list about the OpenBSD developers will be sued.
And when it comes to user space the situation is even more clear.
But then again one has to admit that there are philosophical clashes, especially when it comes to OpenBSD.
If you look at FreeBSD there's a lot more interaction. See OpenZFS. Here FreeBSD decided to corporate. At the same time, even though that is user space, Wayland and other projects are defined as "for Linux".
I believe in general there's a lot more cooperation than you think. It's just that devs aren't really much into making everything some public ceremony. If you share a common fields, you might go to common events or send an email, jump into an IRC, Discord, etc. Especially given the history regarding licenses. People - on both sides (if you can even call it that) - now behave properly and talk things out directly instead of having mud fights.
And if you look at the OpenBSDs Innovation page on their website, compare the list of Programs and subsystems as well as the list of Projects maintained by OpenBSD developers with what people think is great software or with software that's on your system and you don't even think about cause it never ever made problems. And then you have things like sndio which I've seen shipped with more than one game on Steam, because it just works and everyone doing proper updates (Android, RPM, etc.) using bsdiff (granted, that one is in fact older).
The main thing where the BSDs sometimes lag behind (with exceptions such as Apple's ARM processors, etc.) is supporting each and every device of hardware. However, one has to say that your Thinkpad has a better chance of working properly out of the box than any Linux distribution I ever encountered. And while that worked decades ago, I currently don't manage to have any system properly handling audio input/output states. It either switches around in terms of muting, or it forgets that the headphones were connected, outputs to the wrong place, etc. To be fair that also does seem a problem with some other OSs, but on OpenBSD it's always correct. I mention this because it's one example of hardware actually working, and not being wonky. And I don't have to constantly be scared that the next driver version breaks stuff, or that an API is deprecated. Yes, Linux is absolutely excellent on keeping compatibility, but the sad state is that you'll end up with 10 APIs with different parts not working properly.
And don't get me started with epoll vs kqueue. It's one great example of what I mean by half-baked solutions. Or think about how hacky setuid for processes on multi-threaded processes on Linux is. Setting owners of a thread instead of of the whole process? Every other OS gets that one right, but on Linux even big projects like Go struggle:
https://github.com/golang/go/issues/1435
Btw: I say that at someone who is a huge fan of eBPF, even though I have to admit that it really is a double-edged sword. It's not one of these "this is at least better than not having it in all situations" things, even though some people sadly act like it. And on...
Wayland is part of freedesktop.org for instance.
Systemd would be an exception I would admit, but there doesn't seem to be much interest from the BSD to port or attempt to use the systemd fork or openrc.
OpenBSD does have interesting projects, but why would a game needs their own audio server.
Not only that but openbsd isn't exactly releasing big new innovations a cross the board, but usually small security focused or KISS versions of something.
Stability is also what you make of it, while it's true that BSD probably have a higher uptime than say fedora rawhide, it's selection of software and not breaking them is a whole other thing.
I have seen the Linux kernel migrate from ipfwadm to ipchains to iptables to nftables, and this has involved occasional reworking of some simple firewall rules from time to time.
On those later occasions, I did admire pf.
The effective approach for DDOS mitigation is to anycast the attacked subnet to diffuse the attack and intercept it with firewall nodes at the edge which use ASIC/FPGA offload to filter the attack at line rate then forward the filtered stream back to its usual edge.
eBPF direct compilation to NPU instructions is already supported on Linux by some devices, which is one reason eBPF approaches have a lot of potential in this space.
"firewall nodes at the edge which use ASIC/FPGA offload" and most people have that? I can't even think of companies that aren't FAANG or CDN providers that have anycastable ASIC enabled firewalls that can handle high pps/rps.
> eBPF direct compilation to NPU instructions is already supported on Linux by some devices, which is one reason eBPF approaches have a lot of potential in this space.
Ok, I think maybe we are talking about the same thing but what you call "firewalls" are in-house built DDoS scrubbers, which is not what I am talking about. There are also commercial DDoS scrubbers people can use, with or without anycasting.
Back to the topic of this thread: firewalls, as in devices that inspect traffic using their CPU. PF based firewalls, sonicwall, fortigate, paloalto, sourcefire, those. You can build your own ddos scrubber compiling eBPF to NPUs and then afford those. Those are highly specialized purpose specific devices, they don't represent what people mean when they think of firewalls, as in ones you can buy or rent.
It comes down to capacity, they typically can handle throughout just fine but high number of active sessions and very high pps is where most start choking.
The concept of stateful firewall was created to protect clients:
1. Clients initiate flow toward the dangerous wild Internet;
2. The stateful firewall check that returning traffic from dangerous-Internet belongs to a flow initiated by an inside-client. => So it consumes CPU and RAM (session tables) in exchange of protecting inside-clients from dangerous-Internet.
But using a stateful firewall in front of a server seems useless to me:
1. The inside-server behind the firewall MUST reply to all clients, so the stateful firewall must allow all Internet initiated-sessions to reach the server (no usage of a state table here).
2. When the inside-server replies to the outside-clients, the firewall will then consume CPU and RAM to check its state table. So it will consume resources to guarantee that our inside-serve is correctly replying to the outside-clients.. which is pointless.
This could be useful in rare case when you have a very big firewall able to limit the number of sessions toward your small inside-server.
But in case of a DNS server, that are natively able to generate a large number of packets-per-second UDP traffic, this seems to waste firewall CPU and RAM for this useless state table.
I don't understand this requirement, (more specifically the second half of it).
Isn't a firewall literally a device which allows in some connections, rather than all connections?
(And state plays a role in letting the firewall remember which external clients are acting DOSy?)
By my reading the rate-limiting case was covered as the exception in a latter part of the comment ("This could be useful in rare case when you have a very big firewall able to limit the number of sessions toward your small inside-server.")
But in the usual case you'll block connection attempts to the wrong port statelessly, and there's no benefit in tracking the served sessions in an external firewall.
For TCP, if your server doesn't deal well with bogus packets, having a stateful firewall allows dropping data packets without an associated connection. Stateful firewalls may have syncookies where your server os doesn't, in case you're running 20 year old servers or macos servers.
For UDP servers, it could be potentially useful if the rate limiting in the firewall is more flexible than in the server application. But for DNS, usually you're looking at one request -> one response, I don't know why you would want a state for that.
Of course, it is useful to mix stateful and stateless: allow all the service related traffic without state, but allow for incidental outbound traffic with state. Drop everything else. But that's really more about having a clear access control layer than protecting a host from network abuse.
On a FreeBSD server, using pf, I wanted to open one port to only IP addresses from two specific countries[1], and disallow from everywhere else. Everything else is behind a VPN, but I'm managing this server for other people, and, for some reason, a WireGuard VPN is a no (this would have instantly solved the problem).
So then, why a whole country? Because searching for a way to auto-allow their specific /32 public IP addresses each time they changed sounded like a PITA. So, rejected.
So I went and created pf tables for allowing whole-country address ranges, this was the easy part. But then I couldn't get FreeBSD to stop complaining about the tables being too large, even after tweaking some sysctl settings and stuff.
I don't remember the actual size, but at the very least the plaintext file that contained one subnet per line, could fit in my 64GB RAM with a lot of room to spare.
I just gave up for now and told them to use SSH forwarding on a bastion server, which is an option they accept.
[1]: Yes, I know, this isn't bulletproof. I just want to reduce noise from spray-and-pray bs on my SSH and HTTPS logs.
* https://man.freebsd.org/cgi/man.cgi?query=pf.conf#TABLES
It was the size / scalability of the tables/sets that seem to be the issue, as opposed to the functionality of them.
But still, learning nftables has been on my backlog for some time already, so I appreciate it being mentioned. If anything I should try doing this on Linux with nftables and see how it goes, since it sounds like a good way to learn.
OTOH, pf can handle very large state tables with ease. I also use a cluster of pf boxes to front-end a network with 20000+ users and it rides around 500000 to 800000 states every day.
on my big-boi pf NAT cluster, i keep the udp timeout relatively low to deal with state table congestion:
set limit { states 1000000, src-nodes 100000 } set state-defaults pflow, sloppy set state-policy floating set timeout src.track 240 set timeout udp.multiple 120