Author here. There are more conntrack stories to tell...
In this one I tried to do a quick "what happens when table is full", and ended up with a rather rich and complex blog post. In it I discuss:
- EPERM
- implicitly dropped packets
- weird kernel behaviour on loose=0 and ACK being dropped
- iptable policy counters having their gotchas
- problem of syn floods and conntrac
and more.
It's not that this is super breathtaking, but modern deployments depend on conntrack (I'm looking at you k8s). Even though conntrack is notoriously under-documented and misunderstood. I hope that the linked scripts would allow folks to try to reproduce the weird conntrack behaviour and spread the knowledge of underdocumented corner cases.
The ack handling (loose) seem to have some undesired corner cases. Apart from that the problem with conntrack is not semantics but lack of understanding.
This blog post aims to narrow the gap and explain in clear language what you can and what you can not expect from the conntrack layer. Pros and cons.
Good intro. I can relate, I had to debug conntrack dropping packets and performance issues and it's not easy. Cherry on top, there aren't two distributions taking the same settings.
Also have a look into the conntrack hashsize. The conntrack connection table is a hashtable and the hashsize is how many buckets it has. This affect collisions, memory consumption and cache hits.
Then conntrack timeouts settings, that decide when connections can be released from the tables. Connections can be hold by default for minutes even after close. The most trivial performance test can cycle a million connections in a minute and put down any system using conntrack.
I wouldn't have guessed that cloudflare relied heavily on conntrack. It's not that scalable or easy to use IMO.
Conntrack evolved recently and is thought to be fast. Florian Westphal once said that an iptable rule doing -j NOTRACK is slower than hitting conntrack.
The rationale is that iptables need to do indirect jumps, which with spetre mitigations means retpoline which means slow. So... conntrack avoids the indirect jumps and is faster than a couple of indirect jumps that would be taken on the iptables exception path.
But I haven't done the benchmarks so I didn't want to put that in the post.
Anyhow, barring the SYN flood aspect, I think conntrack is fast enough for everybody. The only exceptions might be... dunno... HFT.
conntrack structures can be multi gigabytes when sized to handle a hundred thousand connections per minute and upward. It can cache miss on connection lookup with each new network packet.
What is slower between memory cache miss and indirect jump? I do not know, haven't done a benchmark either.
The conntrack(1) utility also has some fun brokenness. For example, there's a TOCTTOU issue when you flush connection state, as Calico and other k8s CNIs do regularly. The utility queries the kernel state, builds a list in user space, and then iteratively deletes each session. But if a session expires between when its reported and when the utility tries to delete it (for example, hitting the standard UDP session timeout, or a TCP FIN, which happens all the time), the delete fails with ENOENT and the utility immediately exits without flushing the rest of the state.
Calico and other controllers resolve this by calling conntrack(1) in a loop until success or a limit (e.g. 3, 5, or w'ever magic number) is reached. But even then you regularly get one more error than the limit, triggering alarms.
I submitted a patch last year that added a command-line switch for suppressing certain errors, but the netdev mailing-list is high traffic and I never got any feedback. I never expected my patch to be accepted on the first round; not even the approach. I can think of several other alternative ways to address the issue, such as not exiting on error until working through the queue, utilizing exit status codes for reporting the reason, etc. But it's unfortunate I got no feedback whatsoever considering it's a substantial pain point in the wild and a clear (IMO) defect in the implementation--because the error is inevitable and to be expected given the TOCTTOU race.
Of course, if I were writing a controller like Calico I probably wouldn't be shelling out to conntrack(1), but instead making the netlink calls in-process. But I'm not most people, especially in this particular case where everybody else manifestly treats conntrack(1) as the de facto programmatic interface.
I wish iptables had an option to use full-cone NAT (endpoint independent mapping + filtering per rfc5128) instead of always doing full cone (endpoint-dependent). It would dramatically cut down on the number of table entries needed, possibly down to 1 if you use source port binding.
10 comments
[ 0.23 ms ] story [ 17.3 ms ] threadIn this one I tried to do a quick "what happens when table is full", and ended up with a rather rich and complex blog post. In it I discuss:
- EPERM
- implicitly dropped packets
- weird kernel behaviour on loose=0 and ACK being dropped
- iptable policy counters having their gotchas
- problem of syn floods and conntrac
and more.
It's not that this is super breathtaking, but modern deployments depend on conntrack (I'm looking at you k8s). Even though conntrack is notoriously under-documented and misunderstood. I hope that the linked scripts would allow folks to try to reproduce the weird conntrack behaviour and spread the knowledge of underdocumented corner cases.
This blog post aims to narrow the gap and explain in clear language what you can and what you can not expect from the conntrack layer. Pros and cons.
Also have a look into the conntrack hashsize. The conntrack connection table is a hashtable and the hashsize is how many buckets it has. This affect collisions, memory consumption and cache hits.
Then conntrack timeouts settings, that decide when connections can be released from the tables. Connections can be hold by default for minutes even after close. The most trivial performance test can cycle a million connections in a minute and put down any system using conntrack.
I wouldn't have guessed that cloudflare relied heavily on conntrack. It's not that scalable or easy to use IMO.
The rationale is that iptables need to do indirect jumps, which with spetre mitigations means retpoline which means slow. So... conntrack avoids the indirect jumps and is faster than a couple of indirect jumps that would be taken on the iptables exception path.
But I haven't done the benchmarks so I didn't want to put that in the post.
Anyhow, barring the SYN flood aspect, I think conntrack is fast enough for everybody. The only exceptions might be... dunno... HFT.
What is slower between memory cache miss and indirect jump? I do not know, haven't done a benchmark either.
Calico and other controllers resolve this by calling conntrack(1) in a loop until success or a limit (e.g. 3, 5, or w'ever magic number) is reached. But even then you regularly get one more error than the limit, triggering alarms.
I submitted a patch last year that added a command-line switch for suppressing certain errors, but the netdev mailing-list is high traffic and I never got any feedback. I never expected my patch to be accepted on the first round; not even the approach. I can think of several other alternative ways to address the issue, such as not exiting on error until working through the queue, utilizing exit status codes for reporting the reason, etc. But it's unfortunate I got no feedback whatsoever considering it's a substantial pain point in the wild and a clear (IMO) defect in the implementation--because the error is inevitable and to be expected given the TOCTTOU race.
Of course, if I were writing a controller like Calico I probably wouldn't be shelling out to conntrack(1), but instead making the netlink calls in-process. But I'm not most people, especially in this particular case where everybody else manifestly treats conntrack(1) as the de facto programmatic interface.
http://vger.kernel.org/vger-lists.html#netfilter-devel