XDP (eXpress Data Path) is the fastest packet processing framework in linux - but it only works for incoming (ingress) traffic. We discovered how to use it for outgoing (egress) traffic by exploiting a loophole in how the linux kernel determines packet direction. Our technique delivers 10x better performance than current solutions, works with existing Docker/Kubernetes containers, and requires zero kernel modifications.
This post not only expands on the overall implementation but also outlines how existing container and VM workloads can immediately take advantage with minimal effort and zero infrastructure changes.
Forgive me of my ignorance, but is XDP faster than DPDK for packet processing? It seems like DPDK has had a lot of work done for hardware optimizations that allow speeds that I can’t recall XDP being able to do. I have not looked too deeply into this though, so I’m very open to being wrong!
I come from a very different world (optimizing the FreeBSD kernel for the Netflix CDN, running on bare metal) but performance leaps like this are fascinating to me.
One of the things that struck me when reading this with only general knowledge of the linux kernel is: What makes things so terrible? Is iptables really that bad? Is something serialized to a single core somewhere in the other 3 scenarios? Is the CPU at 100% in all cases? Is this TCP or UDP traffic? How many threads is iperf using? It would be cool to see the CPU utilization of all 4 scenarios, along with CPU flamegraphs.
Similar fun as the time I discovered one could use IFB to set qdiscs on incoming traffic (why would one would do that is left as exercise to the reader, but my journey included using the 'plug' qdisc and tcp-checkpoint/restore). The Linux kernel has so many building blocks....
They say "By the time a packet reaches the TC hook, the kernel has already processed it through various subsystems for routing, firewalling, and even connection tracking." but surely this is also true before it reaches the VETH?
Really good, and glad that you're taking this technique further into a docker network plugin. I wouldn't be surprised to see a Kubernetes CNI appear using this approach, seems entirely viable unless I am missing something.
I'll definitely be coming to check you all out at Kubecon.
I think the title is a little disingenuous and the idea of using a redirect is certainly not novel. The solution for XDP egress should be able to handle all host egress including sr-iov traffic. This works with a very specific namespace driven topology.
i really love this one. its a really elegant and well informed solution. one of the nicest finds ive seen in a while was a pleasure reading how it works! thanks a lot
XDP, and the eBPF ecosystem in general, is quite neat. However, a word of caution:
* The BPF verifier's DX is not great yet. If it finds problems with your BPF code it will spit our a rather inscrutable set of error messages that often requires a good understanding of the verifier internals (e.g the register nomenclature) to debug
* For the same source code, the code generated by the verifier can change across compiler versions in a breaking way, e.g. because the new compiler version implemented an optimization that broke the verifier (see https://github.com/iovisor/bcc/issues/4612)
* Checksum updating requires extra care. I believe you can only do incremental updates, not just because of better perf as the post suggests but also because the verifier does not allow BPF programs to operate on unbounded buffers (so checksumming a whole packet of unknown size is tricky / cumbersome). This mostly works but you have to be careful with packets that were generated with csum offload, don't have a valid checksum and whose csum can't be incrementally updated.
As the blog post points out, the kernel networking stack does a lot of work that we don't generally think about. Once you start taking things into your own hands you don't have the luxury of ignorance anymore (think not just ARP but also MTU, routing, RP filtering etc.), something any user of userspace networking frameworks like DPDK will tell you.
My general recommendation is to stick with the kernel unless you have a very good justification for chasing better performance and if you do use eBPF save yourself some trouble and try to limit yourself to readonly operations, if your use case allows.
Also, if you are trying to debug packet drops, newer kernels have started logging this information that you can track using bpftrace which gives you better diagnostics.
Example script (might have to adjust based on kernel version):
This post hits close to home, I've run into all of these myself.
On checksums: Incremental updates are the path of least pain only if the packet’s checksum is valid and not CHECKSUM_PARTIAL. With modern offloads (TSO/GSO/GRO/checksum offload), the checksum visible to XDP is often zero/garbage because the NIC fills it later. In practice, either disable offloads for that traffic or recompute from scratch with bpf_csum_diff() plus bpf_l3_csum_replace() / bpf_l4_csum_replace().
The verifier: This is a fun one, when you make a small change and suddenly the verifier won't allow it.
And the moment you start modifying packets too much yourself, you're on the hook for everything the kernel used to do for you.
I once went down the rabbit hole of building a minimal TCP stack, and the experience was exactly as you'd expect. Getting to 95% done felt quick, but that last 5% was a nightmare (if 100% is even achievable)
21G on tc egress is slightly surprising to me. I'd like to see the program used for the benchmark. Was GSO accounted for? If you pop/pull headers by hand, you'll often kill GSO which will result in a massive loss in throughput like this.
28 comments
[ 3.2 ms ] story [ 50.4 ms ] threadThis post not only expands on the overall implementation but also outlines how existing container and VM workloads can immediately take advantage with minimal effort and zero infrastructure changes.
> XDP for Egress Traffic Flow Diagram
1. Are packages still raw application data when they get to veth0-A without going through the tcp-stack?
2. In container to container communication case, do you plan redirect packages directly from veth0-A to veth0-B
One of the things that struck me when reading this with only general knowledge of the linux kernel is: What makes things so terrible? Is iptables really that bad? Is something serialized to a single core somewhere in the other 3 scenarios? Is the CPU at 100% in all cases? Is this TCP or UDP traffic? How many threads is iperf using? It would be cool to see the CPU utilization of all 4 scenarios, along with CPU flamegraphs.
You can also use XDP for outgoing packets for tap interfaces.
In some scenarios veth is being replaced with netkit for a similar reason. Does this impact how you're going to manage this?
Why doesn’t checksum offload in the NIC take care of that?
I'll definitely be coming to check you all out at Kubecon.
* The BPF verifier's DX is not great yet. If it finds problems with your BPF code it will spit our a rather inscrutable set of error messages that often requires a good understanding of the verifier internals (e.g the register nomenclature) to debug
* For the same source code, the code generated by the verifier can change across compiler versions in a breaking way, e.g. because the new compiler version implemented an optimization that broke the verifier (see https://github.com/iovisor/bcc/issues/4612)
* Checksum updating requires extra care. I believe you can only do incremental updates, not just because of better perf as the post suggests but also because the verifier does not allow BPF programs to operate on unbounded buffers (so checksumming a whole packet of unknown size is tricky / cumbersome). This mostly works but you have to be careful with packets that were generated with csum offload, don't have a valid checksum and whose csum can't be incrementally updated.
As the blog post points out, the kernel networking stack does a lot of work that we don't generally think about. Once you start taking things into your own hands you don't have the luxury of ignorance anymore (think not just ARP but also MTU, routing, RP filtering etc.), something any user of userspace networking frameworks like DPDK will tell you.
My general recommendation is to stick with the kernel unless you have a very good justification for chasing better performance and if you do use eBPF save yourself some trouble and try to limit yourself to readonly operations, if your use case allows.
Also, if you are trying to debug packet drops, newer kernels have started logging this information that you can track using bpftrace which gives you better diagnostics.
Example script (might have to adjust based on kernel version):
On checksums: Incremental updates are the path of least pain only if the packet’s checksum is valid and not CHECKSUM_PARTIAL. With modern offloads (TSO/GSO/GRO/checksum offload), the checksum visible to XDP is often zero/garbage because the NIC fills it later. In practice, either disable offloads for that traffic or recompute from scratch with bpf_csum_diff() plus bpf_l3_csum_replace() / bpf_l4_csum_replace().
The verifier: This is a fun one, when you make a small change and suddenly the verifier won't allow it.
And the moment you start modifying packets too much yourself, you're on the hook for everything the kernel used to do for you.
I once went down the rabbit hole of building a minimal TCP stack, and the experience was exactly as you'd expect. Getting to 95% done felt quick, but that last 5% was a nightmare (if 100% is even achievable)
I'm using Firefox
Video: https://cs.joshstrange.com/Zhxk4kRp
We use it quite a ton for capturing and dashboarding inbound network traffic over at https://yeet.cx
I am really excited for the future of eBPF especially with tcx now being available in Debian 13. The tc API was very hard to work with.
What I don’t really understand is why iptables and tv is so slow.
If the kernel can’t route packets at line speed, how are userspace applications saturating it?