I can see why they rewrote QUIC in Rust and for use in userspace, though going the in-house approach would warrant keeping an eye on the relevant kernel commits like a hawk to avoid missing bug fixes like these. These in-house implementations tend to have less eyeballs than the kernel.
I found it interesting that Cloudflare is not yet using BBR as the default in quiche. CUBIC's recovery in this day and age, and especially in datacenters with large pipes, seems so slooow to me. Almost two seconds with no loss whatsoever till achieving BDP again and then shooting itself in the foot every time it hits the ceiling. Each one of those losses a retransmission.
Looking at the last plot, it seems like the backoff is roughly 1/5 of the total bandwith and it happens every 50 ms or so. Wouldn't it make sense to reduce the backoff and the growth speed if a backoff occurs repeatedly in rapid succession? We want to maximize the area under the curve (transmitted packages), right?
As per the article, CCAs aim to maximize data transfer by inferring the "available bandwidth" of the network. CUBIC relies primarily on packet loss as a congestion signal. For recovery, CUBIC's window size is a cubic function of time since the last congestion event.
After the initial packet loss triggered purposefully the first two seconds in this experiment, the only thing which could cause loss is the network queue (i.e. a simple tail drop, fq-codel, etc) which cannot process packets faster than they can arrive. At this point the link is saturated. The loss becomes a signal for CUBIC to reduce its window. This causes the oscillations you pointed out.
Unlike CUBIC, BBR [0] uses a model-based approach that estimates the available bandwidth and leaves some headroom kind of like you suggest to achieve higher throughput, and doesn't react as aggressively to loss as CUBIC.
What jumps out to me is that this is a success story of using a non-trivial test to illuminate an important but hard to observe bit of algorithm. I appreciate the engineering grit to put in a complex test like this, and follow it up when the graph does not have the expected shape.
Imagine your team does not want to write a test because it's too much work or hard to model - this is a great example to bring up.
I tried to read the article, and had to go back and forth between reading about terms and etc, because I'm not _that_ familiar in the space. But previously I could understand Cloudflare's blog posts.
This one just feels... off. The buildup just doesn't feel right.
The fact that there is an Em Dash (sorry...) in the code tells me that it's at least AI assisted, which explains the vibe the article emanates.
And once I finally made it to the end I read the following:
> If you're interested in congestion control, transport protocols, or contributing to open-source networking code, check out the quiche repository. We're always looking for talented engineers who love digging into problems like these, please explore our open positions.
You don't add that to your blog-posts 5 days after laying off 20% of the company, regardless of whether they're sales people or engineers. If you want to add it, delay the post by 2 weeks.
Equally, there is only 1 role open in Engineering, and it's an intern role, posted yesterday:
I _believe_ i've been having some connectivity issues related to this. Where the site acts like it's down but you refresh the page and it works. The solution is to disable the HTTP-3/QUIC toggle in Cloudflare. Some discussion here: https://xenforo.com/community/threads/page-could-not-be-load...
May not be related, but it might. It only recently started happening to me/our sites in the past couple of days.
The line that stuck with me: "Recovery after congestion collapse is an uncommon regime, but it is exactly the regime a congestion controller exists to handle."
This generalizes well beyond congestion control.
Most control loops have the same property. The path the system follows 99% of the time gets well-exercised; the path it falls into when things go wrong is the path you actually need to be correct on. There's usually no way to discover the bug until you deliberately drive the system into the bad regime and watch it try to climb out.
12 comments
[ 2.9 ms ] story [ 30.3 ms ] threadI found it interesting that Cloudflare is not yet using BBR as the default in quiche. CUBIC's recovery in this day and age, and especially in datacenters with large pipes, seems so slooow to me. Almost two seconds with no loss whatsoever till achieving BDP again and then shooting itself in the foot every time it hits the ceiling. Each one of those losses a retransmission.
After the initial packet loss triggered purposefully the first two seconds in this experiment, the only thing which could cause loss is the network queue (i.e. a simple tail drop, fq-codel, etc) which cannot process packets faster than they can arrive. At this point the link is saturated. The loss becomes a signal for CUBIC to reduce its window. This causes the oscillations you pointed out.
Unlike CUBIC, BBR [0] uses a model-based approach that estimates the available bandwidth and leaves some headroom kind of like you suggest to achieve higher throughput, and doesn't react as aggressively to loss as CUBIC.
[0] https://datatracker.ietf.org/meeting/104/materials/slides-10...
What is a CCA in this context?
> Before we dive into the core problem, a quick refresher on Congestion Control Algorithms (CCAs) may help to set the stage.
Imagine your team does not want to write a test because it's too much work or hard to model - this is a great example to bring up.
This one just feels... off. The buildup just doesn't feel right.
The fact that there is an Em Dash (sorry...) in the code tells me that it's at least AI assisted, which explains the vibe the article emanates.
And once I finally made it to the end I read the following:
> If you're interested in congestion control, transport protocols, or contributing to open-source networking code, check out the quiche repository. We're always looking for talented engineers who love digging into problems like these, please explore our open positions.
You don't add that to your blog-posts 5 days after laying off 20% of the company, regardless of whether they're sales people or engineers. If you want to add it, delay the post by 2 weeks.
Equally, there is only 1 role open in Engineering, and it's an intern role, posted yesterday:
https://www.cloudflare.com/careers/ (filter by Engineering).
Did they lay off their PR team as well?
May not be related, but it might. It only recently started happening to me/our sites in the past couple of days.
Most control loops have the same property. The path the system follows 99% of the time gets well-exercised; the path it falls into when things go wrong is the path you actually need to be correct on. There's usually no way to discover the bug until you deliberately drive the system into the bad regime and watch it try to climb out.