It was prolonged by the fact that Cloudflare didn't react correctly to withdrawn BGP routes to a major peer, that the secondary routes had reduced capacity due to unaddressed problems, and basic nuisance rate limiting had to be done manually.
It seems like they just build huge peering pipes and basically just hope for the best. They've maybe gotten so used to this working that they'll let degraded "secondary" links persist for much longer than they should. It's the typical "Swiss Cheese" style of failure.
Anyone want to tell Cloudflare that BGP advertisements at AWS are automated and their congested network directly cause BGP withdrawals as the automated system detected congestion and decreased traffic to remediate it?
It wouldn't surprise me if the BGP routes in the DCI PNI were manually configured, since this is probably one of the most direct and important connections. I would be surprised if Cloudflare didn't have firsthand knowledge of what happened with AWS during this incident.
I think the withdrawal approach by AWS would normally work, as this action should desaturate the connections. Just really unfortunate that this caused routing through a link that was at half capacity.
> This system will allot network resources on a per-customer basis, creating a budget that, once exceeded, will prevent a customer's traffic from degrading the service for anyone else on the platform
How would this work practically? If a single client is overflowing the edge router queues you are kindof screwed already? Even if you dropped all packets from that client you would need to still process the packets to figure out what client they belong to before dropping the packets?
I guess you could somehow do some shuffle sharding where a single client belongs to a few IP prefixes and when that client misbehaves you withdraw those prefixes using BGP to essentially black hole the network routes for that client. If the shuffle sharding is done right only the problem client will have issues as other clients on the same prefixes will be sharded to other prefixes.
In this specific case, it wasn't requests from the client that caused overload. It was the responses to those requests. So Cloudflare can avoid sending responses, and prevent the problem.
You're right that this doesn't solve all cases, but it would have prevented this case.
> Even if you dropped all packets from that client you would need to still process the packets to figure out what client they belong to before dropping the packets?
In modern Linux you can write BPF-XDP programs to drop traffic at the lowest level in the driver before any computation is spent on them at all. Nearly the first thing the driver does after getting new packets in the rx ring buffer is run your program on them.
There was definitely a recurring pattern at AWS where a single customer would trigger latent bugs/undercapacity resulting in outages. Postmortems would often recommend developing per-customer observability and mitigation.
I'm having trouble understanding the second diagram in the article. I can make sense of a directed graph, but this one has thin horizontal lines with arrows leaving them in both directions. These lines look like dividers, not nodes, so I'm not sure how to interpret it.
> The incident was a result of a surge of traffic from a single customer that overloaded Cloudflare's links with AWS us-east-1. It was a network congestion event, not an attack or a BGP hijack.
And no one knew a single thing about it until the incident. That is the current network management state of the art, let Cloudflare deal.
Braze is my guess. They let customers do a lot of stuff with pushing and pulling data per user and I would guess every customer is in a sandbox. They were also impacted by the incident.
21 comments
[ 3.3 ms ] story [ 85.1 ms ] threadIt was prolonged by the fact that Cloudflare didn't react correctly to withdrawn BGP routes to a major peer, that the secondary routes had reduced capacity due to unaddressed problems, and basic nuisance rate limiting had to be done manually.
It seems like they just build huge peering pipes and basically just hope for the best. They've maybe gotten so used to this working that they'll let degraded "secondary" links persist for much longer than they should. It's the typical "Swiss Cheese" style of failure.
I think the withdrawal approach by AWS would normally work, as this action should desaturate the connections. Just really unfortunate that this caused routing through a link that was at half capacity.
I imagine Cloudflare and AWS were on a Chime bridge while this all went down, they both have a lot at stake here.
How would this work practically? If a single client is overflowing the edge router queues you are kindof screwed already? Even if you dropped all packets from that client you would need to still process the packets to figure out what client they belong to before dropping the packets?
I guess you could somehow do some shuffle sharding where a single client belongs to a few IP prefixes and when that client misbehaves you withdraw those prefixes using BGP to essentially black hole the network routes for that client. If the shuffle sharding is done right only the problem client will have issues as other clients on the same prefixes will be sharded to other prefixes.
You're right that this doesn't solve all cases, but it would have prevented this case.
In modern Linux you can write BPF-XDP programs to drop traffic at the lowest level in the driver before any computation is spent on them at all. Nearly the first thing the driver does after getting new packets in the rx ring buffer is run your program on them.
And no one knew a single thing about it until the incident. That is the current network management state of the art, let Cloudflare deal.