While reading Aleksey’s blog[0] linked from yours it struck me that this sounds a great deal like (real world) traffic engineering. A backup occurs, the traffic grinds to a halt, and after the blockage is removed it takes a very long time for the system to recover. If traffic is heavy enough, even just a small slowdown is enough to cause prolonged pain.
(Having said that, my traffic engineering cluefulness is roughly at the “casual reader glancing at headlines” level, if that, but I found the comparison interesting.)
I’m unfamiliar with HotOS, so I’m curious about the context behind this sentence:
> But it's a HotOS paper, and this sure is a hot topic, so I won't hold the lack of a rigorous investigation of the background against the authors.
In other contexts I’ve seen its opposite called “failure demand”, which I think is an apt notion for this context as well: Increased load on the system, not from original users’ intent, but from the response to failures, including retries from the user due to failures of prior interactions.
I found the paper on metastable failures brought back a lot of systems theory for me.
Interesting article. It seems like if you had some kind of generic load shedding component that could monitor the difference between goodput and throughput and throttle load until the former increases it might help? Of course how to do this while guaranteeing it won't make the situation worse is another thing.
> Fast paths, caches, retries, failover, load balancing, and autoscaling all make the failure state less resource-efficient relative to the normal state, which makes the feedback loop worse. Beware of very high cache hit rates!
This scares me as someone who is working on performance tuning and is adding fast paths every other day....
The paper kinds of reminded me the "Gray failure" paper by the way.
One is reminded of Nassim Taleb's writing on efficiency and fragility. Long-lived systems (e.g. in biology) have a built in surplus, rather than operating everything at peak efficiency, because longevity is often determined not by how well you perform normally, but how well you perform during your worst crisis moments.
I'm certainly reminded of the current semiconductor shortages, not to mention early-2020 toilit paper shortages. People see low stock of an essential product, buy up what they need and some spare to cover an anticipated shortage, and the stock runs out, causing a shortage. Other people then try to buy whaxever they can to build their own buffers, extending the shortages.
Here’s a simple one from my career. A calls B calls C. C makes a bad deploy. B passes through the errors from C and A retries against them. The retries push B’s utilization past the limit and instances start getting OOM killed. The C deploy is reverted. But the retries from A to B aren’t about C anymore, they’re about the fact that B is overloaded. A and B are in a neat little death spiral. We have to stop A, let B recover, and then slowly turn A back on.
After this incident we installed a per-instance rate limit on all services.
A very simple system is sufficient to demonstrate such metastable states. Consider a frontend supported by a database. Requests to the FE come in at a uniform rate below the ability of the database. Now there’s a peak in incoming requests. As parallel queries pile up, they start stepping on each other’s toes and the response time goes up. Given the temporarily stable incoming request rate, this means even more parallel queries and even slower response rate. Eventually the database response rate drops to the incoming request rate. If the peak of traffic subsides, the system sees a stable, usual, incoming request rate but a massively struggling database with a large number of parallel requests. This is the metastable state. When the db is restarted, purging the connections, everything returns to normal. Which can lead to the conclusion of the db platform being at fault. This is why you have queues or connection pools in front of your BE: to prevent dynamic complexity leaking over component boundaries and wreaking havoc all over the system.
11 comments
[ 0.24 ms ] story [ 31.8 ms ] threadI wrote a blog post about this paper, too: https://brooker.co.za/blog/2021/05/24/metastable.html
[0]: http://charap.co/metastable-failures-in-distributed-systems/
(Having said that, my traffic engineering cluefulness is roughly at the “casual reader glancing at headlines” level, if that, but I found the comparison interesting.)
I’m unfamiliar with HotOS, so I’m curious about the context behind this sentence:
> But it's a HotOS paper, and this sure is a hot topic, so I won't hold the lack of a rigorous investigation of the background against the authors.
https://en.m.wikipedia.org/wiki/Goodput
I found the paper on metastable failures brought back a lot of systems theory for me.
This scares me as someone who is working on performance tuning and is adding fast paths every other day....
The paper kinds of reminded me the "Gray failure" paper by the way.
https://blog.acolyer.org/2017/06/15/gray-failure-the-achille...
After this incident we installed a per-instance rate limit on all services.