Even with backoff and jitter I see lots of these failures because a high level retry will re-trigger the entire chain underneath it, with fresh retry budgets, recursively. If you had it truly everywhere it will always eventually slow down, but only after massive numbers of requests further down the call stack.
Circuit breaking tends to help with that (because it usually applies cross-request), though it brings other problems too. But it's generally worth it.
I kinda wonder if there needs to be some kind of "retry cost" for requests, so high level things retry less when previous attempts consumed a lot of resources further down... but I'm not sure how that could be tracked without either reliable communication of failures (ha) or a global storage of some kind (a potentially massive point of failure). Or maybe just inferring from latency is good enough in most cases?
> I kinda wonder if there needs to be some kind of "retry cost" for requests, so high level things retry less when previous attempts consumed a lot of resources further down... but I'm not sure how that could be tracked without either reliable communication of failures (ha) or a global storage of some kind (a potentially massive point of failure)
You could pretty easily get rough awareness of “things down the stack being expensive” without any sort of reliable communication or centralized storage if you tracked it per request.
If you’re already passing something like a request id through to support tracing across your services, more or less the same idea except pass some sort of budget.
All your entrypoints assign a budget based on some reasonable expectation of cost for the request plus whatever margin you want for retries and failures.
Every caller, when making a request, passes the current budget value through to the callee. If the callee returns, it can include a header with the cost so the caller can deduct it from the budget. If you wanted to use the latency idea, you could replace the cost with a scaling factor to multiply by the latency so any service that was seeing latency spikes would automatically get more expensive. If it doesn’t return, the caller has some relatively high fixed cost it deducts instead. (Optimally, a value that’s related to the general magnitude of cost for the request normally.)
If any service tries to initiate a request and the budget is <= 0, it instead throws an exception. It returns the costs and now the caller has reached zero and if it retries it will throw an exception back to its caller and so on and so forth until it gets back to the original consumer.
You’ve basically set an upper limit on how much effort you’ll expend servicing any request… whatever the reason it’s taking more work or longer than expected.
Main issue I’d see though is… that’s a lot of numbers to tune and keep in alignment. While it would probably be good to know that some call you’re making to some other team’s service just got 10x more expensive because they integrated some other thing… probably not by your API suddenly blowing up. Or the reverse, some team optimizes some stuff and reduces the cost of some requests and now you actually have enough budget to retry too many times and you’re almost back where you started.
I generally just lean towards retries being something that happens in one place, at one layer. Or even not at all—the user can see a “Something went wrong. Try again?” and we can retry when they ask us to.
11 comments
[ 2.8 ms ] story [ 60.9 ms ] threadThose million dollar outages put you through your paces.
It would mean I had a product launched in the wild.
Circuit breaking tends to help with that (because it usually applies cross-request), though it brings other problems too. But it's generally worth it.
I kinda wonder if there needs to be some kind of "retry cost" for requests, so high level things retry less when previous attempts consumed a lot of resources further down... but I'm not sure how that could be tracked without either reliable communication of failures (ha) or a global storage of some kind (a potentially massive point of failure). Or maybe just inferring from latency is good enough in most cases?
The method they describe is basically probabilistic circuit breaking based on the measured success rate the client sees.
If you’re already passing something like a request id through to support tracing across your services, more or less the same idea except pass some sort of budget.
All your entrypoints assign a budget based on some reasonable expectation of cost for the request plus whatever margin you want for retries and failures.
Every caller, when making a request, passes the current budget value through to the callee. If the callee returns, it can include a header with the cost so the caller can deduct it from the budget. If you wanted to use the latency idea, you could replace the cost with a scaling factor to multiply by the latency so any service that was seeing latency spikes would automatically get more expensive. If it doesn’t return, the caller has some relatively high fixed cost it deducts instead. (Optimally, a value that’s related to the general magnitude of cost for the request normally.)
If any service tries to initiate a request and the budget is <= 0, it instead throws an exception. It returns the costs and now the caller has reached zero and if it retries it will throw an exception back to its caller and so on and so forth until it gets back to the original consumer.
You’ve basically set an upper limit on how much effort you’ll expend servicing any request… whatever the reason it’s taking more work or longer than expected.
Main issue I’d see though is… that’s a lot of numbers to tune and keep in alignment. While it would probably be good to know that some call you’re making to some other team’s service just got 10x more expensive because they integrated some other thing… probably not by your API suddenly blowing up. Or the reverse, some team optimizes some stuff and reduces the cost of some requests and now you actually have enough budget to retry too many times and you’re almost back where you started.
I generally just lean towards retries being something that happens in one place, at one layer. Or even not at all—the user can see a “Something went wrong. Try again?” and we can retry when they ask us to.