Like many organizations, DoorDash moved to a microservice architecture from a monolith in the last couple of years. Along with the benefits of better business isolation and increased development agility, the move to microservices also brings new kinds of failures that are difficult to handle. I have been working on reliability projects at DoorDash for the last year, and witnessed firsthand how fires arise and are put out, and what DoorDash does to prevent outages from happening. To share what I learnt, my colleague and I have written a comprehensive overview about failures and their mitigation at DoorDash. In particular, we discussed the advantage of using a centralized reliability control software, Aperture, which can launch coordinated mitigation plan across services during an outage. Unlike load shedders and circuit breakers, a centralized reliability software is in principle more effective in taming the outages as it has a global view of the system. What do you think of the centralized methodology for reliability control? Do they add extra value compared to existing localized mechanisms? Blog link: https://doordash.engineering/2023/03/14/failure-mitigation-f...