Hi, author of the post here. Happy to answer any questions, would love to hear from any of you who have used similar approaches to managing failure in distributed systems.
Thanks for the nice and informative post. Can you add a bit more color to extreme chaos failure with maybe a more concrete action and its outcome? I'm assuming you're pre-classifying availability/sla's for the serving parts of the search system and its sub-systems. Also, a real world example of a silent failure observed in your systems will be nice to learn from.
Thanks for the feedback! I do have an example of a silent failure that would be classified as extreme chaos.
Before we went into production, we were having problems with leader election that could cause multiple masters per cluster. This is an extreme chaos state not only because it could result in corrupted or missing data (as a cloud service, we take data integrity very, very seriously), but also because it didn't raise any immediate notifications, only downstream errors.
This problem has since been fixed, but even before we did that, the highest priority was to ensure our system detected this state as quickly as possible. We added a service that monitored cluster leader status and that quickly alerted us whenever anything was unexpected.
Even after fixing the problem, I've set the Search Chaos Monkey to injecting failure into leader election at various points to ensure that the cluster can always recover in an expected way that won't put customer data at risk.
I was wondering how many engineers are actively working on your search chaos monkey. In my experience it takes a lot of man hours to get something such as this up and running, especially taking into account stabilising the azure search product while bringing the automated functional testing up, stabilising the testing system, and then bringing the chaos monkey monkey up and stabilising all three (product, testing system and chaos system).
Hi, kabouseng. Right now, I'm the only engineer working on this.
I mentioned this in a comment on the blog post, but the main reason I was able to put this together by myself and do it relatively quickly was that I was able to leverage pre-existing Azure Search infrastructure.
We have great tools for monitoring clusters, a proven queue-based job process and a good deployment workflow. Not only does it make it easy to set up systems like this, but it also allows us to smoothly and efficiently manage an ever-increasing number of customer services. It's been a good investment for us.
How do other Azure teams inside Microsoft do their testing (SQL Azure, Azure DocDB etc?) I'd assume there's a common platform that allows you to just throw stuff and inject failures? Further, in production, what percent of bugs if any did you find were not caught by Chaos engineering? Where they of a particular class? Did you still see a necessity for functional tests that test all permutations diligently?
There are definitely tools within Azure to perform fault injection, but I have not yet made use of them for our Search Chaos Monkey. Right now I've only taught the monkey how to perform basic Azure operations like restarting VMs and focused mostly on injecting failure into Azure Search components. It's absolutely something that I want to look into for future iterations.
We've found bugs in production that chaos engineering hasn't caught. I can't give you a percentage, but the monkey is great at catching a majority of bugs where it has coverage (places like our provisioning and scaling workflow). Most of the bugs that have slipped through are as a result of lack of coverage. I haven't added any operations that test the service under heavy load (large amount of data or large number of requests/second), and I feel like we've seen more production issues in that area as a result.
8 comments
[ 3.1 ms ] story [ 39.4 ms ] threadBefore we went into production, we were having problems with leader election that could cause multiple masters per cluster. This is an extreme chaos state not only because it could result in corrupted or missing data (as a cloud service, we take data integrity very, very seriously), but also because it didn't raise any immediate notifications, only downstream errors.
This problem has since been fixed, but even before we did that, the highest priority was to ensure our system detected this state as quickly as possible. We added a service that monitored cluster leader status and that quickly alerted us whenever anything was unexpected.
Even after fixing the problem, I've set the Search Chaos Monkey to injecting failure into leader election at various points to ensure that the cluster can always recover in an expected way that won't put customer data at risk.
I was wondering how many engineers are actively working on your search chaos monkey. In my experience it takes a lot of man hours to get something such as this up and running, especially taking into account stabilising the azure search product while bringing the automated functional testing up, stabilising the testing system, and then bringing the chaos monkey monkey up and stabilising all three (product, testing system and chaos system).
Again thanks for the write up.
I mentioned this in a comment on the blog post, but the main reason I was able to put this together by myself and do it relatively quickly was that I was able to leverage pre-existing Azure Search infrastructure.
We have great tools for monitoring clusters, a proven queue-based job process and a good deployment workflow. Not only does it make it easy to set up systems like this, but it also allows us to smoothly and efficiently manage an ever-increasing number of customer services. It's been a good investment for us.
We've found bugs in production that chaos engineering hasn't caught. I can't give you a percentage, but the monkey is great at catching a majority of bugs where it has coverage (places like our provisioning and scaling workflow). Most of the bugs that have slipped through are as a result of lack of coverage. I haven't added any operations that test the service under heavy load (large amount of data or large number of requests/second), and I feel like we've seen more production issues in that area as a result.
Here is a handy list of top requests: http://visualstudio.uservoice.com/forums/121579-visual-studi...