Hi, kabouseng. Right now, I'm the only engineer working on this. I mentioned this in a comment on the blog post, but the main reason I was able to put this together by myself and do it relatively quickly was that I was…
Thanks for the feedback! I do have an example of a silent failure that would be classified as extreme chaos. Before we went into production, we were having problems with leader election that could cause multiple masters…
There are definitely tools within Azure to perform fault injection, but I have not yet made use of them for our Search Chaos Monkey. Right now I've only taught the monkey how to perform basic Azure operations like…
Hi, author of the post here. Happy to answer any questions, would love to hear from any of you who have used similar approaches to managing failure in distributed systems.
Hi, kabouseng. Right now, I'm the only engineer working on this. I mentioned this in a comment on the blog post, but the main reason I was able to put this together by myself and do it relatively quickly was that I was…
Thanks for the feedback! I do have an example of a silent failure that would be classified as extreme chaos. Before we went into production, we were having problems with leader election that could cause multiple masters…
There are definitely tools within Azure to perform fault injection, but I have not yet made use of them for our Search Chaos Monkey. Right now I've only taught the monkey how to perform basic Azure operations like…
Hi, author of the post here. Happy to answer any questions, would love to hear from any of you who have used similar approaches to managing failure in distributed systems.