This is an excellent cautionary article... I still don't get all the love for AWS after they have shown that they have less uptime than many dedicated server providers.
Reliability is the greatest feature you can offer, as a downed service can't be used by anyone.
The big trade-off with AWS is it allows poor people to afford lavish computing resources for small amounts of time, at the expense of things like performance and reliability. If you have slow, steady growth metrics it makes much more sense to host yourself. Planning for peak capacity and buying only what you need is cheaper than dynamically scaling what you need.
The product is geared to a specific kind of customer, and frankly I have no idea why companies like Netflix use them. Hell, Netflix actually buys whole instance blocks [servers] just to work around the shared i/o of EC2. You'd think just buying a real dedicated server would be cheaper and easier.
Well, that plus this bit from the EC2 pricing page:
> Reserved Instances are eligible for volume discounts, which can save you an additional 10% when you spend more than $250,000 on Reserved Instances or an additional 20% when you spend more than $2,000,000 on Reserved Instances. Customers who spend more than $5,000,000 on Reserved Instances should contact us for additional discounts.
> The product is geared to a specific kind of customer, and frankly I have no idea why companies like Netflix use them.
I'd imagine Netflix sees enormous fluctuations in required server capacity. Not many people watch movies at work, but when they come home, or when it's a weekend, traffic probably spikes tremendously.
Yes, perhaps we should think of it this way: Netflix wants to run blocks of dedicated servers, but to defray their costs by leasing their excess capacity to thousands of other customers during non-peak hours.
And a cheap way to build out the technical, marketing, and billing architecture for that is to partner with AWS, which has built all that already.
On the other hand, though, if half the internet is offline with an AWS problem, chances are pretty decent a lot of folks will just go "oh, the Internet is having troubles again".
The point about high-availability platforms is that your customer won't go "Oh, the Internet is having troubles again". They'll probably call you, or perhaps start losing money at an eye-watering rate.
The problem is AWS makes representations about how available certain components will be, and the independence of various features like AZs, which turn out to be lies in practice. When a higher level system builds on top of AWS and relies on these representations, the higher level system fails.
Carriers lie about route diversity on specifically ordered diverse circuits all the time. I rarely physically trace every power cable in a colo to make sure A and B are independently fed; if someone has done it correctly 30 times before, 31 is probably going to be ok. If it is critical, or something which might be difficult, I may check, but since I am confident it will pass, I'll bundle a bunch of checks together so it only passes if all the checks pass.
I think the real reason people like Amazon cloud is for people like me who are decent programmers but who don't know or don't want to know how the system works internally. This guy sounds like he has a lot of what I would call "ops" knowledge which is great for him, but if I decided to start running my own website today I probably wouldn't have the chance to learn all of that stuff.
Cloud providers are trying to abstract it all away while in reality the real world factors cant be abstracted. E.g. latency if servers aren't close enough, I/O performance due to virtualization, disk latency if you are using block storage, etc... have to still be understood. What I've noticed with people using the cloud is that they have to still understand how the cloud services work and many times make up for it in app logic. Look at Netflix for example having to make their entire infrastructure extremely tolerant to latency due to the spiky nature of AWS. At the end of the day, you will have to deal with these factors. You can either do it by learning how the cloud works and figure it out there or by running your own infrastructure. One important point the article makes is that if you roll out your own you can at least control those factors.
It's not to say that clouds aren't useful. They provide a lot of value to many apps but it's not a silver bullet that makes it so you can be ignorant of infrastructure. Maybe one day it will be but as of right now it's important to understand the tradeoffs if you plan on using the cloud.
don't want to know how the system works internally.
I see this sentiment a lot. People seem to be proud of technical ignorance ("We don't even know where are servers are! lol!") because they are too busy doing whizbang gollygee Important Work like making the world's four billionth PHP CRUD app and setting up a DB without durability guarantees.
"But we don't have time to learn... we are busy being cool and popular!" Sometimes you have to slow down and actually learn things. Sometimes you should be an intern for a year before being a CEO. (It sounds like you've already resigned to not learn things: I probably wouldn't have the chance to learn all of that stuff.)
what I would call "ops" knowledge which is great for him
That's kinda impressive: dismissal condescension towards someone who knows more than you. It provides the ego with a nice padding. "They know more than I do, but what they know is silly and useless in these modern cloud-based times. I feel sorry they actually had to learn how things work."
Programming means manipulating a system. Basic/beginner programmers pretend the entire system is the API of their favorite (read: ruby) programming language. You can get pretty far not learning more than that, but you will always be limited. You will always be pretending to understand more than you actually do.
If you aren't a "computer person," you'll stop at the basic level. You'll spin your wheels cranking out things that feel the same forever. What separates a "computer person" from people just in it for the glory or money is their insatiable knowledge hungerness. They want to be good, better, and then optimally best.
Intermediate programmers realize (and have a mental model of) how everything works together including a basic CPU, bus, memory, network (from hardware up to packet switching through routing and DNS), your language stack and heap, calling conventions, directly attached storage, network attached storage, and half a dozen other things depending on speciality (graphics? wireless? robots? distributed system? embedded system? web?).
And there's no shame in standing on other peoples shoulders, but if you're building a web application you really need to think about having at least person with a devops background. Hell, even if they know how to compile from source and monitor properly that'd put them in a good place to go forward.
> There's no shame in programming in high level languages to deliver more value more quickly. Ditto with operations, ditto with everything.
There is shame in not understanding how that high-level language actually works, because it means you'll invariably be a worse programmer than the guy that does understand what he's actually doing.
I think it's the difference between a user and an engineer. As a user, you... use. You don't care how it works. You drive a car; you don't design a car.
Things get fuzzier in software development. Programmers are supposed to be users of everything they aren't programming. Things are supposed to be black boxes with fully isolated layers of abstraction. Ha. Doesn't work. To be competent, you must be aware of, at a minimum, what pieces are under you, at an intermedium, how they work, and at a maximum, how to create them yourself.
I'm only talking about people who want to be great technical workers though. If you just want to make a MVP to get funding then hire dweebs, who wasted their life learning computers, to make you rich, feel free to do that instead.
How many software systems are even 1% as battle tested as a "CPU"? Even the most well designed software system doesn't have the kind of test coverage of a hardware.
CPU's are complex and far from perfect, often shipping with hundreds of 'errata' at the design level plus a wide assortment of random manufacturing errors in each chip. So, there is software that has fewer bugs and better coverage than a modern CPU. ex: Grep
> There's no shame in programming in high level languages to deliver more value more quickly. Ditto with operations, ditto with everything.
Agreed. That isn't the same as the "don't want to know how the system works" mentality. High level languages are great tools for "encoding and automating processes you understand so you Don't Repeat Yourself, or perhaps Don't Need To Do It The First Time". They are great for "making all the initial decisions for you so you can get started and come back to those later".
They are horrid tools for "allowing you to be ignorant".
Now, I don't think everyone should know everything about computer systems (heck, just asking someone to know everything about C++ is a tall order ;-), and I certainly don't think you should have memorized all the configuration logic for hardware and software. However, if you are a technology professional, you really should see a key part of your professional value as your understanding of as much of how the system functions as possible. The understanding will never be total; it will never be complete; it ought to be an area of concern.
If it isn't, you aren't an technology professional. You might be an entrepreneur. You might even be a great product designer. You aren't an engineer.
That might sound like it is about semantics. It isn't. Computer systems are basically towers of babel where each brick is another leaky abstraction. Engineers are the people who build, maintain, and replace that tower. You can't do the job well if you only know about the roof.
Entrepreneurs have such a huge challenge just searching for a viable business model, that they tend to see all other problems as irrelevant. While the problems are usually smaller, possibly even trivial by comparison, they aren't irrelevant. If you are a tech company, doing technology right is more than just a tiny bit relevant.
Think of it this way: SQL provides a tremendously high level of abstraction to data systems. This makes it terribly efficient for quickly coming up with queries you want answers for. However, if something ever goes wrong, or if you are writing application code that is going to be run over and over again, you want someone who understands a lot more about the system than just how to translate a question in to SQL. That goes double if somehow the query is vital to your business' ongoing operations.
This is not a question of emotions. If you don't know how something works you won't be able to use it as well. If that's fine with you, okay.
If you want to drive a race car, you don't have to be a mechanic. You do have to understand some basic physics and simple parts of the car. For example, when you turn sharply into a corner, the wheel's camber helps determine how much of the tire's surface stays on the road, giving you more grip, keeping you in the corner at a higher G-force. Looking at the wear on the tire you can tell that your camber might need to be adjusted, and that might make 1/10th of a second difference, which is the difference between winning and losing.
If you want to program like a grandmother driving to the grocery store, all you need is gas. But if you want to program like a race car driver, try to learn how the internals work.
I'd also add that if you want to program for people like Google or Amazon, you need to understand how the internals work, or you will be seriously embarrassed at the interview. (From personal experience)
I do think it's true that good programmers have a passion for understanding as much as they can. But there are so many levels and it's impossible to master everything. Not only that, but things are constantly evolving. If you set about to master everything you can drive yourself crazy.
I have a friend who is an incredible engineer, he writes high performance code at the C and Assembly level. He knows almost nothing about higher level code. You could say in a way that he's lazy because he wouldn't know how to write a simple web page, but that would be missing the point. He's critical for the work that he does.
Being a jack of all trades is helpful at times, but a team of people who specialize can go into depth that just isn't possible for one individual.
Personally I'm interested in the design of a many systems - but I can see a case for wanting other systems to present a single well-defined, correctly implemented interface.
There was a time when, to make a program with sound, programmers had to handle different sound cards differently. You used to get settings in games to choose whether your card was Sound Blaster 16 Compatible.
Were those the good old days, or the bad old days?
What you see as celebrating ignorance, I see as celebrating simplicity: "We don't _need_ to know how it works any more!"
For many use cases, this is perfectly acceptable (and the article rightly notes this).
"That's kinda impressive: dismissal condescension towards someone who knows more than you."
I saw nothing dismissive about that statement whatsoever. I read it as "they know more than I do about that particular subject, but what they know is not useful for me to know at this moment in time".
There is value in specialization. You denigrate the OP with your condescending "too busy doing whizbang gollygee Important Work" and mischaracterize him as using words like "silly and useless", but the OP never said that. All he said is that he's focused on one part of the work and doesn't know that much about the others.
I've been considering these choices a lot. In my own career, I've made a conscious choice to pursue lots of different things that I found interesting, but I've become a generalist, pretty good at lots of different things but lacking the deep domain knowledge in any one of them that a specialist gains. You seem to be arguing that specialists should not exist, everyone should go for broad rather than deep. I think there's a place for both, and becoming a specialist often requires the discipline to stay focused and not pursue every interest and curiosity that rears it's distracting head.
I think there's value in a programmer knowing the whole stack, from language APIs to hardware implementations, and being able to understand the consequences of every line of code. But time is finite and someone at the beginning of their careers must make choices on where to build their expertise.
Very well put. Every good programmer knows the underlying parts of what he is using, and unfortunately the new trend is that programmers don't need to know this stuff anymore.
It's true that in some cases they don't have to, but they should, and for some reason people get offended when you tell them that...
Fortunately everyone here reads Hacker News so we already know that AWS has downtime that can affect whole regions, and that platforms like Heroku don't yet provide resiliency that spans regions.
So if you don't really care about occasional downtime, you don't need to bother learning all that hard stuff. It's perfectly ok with me if your service is down.
This article reads to me like someone who lives in an old business model watching the internet come to town and screaming that it will never catch on! Will there be a place for legacy architecture like this for many years to come? Yes. But each time these cloud architectures fail, we are learning how to deal with that failure and improve our collective availability to remain up through out disaster.
My thinking is that I want experts working on the low-level configurations (power, racks, UPSes, etc). I don't have that expertise myself. Now, I could try to hire some experts to set up highly available hardware from me... or... I could rent some that has already been set up. I think the basic assumption you have to buy into is that your cloud provider knows how to do this better than I do.
I think if we look at a AWS reliability, the interesting and sorta-unanswerable question is: would these applications have been more, or less, reliable, at the same cost, if they were running the infrastructure themselves? My guess: less. It's so easy to screw this stuff up.
I like this article. I used to work on HA systems and I saw systems that were really HA all the way down to the electrons. Right now, I would rather have someone like you who enjoys building HA architecture and I would rather be writing software.
The problem is that you are good(expensive) and most people can't afford to build out and maintain a true HA system so they go for the cheap alternative and in that case, you get what you pay for.
As stated in the article, it's not wrong to do what you're saying, but the article is talking about people that can do that and should do that.
Quoting the article: When it gets to the point where you start measuring downtime in dollars rather than “time I would’ve been doing something else” it is time to move your critical infrastructure to something you know.
This post seems like somebody looking for an excuse to attack "the cloud". AWS's issues last week were caused by a whole datacenter going down, due to an electrical storm. No matter what your configuration is, whether you use cloud hosting or traditional hosting, losing a datacenter is going to throw a wrench in things. Even if your failover works perfectly, the increased load on whatever you fail to might bring things down.
The procedure for high availability is the same for cloud hosts as traditional hosts: multiple datacenters and multiple providers. If somebody made the decision to only host only with amazon and only in US-East, the traditional hosting alternative to that is not a high-availability network spanning 5 datacenters. You're comparing apples to oranges. You can host with AWS across multiple regions, just like you can host with any other provider in multiple datacenters.
At some level, I can agree with this sentiment. You don't know where the servers are, you can't see the cables, and a lot of things are just plainly out of your control.
TL;DR: AWS can realistically provide high-enough availability for most; but I think the article makes good points for those looking to get deep into the nines.
However, I think the real issue is that while you can make fairly HA systems on AWS, it quickly gets expensive. A huge pull for AWS (IMO) is how cheap it is for an org to spin up four or five servers of various specs, and keep them running or shut them down. In order to achieve some aspect of HA through AWS, you need to mirror your entire setup in three places (not to mention some archives in S3, and ideally some archives on non-AWS). The cost savings start to quickly degrade as you configure your AWS-hosted system for real HA (or real close to HA).
IMO, many of the organizations that use AWS are in a position where they probably wouldn't have true HA configured if they were using VPS or physical hardware, so it's not so much of a concern. For the larger organizations who have more funds and more dedicated Ops employees, we've only seen a couple of issues that should have affected availability, and usually availability drops because they cut corners. Granted - at least two of the recent outages showed errors on the AWS-side, and decisions that Ops made for AWS regions/AZs should have been fine, but weren't. That in mind, any HA sysadmins out there would have shuttered to think of relying on two different mechanisms that were potentially in the same data center (that is, backup servers or load balanced servers in the same region).
43 comments
[ 8.4 ms ] story [ 106 ms ] threadReliability is the greatest feature you can offer, as a downed service can't be used by anyone.
The product is geared to a specific kind of customer, and frankly I have no idea why companies like Netflix use them. Hell, Netflix actually buys whole instance blocks [servers] just to work around the shared i/o of EC2. You'd think just buying a real dedicated server would be cheaper and easier.
> Reserved Instances are eligible for volume discounts, which can save you an additional 10% when you spend more than $250,000 on Reserved Instances or an additional 20% when you spend more than $2,000,000 on Reserved Instances. Customers who spend more than $5,000,000 on Reserved Instances should contact us for additional discounts.
I'd imagine Netflix sees enormous fluctuations in required server capacity. Not many people watch movies at work, but when they come home, or when it's a weekend, traffic probably spikes tremendously.
And a cheap way to build out the technical, marketing, and billing architecture for that is to partner with AWS, which has built all that already.
With black-box cloud services such as AWS, Azure, you just don't know what is happening.
Peace of mind is also important!
Carriers lie about route diversity on specifically ordered diverse circuits all the time. I rarely physically trace every power cable in a colo to make sure A and B are independently fed; if someone has done it correctly 30 times before, 31 is probably going to be ok. If it is critical, or something which might be difficult, I may check, but since I am confident it will pass, I'll bundle a bunch of checks together so it only passes if all the checks pass.
It's not to say that clouds aren't useful. They provide a lot of value to many apps but it's not a silver bullet that makes it so you can be ignorant of infrastructure. Maybe one day it will be but as of right now it's important to understand the tradeoffs if you plan on using the cloud.
I see this sentiment a lot. People seem to be proud of technical ignorance ("We don't even know where are servers are! lol!") because they are too busy doing whizbang gollygee Important Work like making the world's four billionth PHP CRUD app and setting up a DB without durability guarantees.
"But we don't have time to learn... we are busy being cool and popular!" Sometimes you have to slow down and actually learn things. Sometimes you should be an intern for a year before being a CEO. (It sounds like you've already resigned to not learn things: I probably wouldn't have the chance to learn all of that stuff.)
what I would call "ops" knowledge which is great for him
That's kinda impressive: dismissal condescension towards someone who knows more than you. It provides the ego with a nice padding. "They know more than I do, but what they know is silly and useless in these modern cloud-based times. I feel sorry they actually had to learn how things work."
Programming means manipulating a system. Basic/beginner programmers pretend the entire system is the API of their favorite (read: ruby) programming language. You can get pretty far not learning more than that, but you will always be limited. You will always be pretending to understand more than you actually do.
If you aren't a "computer person," you'll stop at the basic level. You'll spin your wheels cranking out things that feel the same forever. What separates a "computer person" from people just in it for the glory or money is their insatiable knowledge hungerness. They want to be good, better, and then optimally best.
Intermediate programmers realize (and have a mental model of) how everything works together including a basic CPU, bus, memory, network (from hardware up to packet switching through routing and DNS), your language stack and heap, calling conventions, directly attached storage, network attached storage, and half a dozen other things depending on speciality (graphics? wireless? robots? distributed system? embedded system? web?).
Advanced programmers do not exist.
I like that sentiment a lot. We have finite time and finite brain cycles. Stand proudly on others' shoulders and build something bigger.
There's no shame in programming in high level languages to deliver more value more quickly. Ditto with operations, ditto with everything.
There is shame in not understanding how that high-level language actually works, because it means you'll invariably be a worse programmer than the guy that does understand what he's actually doing.
Things get fuzzier in software development. Programmers are supposed to be users of everything they aren't programming. Things are supposed to be black boxes with fully isolated layers of abstraction. Ha. Doesn't work. To be competent, you must be aware of, at a minimum, what pieces are under you, at an intermedium, how they work, and at a maximum, how to create them yourself.
I'm only talking about people who want to be great technical workers though. If you just want to make a MVP to get funding then hire dweebs, who wasted their life learning computers, to make you rich, feel free to do that instead.
No? Well perhaps, not needing to dig into how things work is a useful feature of a well designed system.
EX: Pages 19-73 with 2 to 4 errata per page for a single popular CPU design: http://download.intel.com/design/processor/specupdt/320836.p... And that's just limited to stuff Intel knows about.
Agreed. That isn't the same as the "don't want to know how the system works" mentality. High level languages are great tools for "encoding and automating processes you understand so you Don't Repeat Yourself, or perhaps Don't Need To Do It The First Time". They are great for "making all the initial decisions for you so you can get started and come back to those later".
They are horrid tools for "allowing you to be ignorant".
Now, I don't think everyone should know everything about computer systems (heck, just asking someone to know everything about C++ is a tall order ;-), and I certainly don't think you should have memorized all the configuration logic for hardware and software. However, if you are a technology professional, you really should see a key part of your professional value as your understanding of as much of how the system functions as possible. The understanding will never be total; it will never be complete; it ought to be an area of concern.
If it isn't, you aren't an technology professional. You might be an entrepreneur. You might even be a great product designer. You aren't an engineer.
That might sound like it is about semantics. It isn't. Computer systems are basically towers of babel where each brick is another leaky abstraction. Engineers are the people who build, maintain, and replace that tower. You can't do the job well if you only know about the roof.
Entrepreneurs have such a huge challenge just searching for a viable business model, that they tend to see all other problems as irrelevant. While the problems are usually smaller, possibly even trivial by comparison, they aren't irrelevant. If you are a tech company, doing technology right is more than just a tiny bit relevant.
Think of it this way: SQL provides a tremendously high level of abstraction to data systems. This makes it terribly efficient for quickly coming up with queries you want answers for. However, if something ever goes wrong, or if you are writing application code that is going to be run over and over again, you want someone who understands a lot more about the system than just how to translate a question in to SQL. That goes double if somehow the query is vital to your business' ongoing operations.
If you want to drive a race car, you don't have to be a mechanic. You do have to understand some basic physics and simple parts of the car. For example, when you turn sharply into a corner, the wheel's camber helps determine how much of the tire's surface stays on the road, giving you more grip, keeping you in the corner at a higher G-force. Looking at the wear on the tire you can tell that your camber might need to be adjusted, and that might make 1/10th of a second difference, which is the difference between winning and losing.
If you want to program like a grandmother driving to the grocery store, all you need is gas. But if you want to program like a race car driver, try to learn how the internals work.
I'd also add that if you want to program for people like Google or Amazon, you need to understand how the internals work, or you will be seriously embarrassed at the interview. (From personal experience)
I have a friend who is an incredible engineer, he writes high performance code at the C and Assembly level. He knows almost nothing about higher level code. You could say in a way that he's lazy because he wouldn't know how to write a simple web page, but that would be missing the point. He's critical for the work that he does.
Being a jack of all trades is helpful at times, but a team of people who specialize can go into depth that just isn't possible for one individual.
There was a time when, to make a program with sound, programmers had to handle different sound cards differently. You used to get settings in games to choose whether your card was Sound Blaster 16 Compatible.
Were those the good old days, or the bad old days?
What you see as celebrating ignorance, I see as celebrating simplicity: "We don't _need_ to know how it works any more!"
For many use cases, this is perfectly acceptable (and the article rightly notes this).
"That's kinda impressive: dismissal condescension towards someone who knows more than you."
I saw nothing dismissive about that statement whatsoever. I read it as "they know more than I do about that particular subject, but what they know is not useful for me to know at this moment in time".
or that people shouldn't attempt things without expertise?
I've been considering these choices a lot. In my own career, I've made a conscious choice to pursue lots of different things that I found interesting, but I've become a generalist, pretty good at lots of different things but lacking the deep domain knowledge in any one of them that a specialist gains. You seem to be arguing that specialists should not exist, everyone should go for broad rather than deep. I think there's a place for both, and becoming a specialist often requires the discipline to stay focused and not pursue every interest and curiosity that rears it's distracting head.
I think there's value in a programmer knowing the whole stack, from language APIs to hardware implementations, and being able to understand the consequences of every line of code. But time is finite and someone at the beginning of their careers must make choices on where to build their expertise.
It's true that in some cases they don't have to, but they should, and for some reason people get offended when you tell them that...
Fortunately everyone here reads Hacker News so we already know that AWS has downtime that can affect whole regions, and that platforms like Heroku don't yet provide resiliency that spans regions.
So if you don't really care about occasional downtime, you don't need to bother learning all that hard stuff. It's perfectly ok with me if your service is down.
I think if we look at a AWS reliability, the interesting and sorta-unanswerable question is: would these applications have been more, or less, reliable, at the same cost, if they were running the infrastructure themselves? My guess: less. It's so easy to screw this stuff up.
The problem is that you are good(expensive) and most people can't afford to build out and maintain a true HA system so they go for the cheap alternative and in that case, you get what you pay for.
"Right" means something different to you than me. I build real-time systems and quasi real-time systems across RTOSs and nixes.
They relied on a single cloud provider. If you want seamless resiliency you need redundant pairs at the very least across all failure scenarios.
Can you go overboard on this? Absolutely.
But did they do _everything_ to mitigate this ahead of time? Absolutely not.
Did they do this "right" according to what their business requirements (/risk profile) were? Who knows.
Quoting the article: When it gets to the point where you start measuring downtime in dollars rather than “time I would’ve been doing something else” it is time to move your critical infrastructure to something you know.
:)
The procedure for high availability is the same for cloud hosts as traditional hosts: multiple datacenters and multiple providers. If somebody made the decision to only host only with amazon and only in US-East, the traditional hosting alternative to that is not a high-availability network spanning 5 datacenters. You're comparing apples to oranges. You can host with AWS across multiple regions, just like you can host with any other provider in multiple datacenters.
TL;DR: AWS can realistically provide high-enough availability for most; but I think the article makes good points for those looking to get deep into the nines.
However, I think the real issue is that while you can make fairly HA systems on AWS, it quickly gets expensive. A huge pull for AWS (IMO) is how cheap it is for an org to spin up four or five servers of various specs, and keep them running or shut them down. In order to achieve some aspect of HA through AWS, you need to mirror your entire setup in three places (not to mention some archives in S3, and ideally some archives on non-AWS). The cost savings start to quickly degrade as you configure your AWS-hosted system for real HA (or real close to HA).
IMO, many of the organizations that use AWS are in a position where they probably wouldn't have true HA configured if they were using VPS or physical hardware, so it's not so much of a concern. For the larger organizations who have more funds and more dedicated Ops employees, we've only seen a couple of issues that should have affected availability, and usually availability drops because they cut corners. Granted - at least two of the recent outages showed errors on the AWS-side, and decisions that Ops made for AWS regions/AZs should have been fine, but weren't. That in mind, any HA sysadmins out there would have shuttered to think of relying on two different mechanisms that were potentially in the same data center (that is, backup servers or load balanced servers in the same region).