I've been getting the "We're sorry!" error for at least 15 minutes. No idea about regions or anything, I was just about to look up their docs about moving accounts between orgs etc. Anyone affected?
Edit: Seems to be flapping between the 'sorry' error and a blank page. Thoughts and prayers with the SREs, if they call 'em that over there.
Odd that a major landing page like aws.amazon.com isn't super HA. At a previous $work, the "marketing" page (ie: main dot com site) was horribly mis-managed and had countless outages due to various embarrassing reasons. Every time it was down, customers complained that the product was down (it wasn't, but people used the dot com page to get to the login page).
The only thing that was down was the marketing home page at aws.amazon.com, which is not a “service” and is not on the SHD. All of the actual services are fine, hence the green.
I wonder why the error message is also in … French? (I’m not located anywhere French speaking).
> Désolés!
> Une erreur s'est produite lorsque nous avons tenté de traiter votre requête. Soyez assuré que nous travaillons déjà à la résolution du problème que nous pensons trouver très rapidement.
Japanese has multiple politeness levels, in this particular case the message is using keigo which is the most polite. At a guess I'd say that the machine translation is identical all the way down.
Thanks. I know nothing of Japanese so this was confusing to me.
I'm thinking about it, and I guess there's "politeness levels" in English, but probably not as much so, and usually in a form of passive aggresisveness.
That's fascinating. So the "style" of language they use has a huge impact on the message, possibly more so than the actual words? Very McLuhanesque :)
You even conjugate the verbs differently for your customers.
Richard Feynman talked about this point really frustrating him and turning him off Japanese entirely. He just wanted to use words to get things done...granted for informational purposes it's nice to not have to conjugate the verb for download based on who downloaded it.
I kind of get that sentiment. Fundamentally I think it's beautiful and I'm delighted by such variety in human languages. But I spent 10 years learning French and it drove me nuts how many different ways you conjugate verbs.
When I've worked with AWS and other providers before, they've typically given early indications about potential root causes under NDA. If you're a small customer, you probably get to wait with the rest of us but the big folks probably have some idea of the root cause.
it's usually a month or more after a large outage to see the full breakdown on what happened. people expecting to see it the same week are kinda not living in reality.
Fair I guess I would have expected more in the way of official comms though. Most of what I've seen is based on deduction and speculation rather than from the horse's mouth
I think this adds some momentum to the pendulum swinging back the other way. Maybe cloud teams can patch your services better than your in-house team can (See 2 critical issues in Azure the last 3 months, _caused_ by MS itself).
Maybe the cloud has a higher uptime than your on-premise infrastructure (see the AWS, Azure outages). Make sure to compare the actual outage time v.s. the stats doctored by various political pressures and weaselly worded SLAs (how do you mean you had an outage? Only 49% of your requests were failing!).
Even if you can manage more uptime on your own than through the cloud (which I doubt), being on the cloud means downtime is correlated with downtime of other services. That's usually a good thing.
Your customers will be more understanding if your outage is part if a wider outage that makes national news.
Any services you integrate with are likely down too. If two services with 99% uncorrelated uptime together drops to about 98%. It doesn't drop if the downtime is perfectly correlated.
Even if you don't directly integrate, your customer's workflow might. They see many services down and say, "cool, time to get caught up on laundry". If only you're down it's more aggrevating.
My desktop PC has better uptime than the cloud right now. So does my datacenter. In fact, I am not sure I have anything electronic that doesn't have better uptime than AWS or Azure.
> Even if you can manage more uptime on your own than through the cloud (which I doubt)
Getting higher uptime is super easy for smallish inhouse deployments.
You just don't install any updates and let the server run, trusting your VPN to shield you from possible security issues.
The maintenance burden is the reason why people often prefer the cloud services, not the uptime. Because maintaining the instance with updates, reading all patch notes and steps for migration, keep every health metric monitored and respond quickly on issues without getting stuck googling for possible reasons is quite a bit of work and quickly forces you to employ n+1 people.
This is particularly true with modern fully solid state hardware. Uptimes of decades are likely possible if you don't mess with anything.
We run some bare metal servers. They just never go down. Solid continuous pings for years as monitored from elsewhere on the Internet. That's because they're just boxes on a rack somewhere running an OS and some steady-state services (ZeroTier roots). Simplicity is more robust than complexity.
SaaS is definitely about the pain of managing and upgrading software, but it's also about OPEX vs CAPEX. Many companies will pay more for things to put them in the OPEX column for various entirely synthetic accounting, investor relations, and tax reasons.
I do wonder if the pendulum there will swing back though since if you price out cloud vs. physical hardware the market has become extremely distorted. Many companies spend enough on AWS to buy an entire rack of hardware at a different data center every month and pay 2-3 employees to manage it. That hardware would be up to 100X as fast and powerful as what they rent at AWS and bandwidth would be almost free. That's a really distorted market. The amortized costs should not be this different.
>Many companies spend enough on AWS to buy an entire rack of hardware at a different data center every month and pay 2-3 employees to manage it. That hardware would be up to 100X as fast and powerful as what they rent at AWS and bandwidth would be almost free. That's a really distorted market. The amortized costs should not be this different.
I think the issue here is that it's not zero-cost to switch. Your processes will adapt to some implicit assumptions that aren't true outside AWS, Azure, or whatever vendor you locked yourself into.
If we somehow managed to have a completely standardized interface here, the market would be more competitive.
How do the attackers ddos an inhouse deployment that cannot be accessed from the internet without joining a VPN?
They can try to saturate the VPN host maybe, but that's going to be challenging considering that it's going to be limited to connection requests without valid credentials.
and these are likely set to be ignored on multiple failed attempts through fail2ban or similar tooling
You have a connection to the internet with an IP address do you not? That connection can easily be saturated with traffic and fail2ban would not have a chance to do anything.
Wha? We host everything on our hardware (which is nothing special) and haven't had any downtime in this year (yet). And we're just another run of the mill dev shop, very far from "superstars" who work on these (supposedly extremely stable) platforms.
Two questions I have regarding your in-house hardware:
1. How easy can I access your physical servers ?
2. What happens if there is a catastrophic failure, for example local power outage or a major flooding
3. How secure is your server? Are you regularly patching your operation systems
4. If I want to run a project that requires double the capacity of your current hardware for a specific project, how long is it going to take to get it spun up?
I'm running about 10 servers myself in production. They just do transcoding, so aren't mission-critical, but...
- Regularly patching is automated and took about 30 seconds to configure, using an automated script
Regarding running your own physical servers, that is a different ballgame, but for all of my projects, if I need to:
- I can pretty easily spin up VPSs / bare metal servers anywhere (netcup, linode, hetzner, etc) and provision there while I wait for new hardware to come in
- If you want to double the capacity of your current hardware, you'll have to order it and wait, but it's cheap (vs the major cloud providers) to way over provision if you're running your own physical hardware, so you can pretty easily have 2-4x extra capacity and still come out with extra money in your pocket.
I host in the cloud, but I think people vastly over estimate how much it saves 90% of cloud customers.
There are many approaches which don't depend on AWS, and not all of them mean hosting your own physical servers, and they certainly don't mean you don't have an off-site backup policy.
There are well-understood answers to all your questions, they are not too difficult, they just cost money - some businesses choose not to spend that money, some weigh cost-benefit and go for AWS, some decide to go for in-house servers, some go for hosted Virtual Servers, some go for serverless.
> they just cost money - some businesses choose not to spend that money, some weight cost-benefit and go for AWS, some weigh cost benefit and decide to go for in house, some go for hosted Virtual Servers, some go for Serverless.
I'd also say some choose not to spend the money, but fail to consider the cost of that choice.
For example: Doing old-school manual deployments that require herculean efforts to update at off-hours on the weekends burns people out, and makes it hard to attract new talent. In other words, you've made the decision to spend more money on finding and retaining people. But it's definitely way cheaper to pay for colocating a single Dell server you bought 3 years ago than what you'd spend in the same time on AWS.
And if your hardware never dies, paying for the redundancy might seem silly. A lot like paying for fire insurance despite the fact your house has never even burned down.
Of the options I mentioned, you seem to have picked only self-purchased hardware with no redundancy and no backups (your addition) to compare with, and also throw in manual deployment (why?).
Most businesses are somewhere between a forgotten old dell server in the closet and fully hosted multi-region auto-scaling fully bought in to AWS, and that's OK!
i am not saying everyone needs to go for AWS and there are many good reasons for self hosting However, if the only reason for self hosting is a higher uptime than AWS or GCP, it is general false economy, and you are likely taking short cuts that you are not aware of.
This is a false dichotomy (I listed many options, not just self hosting, not many businesses fully self-host these days), and uptime is provably significantly better away from the big hosted services which are constantly churning features, config and hardware resulting in outages for everyone hosted on them.
1: Access to the data center requires either an access card plus biometric verification (fingerprint in one DC in my case, retina scan in the other), or an ID-verified appointment. Then, you still need to know where my server is, plus the access code for the rack (or, you need to be an average lockpicker, but beware, there's cameras...).
2: Each data center has dual-provider AC feeds, plus generators, and provides A/B feeds to my rack. I've not had a dual-feed outage in the last 20 years or so.
3: No cloud provider that I'm aware of guarantees server security or does automated patching (for servers, not services). So, keeping your server up-to-date seems equally important for both cloud and non-cloud scenarios?
4: At least two weeks, I guess? I have sufficient VM host capacity to accommodate 30% unplanned growth, but 100% would require new hardware. So: ordering two servers, installing these in two data centers. But if the new project also requires significant bandwidth, getting new Internet connections in might take longer.
Look, I'm definitely not denying that "the cloud" makes it easier to scale fast, but scaling fast is not an overriding concern for most businesses. Cost is, and self-hosting, even with a pretty redundant infrastructure, is still much cheaper than AWS.
When I worked as an IT person for a cheap hotel back in the day, I set up a single Compaq PC as the only server (Samba NT DC, file sharing, IP masquerading, web/mail/DNS server, etc) in a closet. It was not on any battery backup, there was no modem. It ran for years without ever rebooting. In a city where power outages were normal.
Similarly, I've had EC2 instances run for years without ever being rebooted or going down. One of them's still running today after 5 years.
But none of those services were being used 24/7; if the internet went down for an entire weekend, or a hard drive was a little bit corrupt but kept running programs in memory, I probably would never have noticed. I've also had EC2 instances literally just fall off the map and sort of disappear, and had them manually replaced without notice by AWS, had virtual drives fail and corrupt, and had calls to services fail. And I've had my desktop's power supply get fried by a power surge.
Without a lot of experience, running systems seemed easy. But as time went on I learned that it can be easy, and it also can go down if somebody blows on it the wrong way. What we see as being reliable may just be chance. The only way to guarantee reliability is to expect that things are going to go down, and design and build it accordingly.
What AWS makes easy is they give you all the components for reliability, but you have to do the plumbing yourself. I'll bet you the people whose services went down did not properly design for reliability, as they were probably running in one region, in one set of AZs, and relied on distributed system operations that can fail, and didn't properly account for how to deal with those failures. One product I maintain on AWS did not go down, but another did.
Also, the more components a system has, the higher probability there is of failure. Big systems are actually more error prone than small ones.
For any individual company able to offload the blame, that's great. It's not so great if half the countries' doorbells, robot cleaners, various home streaming service setups, the baby camera, the fridge, the smart TV and your phone stop working... All at the same time.
In my opinion, the 'downtime' really should be measured in $NUM_SERVICES_STOPPED X $TIME, instead of just $TIME. And in this case I think any long time Amazon outage is orders of magnitudes worse than your regular old slow IT company outage.
Make things that fail with everyone else. Your customers will just have to get over it because everyone else is down.
News flash:
This is exactly why "overconsolidation" is a bad thing. The bigger, more complex, and integrated the player, the more devastating the eventual failure is as reality dictates you will build the most complex system possible until you outstrip your ability to mentally simulate, reason about, and debug it. You cannot create a thriving, resilient business ecosystem
With everyone flocking to the same players.
Customers must come first. Not your books. Working, resilient solutions. Anything else is LARPing, and kicking the catastrophe can down the road to someone else to pay.
If you happen not to be on the major cloud platform where everybody is enjoying donwtimeshare, we provide you with some downtime monkeys that will pull the lever EXACTLY when your platform should go down!
Let no more have your integrations with systems in the cloud frustrate your customers when they are unreachable - let the news explain the downtime. JOIN the Downtime Umbrella NOW and receive 5 downtime lever pulls for FREE! /s
> being on the cloud means downtime is correlated with downtime of other services. That's usually a good thing.
A long time ago, I used to circulate snarky little emails at work.
One of them was responding to this very concept.
My managers were throwing out our working and mature UNIX servers (implementing DNS, Mail, and other services) in favour of NT.
The new system crashed a lot, we had some security breaches, but at least it was 'industry standard.'
Managements' response was that with the UNIX stuff we had no one to pin our outages on. Now we could blame Microsoft, and call their support line.
I circulated an email making fun of this justification, which promoted a fictional product called 'Blame Studio' which would help you map out the blame path for any of your products or services.
It would help to make sure that none of the blame ever landed on you, but rather was always redirected onto some other company.
Another key point of this: Selecting your downtime. If you own your own stuff, your major changes and maintenance are done at times favorable for your business. AWS or Azure configuration fat-fingers happen when best for their business, not yours.
Agreed. In fact, I've heard government decision makers actually swayed to go to the cloud due to outages with logic along the lines of, "If Amazon can't keep AWS running, what chance does our 30 person, chronically underfunded, team have?" There is also safety in having the outage be somebody else's fault, and having that outage effect a huge swath of the economy - "Sorry customer, but it's not just us." Reminds me a bit of the "nobody gets fired for picking IBM" mentality.
One problem facing on-prem orgs today is the sad state of commodity server hardware; poor quality control, buggy firmware (and vendors who won’t help), BMCs with massive attack surfaces (that have already claimed one VPN provider), and commodity network switches that can push lots of packets but aren’t terribly flexible for people who are pushing more challenging payloads around their DC (video, etc).
"Hyperscalers" like Amazon, Microsoft, Facebook and Google build their own hardware and are able to avoid many of these problems. Unfortunately, none of this stuff is available off the shelf to mere mortals.
There’s a startup trying to fix this problem (Oxide) which I think is launching their racks next year. Will be interesting to see what happens.
> Maybe the cloud has a higher uptime than your on-premise infrastructure (see the AWS, Azure outages).
Not even so sure about that. I've had a ton more downtime ("degraded" in AWS speak) with AWS than any self-hosted systems. And that's with more than half my career on self-hosted.
If a major disaster strikes, like the whole rack catching fire and melting everything, then it's true that AWS could recover quicker than self-hosted. But most problems are not of that sort.
Mine has exceeded AWS' uptime in the last 5 years. I think. I wish I had the actual numbers. If not, it's only because I actually didn't care very much about uptime, and did an in-place system upgrade on the live server late in the evening, and when it broke I waited till the next morning to fix it.
Also, the rather rare and brief maintenance window from the provider is always middle of the night for all my customers.
We're sorry!
An error occurred when we tried to process your request. Rest assured, we're already working on the problem and expect to resolve it shortly.
and now:
This page isn’t working
aws.amazon.com is currently unable to handle this request.
HTTP ERROR 503
Anyone feel like Big Tech outages are happening more frequently recently? In recent months, we’ve seen Amazon Web Services, Facebook, Gmail, and Twitter go down.
Are we at the point where the people who maintain the infrastructure are now completely different than the ones who built it, and are struggling to keep it running because they don’t understand it as well?
I think also just finally getting to the point where they have a tech debt burden comparable to everyone else. Starting fresh has its advantages...for a while.
Even on HN there are some former AWS employees who talk about how its all stitched together and flying on a wing a prayer. Apparently the on-call is just a traumatizing experience. It will take real damage to revenue for the management to pay that debt off.
Is there a case of an organization ever paying off "tech debt" (I refuse the term, I call it incomplete software)? I've only ever seen it snowball until the product falls into the sea and they start again fresh.
I've never seen it. It's kind of like credit card debt. The people that have it get progressively more to pay off the previous debt until they reach bankruptcy.
So I do think tech debt is accurate, in that the average human deludes themselves into thinking they will pay it off at some point.
It seems like they could at least cordon some of it off into a corral of "legacy products". Or maybe a concept of "next gen" regions that have only a subset of products.
You never pay it off. It's like "the war on crime" or "the war on drugs". Don't call it a war if there is no clear win. You can never win the war, but you MUST win the battles in order to not lose it.
A well-maintained car can drive for a very long time, but for a poorly maintained car, restoring it to like-new condition rarely makes financial sense. It's usually cheaper to just buy a new car, and/or do the bare-minimum maintenance until it dies completely (then buy a new car).
I'm not sure tech debt can ever be fully paid down. You can pay off a bit, and you can stop the debt from accumulating further, but the only way to realistically unburden yourself of a really horrible, big pile of tech debt is to just rebuild the thing from scratch. Or don't, and just keep making money while you wait for the business to fail. (From an investor's perspective, this is the equivalent of riding the car into the ground.)
I think the term is fine. I don't don't know if is the "ever accumulating" version is correct as at some point it levels off in my experience. At some point things break and you have to take care of it.
I was thinking something similar. The FAANG interview has been optimized to punish candidates with actual experience in these problems in favor of those that can solve leetcode problems and regurgitate design architectures from youtube. This filters out a lot of engineers with grit that can actually work through tough real world (IT) problems.
Those people, the gritty ones who can't make it through the interview hoops are busy integrating companies into the cloud providers. Now they're taking it easy whilst the new kids are getting a first class lesson in what grit is.
I don't even work at FAANG but even we have our own in-house grown compute platforms with layers and layers of abstractions and enormous complexity which were built over several years by hundreds of engineers. These don't resemble public cloud or open source solutions at all, ideas are similar at best. You can have a lot of real world small company knowledge but the moment you walk through the door this experience is worthless, all that counts is how fast you can adapt and how good your understanding of the basics is.
Look at some of the code that was open sourced by Yahoo years ago. Other large engineering organizations today have technology which far exceeds the complexity of what was made public by Yahoo a decade ago. Unfortunately Yahoo is pretty much the only example of a large company open sourcing big portions of their platform code.
Every service you mentioned (maybe sans Gmail, the only additions I remember in the last twelve years are a new UI and Hangouts) has bolted on so many features over the last years: AWS was virtual machines + SDNs in the beginning as Amazon only intended to sell spare capacity on their own servers, now it's a global one-stop-shop for everything that can be done on the Internet. Facebook was a social media feed, now it's event coordination, groups, chat, image and media hosting at global scale.
And apparently, no one at these organizations ever thought about re-working their infrastructure with "lessons learned over the last decade" in mind. Every new feature was simply bolted on, on top of an infrastructure that was hardly even envisioned to ever become the scale they are today. And that sort of refactoring costs serious amounts of money and developer time, not to mention that it doesn't make sense to develop new features on a code base that's going to be shut down in a year, so management doesn't approve it out of a fear they will be "out-featured" by a competitor and cannot react (=copy, like Instagram's Stories that were a clear rip-off from Snapchat) in time or that their own PKI/OKR goals and with it their bonus payments won't get hit.
That mindset/scale issue is also why IBM mainframes are still so common, why travel PIRs seem to be stuck in formats over half a century old or why "put CSV files on an FTP server" is the standard on bank transfers... big corporate/government clients pay a shitload of money for virtualized mainframes on new hardware that still can run the 70s-era code and even more money for people able to speak COBOL, because that is still cheaper than the alternative - reworking everything from scratch on a modern foundation, testing data integrity and edge cases, revise interfaces to hundreds or thousands of clients. Hell, even Internet standards have the same problem... we are still using protocols like BGP that have been around since before I was born, and tacked on security only a few years ago after a couple of fat-finger incidents.
Modernization in such entities only tends to happen when laws or regulatory frameworks change, and then it can become a real shitshow for those at the lowest rungs of the IT ladder that have to implement them - simply take same-sex marriages and try to shoehorn them into a database that was labeled for "husband and wife", or trans/inter people with gender data represented by a boolean field.
As the layers of the stack increase in depth the probability that any one piece breaks the system approaches one. There will be a Singularity but it will be a horizontal asymptote instead of a vertical one.
maybe we actually have a case for decentralized “web3” networks. At least ones which really need reliability, but can’t can use whatever e.g. airplanes use (telehealth? online exams?)
Of course you need to make sure the network is actually decentralized (see Solana). Or maybe you can just rely on Amazon and GCP and Azure, because surely they won’t all fail at the same time.
The whole of Amazon isn’t falling, just some services in one region, it is easy enough to build region redundant services but most people don’t because the few hours of downtime per year don’t seem worth it.
You usually just don’t need 100% uptime. “Sorry were closed, come back later” is fine.
I wonder if we're hearing the full story. Could they be being attacked by foreign hacker farms in Russia and China just to make the US infrastructure look bad?
256 comments
[ 2.9 ms ] story [ 271 ms ] threadEdit: Seems to be flapping between the 'sorry' error and a blank page. Thoughts and prayers with the SREs, if they call 'em that over there.
[0] https://status.aws.amazon.com/
> Désolés!
> Une erreur s'est produite lorsque nous avons tenté de traiter votre requête. Soyez assuré que nous travaillons déjà à la résolution du problème que nous pensons trouver très rapidement.
> 申し訳ありません。
> リクエストの処理中に問題が発生しました。 現在問題を調査しておりますので、解決するまでもう少々お待ちください。
I mean, you could say they mean we're sorry and please wait, but that's also just wrong in a whole other way.
I'm thinking about it, and I guess there's "politeness levels" in English, but probably not as much so, and usually in a form of passive aggresisveness.
That's fascinating. So the "style" of language they use has a huge impact on the message, possibly more so than the actual words? Very McLuhanesque :)
Richard Feynman talked about this point really frustrating him and turning him off Japanese entirely. He just wanted to use words to get things done...granted for informational purposes it's nice to not have to conjugate the verb for download based on who downloaded it.
it's usually a month or more after a large outage to see the full breakdown on what happened. people expecting to see it the same week are kinda not living in reality.
That direct URL aws.amazon.com gives me a weird error, but everything else seems to be fine.
Maybe the cloud has a higher uptime than your on-premise infrastructure (see the AWS, Azure outages). Make sure to compare the actual outage time v.s. the stats doctored by various political pressures and weaselly worded SLAs (how do you mean you had an outage? Only 49% of your requests were failing!).
Your customers will be more understanding if your outage is part if a wider outage that makes national news.
Any services you integrate with are likely down too. If two services with 99% uncorrelated uptime together drops to about 98%. It doesn't drop if the downtime is perfectly correlated.
Even if you don't directly integrate, your customer's workflow might. They see many services down and say, "cool, time to get caught up on laundry". If only you're down it's more aggrevating.
Yes they definitely are.
Getting higher uptime is super easy for smallish inhouse deployments.
You just don't install any updates and let the server run, trusting your VPN to shield you from possible security issues.
The maintenance burden is the reason why people often prefer the cloud services, not the uptime. Because maintaining the instance with updates, reading all patch notes and steps for migration, keep every health metric monitored and respond quickly on issues without getting stuck googling for possible reasons is quite a bit of work and quickly forces you to employ n+1 people.
We run some bare metal servers. They just never go down. Solid continuous pings for years as monitored from elsewhere on the Internet. That's because they're just boxes on a rack somewhere running an OS and some steady-state services (ZeroTier roots). Simplicity is more robust than complexity.
SaaS is definitely about the pain of managing and upgrading software, but it's also about OPEX vs CAPEX. Many companies will pay more for things to put them in the OPEX column for various entirely synthetic accounting, investor relations, and tax reasons.
I do wonder if the pendulum there will swing back though since if you price out cloud vs. physical hardware the market has become extremely distorted. Many companies spend enough on AWS to buy an entire rack of hardware at a different data center every month and pay 2-3 employees to manage it. That hardware would be up to 100X as fast and powerful as what they rent at AWS and bandwidth would be almost free. That's a really distorted market. The amortized costs should not be this different.
I think the issue here is that it's not zero-cost to switch. Your processes will adapt to some implicit assumptions that aren't true outside AWS, Azure, or whatever vendor you locked yourself into.
If we somehow managed to have a completely standardized interface here, the market would be more competitive.
https://www.openstack.org
They can try to saturate the VPN host maybe, but that's going to be challenging considering that it's going to be limited to connection requests without valid credentials.
and these are likely set to be ignored on multiple failed attempts through fail2ban or similar tooling
1. How easy can I access your physical servers ? 2. What happens if there is a catastrophic failure, for example local power outage or a major flooding 3. How secure is your server? Are you regularly patching your operation systems 4. If I want to run a project that requires double the capacity of your current hardware for a specific project, how long is it going to take to get it spun up?
- Regularly patching is automated and took about 30 seconds to configure, using an automated script
Regarding running your own physical servers, that is a different ballgame, but for all of my projects, if I need to:
- I can pretty easily spin up VPSs / bare metal servers anywhere (netcup, linode, hetzner, etc) and provision there while I wait for new hardware to come in - If you want to double the capacity of your current hardware, you'll have to order it and wait, but it's cheap (vs the major cloud providers) to way over provision if you're running your own physical hardware, so you can pretty easily have 2-4x extra capacity and still come out with extra money in your pocket.
I host in the cloud, but I think people vastly over estimate how much it saves 90% of cloud customers.
There are well-understood answers to all your questions, they are not too difficult, they just cost money - some businesses choose not to spend that money, some weigh cost-benefit and go for AWS, some decide to go for in-house servers, some go for hosted Virtual Servers, some go for serverless.
Why does everything have to be built one way?
I'd also say some choose not to spend the money, but fail to consider the cost of that choice.
For example: Doing old-school manual deployments that require herculean efforts to update at off-hours on the weekends burns people out, and makes it hard to attract new talent. In other words, you've made the decision to spend more money on finding and retaining people. But it's definitely way cheaper to pay for colocating a single Dell server you bought 3 years ago than what you'd spend in the same time on AWS.
And if your hardware never dies, paying for the redundancy might seem silly. A lot like paying for fire insurance despite the fact your house has never even burned down.
Of the options I mentioned, you seem to have picked only self-purchased hardware with no redundancy and no backups (your addition) to compare with, and also throw in manual deployment (why?).
Most businesses are somewhere between a forgotten old dell server in the closet and fully hosted multi-region auto-scaling fully bought in to AWS, and that's OK!
1: Access to the data center requires either an access card plus biometric verification (fingerprint in one DC in my case, retina scan in the other), or an ID-verified appointment. Then, you still need to know where my server is, plus the access code for the rack (or, you need to be an average lockpicker, but beware, there's cameras...).
2: Each data center has dual-provider AC feeds, plus generators, and provides A/B feeds to my rack. I've not had a dual-feed outage in the last 20 years or so.
3: No cloud provider that I'm aware of guarantees server security or does automated patching (for servers, not services). So, keeping your server up-to-date seems equally important for both cloud and non-cloud scenarios?
4: At least two weeks, I guess? I have sufficient VM host capacity to accommodate 30% unplanned growth, but 100% would require new hardware. So: ordering two servers, installing these in two data centers. But if the new project also requires significant bandwidth, getting new Internet connections in might take longer.
Look, I'm definitely not denying that "the cloud" makes it easier to scale fast, but scaling fast is not an overriding concern for most businesses. Cost is, and self-hosting, even with a pretty redundant infrastructure, is still much cheaper than AWS.
Similarly, I've had EC2 instances run for years without ever being rebooted or going down. One of them's still running today after 5 years.
But none of those services were being used 24/7; if the internet went down for an entire weekend, or a hard drive was a little bit corrupt but kept running programs in memory, I probably would never have noticed. I've also had EC2 instances literally just fall off the map and sort of disappear, and had them manually replaced without notice by AWS, had virtual drives fail and corrupt, and had calls to services fail. And I've had my desktop's power supply get fried by a power surge.
Without a lot of experience, running systems seemed easy. But as time went on I learned that it can be easy, and it also can go down if somebody blows on it the wrong way. What we see as being reliable may just be chance. The only way to guarantee reliability is to expect that things are going to go down, and design and build it accordingly.
What AWS makes easy is they give you all the components for reliability, but you have to do the plumbing yourself. I'll bet you the people whose services went down did not properly design for reliability, as they were probably running in one region, in one set of AZs, and relied on distributed system operations that can fail, and didn't properly account for how to deal with those failures. One product I maintain on AWS did not go down, but another did.
Also, the more components a system has, the higher probability there is of failure. Big systems are actually more error prone than small ones.
For any individual company able to offload the blame, that's great. It's not so great if half the countries' doorbells, robot cleaners, various home streaming service setups, the baby camera, the fridge, the smart TV and your phone stop working... All at the same time.
In my opinion, the 'downtime' really should be measured in $NUM_SERVICES_STOPPED X $TIME, instead of just $TIME. And in this case I think any long time Amazon outage is orders of magnitudes worse than your regular old slow IT company outage.
Make things that fail with everyone else. Your customers will just have to get over it because everyone else is down.
News flash: This is exactly why "overconsolidation" is a bad thing. The bigger, more complex, and integrated the player, the more devastating the eventual failure is as reality dictates you will build the most complex system possible until you outstrip your ability to mentally simulate, reason about, and debug it. You cannot create a thriving, resilient business ecosystem With everyone flocking to the same players.
Customers must come first. Not your books. Working, resilient solutions. Anything else is LARPing, and kicking the catastrophe can down the road to someone else to pay.
Let no more have your integrations with systems in the cloud frustrate your customers when they are unreachable - let the news explain the downtime. JOIN the Downtime Umbrella NOW and receive 5 downtime lever pulls for FREE! /s
"my outage means the customer is probably also down so they maybe don't care" is not a viable way to run a business in my mind.
A long time ago, I used to circulate snarky little emails at work.
One of them was responding to this very concept.
My managers were throwing out our working and mature UNIX servers (implementing DNS, Mail, and other services) in favour of NT.
The new system crashed a lot, we had some security breaches, but at least it was 'industry standard.'
Managements' response was that with the UNIX stuff we had no one to pin our outages on. Now we could blame Microsoft, and call their support line.
I circulated an email making fun of this justification, which promoted a fictional product called 'Blame Studio' which would help you map out the blame path for any of your products or services.
It would help to make sure that none of the blame ever landed on you, but rather was always redirected onto some other company.
What would be more valuable to you:
1. 99.5% uptime where unscheduled downtime is max 10 minutes vs.
2. 99.5% uptime where unscheduled downtime comes in 2-5 hour chunks.
These outages have almost zero impact on anyone deciding to use a cloud provider.
"Hyperscalers" like Amazon, Microsoft, Facebook and Google build their own hardware and are able to avoid many of these problems. Unfortunately, none of this stuff is available off the shelf to mere mortals.
There’s a startup trying to fix this problem (Oxide) which I think is launching their racks next year. Will be interesting to see what happens.
Not even so sure about that. I've had a ton more downtime ("degraded" in AWS speak) with AWS than any self-hosted systems. And that's with more than half my career on self-hosted.
If a major disaster strikes, like the whole rack catching fire and melting everything, then it's true that AWS could recover quicker than self-hosted. But most problems are not of that sort.
https://<region>.console.aws.amazon.com/
Google "site: aws.amazon.com" and try any of the links.
Also, the rather rare and brief maintenance window from the provider is always middle of the night for all my customers.
at first:
and now:Are we at the point where the people who maintain the infrastructure are now completely different than the ones who built it, and are struggling to keep it running because they don’t understand it as well?
So I do think tech debt is accurate, in that the average human deludes themselves into thinking they will pay it off at some point.
[1] https://news.ycombinator.com/item?id=29039611
I'm not sure tech debt can ever be fully paid down. You can pay off a bit, and you can stop the debt from accumulating further, but the only way to realistically unburden yourself of a really horrible, big pile of tech debt is to just rebuild the thing from scratch. Or don't, and just keep making money while you wait for the business to fail. (From an investor's perspective, this is the equivalent of riding the car into the ground.)
Look at some of the code that was open sourced by Yahoo years ago. Other large engineering organizations today have technology which far exceeds the complexity of what was made public by Yahoo a decade ago. Unfortunately Yahoo is pretty much the only example of a large company open sourcing big portions of their platform code.
Twitter used to go down so frequently the "fail whale" was a whole cultural thing.
Big AWS outages are rare, but hardly new.
https://www.theregister.com/2017/03/01/aws_s3_outage/
https://arstechnica.com/information-technology/2012/10/amazo...
Every service you mentioned (maybe sans Gmail, the only additions I remember in the last twelve years are a new UI and Hangouts) has bolted on so many features over the last years: AWS was virtual machines + SDNs in the beginning as Amazon only intended to sell spare capacity on their own servers, now it's a global one-stop-shop for everything that can be done on the Internet. Facebook was a social media feed, now it's event coordination, groups, chat, image and media hosting at global scale.
And apparently, no one at these organizations ever thought about re-working their infrastructure with "lessons learned over the last decade" in mind. Every new feature was simply bolted on, on top of an infrastructure that was hardly even envisioned to ever become the scale they are today. And that sort of refactoring costs serious amounts of money and developer time, not to mention that it doesn't make sense to develop new features on a code base that's going to be shut down in a year, so management doesn't approve it out of a fear they will be "out-featured" by a competitor and cannot react (=copy, like Instagram's Stories that were a clear rip-off from Snapchat) in time or that their own PKI/OKR goals and with it their bonus payments won't get hit.
That mindset/scale issue is also why IBM mainframes are still so common, why travel PIRs seem to be stuck in formats over half a century old or why "put CSV files on an FTP server" is the standard on bank transfers... big corporate/government clients pay a shitload of money for virtualized mainframes on new hardware that still can run the 70s-era code and even more money for people able to speak COBOL, because that is still cheaper than the alternative - reworking everything from scratch on a modern foundation, testing data integrity and edge cases, revise interfaces to hundreds or thousands of clients. Hell, even Internet standards have the same problem... we are still using protocols like BGP that have been around since before I was born, and tacked on security only a few years ago after a couple of fat-finger incidents.
Modernization in such entities only tends to happen when laws or regulatory frameworks change, and then it can become a real shitshow for those at the lowest rungs of the IT ladder that have to implement them - simply take same-sex marriages and try to shoehorn them into a database that was labeled for "husband and wife", or trans/inter people with gender data represented by a boolean field.
Of course you need to make sure the network is actually decentralized (see Solana). Or maybe you can just rely on Amazon and GCP and Azure, because surely they won’t all fail at the same time.
You usually just don’t need 100% uptime. “Sorry were closed, come back later” is fine.