This is a relevant comment. I agree. Who will hold them accountable for violating uptime guarantees? Nobody? Then what's the point or purpose of an uptime guarantee? Marketing value?
If it’s in a contract, companies can sue. And big enough customers who have lost enough money due to the outage would definitely threaten to sue to recover that money.
Wouldn't the contract specify the remedy? Or do they actually on paper really promise uptimes that on a single isolated data-center level no one can keep?
We just ask nicely. Never really had a problem getting a huge % discount on the bill because of an outage. Extra bonus for us since there's no bottom line impact since we can tolerate some downtime (just annoying for engineering).
His point is not about aws inability to have 0 downtime. His point is some people apparently pay based on a guarantee for uptime that is not delivered. He is criticizing a company not providing the service it sells and our inability to hold them accountable
Exactly. Try telling your boss when you paid a premium for uptime available that you just wasted a whole lot of money because you aren't getting the uptime you paid for. If AWS can't actually guarantee uptime (maybe no one can), then they need to have in their terms an automatic credit on a per-minute basis for uptime that is not delivered but otherwise paid for in a "guarantee".
Yes but only if you initiate a claim and follow their steps. Check out these onerous terms:
Credit Request and Payment Procedures
To receive a Service Credit, you must submit a claim by opening a case in the AWS Support Center. To be eligible, the credit request must be received by us by the end of the second billing cycle after which the incident occurred and must include:
1. the words “SLA Credit Request” in the subject line;
2. the dates, times, and affected AWS region of each Unavailability incident that you are claiming;
3. the resource IDs for the affected Included Service ; and
4. your request logs that document the errors and corroborate your claimed outage (any confidential or sensitive information in these logs should be removed or replaced with asterisks).
If the Monthly Uptime Percentage of such request is confirmed by us and is less than the Service Commitment, then we will issue the Service Credit to you within one billing cycle following the month in which your request is confirmed by us. Your failure to provide the request and other information as required above will disqualify you from receiving a Service Credit.
This is backwards thinking. Why require customers to file a claim for what are obvious outages? Instead, AWS should automatically apply credits to those accounts that have paid for guaranteed uptime without requiring this whole silly claims process.
The mechanism can be really simple. If AWS themselves posts an outage to their status page and/or some third-party service posts an outage then credits are immediately applied to the services where there are outages for those that paid for high level uptime guarantees without requiring any claims process. It can easily be done if they want to do it that way.
Of course from a business perspective I understand why they're doing it the way that they are. If they can make customers jump through hoops, then only those who really care will follow through. Meanwhile the uptime guarantee can continue as an empty promise.
It's much more likely that the reason is someone going "well what if people want to abuse this?" without any evidence that they would.
Also: requiring your customers to ask for their money back when you know that you didn't deliver the service promised and all other billing is automated.. come on.
When my ISP was unable to provide connectivity for an extended period, it automatically compensated me. I didn't have to do anything. The relevant system was being monitored, the ISP knew exactly when it was out of service, and I was credited accordingly with an apology and a note on my next bill showing the reduction. It doesn't seem unreasonable to expect the biggest name in the cloud to do something similar to support its customers when it screws up.
Why is criticizing on a random forum "holding them accountable". Its pretty simple to hold them accountable, ask your service rep for refund based on contract SLA. Anything else is just soapbox grandstanding for the purpose of internet karma points
Ughh, he claims to have been scammed by AWS because their services are having an outage, and I'm missing the point?
This stupid hyperboles need to be shot down. I'm sick and tired of the victim mentality and hyperboles. Every time something inconvenient happens, people scream and shout at the top of their lungs like the world has wronged them.
NO, YOU DID NOT GET SCAMMED BY AMAZON BECAUSE THEY HAVE A SERVICE OUTAGE.
Simple as that. People need to calm down and stop acting like the world owes them something. Unforeseen events happen. Take a breath, no one scammed you. If you have an SLA and contract, follow the steps and process to get reimbursed. Anything more is just worthless bickering and victim mentality complaining.
> NO, YOU DID NOT GET SCAMMED BY AMAZON BECAUSE THEY HAVE A SERVICE OUTAGE.
Many people have pushed for cloud services because they are supposed to be more reliable than setting up a system in a rack in a datacenter. AWS will constantly point to their uptime guarantee, except it isn't a guarantee. It's just a sales tactic that misrepresents the historical uptime of AWS.
The larger point is that if 99.5% is the real expected uptime, it's vastly cheaper to have a solution that is not AWS, even before you factor in the cost savings of learning their security model and completely opaque billing system.
Advertising a product with features it does not have is the classic definition of a scam.
You could, at rush hour, drive to microcenter, buy all the components, drive to home depot and buy a generator and gas can, go back to the office and assemble everything, fill and turn on the generator...in less time than this outage has lasted. I think generally more people are concerned about the scope and length of the outage, rather than that outages occur. And in this instance the fact that they aren't admitting to their downtime at all...
7:30 AM PST: We are currently blue on Kinesis, Cognito, IoT Core, EventBridge and CloudWatch given an increase in errors for Kinesis in the US-EAST-1 Region. It's not posted on SHD as the issue has impacted our ability to post there. We will update this banner if there continue to be issues with the SHD.
This has happened a few times before, actually. Dogfooding is good, but not for status pages!
Cloudflare does it right for their status page (https://www.cloudflarestatus.com). They don't use Cloudflare itself for it (you can tell because /cdn-cgi/trace returns nothing), the actual backend is Atlassian Statuspage, their TLS certificate is issued by Let's Encrypt instead of Cloudflare itself, and it's on a completely separate domain for DNS purposes.
Reminds me of a recent outage from IBM Cloud, where the VPN was hosted on IBM Cloud so employees couldn’t log in to fix it, and the email is hosted on IBM Cloud so support teams couldn’t email customers to let them know and even access to their Twitter was behind the non-functional VPN so they couldn’t tweet during the outage either.
>[customers] were texting with their account managers, because the account managers had no access to any internal systems. Reportedly, the corporate VPN was not working. My thesis is... everything was single-tracking through a corporate VPN that itself was subject to this disruption... their traditional tweets have been done through an enterprise social media client called Sprinklr
Banner on top of https://status.aws.amazon.com/ just has an update from 8:36AM PST -- just removed -- even thought it's only 7:42AM PST. I guess it's really manual firefighting there.
Isn't it common practice to host your status board on someone else's infrastructure?
In 2017 there was an S3 issue that supposedly affected their ability to post. I believe they said that they were updating how they posted to the status board so that there would no longer be a dependency on S3. Well, I guess whatever they're dependent on now broke.
It's common practice for small players but Amazon, Microsoft Azure and Google Cloud host their status pages on their own servers because they value the marketing aspect higher than a functioning status page for their customers.
I find it surprising how many people forget how much underlying business motives drive pretty much every action they make and how this is quickly forgotten by many.
No matter how much you value science and engineering, it ultimately doesn't matter to the business unless that aligns directly with their revenue stream. Sometimes it does, sometimes it doesn't.
Yes. But I wonder if self-hosting their status page is really the correct decision from a marketing perspective. The people who consumes the status page on say Google Cloud probably know that Google self-hosting it is a bad decision from a technical point of view. So to the only people who care, their choice appear stupid.
So I don't really understand what they gain by doing it. I think maybe I am wrong about it being a marketing concern and that the choice is more related to internal politics and incompetent management.
The point is to manage potential external liabilities. A business doesn't want any sort of liability they have automatically costing them if they can avoid it. They're more than happy to have anything that profits them automatically generate revenue, but if something could potentially lose them thousands or millions, they want to make sure there's a human-in-the-loop from management to check off. Not meeting SLAs or service outages are a good way to cost them money.
Few companies really respect their engineering teams/divisions in any sensible form from my experience, though I'm biased (even in heavy R&D environments). You're simply a means to an ends.
I understand your point though (and identify with it), but I find any mechanism/option that provides a way of containing potentially damaging information is going to be pushed by management over the option to release damaging information that a responsible engineer may want to disclose.
You're in a culture where admitting fault or liability is like pulling teeth and ripping finger nails off. It shouldn't be IMHO (we should own up to our mistakes and be reasonably forgiven), but that's unfortunately not the culture we have.
Reminds me of: "When a measure becomes a target, it ceases to be a good measure"
When you're advertising uptime/availability, you're motivated not to report downtime/unavailability. Then the value of such reports is lost; developers start banging their heads trying to figure out if it's a service outage or a bug in their software (yes, informed by personal experience).
I operate StatusGator, which is a service that aggregates status pages so I'm ALL TOO familiar with the AWS status page.
The main change they made in 2017 was the ability to post a message at the top of the page that is independent of the status of the individual items below. IIRC, it was the items they couldn't update. So that is kind of a hack, but it works.
It would be ideal if it was host entirely on completely separate infrastructure, and even a separate domain, but I won't hold my breath. Theirs is still more reliable than, for example, the IBM Cloud status page which was hard down during their epic outage back in June.
I think the issue is that Kinesis is a single point of failure for a ton of systems. When it goes down, loads of other system's workflows can't operate. AWS is famous for eating their own dog food and someone just poisoned it.
EventBridge has been struggling for about the past 14 hours as well, which means Cloudwatch Events is not too happy; and, I have the impression CWE underpins a surprising diversity of other things at AWS.
Can anyone explain why status pages are so difficult. Theres even statups like status.io dedicated to this one thing.
It really does seem that anytime there is an outage more often than not the status page is showing all green traffic lights. Making it redundant as a tool to corroborate whats happening.
How did AWS status page compare with status.io/aws?
And not just outages, but security incidents. I’ve worked at/with/for many companies as both an employee and a consultant where the top priority wasn’t to have fewer security incidents, but to have fewer security incidents that would require disclosure.
Publicly disclosing an incident to a customer is embarrassing and potentially damaging but almost equally as damaging is telling other teams you had an incident. Now anything that goes wrong is your fault by default because “it’s probably related to that incident” and any new security policies are blamed on the other team: “we wouldn’t have to do that if Ops didn’t mess up last month”.
The answer to “is this service suffering an outage” is seriously complex and hard to determine. The answer to “is this a security incident” is 10x harder and 100x more political because the industry is still just so wildly immature.
Additionally, you're penalized for doing it "right", because you're often competing against companies which rarely say that anything's wrong (ahem, Mailchimp). You look worse, because you're being transparent about service status, which creates the perception that you're generally less stable.
Many companies tie uptime and outages to performance reviews, either directly or indirectly.
Admitting that your services are down could be costly to your career progression and bonus. When people know this, they go to great lengths to avoid admitting fault. Updating the status page is the first admission of fault. The longer the status page shows an outage, the worse it gets.
I worked with an ex-Amazon engineer at a previous company. After each outage, he would spend days or weeks writing long reports explaining how the outage was not his fault. He didn't care so much about downtime so much as not getting blamed for outages. Predictably, this was terrible for team morale and most of his team members ended up quitting.
If anyone else finds themselves in this position, the solution is have another team responsible for monitoring uptime, and to rate teams on how quickly they acknowledge outages. Once the response time and accuracy of your status page becomes a performance metric, people are less likely to play games with it.
It is kind of perplexing that AWS dogfoods its own status page. I remember during the massive S3 outage a few years ago that their status page remained green almost the entire time because the red/green/blue icons for the status was stored in... wait for it... S3.
They did. It came up in the post incident report, and senior leadership kicked off work to have it run on its own distinct infrastructure so that this wouldn't happen again.
K so they avoided that problem, but something similar has obviously gone wrong again, considering that Kinesis had been partially or fully down for almost an hour before the status page got their first update.
And the fact remains that currently an outage of AWS's own infrastructure is impacting AWS's ability to status updates on its own status dashboard. It's just seems so... amateurish.
That's incredibly annoying, given the mandate the replacement service had.
I'd be curious to be a fly on the wall during the next Ops meeting when it comes up that yet again the status dashboard got made in a way that makes it hard to update during an outage.
Part of the problem is, engineers love shiny things.
Status pages are fundamentally boring things. Who wants to work on them?
It's always tempting to complicate something simple because in part "ooh shiny", and you can always find reasons to justify why. It takes some strong engineering leadership to effectively argue against complicating things, and not be just a constant pain in the arse to everyone and every thing.
The kinds of people that are that good, tend to be people that aren't going to want to do something so boring as build and maintain the infrastructure for hosting status pages.
>Part of the problem is, engineers love shiny things.
Status pages are fundamentally boring things. Who wants to work on them?
I would work on a status page. It's a interesting problem, creating tests that prove services are viable at a place like AWS would be fun. However what I don't want to deal with is some director of so and so I never heard of yelling at me at 3 in the morning because my status page reported that his service was down accurately. I suspect that plays more into the problem. The status page is a political implement not a technical one.
Congratulations, you're already complicating the status page.
The status page shouldn't be figuring out what the status of any service is. It's impossible to do without a lot of contextual information about a service and understanding how to evaluate service impact, something that is continually in flux.
It just needs to be a page that is updated manually. AWS has a 24x7 incident management team that could / should do it.
I'm afraid you're shifting the complexity to a manual process.
I agree that it doesn't have to, and perhaps should not, be fully automated. But automating some parts will help not waste time on last minute arguments.
> Updated manually by whom?
> I'm afraid you're shifting the complexity to a manual process.
You're right, that's 100% what I'm doing. Why? Because it shouldn't be that complicated to update an overall health status page during an outage event, and it shouldn't take other tools and services within AWS to do it.
A common pattern in cloud providers (including AWS) is that services have some kind of tiering, whereby you can't pick up a dependency on any service on a lower tier than yourselves. Tier 2 services can't rely on Tier 3 services, etc. Services like, say, IAM, would be right at the very top. It can't rely on EBS, ELB etc. Everything has to be created in-service, because everything ultimately has to rely on authentication working.
If they're going to keep an overall status page going, it needs to be seen as a top tier service, just like identity is. That's where they were headed towards when I left AWS about 5 1/2 years ago. It had been spurred by a previous major incident couldn't be reflected in the status dashboard because of a failure in a dependency.
> I agree that it doesn't have to, and perhaps should not, be fully automated. But automating some parts will help not waste time on last minute arguments.
I go in to a bit more detail in another comment within this discussion, but a status page does not even close to accurately capture the ways that cloud environments fail, which are very, very rarely affecting more than a small percentage of customers, and even then often in some very specific way under specific circumstances. That's why AWS built the personalised status page service. They want to ensure that customers have an accurate way of telling what is going on with services they're consuming, rather than the confusing situation of checking an overall status site that doesn't really reflect their experience and never could.
Situations like today's where it at least (from the outside) seemed like Kinesis was completely down, would be a good example of something that should be reflect in the main overall status page.
The status page should be manual, and should be something the incident management team can do (and have political ability to force it to happen, rather than being subject to the whims of service directors)
Except they posted this:
7:30 AM PST: We are currently blue on Kinesis, Cognito, IoT Core, EventBridge and CloudWatch given an increase in error rates for Kinesis in the US-EAST-1 Region. It's not post on SHD as the issue has impacted our ability to post there. We will update this banner if there continue to be issues with the SHD.
Any time companies have SLA's where money is on the line if they admit they're having an outage, they're going to be delayed on updating a status page.
It says on this outage page (as of 11:11 ET) that the problem with Kinesis is also causing problems updating this outage dashboard which may explain the delay?
Status pages, like SLAs, are sales tool - not engineering tools. At best, they are there to help decision makers go through their checklist. At worse, they exist to deceive.
> Can anyone explain why status pages are so difficult.
What is an outage? When does an outage reach sufficient scale that updating the status page is the right thing to do?
I used to work for AWS, and now work for another cloud provider.
One thing that's hard to communicate is the sheer scale that these services operate at, what that means architecturally, and how they tend to break.
Outages, even just slight degradation, occurring on a whole service scale are very rare. I would argue from my experiences there that most incidents affect less than 10% of any given service's customers. Whether it gets noticed in part depends on who is encompassed by that percentage.
What is very often the case is that a subset of customers get impacted to some degree during any given incident. That can be even things like single percentage of customers or less, but be an incident that has all hands to deck and the entire management chain of the service aware and involved in.
At what percentage do you draw the line and say "Yes we need this many percentage of our customers to be affected before we post a green-i" (AWS terminology for the first stage of failure notification).
How do you communicate that effectively to customers, in such a way that doesn't suggest your service is unreliable when it really isn't.
The moment you post a green-i or above, customers start blaming you and your service for problems with their infrastructure that are not caused by it. If you're looking to use a service and go look at the status history and see it filled with green-i or similar, are you likely to trust it? No. Even if those green-i's were for impacts on a limited subset of customers.
AWS wrestled with this a bunch about 5-6 years ago. There were no end of discussions during the weekly ops meetings with senior leadership, directors and engineers across the company. Everyone wants to do the right thing and make sure customers get an accurate picture about the health of the service, without giving the wrong impression.
In the end they opted to move towards having personal notifications for outages, and build tooling to help services quickly identify which customers are being affected by any particular incident and provide personalised status pages for them that can be way more accurate than any generalised status page.
Not really. People will automatically assume they were in that impacted percentage and that what was happening with their stuff was entirely AWS's fault.
Exactly this. I work for a cloud provider and there has been a ton of push in the last year or so to develop customer communication teams and involve them at the first inkling of an outage. We can identify the subset of customers affected and contact them directly. Just publically saying there’s an outage would cause much more chaos.
Try undercutting it. At some point you’ll learn that the problem isn’t that simple, operations is a key part of the product and isn’t free, and people expect support for important services.
Except, the option to ping a service in order to programmatically inform a status page is almost never used. The dirty secret of status pages is that they are almost always manually updated, typically only when a very high bar is met, and after senior managers, sometimes even comms people, approve it.
They aren't difficult. Amazon has no interest in having a working status page. Amazon would prefer the appearance of always green checkmarks over actually having a status page.
This is why my instinct is to check Twitter feeds of the related service first. So far in several years of experience it has been more informative and helpful than a status has ever been. It's a sad state.
It's not that easy to quantify how down a service is at the scale of AWS, for example Cognito has issues, does it means every services that rely on that have issues, what is the impact etc...
I work at AWS. I can tell you surely enough it's not pretty or easy to work with. Design and architecture are great here but implementation of that is pretty crap...
Because business authorised it's use. The final say on using AWS doesn't belong to tech but busines and AWS is very good at the sales game. I went to one of their conferences and it was mostly business people and sales pitches.
Thats too bad, I always imagined the backend was as magical as what AWS users see. I still wish I could have a peek at how S3 works or IAM. Not enough to get a job at AWS - I know they'd fire me the first time I left early for a parent teacher conference or took a sick day, so why put myself in that position.
The only thing magical about AWS' backend is how much manpower they can throw at things.
Amazon doesn't have a good engineering culture. It's all about shipping things as fast as possible. People get promoted an leave for other teams, and the new folk gets burned out due to on-call load while trying to fix crappy software they have inherited.
Always seems to be the case -- this happened before where the status pages updates were stored in ... S3. It goes beyond coincidence when this happens several times in a row.
I think the other explainations sound plausible. There is no technical difficulty here that AWS can't solve -- it's political. Having an outage with a status page makes you liable for your SLAs.
But some global services run through us-east-1 - eg Cloudfront is now broken too. So this is also affecting users who don't actually run anything in us-east-1 explicitly (or in the US at all)
I'm not seeing any issues here yet with S3 images/website buckets stored in eu-west-1 and served by CloudFront.
You're right that there's definitely some internal coupling though:
> If you want to require HTTPS between viewers and CloudFront, you must change the AWS Region to US East (N. Virginia) in the AWS Certificate Manager console before you request or import a certificate.
Existing cloudfront is indeed fine. But creating or deleting distributions fails now.
(I think it's also pretty rare for an already configured cloudfront to suffer from issues on the control planes. Cloudfront configuration updates are painfully slow even under normal circumstances, and that's probably because the configuration is heavily replicated to all POPs)
> This is also causing issues with Amplify, API Gateway, AppStream2, AppSync, Athena, Cloudformation, Cloudtrail, Cloudwatch, Cognito, DynamoDB, IoT Services, Lambda, LEX, Managed BlockChain, S3, Sagemaker, and Workspaces.
Indeed, we had the first AWS Kinesis issues already at 13:50 (UTC). Now it's still ongoing after two hours. The status page didn't even update in the first 45 min or so...
That's typical. The AWS status page is a marketing gimmick whose job is to stay green, not a good faith attempt to assess and report status. If there's an outage, seeing it accurately reflected on the status page is the exception, not the rule.
Updating the status dashboard is pretty low priority for operators trying to resolve this issue. It requires escalation up the management chain and careful wording.
By design. If it was a good faith attempt to report status, it would be automatically updated from a flock of canaries instead of through a slow, political process.
Even that would be meaningless at the scale of AWS.
"A top of rack switch let out the blue smoke and it'll be ~30 before we can re-rack it" would impact what fraction of a fraction of a percent of canaries? Irrelevant to me, unless of course my VM lives on a box backed by that switch. ;)
The status dashboard exists for us to laugh at when things break and to convince C*Os that everything is fine. That's it.
Ehhh... the ratio of "bump in the night" problems that affect just me to genuine outages that cross regions and affect others is about 1:1, and then about 1 in 4 or 5 of the cross-region problems blow up to the scale where they feel forced to update the dashboard. So I disagree, I think a canary flock would be both meaningful and useful.
As you point out, though, the status dashboard isn't truly meant to be either of those things. I don't have any illusions about it ever changing.
It's a shame Amazon doesn't have thousands of employees to divide these tasks between different people, as it is only these busy operators who could update this status page.
If you're right, why have the status page then? It is useless by your definition yes?
Just because it has a lag from “issues reported” to “confirmed outage” doesn’t mean it’s useless. Non-green means there are issues and Amazon is aware of them.
My comment was in the context of the assertion that a team that knows the service is down and is fixing it is too busy to update the status, therefore no one else can update this status. Certainly it is understandable that if the issue is unknown that status cannot be updated.
Literally 99.9% of the employees have no more knowledge than you about the inner workings of a given AWS service. This isn't to forgive their lack of updating the status page, but large engineering orgs are never the knowledge monoliths you might imagine they are.
This isn't a question of knowing a service is down. We're assuming the team that is fixing the service, knows it is down. It was a question of not having the resources to direct literally any other person in the org to log into an admin panel and flip a toggle from green to red.
I was merely addressing the "thousands of employees" non sequitur. Org structure means that the raw number of employees is a meaningless metric. The only people who are going to potentially flip that switch are going to have some sort of direct responsibility for the product. That number is going to be very similar whether it's a large company like Amazon or a smaller one like, say, Heroku or Dreamhost.
The issue has nothing to do with a lack of manpower to flip the switch, whether it's 1000s or 5 people, is the point.
People responsible for the product should not have say over the switch being flipped, for obvious reasons (illustrated in other comments in this thread).
> Updating the status dashboard is pretty low priority for operators trying to resolve this issue
Which is why, during incident responses, there has to be people in charge of communication. Both internal and external communication, and some of this can be further delegated.
That's a poor excuse.
> It requires escalation up the management chain and careful wording
Careful wording is more important for external stakeholders who might not have the full context. If one is stepping in eggshells with internal management too, that's bad management. Incident communication should be factual and concise.
> Incident communication should be factual and concise.
Could not agree more. It's immensely frustrating working with organisations that spend more time trying to cover up the cause of a outage to external stakeholders than actually fixing the root cause.
The same organisations tend try and blame individuals for outages.
I think both are a symptom of businesses that embrace the "blame culture"
Fargate console is reporting no capacity in us-east-1 which is a bummer because I've lost several services that got spun down apparently due to missing cloudwatch data. But EC2 appears to be working though its taking noticeably longer to create resources. I think the take-away for a lot of people is that multiple availability zones is not a substitute for proper BCP that encompasses multiple regions or cloud providers.
We're also seeing issues with FarGate ECS -- the task we had with auto-scaling scaled down to 0. The one we had with a fixed number of workers is fine.
Thanks for this. My Lambda@Edge function was not working and I thought I broke something my permissions even though I had not touched that for atleast a month. This is the very "helpful" error message
The Lambda function associated with the CloudFront distribution is invalid or doesn't have the required permissions. We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.
If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.
That's a tough one -- I'm usually with you that it's always either DNS or cert expiry, but my go-to "it's always ..." when discussing AWS is: it's always security groups
Heh, maybe they accidentally locked themselves out of IAM, since those are great fun to troubleshoot, also
369 comments
[ 2.0 ms ] story [ 257 ms ] threadI guess the lawyers of those who paid for uptime guarantees...
host on-premise then, and deal with power outages, and infrastructure management, and failing hardware
shit happens, drink a coffee and sit back for 30 mins
https://aws.amazon.com/legal/service-level-agreements/
Credit Request and Payment Procedures
To receive a Service Credit, you must submit a claim by opening a case in the AWS Support Center. To be eligible, the credit request must be received by us by the end of the second billing cycle after which the incident occurred and must include:
1. the words “SLA Credit Request” in the subject line;
2. the dates, times, and affected AWS region of each Unavailability incident that you are claiming;
3. the resource IDs for the affected Included Service ; and
4. your request logs that document the errors and corroborate your claimed outage (any confidential or sensitive information in these logs should be removed or replaced with asterisks).
If the Monthly Uptime Percentage of such request is confirmed by us and is less than the Service Commitment, then we will issue the Service Credit to you within one billing cycle following the month in which your request is confirmed by us. Your failure to provide the request and other information as required above will disqualify you from receiving a Service Credit.
There's most likely a reason for this.
Like, maybe in the past AWS customers have tried claiming for SLA credits for incidents that didn't impact them, in order to reduce their bill.
The mechanism can be really simple. If AWS themselves posts an outage to their status page and/or some third-party service posts an outage then credits are immediately applied to the services where there are outages for those that paid for high level uptime guarantees without requiring any claims process. It can easily be done if they want to do it that way.
Of course from a business perspective I understand why they're doing it the way that they are. If they can make customers jump through hoops, then only those who really care will follow through. Meanwhile the uptime guarantee can continue as an empty promise.
Also: requiring your customers to ask for their money back when you know that you didn't deliver the service promised and all other billing is automated.. come on.
This stupid hyperboles need to be shot down. I'm sick and tired of the victim mentality and hyperboles. Every time something inconvenient happens, people scream and shout at the top of their lungs like the world has wronged them.
NO, YOU DID NOT GET SCAMMED BY AMAZON BECAUSE THEY HAVE A SERVICE OUTAGE.
Simple as that. People need to calm down and stop acting like the world owes them something. Unforeseen events happen. Take a breath, no one scammed you. If you have an SLA and contract, follow the steps and process to get reimbursed. Anything more is just worthless bickering and victim mentality complaining.
Many people have pushed for cloud services because they are supposed to be more reliable than setting up a system in a rack in a datacenter. AWS will constantly point to their uptime guarantee, except it isn't a guarantee. It's just a sales tactic that misrepresents the historical uptime of AWS.
The larger point is that if 99.5% is the real expected uptime, it's vastly cheaper to have a solution that is not AWS, even before you factor in the cost savings of learning their security model and completely opaque billing system.
Advertising a product with features it does not have is the classic definition of a scam.
Never trust that. Deploy in multiple regions (and AZs within those regions) if you really cannot tolerate any downtime.
Was posted 8 minutes ago.
Is that not a massive catch-22 for a service dashboard?
Cloudflare does it right for their status page (https://www.cloudflarestatus.com). They don't use Cloudflare itself for it (you can tell because /cdn-cgi/trace returns nothing), the actual backend is Atlassian Statuspage, their TLS certificate is issued by Let's Encrypt instead of Cloudflare itself, and it's on a completely separate domain for DNS purposes.
Do you have a link for more details?
The relevant bit:
>[customers] were texting with their account managers, because the account managers had no access to any internal systems. Reportedly, the corporate VPN was not working. My thesis is... everything was single-tracking through a corporate VPN that itself was subject to this disruption... their traditional tweets have been done through an enterprise social media client called Sprinklr
"amazon-cognito-identity-js": "^3.2.2" "aws-amplify": "^2.2.2"
Then the status page would be almost entirely useless ...
And request timeouts against cognito-idp.us-east-1.amazonaws.com
And the cognito console won't load
In 2017 there was an S3 issue that supposedly affected their ability to post. I believe they said that they were updating how they posted to the status board so that there would no longer be a dependency on S3. Well, I guess whatever they're dependent on now broke.
No matter how much you value science and engineering, it ultimately doesn't matter to the business unless that aligns directly with their revenue stream. Sometimes it does, sometimes it doesn't.
So I don't really understand what they gain by doing it. I think maybe I am wrong about it being a marketing concern and that the choice is more related to internal politics and incompetent management.
If they self-host it, it signals that they're overconfident in their ability to maintain an accurate status page.
Given these two options, which do you think a budget manager will have an easier time signing off on and defending upward?
Few companies really respect their engineering teams/divisions in any sensible form from my experience, though I'm biased (even in heavy R&D environments). You're simply a means to an ends.
I understand your point though (and identify with it), but I find any mechanism/option that provides a way of containing potentially damaging information is going to be pushed by management over the option to release damaging information that a responsible engineer may want to disclose.
You're in a culture where admitting fault or liability is like pulling teeth and ripping finger nails off. It shouldn't be IMHO (we should own up to our mistakes and be reasonably forgiven), but that's unfortunately not the culture we have.
When you're advertising uptime/availability, you're motivated not to report downtime/unavailability. Then the value of such reports is lost; developers start banging their heads trying to figure out if it's a service outage or a bug in their software (yes, informed by personal experience).
Luckily my company decided against multi-az for the cost savings so I spent all day firefighting.
The main change they made in 2017 was the ability to post a message at the top of the page that is independent of the status of the individual items below. IIRC, it was the items they couldn't update. So that is kind of a hack, but it works.
It would be ideal if it was host entirely on completely separate infrastructure, and even a separate domain, but I won't hold my breath. Theirs is still more reliable than, for example, the IBM Cloud status page which was hard down during their epic outage back in June.
- 7 cloudfront distributions created today are still in "InProgress", a few already for more than one hour
- The support case I created about it doesn't show up in my support portal. Direct link to it does work though
It really does seem that anytime there is an outage more often than not the status page is showing all green traffic lights. Making it redundant as a tool to corroborate whats happening.
How did AWS status page compare with status.io/aws?
Failure happens at the speed of computing but agreeing that something is failing in a way that customers need to be told about is a slower process.
Even when status pages are fully automatic (rather than manually updated), there will tend to be gaming of the metrics that constitute that.
Ideally you would just be monitoring your SLOs and publishing that to customers... that doesn't seem to be how it works, anywhere.
Publicly disclosing an incident to a customer is embarrassing and potentially damaging but almost equally as damaging is telling other teams you had an incident. Now anything that goes wrong is your fault by default because “it’s probably related to that incident” and any new security policies are blamed on the other team: “we wouldn’t have to do that if Ops didn’t mess up last month”.
The answer to “is this service suffering an outage” is seriously complex and hard to determine. The answer to “is this a security incident” is 10x harder and 100x more political because the industry is still just so wildly immature.
Admitting that your services are down could be costly to your career progression and bonus. When people know this, they go to great lengths to avoid admitting fault. Updating the status page is the first admission of fault. The longer the status page shows an outage, the worse it gets.
I worked with an ex-Amazon engineer at a previous company. After each outage, he would spend days or weeks writing long reports explaining how the outage was not his fault. He didn't care so much about downtime so much as not getting blamed for outages. Predictably, this was terrible for team morale and most of his team members ended up quitting.
If anyone else finds themselves in this position, the solution is have another team responsible for monitoring uptime, and to rate teams on how quickly they acknowledge outages. Once the response time and accuracy of your status page becomes a performance metric, people are less likely to play games with it.
You'd think they would have learned from that.
If you look at where the content on https://status.aws.amazon.com/ is actually hosted from you'll see things like the status icons are all hosted under the same domain, e.g. https://status.aws.amazon.com/images/status1.gif https://status.aws.amazon.com/images/status0.gif etc.
If you look at the source code for the site, you'll again see that everything is hosted from the same domain.
One of their main goals was to ensure that it could never go wrong that way again.
And the fact remains that currently an outage of AWS's own infrastructure is impacting AWS's ability to status updates on its own status dashboard. It's just seems so... amateurish.
I'd be curious to be a fly on the wall during the next Ops meeting when it comes up that yet again the status dashboard got made in a way that makes it hard to update during an outage.
Status pages are fundamentally boring things. Who wants to work on them?
It's always tempting to complicate something simple because in part "ooh shiny", and you can always find reasons to justify why. It takes some strong engineering leadership to effectively argue against complicating things, and not be just a constant pain in the arse to everyone and every thing.
The kinds of people that are that good, tend to be people that aren't going to want to do something so boring as build and maintain the infrastructure for hosting status pages.
I would work on a status page. It's a interesting problem, creating tests that prove services are viable at a place like AWS would be fun. However what I don't want to deal with is some director of so and so I never heard of yelling at me at 3 in the morning because my status page reported that his service was down accurately. I suspect that plays more into the problem. The status page is a political implement not a technical one.
The status page shouldn't be figuring out what the status of any service is. It's impossible to do without a lot of contextual information about a service and understanding how to evaluate service impact, something that is continually in flux.
It just needs to be a page that is updated manually. AWS has a 24x7 incident management team that could / should do it.
I'm afraid you're shifting the complexity to a manual process.
I agree that it doesn't have to, and perhaps should not, be fully automated. But automating some parts will help not waste time on last minute arguments.
You're right, that's 100% what I'm doing. Why? Because it shouldn't be that complicated to update an overall health status page during an outage event, and it shouldn't take other tools and services within AWS to do it.
A common pattern in cloud providers (including AWS) is that services have some kind of tiering, whereby you can't pick up a dependency on any service on a lower tier than yourselves. Tier 2 services can't rely on Tier 3 services, etc. Services like, say, IAM, would be right at the very top. It can't rely on EBS, ELB etc. Everything has to be created in-service, because everything ultimately has to rely on authentication working.
If they're going to keep an overall status page going, it needs to be seen as a top tier service, just like identity is. That's where they were headed towards when I left AWS about 5 1/2 years ago. It had been spurred by a previous major incident couldn't be reflected in the status dashboard because of a failure in a dependency.
> I agree that it doesn't have to, and perhaps should not, be fully automated. But automating some parts will help not waste time on last minute arguments.
I go in to a bit more detail in another comment within this discussion, but a status page does not even close to accurately capture the ways that cloud environments fail, which are very, very rarely affecting more than a small percentage of customers, and even then often in some very specific way under specific circumstances. That's why AWS built the personalised status page service. They want to ensure that customers have an accurate way of telling what is going on with services they're consuming, rather than the confusing situation of checking an overall status site that doesn't really reflect their experience and never could.
Situations like today's where it at least (from the outside) seemed like Kinesis was completely down, would be a good example of something that should be reflect in the main overall status page.
The status page should be manual, and should be something the incident management team can do (and have political ability to force it to happen, rather than being subject to the whims of service directors)
(SHD being the Service Health Dashboard)
> You'd think they would have learned from that.
They did.
The page has been updated numerous times since the start of this incident.
> This issue has also affected our ability to post updates to the Service Health Dashboard.
Just seems so ridiculous that they have trouble reporting the impaired status of their system due to... the impaired status of that same system.
Which makes me wonder, why do we all rely on status pages rather than solve the problem ourselves in ways that don't require us to rely on the vendor?
What is an outage? When does an outage reach sufficient scale that updating the status page is the right thing to do?
I used to work for AWS, and now work for another cloud provider.
One thing that's hard to communicate is the sheer scale that these services operate at, what that means architecturally, and how they tend to break.
Outages, even just slight degradation, occurring on a whole service scale are very rare. I would argue from my experiences there that most incidents affect less than 10% of any given service's customers. Whether it gets noticed in part depends on who is encompassed by that percentage.
What is very often the case is that a subset of customers get impacted to some degree during any given incident. That can be even things like single percentage of customers or less, but be an incident that has all hands to deck and the entire management chain of the service aware and involved in.
At what percentage do you draw the line and say "Yes we need this many percentage of our customers to be affected before we post a green-i" (AWS terminology for the first stage of failure notification).
How do you communicate that effectively to customers, in such a way that doesn't suggest your service is unreliable when it really isn't.
The moment you post a green-i or above, customers start blaming you and your service for problems with their infrastructure that are not caused by it. If you're looking to use a service and go look at the status history and see it filled with green-i or similar, are you likely to trust it? No. Even if those green-i's were for impacts on a limited subset of customers.
AWS wrestled with this a bunch about 5-6 years ago. There were no end of discussions during the weekly ops meetings with senior leadership, directors and engineers across the company. Everyone wants to do the right thing and make sure customers get an accurate picture about the health of the service, without giving the wrong impression.
In the end they opted to move towards having personal notifications for outages, and build tooling to help services quickly identify which customers are being affected by any particular incident and provide personalised status pages for them that can be way more accurate than any generalised status page.
Cheaper than GCP. Still less crappy than Azure.
Amazon doesn't have a good engineering culture. It's all about shipping things as fast as possible. People get promoted an leave for other teams, and the new folk gets burned out due to on-call load while trying to fix crappy software they have inherited.
Why don't the new folk iteratively refactor their systems to remove operational burden? Isn't that part of owning any codebase you didn't write?
Last sentence of the alert at the top of the page.
I think the other explainations sound plausible. There is no technical difficulty here that AWS can't solve -- it's political. Having an outage with a status page makes you liable for your SLAs.
You're right that there's definitely some internal coupling though:
> If you want to require HTTPS between viewers and CloudFront, you must change the AWS Region to US East (N. Virginia) in the AWS Certificate Manager console before you request or import a certificate.
From https://docs.aws.amazon.com/AmazonCloudFront/latest/Develope...
(I think it's also pretty rare for an already configured cloudfront to suffer from issues on the control planes. Cloudfront configuration updates are painfully slow even under normal circumstances, and that's probably because the configuration is heavily replicated to all POPs)
Well, this is a major outgage
edit: not sure why my question deserved a downvote...
EDIT: 15 minutes later and the board is looking worse again.
"A top of rack switch let out the blue smoke and it'll be ~30 before we can re-rack it" would impact what fraction of a fraction of a percent of canaries? Irrelevant to me, unless of course my VM lives on a box backed by that switch. ;)
The status dashboard exists for us to laugh at when things break and to convince C*Os that everything is fine. That's it.
As you point out, though, the status dashboard isn't truly meant to be either of those things. I don't have any illusions about it ever changing.
It's a shame Amazon doesn't have thousands of employees to divide these tasks between different people, as it is only these busy operators who could update this status page.
If you're right, why have the status page then? It is useless by your definition yes?
Its even more frustrating when you are aware of problems early on and start talking to support and THEY don't even know about problems yet.
Maybe the thousands of people is what prevents status from being updated, everyone tries to hide their own faults internally even
People responsible for the product should not have say over the switch being flipped, for obvious reasons (illustrated in other comments in this thread).
Which is why, during incident responses, there has to be people in charge of communication. Both internal and external communication, and some of this can be further delegated.
That's a poor excuse.
> It requires escalation up the management chain and careful wording
Careful wording is more important for external stakeholders who might not have the full context. If one is stepping in eggshells with internal management too, that's bad management. Incident communication should be factual and concise.
Could not agree more. It's immensely frustrating working with organisations that spend more time trying to cover up the cause of a outage to external stakeholders than actually fixing the root cause.
The same organisations tend try and blame individuals for outages.
I think both are a symptom of businesses that embrace the "blame culture"
The Lambda function associated with the CloudFront distribution is invalid or doesn't have the required permissions. We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner. If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.
Heh, maybe they accidentally locked themselves out of IAM, since those are great fun to troubleshoot, also