I wonder if this is a coincidence or if us-east-1 is simply down enough that I'm just experiencing selection bias; but I posted a poll on twitter earlier today: https://twitter.com/dijit/status/1668678588713824257
Contents:
> Has anyone ever actually had customers accept an outage because AWS was down; or is this just cloud evangelicalism copium?
I guess a demanding customer would have said 'you should have implemented disaster recovery so you could failover to us-east-2' but that's easier said than done. The more regional AWS services you adopt, the bigger the impact is. How does one recover from a regional outage if their pipeline is in that region?
I’ve worked for several systemically important megacorps where certain things had to not only run cross region but also cross provider. It’s absurdly difficult, and only should be done if you need five or more 9’s of availability. Almost nothing actually does.
What I did once I was in the position of _having_ to provide that level of support, was to run the pipeline in a third region, different from the "prod" ones. That way, worst case you can't do deployments during the outage...
Another alternative studied was to use a thirdparty ci/cd service, outside of our network. It was discarded bc you never know where that would actually run
> It was discarded bc you never know where that would actually run
Yep, I considered that switching to GitHub Actions would _theoretically_ eliminate the need for disaster recovery for CI/CD (since the handling of disasters is out of your hands) but in practice their SLA is far worse than just running CodePipeline in a single region.
Yeah, that's why we went with a third region instead. But, at the end of the day, if _only_ changes are affected for a couple of hours, that wouldn't impact the service that much
AZs don't really help when it's AWS' own services across the entire region that break. Anecdotally, we have had customers accept outages that were out of our control without penalty.
Wild, that wouldn't have flown with datacenter providers having issues for my previous companies.
AWS really does have an easier time than old school datacenter providers. I guess the complexity is higher but it's shocking that they can charge so much yet we hold them to a lower standard.
DCs are pretty static and offer way fewer services than AWS or any other public cloud.
I worked for one for some time and whenever we had issues, some people would call and ask if we were going bankrupt. It gave me a feeling they also have way smaller customers that might not understand the underlying stack.
If all you use in AWS is static EC2 instances you would have to go back a looooong time to find an outage which affected their availability. Even in us-east-1.
Outage rates are also wildly different. When you're using dozens of managed services and have a few prod-impacting outages with any reasonable (cross-AZ) design, customers are less sensitive then when they are dependent on dozens of products that hav independent failure modes with potentially cascading impact.
AZs also don't help with natural disasters at all. I believe AWS is the only one doing geographically distributed AZs, for the others it just means different connections and placed somewhere else in the building.
edit: turns out AWS is the one with geo distribution, not Azure
Aws azs are also distributed geographically within a region w separate power and network lines. From the docs “ Availability Zones are distinct locations within an AWS Region that are engineered to be isolated from failures in other Availability Zones.”
Ah, you are probably right. I was thinking of the incident a few weeks back where the fire suppression took out multiple AZs, but that was actually GCP.
My employer lets customers choose which of our supported regions to run in and exempts cloud provider outages from our SLA (we’re on the hook for staying up for single AZ outages, but not multi AZ or region outages). We provide tools to help customers replicate their data so they can be multi-region or even multi provider if they want to.
it's important to inform customers about the resiliency of their systems and let them pick how far they are going to invest for it.
then you get to eat popcorn when stuff explodes.
* single server event. $
* multi server event. $$
* single az event. $$$
* multi az event. $$$$
* global provider event. $$$$$
* cross provider event. $$$$$$
* alien invasion. $$$$$$$$$$$$$$
And you thought the current time zone confusion was bad. Now you have two sets of time zones, and a varying delay of about 5 to 21 minutes between them. Oh, the joy!
Multi-planetary-AZ DB cluster deployments. Putting the emphasis on "Eventual" in eventual consistency.
Go for a walk before retrying reading from this replica!
Back when we had servers in an onsite DC we lost a raid card and the system I was developing went down. We had the fancy support so a tech was out with the card replaced in a couple hours, then we had to restore from tape backup. All in all, a non-critical system was down for most of a business day. My bosses boss stormed in, upset he couldn't pull a report, and asked how do we prevent this in the future. I responded at a minimum we had to double the cost for a hot standby, and he said 'never mind' and walked out.
> Has anyone ever actually had customers accept an outage because AWS was down...
Whether customers "accept" it or not just comes down to what's in your SLA, if you have one in the first place, and if they are on a contract tier that it applies to. [Many servies provide no SLA for hobby / low tiers, beta features, etc.]
Firebase Auth, for instance, offers no SLA at all [1].
I would be curious to see statistics across a range of SLAs for what % include a force majeure or similar clause which excludes responsibility for upstream outages. I would expect this to be more common with more technical products / more technical customers.
certificate manager also down (I know because I tried to update an ssl cert for cloudfront which only allows US-East-1 ssl certs, maybe someone will eventually fix that to allow any region to have the ssl cert for cloudfront)
I had a similar reaction. Oh no WTF, how did i break that?! Then my buddy texted me about us-east-1 being down. Then i thought "Oh thank god, this shitshow is someone else's fault."
Had our login and other features go down less than an hour after I altered our prod scheme and thought I did something wrong, what a relief it was to see this
I'm so glad my demo today was specifically about local inference on... Windows. I guess working I finally found an upside to doing ML outside Linux ; we don't have Windows VMs on AWS!! :)
I worked with a devops person who moved everything we had set up in other regions _to_ US-East-1 because that is where you are supposed to run stuff. According to him, the other regions were just for DR stuff.
Yep, it has issues so frequently. I wonder how many companies/teams start using AWS and blindly choose us-east-1 without realizing what they're getting into.
<rant>
It's also quite annoying sometimes that some things _need_ to be in us-east-1, and if e.g. you are using Terraform and specify a different default region, AWS will happily let you create useless resources in regions that aren't us-east-1 that then mysteriously break stuff because they aren't in this one blessed region. AWS Certificate Manager (ACM) certificates are like this, I believe.
ACM certificates themselves can be had in any region (and you can use them for stuff like ELBs), but since the Cloudfront control plane is in us-east-1, if you want Cloudfront (and IIRC, also if you want custom domain names for an S3 bucket, but don't quote me on that) you'll have to create an additional certificate in us-east-1.
The second law of thermodynamics guarantees that there will always / eventually be downtime… and that’s okay. Design for downtime. Shameless self-plug for https://heiioncall.com/ for free website / HTTP endpoint monitoring and cron job monitoring if you want to know about your own app’s downtime.
EDIT: thanks to those of you who have signed up in the past few minutes! Let us know if you have any feedback.
"eventually" is doing a lot of work in that sentence. Our sun will last another 5 billion years, and the heat death is something like 100 trillion years away.
Though a lot of practical thermal-related causes of electronics failure seem to operate on timescales of years to decades, like electromigration https://en.m.wikipedia.org/wiki/Electromigration or even just cooling fan bearing failure. And I don’t think it would be a huge stretch to point to electromigration as a case of diffusion, a natural entropy increasing process, re-randomizing the arrangement of atoms within a transistor (and therefore making it fail eventually).
The inconsistency with timezones across different services in the AWS console has always baffled and annoyed me. Some places have a time without a timezone and I can never tell right away if it's utc, local time, or region time.
> The inconsistency [of everything, everywhere] in the AWS console
ftfy
AWS is powerful and very popular, but for the console, "it functions" must be the only condition the UI has to satisfy. Should every page use a unique table and sorting widget and UI language? Yes, please!
I'm assuming this helps them move fast, not having to coordinate with anybody or wait for a UI designer to tell them how it should look. But it's striking when compared to GCP.
Semi-related: if you ever feel the need to report times to a global audience, not only make sure to always report the timezone (even if it is the same as the user's), but also use UTC offsets rather than timezone names.
Life is too short to remember what each timezone name means and converting to it, UTC offsets are much easier on the mental calculator.
UTC is human readable even if it is not calculated correctly. yes, i'm saying that if you can read epoch seconds, you're not human. 1970-01-01 00:00:00 is always a give away that something is a foot
"anchored on" then? I might be wrong but we're both talking about showing time as distance from the same starting point are we not? One's just more human readable so that's why I say why not just use that? Seconds since can be miscalculated too, especially if current time isn't known/reliable
I mostly struggle with Irish Standard Time (used for DST in Ireland) and Indian Standard Time which have the same acronym. :(
Thankfully, I learnt a long time ago to use ISO 8601 and UTC for dates and times. I still revert to PST/PDT if my audience is primarily left coast based.
And I can't say it's ever actually caused a problem, but something about Indian Standard Time being a half-hour offset from UTC has always bothered me so much... But now we're fully off-topic.
Everyone knows "Mountain Time". It is when you go to the mountains on vacation, and don't spend much time adhering to a strict schedule, instead taking leisurely strolls around the fields and promising vague things like "I'll try to be back for dinner".
And if you've been American since birth, and live in Arizona, one might still not know, since PDT and Mountain Time alternate covering Arizona seasonally. ("Ask me how i know.")
It can also varies within Arizona... one of the most confusing times in my life was driving from California through the Navajo Reservation in AZ on my way to an appointment. Was my cell phone giving me the local time on the reservation? Was it connecting to a cell tower just outside the reservation, giving me DST-less Arizona time? Or a tower slightly further away in Utah (DST?) Or was it giving me the time on the Hopi reservation, which is an enclave totally surrounded by the Navajo Reservation which uses AZ time?
3 years ago, when I started work for my current employer, I noticed in Slack that everyone was reckoning time in "Standard Time" year-round. Now imagine my chagrin because I live in Arizona, and "Mountain Standard Time" does not change for DST. Therefore, all my coworkers were citing nonsensical, nonexistent time zones and it was messing up my ability to convert back and forth.
Come to find out that this was some sort of entrenched, company-wide standard that was deliberately imposed. I made a lot of noise about this and appealed to some rather highly-placed directors, because I felt like it was wildly inaccurate and deceiving people; if you schedule a meeting in EDT but you say it's in EST, and we have employees all around the world, who's going to know? You're inviting off-by-one errors. Especially with me who lives permanently in MST.
3 years on, I've been unable to change this fundamentally; while a few people acknowledge DST, 90% of the company still adheres to this crazy false standard.
How does this generate off-by-one errors? I am also part of a company with employees in pretty much every timezone, but when they create a meeting the meeting invitation is programmed with the correct timezone so in my Calendar it always shows what time the meeting is going to be for me. I never even have to think what timezone the organizer is...
It's the same at my company. Teams and Zoom both automatically schedule meetings in every attendees' own time zone. Maybe that person's company still does phone meetings or something.
We don't use any automatic scheduling with Zoom or Google Calendar. Management doesn't send invites to those meetings, they just publish the link on Slack and we have to figure out how to get it into our calendars.
Trust me, at least once I missed a meeting because I was late by an hour due to time zone confusion.
The off-by-one error occurs when you announce an event in Standard time but really mean Daylight time, or vice versa. While those local to the time zone will often automatically correct this mistake either consciously on unconsciously, those in other time zones (especially where Daylight time isn't used or is on a different schedule) will tend to rely on time conversion tools which will take a literal interpretation of the scheduled time and result in the person being an hour early or an hour late.
The fact that you have to announce timezones is already an error. If I need to schedule a meeting I don't need to select timezones, they're already selected from the timezone I'm part of. There's never room for error by "picking" the wrong thing, since there's nothing to pick. And if my system is programmed with the wrong timezone, then every single meeting will be off-by-N and my calendar will show the wrong time as "now". It would be impossible to miss such an error.
I think your company needs better tools to handle meetings.
I just had someone asking me if I'm available at 5pm EST.
Also, your clock can get confused driving North from PHX to Zion National Park.
In summer you start in Mountain Standard Time, drive into the Navajo Nation which does observe Mountain Daylight Time, containing through the Hopi Reservation, which is Mountain Standard Time. Then you end up back in Navajo Nation with Mountain Daylight Time. You keep on driving towards Page which is in Mountain Standard Time. However, when you cross the state-border of AZ/UT you're back in Mountain Daylight Time.
Even in Australia, AEST has a DST flavour and a non-DST one. Queensland does not observe DST while the other states do. You can drive around a roundabout at the border and switch timezones for fun. Or go down there to celebrate the new year twice.
Or go from rabbits being OK to some 5 figure fine if you're caught with one :)
It's also not too complicated to add a few lines of javascript that show the datet/time in the user's local time zone (via Date.getTimezoneOffset) as well.
We do this in all of our web apps. It's pretty simple and dramatically improves UX when you have customers that are doing a lot of scheduling.
Showing both at the same time is peak design for me personally. UTC compares for relative sequencing, local time for "was that before or after I ate lunch".
GCP's various products have gotten a lot better at this lately, but just a few months ago I could click around between various dashboards and explorers, some showing the time in UTC, some in your browser's tz, and some in your profile's tz (if I recall correctly). Some of them were showing the tz, and for some you had to guess. Sometimes you had multiple tzs on the same page. Sometimes the date picker for a control was in one tz and the widget it was controlling in another (leading to quite a lot of confusion).
The worst offence IMO was not showing the tz at all. Especially given the overall lack of consistency.
still show the timezone it's displayed in, users aren't always fully aware of which time zone their browser is believing they are in (sounds stupid at first but imagine you just traveled somewhere and are temporary used to that time zone but e.g. your laptop is fixed set to your home time zone, or maybe it's not and you just thought it is, or maybe you are just a bit confused because you switched time zones 4 times the last 24 hours etc.)
Also report it using IATA time zones (America/Los_Angeles) at least in addition (I'd argue instead of) those abbreviations which are completely unstandardized and not unique.
If the world were fair, we’d be calling these Eggert time zones, as Paul Eggert (longtime tzdata maintainer until the copyright trolls came) invented them; but it isn’t.
(You probably still meant IANA the Internet org not IATA the aviation one.)
Nothing worse than people who say "9 AM my time" I suppose it's OK if it's Pacific vs Mountain but even there Arizona doesn't observe Daylight, and parts of Eastern Oregon are Mountain, not Pacific.
Never mind dealing with India, Australia, etc etc.
OK to use local time in your statement, just say what that time is.
> Life is too short to remember what each timezone name means and converting to it, UTC offsets are much easier on the mental calculator.
Many people also get the timezone names completely wrong. I've had multiple scheduling email exchanges where someone says X pm EST not realizing that at the time it's currently EDT and that EST ≠ EDT.
And yet, for some reason, the two-letter abbreviations (e.g., ET) that are technically correct year-round, never seem to have caught on in the wild.
I've given up on the abbreviations and just say "Eastern" now to avoid confusion.
Fair enough, but please only use that with strictly USA audiences. (And remember that public information likely will not be targeted to strictly USA audiences)
Names don't carry any information intrinsically, they are only a reference to the actual information, and the offset information is pretty short, so why not just provide the information directly?
"X pm GMT-3" only requires the reader to know their own timezone offset, unlike "X pm Brasilia time" (which is inaccurately known as São Paulo time outside Brazil) or "X pm BRT", which requires the reader to both know what that timezone means, and their own (or, more likely, requires them to look the conversion up).
(And if the difference between GMT and UTC is significant, I hope it didn't take my comment to convince you about using offsets :> )
The outage is in Virginia so PDT isn't even local time. On their status page they are asking users to access the console via a region specific endpoint like https://us-west-2.console.aws.amazon.com. Wonder if the PDT timestamp is because they have to serve the status page from US West right now.
Technically PDT is always 7 hours behind UTC. PST is always 8 hours behind. We just change which one we use twice a year. Pacific time makes sense when you realize Fremont is the center of the universe.
Indeed. There are Americans who will tell me PST, when they meant PDT but forgot to mention that. Now I have to track the American DST calendar as well as European DST calendar to do the conversion.
There are also people who tell me GMT (because they think that term means "the time in London") when they meant BST (because in summer, London doesn't operate on GMT).
Yup, this is why I always say “US ET” (I'm on the east coast). I don't trust myself or anyone else to get it right, and if the other party is converting anyway, their conversion tool (google?) should be able to handle that. (Of course, the date is necessary but implicit, but that's usually fine too.)
After spending some time in the Canary Islands I realized how nice it was to be in UTC all the time and now I have my laptop clock set to UTC. Still contemplating whether I should set Google Calendar and my smartwatch to UTC as well. 8-)
Imagine you have two browser opening the same page, one showing UTC, another showing your local time.
There are no indicator showing which time zone is used. You have to mentally correlate "this browser windows has logged in..." with the time shown on screen.
The fact that which timezone is used in the announcement is a sign of progress... AWS announced it pretty quickly, gave nice updates, and seems to have fixed the problem quickly enough. I'm interested to see the postmortem...
When I was with AWS I advocate for ISO8601 "Z" whenever I could or need to influence, say internal systems.
If all systems talk this we'd save tens of thousands of man hours. Just do the conversion for us mortals, or other necessities. Tech side of incidents is definitely "system", I'd argue more often than not consumers of AWS are also tech side with systems in UTCs so health dashboards should also be a UTC first system. Doubt this could get prioritized tho
the last big us-east-1 outage was ... DNS - and it's usually DNS or software-defined core networking causing these cascading failures
Loss of DNS causes inter-service api calls to fail, then IAM and all other services fail. Anything not built to handle those situations with backoff causes a 'stampeding herd' of failure/retry and exacerbate the outage
I have a suspicion that AWS uses some regions as canaries. Because we control both ends of things, I have personally noted that certain AWS functions clearly break in Australia first.
When I worked there, there were few hard and fast rules. Every team had its own release processes, so there was a lot of variance. It has been a couple of years, so this may have changed.
Typically, a team would group their regions into batches and deploy their change to one batch at a time. Usually they follow a geometric progression, so the first batch has one region, the second batch has two regions, the third batch has four regions, and so on. This batching was performed for the sake of time; nobody wants to wait a month for a single change to finish rolling out.
One reason not to deploy to us-east-1 in the first batch is so you don't blow up your biggest region. The fewer customers you break, the better.
One reason not to deploy to us-east-1 in the last batch is that there are a lot of batches. If a problem is uncovered after deploying the last batch, then someone has to initiate rollbacks for every single region.
Some teams tried to compromise and put us-east-1 in one of the earlier batches.
No definitely not. Usually pipelines deploy over 1-2 week periods, and they don't deploy on Fridays/holidays/high-traffic periods like December.
Deployments start off very conservative, maybe 1-2 small regions on the first day of deployments. As you gain confidence, the pipeline deploys to more regions/bigger regions.
A pipeline that deploys to 22 regions over one week might go from 2 small regions on monday, 4 small/medium regions on tuesday, 8 medium/large regions on wednesday, 8 regions on thursday.
us-east-1 is usually going to be deployed to on the wednesday/thursday in this example, but that isn't always the case because sometimes deployments are accelerated for feature launches (especially around re:invent), or retried because of a failure.
There are best practice guides within Amazon that very closely detail how you should deploy, although it is up to the teams to follow them, which they usually do an okay job of.
Usually us-east-1 is deployed to after several smaller regions. Usually it'll fall in the middle of the week depending on the pipeline.
Just because a feature is there on launch day doesn't mean it was deployed to first. Features are often hidden behind flags that are switched for launch.
I'm well aware of that, but the point is that when the feature is ungated to the public, it's in us-east-1 and gets all that load, and more load than the rest because of the fact that a lot of big customers are based in us-east-1, including much of Amazon itself.
Those are not single-region services. Changes must be executed there, but the data is replicated globally. If you don’t need to make changes in the context of those services, they will keep working in the other regions even during an incident in the primary region.
us-east-1 is the largest region, so it is where changes meet scale.
It is also a massively complex beast in itself spanning dozens of datacenters with massive amounts of fiber between them. Much more fragile than having everything in a single building and as you scale up the number of components you increase the rate of failure.
Touché. Still I'd rate the overall reliability of AWS higher than Azure; and even if that weren't the case, security issues make Azure look like a very poor choice.
I don't think this article has any value. Are you only counting region wide outages? US east is probably 10x the size of any other region with more AZ's than any other region.
330 comments
[ 3.2 ms ] story [ 419 ms ] threadContents:
> Has anyone ever actually had customers accept an outage because AWS was down; or is this just cloud evangelicalism copium?
> [ ] Yeah, outages free pass
> [ ] No, they say to use AZ's
Using 3 AZs in us-east-1 won't save you.
I guess a demanding customer would have said 'you should have implemented disaster recovery so you could failover to us-east-2' but that's easier said than done. The more regional AWS services you adopt, the bigger the impact is. How does one recover from a regional outage if their pipeline is in that region?
Another alternative studied was to use a thirdparty ci/cd service, outside of our network. It was discarded bc you never know where that would actually run
Yep, I considered that switching to GitHub Actions would _theoretically_ eliminate the need for disaster recovery for CI/CD (since the handling of disasters is out of your hands) but in practice their SLA is far worse than just running CodePipeline in a single region.
AWS really does have an easier time than old school datacenter providers. I guess the complexity is higher but it's shocking that they can charge so much yet we hold them to a lower standard.
I worked for one for some time and whenever we had issues, some people would call and ask if we were going bankrupt. It gave me a feeling they also have way smaller customers that might not understand the underlying stack.
edit: turns out AWS is the one with geo distribution, not Azure
then you get to eat popcorn when stuff explodes.
- memo from Enterprise Sales Dept.
Short of alien invasion level are strategic military resistance levels to global/regional wars with differing levels of weapons and devastation.
If your customers are tech, they're too busy running around with their hair on fire too.
Whether customers "accept" it or not just comes down to what's in your SLA, if you have one in the first place, and if they are on a contract tier that it applies to. [Many servies provide no SLA for hobby / low tiers, beta features, etc.]
Firebase Auth, for instance, offers no SLA at all [1].
I would be curious to see statistics across a range of SLAs for what % include a force majeure or similar clause which excludes responsibility for upstream outages. I would expect this to be more common with more technical products / more technical customers.
[1]: https://stackoverflow.com/a/60500860/149428
This seems like an odd limitation. Do you know the technical reason?
And what a surprise it's US-EAST-1 again...
<rant>
It's also quite annoying sometimes that some things _need_ to be in us-east-1, and if e.g. you are using Terraform and specify a different default region, AWS will happily let you create useless resources in regions that aren't us-east-1 that then mysteriously break stuff because they aren't in this one blessed region. AWS Certificate Manager (ACM) certificates are like this, I believe.
</rant>
Sigh.
EDIT: thanks to those of you who have signed up in the past few minutes! Let us know if you have any feedback.
Though a lot of practical thermal-related causes of electronics failure seem to operate on timescales of years to decades, like electromigration https://en.m.wikipedia.org/wiki/Electromigration or even just cooling fan bearing failure. And I don’t think it would be a huge stretch to point to electromigration as a case of diffusion, a natural entropy increasing process, re-randomizing the arrangement of atoms within a transistor (and therefore making it fail eventually).
AWS can do so many things, reporting critical outage updates in UTC is not one of those things.
Thank you for reminding me about one of my biggest mildest annoyances from working at AWS.
ftfy
AWS is powerful and very popular, but for the console, "it functions" must be the only condition the UI has to satisfy. Should every page use a unique table and sorting widget and UI language? Yes, please!
I'm assuming this helps them move fast, not having to coordinate with anybody or wait for a UI designer to tell them how it should look. But it's striking when compared to GCP.
Life is too short to remember what each timezone name means and converting to it, UTC offsets are much easier on the mental calculator.
UTC is human readable even if it is not calculated correctly. yes, i'm saying that if you can read epoch seconds, you're not human. 1970-01-01 00:00:00 is always a give away that something is a foot
time encoded as a float of trecenti-seconds since year -8435 of the Georgian Calendar
why?
'caus it hurt to even just think about implementing that anywhere
Thankfully, I learnt a long time ago to use ISO 8601 and UTC for dates and times. I still revert to PST/PDT if my audience is primarily left coast based.
Heh. After the first few instance of confusion, we switched to saying Bangalore time and Dublin time.
Come to find out that this was some sort of entrenched, company-wide standard that was deliberately imposed. I made a lot of noise about this and appealed to some rather highly-placed directors, because I felt like it was wildly inaccurate and deceiving people; if you schedule a meeting in EDT but you say it's in EST, and we have employees all around the world, who's going to know? You're inviting off-by-one errors. Especially with me who lives permanently in MST.
3 years on, I've been unable to change this fundamentally; while a few people acknowledge DST, 90% of the company still adheres to this crazy false standard.
Trust me, at least once I missed a meeting because I was late by an hour due to time zone confusion.
I think your company needs better tools to handle meetings.
Also, your clock can get confused driving North from PHX to Zion National Park.
In summer you start in Mountain Standard Time, drive into the Navajo Nation which does observe Mountain Daylight Time, containing through the Hopi Reservation, which is Mountain Standard Time. Then you end up back in Navajo Nation with Mountain Daylight Time. You keep on driving towards Page which is in Mountain Standard Time. However, when you cross the state-border of AZ/UT you're back in Mountain Daylight Time.
My clock threw a segmentation fault.
I encourage everyone at my company to do the same. Easy way to eliminate errors while typing 1 less key stroke!
I literally had cases when I was woken up in the middle of the night for production issue because some people are too sloppy about this kind of thing.
Or go from rabbits being OK to some 5 figure fine if you're caught with one :)
https://en.wikipedia.org/wiki/State_of_Origin_series
Showing both at the same time is peak design for me personally. UTC compares for relative sequencing, local time for "was that before or after I ate lunch".
GCP's various products have gotten a lot better at this lately, but just a few months ago I could click around between various dashboards and explorers, some showing the time in UTC, some in your browser's tz, and some in your profile's tz (if I recall correctly). Some of them were showing the tz, and for some you had to guess. Sometimes you had multiple tzs on the same page. Sometimes the date picker for a control was in one tz and the widget it was controlling in another (leading to quite a lot of confusion).
The worst offence IMO was not showing the tz at all. Especially given the overall lack of consistency.
(You probably still meant IANA the Internet org not IATA the aviation one.)
Never mind dealing with India, Australia, etc etc.
OK to use local time in your statement, just say what that time is.
- the keynote will start when this post is 5 hours old
- the rocket launch is scheduled to when this comment is 30 hours old
Many people also get the timezone names completely wrong. I've had multiple scheduling email exchanges where someone says X pm EST not realizing that at the time it's currently EDT and that EST ≠ EDT.
And yet, for some reason, the two-letter abbreviations (e.g., ET) that are technically correct year-round, never seem to have caught on in the wild.
I've given up on the abbreviations and just say "Eastern" now to avoid confusion.
Names don't carry any information intrinsically, they are only a reference to the actual information, and the offset information is pretty short, so why not just provide the information directly?
"X pm GMT-3" only requires the reader to know their own timezone offset, unlike "X pm Brasilia time" (which is inaccurately known as São Paulo time outside Brazil) or "X pm BRT", which requires the reader to both know what that timezone means, and their own (or, more likely, requires them to look the conversion up).
(And if the difference between GMT and UTC is significant, I hope it didn't take my comment to convince you about using offsets :> )
There are also people who tell me GMT (because they think that term means "the time in London") when they meant BST (because in summer, London doesn't operate on GMT).
I have found this site very helpful for linking people to when they are confused or using them incorrectly:
https://time.is/PT
https://time.is/PST
https://time.is/PDT
The time zone comparison feature is nice as well:
https://time.is/compare/0800AM_14_June_2023_in_Cincinnati/Lo...
https://health.aws.amazon.com/health/status#settings
As I'm logged in, it persists across browser sessions.
Imagine you have two browser opening the same page, one showing UTC, another showing your local time.
There are no indicator showing which time zone is used. You have to mentally correlate "this browser windows has logged in..." with the time shown on screen.
If all systems talk this we'd save tens of thousands of man hours. Just do the conversion for us mortals, or other necessities. Tech side of incidents is definitely "system", I'd argue more often than not consumers of AWS are also tech side with systems in UTCs so health dashboards should also be a UTC first system. Doubt this could get prioritized tho
If you login, you can specify what timezone for timestamps and for the text to be parsed into your timezone preference.
Loss of DNS causes inter-service api calls to fail, then IAM and all other services fail. Anything not built to handle those situations with backoff causes a 'stampeding herd' of failure/retry and exacerbate the outage
Review the AWS statements about outages here - https://aws.amazon.com/premiumsupport/technology/pes/
I have always stayed away from that region because it seems significantly less reliable than other regions.
@dijit is right: https://news.ycombinator.com/item?id=36315736
Typically, a team would group their regions into batches and deploy their change to one batch at a time. Usually they follow a geometric progression, so the first batch has one region, the second batch has two regions, the third batch has four regions, and so on. This batching was performed for the sake of time; nobody wants to wait a month for a single change to finish rolling out.
One reason not to deploy to us-east-1 in the first batch is so you don't blow up your biggest region. The fewer customers you break, the better.
One reason not to deploy to us-east-1 in the last batch is that there are a lot of batches. If a problem is uncovered after deploying the last batch, then someone has to initiate rollbacks for every single region.
Some teams tried to compromise and put us-east-1 in one of the earlier batches.
Deployments start off very conservative, maybe 1-2 small regions on the first day of deployments. As you gain confidence, the pipeline deploys to more regions/bigger regions.
A pipeline that deploys to 22 regions over one week might go from 2 small regions on monday, 4 small/medium regions on tuesday, 8 medium/large regions on wednesday, 8 regions on thursday.
us-east-1 is usually going to be deployed to on the wednesday/thursday in this example, but that isn't always the case because sometimes deployments are accelerated for feature launches (especially around re:invent), or retried because of a failure.
There are best practice guides within Amazon that very closely detail how you should deploy, although it is up to the teams to follow them, which they usually do an okay job of.
* Largest (DDoS'd most, most complex, scaling issues etc)
* Oldest (More time for weird idiosyncrasies to take hold)
* Where most testing happens
* Where new products are deployed first
Just because a feature is there on launch day doesn't mean it was deployed to first. Features are often hidden behind flags that are switched for launch.
* The only place where the IAM dashboard can be accessed from. I need to access it NOW. I can't.
IAM, Cloudfront ACM certs, etc
It is also a massively complex beast in itself spanning dozens of datacenters with massive amounts of fiber between them. Much more fragile than having everything in a single building and as you scale up the number of components you increase the rate of failure.
https://statusgator.com/blog/is-north-virginia-aws-region-th...