That joke is getting pretty tired by now and restating it seems unlikely to be helpful for anyone who's affected by the rate-limited API calls in question.
It's not a joke. The us-east-1 region is one of if not the oldest. Some of the infrastructure is aging because it's hard to phase out. Amazon has learned a lot since then and the newer regions have reaped all the benefits, where Virginia hasn't. There are components that exist in that region and don't exist in others but as a rule, I use those as remote resources and keep all my critical infra in Ohio, if I need to be near east.
Have you actually measured it? I’ve had production workloads in us-east-1 for over a decade and haven’t seen it be notably different than the other regions. There’ve been a handful of times where there’s been a truly regional outage (17 minutes of network connectivity in 2011) and a couple of broader service outages (S3, IAM), but it’s not like the other regions never have problems.
In general most of the time when I’ve heard people say “us-east-1 is unreliable” it’s ended up that they’re really telling you that they were in just one AZ, thought the RDS multiple AZ mode was too expensive, etc. and standard architectural guidance would have taken care of it.
EC2 existed in us-east-1 only for 2 years before it was launched in a second region.
Those 2 years were already enough to pick up some legacy cruft and special cases. us-east-1 is also the first region for various other old services, like s3 and route53.
I think it's also the largest region across many axis, which naturally means it will hit various scaling issues before other regions.
Not sure if it is related but it is usually one of the first to get or be trialing new features. I believe a few global services are run out of us-east-1 as well.
Agree. 10s of millions in infrastructure running in us-east-1 here. Honestly it seems to be the only reliable way to teach people about failure and the importance of reserving capacity.
interesting they use blue as the color that indicate an "issue". at a glance, looks like nothing to see here. i have a feeling that if the entire AWS infrastructure were to go offline somehow they would refer to that as 'increase error rates'
Downtime is kinda hard to grok for huge distributed services like this.
If you’re down when anyone gets errors you’ll probably always be down — not really useful.
If you’re down when everyone gets errors you’ll probably always be up — again not really useful.
From a customer perspective an increase in errors means nothing if it doesn’t affect you or everything if it does so even a flat percentage is likely not all that useful except as a probability that any issues you’re seeing are your fault.
Maybe they do, but that's sort of the point? I've had to file numerous <s>bugs</s> support tickets with cloud providers which amounted to something like "99% of requests to S3 are failing and your status page is 'This is fine.' HALP"; and the response comes "oh, it was just your shard/node/backend/puzzle piece", like the fact that it was some private implementation detail really matters to me, a customer, when I'm seeing basically 100% failure.
But if you measure something as a percentile, and it's just my "shard" or whatever, I'm like 0.0001% of S3 traffic, so you'll never notice it at that scale. A small constant stream of errors is somewhat acceptable, so long as they're just the randomly distributed noise that is random failures and bit flips. But persistent errors that block forward progress or some customer having 0% success is what causes SWEs to ask themselves "I wonder what GCP is like.", and it becomes very clear that their monitoring doesn't indicate when the service is on fire for someone, only when it is on fire for everyone.
Azure had an outage yesterday (that was only visible to customers / wasn't reflected on the public status page): there's apparently no more capacity in West US for certain VMs. And, well, it is sort of a basic supposition that our cloud provider can provide a cloud, when asked. And region hopping is not simple: a whole host of resources just aren't compatible if you're not in the same region. (E.g., a vnet requires the VMs be in the same region.) (And that was "elevated" error rates, or something, again. Where, as a customer, "elevated" would appear to mean "100%".) Not a great start to the week.
Someone posted Rachel-by-the-Bay's "Your nines are not my nines" article in a sibling comment, which is basically this. People way discount the TCO of the cloud, IMO.
I'd also add I've worked at companies where every 5xx was hunted down. It was easy to get complacent, chalk them up to "oh, probably just a network blip" and we thankfully had a staff eng. who was very good and log diving and showing that no, it wasn't a blip, it was a bug in our code, and we should fix that bug.
Interesting, I just noticed today Glassdoor changed the Negative Experience chart color from red to a more neutral bluish color.
Maybe it's a recent trend in the industry to distance themselves from negative traits? Amazon doesn't want you to think they're bad, Glassdor doesn't want their customers (corporations) to be seen in a bad light?
I run mostly us-west-2 - and when talking about AWS I've always felt reliability was pretty good. us-east-1 really is the one that seems to get hammered the hardest on the error counts.
As much as I can remember, us-west-2 gets new instances and features at the same time as us-east-1. (And IIRC eu-west-1 gets new stuff as well, but I don't pay much attention to that.) Was there anything in the past few years where this hasn't been true?
Interestingly I couldn’t submit this URL to HN because it would take me to an original story that was posted 6 months ago. So I had to put it a random query string at the end of the URL like this
GCP also experienced problems. A really nice one were if you deleted a service account you end up with restart loop:
> We are experiencing an issue with Google Kubernetes Engine. Removing Service Account from GKE might lead to infinite cluster master restarts. Please refrain from removing GKE service accounts.
There actually are several good reasons to run in us-east-1.
Something I hear a good lot is the latency argument - if you're a startup based in Boston, and your roundtrip to us-west-2 is 80ms (this is actually my roundtrip to us-west 2 right now), it doesn't matter - your customers on the west coast will see 80ms if you go into us-east-1, too. That's true, but your first customers will probably be local, and you almost certainly don't have the resources to be doing true multi-region deployments right out of the gate. So my personal feeling is - deploy into us-east-1 to give your first customers a good experience - _fully understanding that you are taking on some extra risk_, and pay it down as tech debt in time.
Another plays off this, but in a very different way: a hybrid cloud deployment, where the public cloud is used as an on-demand extension of the datacenter. Something I once saw was that the 80ms round trip from a Boston datacenter to us-west-2 actually expanded to massive connection latencies: 80ms for a DNS lookup, and then another 500+ms (!) for the round-trips for TCP and TLS handshakes to take place, all before a SQL query or REST call actually started. That was a complete non-starter.
Does no one else think it’s odd that the general assumption of the tech community is to avoid an entire region of the largest cloud provider in the world, and then to assume these “newer” regions won’t have stability issues in a few years?
Another reason is that us-east-1 has the most features and services and is usually one of the first regions to get new services. That is perhaps less of an incentive now than it was a few years ago, and probably contributes to the instability in us-east-1, but if you need a newer service or feature that constrains your choice of region. And if once you've committed, switching to another region can be expensive.
> Existing instances and networks continue to work normally.
This suggests that one can minimize certain failure modes by making the configuration as static as possible, i.e. creating a fixed number of long-running instances. But then, using immutable, ephemeral instances can make for a more resilient system when the EC2 control plane is working normally. There are always tradeoffs.
51 comments
[ 2.8 ms ] story [ 114 ms ] threadIn general most of the time when I’ve heard people say “us-east-1 is unreliable” it’s ended up that they’re really telling you that they were in just one AZ, thought the RDS multiple AZ mode was too expensive, etc. and standard architectural guidance would have taken care of it.
Those 2 years were already enough to pick up some legacy cruft and special cases. us-east-1 is also the first region for various other old services, like s3 and route53.
I think it's also the largest region across many axis, which naturally means it will hit various scaling issues before other regions.
Took out a few of our production databases, seems like Australia is the beta testing region for at least a few services.
At least in the US you have the choice of other regions.
Blue is more distinguishable from green by colour-blind folks. Red is not distinguishable.
If you’re down when anyone gets errors you’ll probably always be down — not really useful.
If you’re down when everyone gets errors you’ll probably always be up — again not really useful.
From a customer perspective an increase in errors means nothing if it doesn’t affect you or everything if it does so even a flat percentage is likely not all that useful except as a probability that any issues you’re seeing are your fault.
[1] https://news.ycombinator.com/item?id=20451714
This person worked on Alexa and wrote an article about it: https://medium.com/@djsmith42/how-to-metric-edafaf959fc7
But if you measure something as a percentile, and it's just my "shard" or whatever, I'm like 0.0001% of S3 traffic, so you'll never notice it at that scale. A small constant stream of errors is somewhat acceptable, so long as they're just the randomly distributed noise that is random failures and bit flips. But persistent errors that block forward progress or some customer having 0% success is what causes SWEs to ask themselves "I wonder what GCP is like.", and it becomes very clear that their monitoring doesn't indicate when the service is on fire for someone, only when it is on fire for everyone.
Azure had an outage yesterday (that was only visible to customers / wasn't reflected on the public status page): there's apparently no more capacity in West US for certain VMs. And, well, it is sort of a basic supposition that our cloud provider can provide a cloud, when asked. And region hopping is not simple: a whole host of resources just aren't compatible if you're not in the same region. (E.g., a vnet requires the VMs be in the same region.) (And that was "elevated" error rates, or something, again. Where, as a customer, "elevated" would appear to mean "100%".) Not a great start to the week.
Someone posted Rachel-by-the-Bay's "Your nines are not my nines" article in a sibling comment, which is basically this. People way discount the TCO of the cloud, IMO.
I'd also add I've worked at companies where every 5xx was hunted down. It was easy to get complacent, chalk them up to "oh, probably just a network blip" and we thankfully had a staff eng. who was very good and log diving and showing that no, it wasn't a blip, it was a bug in our code, and we should fix that bug.
Maybe it's a recent trend in the industry to distance themselves from negative traits? Amazon doesn't want you to think they're bad, Glassdor doesn't want their customers (corporations) to be seen in a bad light?
https://status.aws.amazon.com/?x=1000
@Dang, any way HN can allow new submissions to the same URL? Thank you .
> We are experiencing an issue with Google Kubernetes Engine. Removing Service Account from GKE might lead to infinite cluster master restarts. Please refrain from removing GKE service accounts.
"Well, then don't do that"
Something I hear a good lot is the latency argument - if you're a startup based in Boston, and your roundtrip to us-west-2 is 80ms (this is actually my roundtrip to us-west 2 right now), it doesn't matter - your customers on the west coast will see 80ms if you go into us-east-1, too. That's true, but your first customers will probably be local, and you almost certainly don't have the resources to be doing true multi-region deployments right out of the gate. So my personal feeling is - deploy into us-east-1 to give your first customers a good experience - _fully understanding that you are taking on some extra risk_, and pay it down as tech debt in time.
Another plays off this, but in a very different way: a hybrid cloud deployment, where the public cloud is used as an on-demand extension of the datacenter. Something I once saw was that the 80ms round trip from a Boston datacenter to us-west-2 actually expanded to massive connection latencies: 80ms for a DNS lookup, and then another 500+ms (!) for the round-trips for TCP and TLS handshakes to take place, all before a SQL query or REST call actually started. That was a complete non-starter.
This suggests that one can minimize certain failure modes by making the configuration as static as possible, i.e. creating a fixed number of long-running instances. But then, using immutable, ephemeral instances can make for a more resilient system when the EC2 control plane is working normally. There are always tradeoffs.