us-east-2 customer here, having a variety of "strange" issues including inability to reach an RDS database, other users in my firm having VPN reconnect trouble to that region.
"[10:11 AM PDT] We are investigating network connectivity issues for some instances and increased error rates and latencies for the EC2 APIs within the US-EAST-2 Region."
[10:25 AM PDT] We can confirm that some instances within a single Availability Zone (USE2-AZ1) in the US-EAST-2 Region have experienced a loss of power. The loss of power is affecting part of a single data center within the affected Availability Zone. Power has been restored to the affected facility and at this stage the majority of the affected EC2 instances have recovered. We expect to recover the vast majority of EC2 instances within the next hour. For customers that need immediate recovery, we recommend failing away from the affected Availability Zone as other Availability Zones are not affected by this issue.
And the reason that works is because HN is mostly hosted on its own stuff, without weird dependencies on anything beyond "the servers being up" and "TCP mostly working."
Living a bit more dangerously at the moment as HN is still running temporarily on AWS. (I'd link to the threads about this from a few weeks ago but am on my phone ATM.)
Availability zones are not guaranteed to have the same name across accounts (ie. us-east-2a in one account might be us-east-2d in another). You would need to use the AZ-ID to determine if they are the same.
I wonder if this is done because people have a tendency or something to always create resources in 'A' (or some other AZ) and this helps spread things around.
And if I would have read the page the link points to better, that's exactly the reason
AWS availability zones (so like us-west-2b rather than us-west-2) are not the same between accounts. us-west-2b for you is something different than us-west-2b for everyone else.
It's a joke but I only knew that because Snap is/was (as of S1) hosted on GCP and not AWS. Crackle happens to be the name of a video on demand company.
It's a reference to the breakfast cereal of the same name.
Pedantic clarification for the unfamiliar: the breakfast cereal is named Rice Krispies while Snap, Crackle, and Pop are the names of the cartoon mascots on the box.
Depends what it’s stuck doing, but you might ctrl-c it and later manually unlock the state file (by carefully coordinating with colleagues and deleting the dynamo DB lock object if you’re using the s3 backend) when the outage is over.
TF makes API calls to the underlying cloud. If those hang, you'll have to wait for them to time out.
Whether TF can update the state & release its locks would depend on where those were hosted. If they're in the downed AZ, then ofc. it can't do that, and manual intervention will be required afterwards. I forget if you can make those objects regional when stored in AWS or not. (You can in some other storages.)
Fun fact, for a lot of providers, it'll hang on any error, not just cloud ones. I presume it's due to the gRPC communication mechanism and the terraform binary blocking until the provider answers "yes or no" to the request
I think any system is susceptible to problems like this if the underlying hardware becomes unavailable. Using dynambodb to obtain locks on s3 is a pretty common pattern in AWS development. This has more to do with AWS than Terraform.
Nothing is perfect, there’s probably good reason for this behaviour … but it is rarely something that happens anyway. and you know, deleting a key for the state lock (one that explicitly tells you when and who created it) ain’t that hard or a that big of a deal.
Control+C (once!) is usually enough to cause it to abort without any ill effects to the state file. If it really got stuck and you have to kill it, then sure, you might have to mess with it a bit.
Haha literally had this same thought. us-east-2 is our default region for most stuff and so far that's been good. I think this is the first AWS downtime in last couple years that hit our systems directly
I interviewed there a few months ago for DevOps, and one of the people I interviewed with said that most of Zoom was in AWS (they liked that I had AWS stuff on my resume).
Suppose it's time to setup multi-az and pay to insure against AWS' own failures. I don't know why I previously thought their EC2 uptime claims were sufficient. Lesson learned.
What are you hosting? Multi-AZ seems like a bare minimum for basic reliability. That said it's not a panacea. There's all sorts of cascading/downstream "weirdness" that can result on AWS's own services through the loss of an AZ.
Are you sure you understand their uptime claims? They offer a 99.99% SLA for regional availability, but only 99.5% for individual instances (and even then, they only owe you a 10% service credit for affected instances)
99.5% availability allows up to about 3 and a half hours of downtime a month. 99.99% means around 4 minutes a month. So if you can't handle hours of downtime, you should definitely be multi-AZ.
253 comments
[ 4.6 ms ] story [ 267 ms ] threadEdit: 2 minutes after I post this it starts working.
Always check HN before trying to diagnose weird issues that shouldn't be connected
Physical identifiers for availability zones are like "use2-az1", ref.
https://docs.aws.amazon.com/ram/latest/userguide/working-wit...
https://docs.aws.amazon.com/prescriptive-guidance/latest/pat...
You'll need to see which availability zone ID (e.g., use2-az3) corresponds to each zone in your account: https://aws.amazon.com/premiumsupport/knowledge-center/vpc-m...
edit: AWS identified this as a power loss in a single zone, use2-az1.
And if I would have read the page the link points to better, that's exactly the reason
I don't work on cloud stuff, so I'm genuinely unsure if this is a joke.
It's a reference to the breakfast cereal of the same name.
> Informational
lol
Whether TF can update the state & release its locks would depend on where those were hosted. If they're in the downed AZ, then ofc. it can't do that, and manual intervention will be required afterwards. I forget if you can make those objects regional when stored in AWS or not. (You can in some other storages.)
… what would you expect to happen here?
https://www.oracle.com/customers/zoom/
I'm not sure if Zoom has any Critical infra in AWS though.
https://aws.amazon.com/compute/sla/
99.5% availability allows up to about 3 and a half hours of downtime a month. 99.99% means around 4 minutes a month. So if you can't handle hours of downtime, you should definitely be multi-AZ.
Datacenters do end up completely dying now and then, you really want to have a good strategy in that case. Or not, if that's not required.