HN Explorer

23 comments

[ 3.2 ms ] story [ 46.0 ms ] thread
cdoxsey
Seems to have hit pagerduty pretty hard... your poor on-call guy may have no idea his app is busted.
revx
Interesting. I got a page during the incident, so at least some of it was working.
godelmachine
Pager as in - the electronic device that existed before the cellphone?
kondro
These things happen to every datacentre. That’s why AWS has multiple, separately managed regions that you can use to keep your app up with.
memer
and pay double? no way
kondro
Hardware will always fail at some point. You can either add extra resources elsewhere, or deal with that inevitable downtime. Both are reasonable, depending on the impact of downtime.
cratermoon
It's not paying double. Take the total number of instances serving the app and divide them across data centers and availability zones within datacenters. This might require adding a few of servers if the app is replicated across a very small number of instances, but for apps that already have a significant number of instances to handle the capacity, it's just a matter of rearranging things.

Depending on how quickly new instances of the app can be brought up to, it may be worthwhile to run a bit more capacity so that the remaining instances don't melt down should all the instances in one zone become unavailable.

thalesac
That's why my aws console wasn't working properly this afternoon. It seems that Amazon is not using it's services correctly
ggggvg
Nonsense. The console is hosted out of us-east-1. Also, please learn to spell before posting. Thanks.
bklyn11201
But AWS has the concept of availability zones that they equate to a data center. AWS refers to an AZ as an "isolated location". It's becoming clear with the S3 outages and this outage that isolation is not as robust as expected so now we see more reference to cross-region replication. But unlike cross-AZ services like S3 or DynamoDB, you need to build your own cross-region replication and failover.
paulddraper
Yep. My take: humans fail at more than hardware does, and AWS takes a lot of action per region.
killahpriest
Everything is closed.

Direct connect: October 18, 2017 at 4:37:09 PM UTC-5

Internet connectivity: October 18, 2017 at 4:33:29 PM UTC-5

EC2 VPC network health: October 18, 2017 at 4:13:00 PM UTC-5

---

Personal Health Dashboard seems like its being udpated faster.

https://phd.aws.amazon.com/

---

Opened at: October 18, 2017 at 4:10:19 PM UTC-5

DirectConnect operational issue

02:10 PM PDT We are investigating network connectivity issues affecting Direct Connect customers using the US-WEST-2 Region.

---

Opened at: October 18, 2017 at 3:59:42 PM UTC-5

Network Connectivity

01:59 PM PDT We are investigating Network Connectivity issues in the US-WEST-2 Region.

---

Opened at: October 18, 2017 at 3:59:00 PM UTC-5

EC2 VPC network health internet issue

Beginning at Wed, 18 Oct 2017 20:59:00 GMT, some instances are experiencing elevated packet loss between the us-west-2a Availability Zone and the Internet. We are now investigating this issue.

bagels
Title says AWS West, but it's specifically us-west-2 region (Oregon).
Meekro
I just noticed Stripe's API returning intermittent 500 errors for the first time in forever. Now I know why! =)
muliwuli
yeah so did we.. and then also from auth0. aws status page was next thing to check.
xer0x
We noticed less network connections around 1:30ish...
geetfun
Yeah noticed this in our production app today. Thanks AWS. Thank goodness we had multi AZ DB running and other failovers.
godelmachine
Would they be publishing a post-mortem? If yes, where do I get to read it? Someone be kind enough to enlighten me. Sincerely,
anonth
Yes,if you can clear their hiring bar, post mortems for all such incidents are shared among AWS employees... Some of them are shared company wide