390 comments

[ 4.3 ms ] story [ 135 ms ] thread
Yes, we're seeing issues with Dynamo, and potentially other AWS services.

Appears to have happened within the last 10-15 minutes.

The RDS proxy for our postgres DB went down.
We're seeing issues with RDS proxy. Wouldn't be surprised if a DNS issue was the cause, but who knows, will wait for the postmortem.
This is widespread. ECR, EC2, Secrets Manager, Dynamo, IAM are what I've personally seen down.
I can't even see my EKS clusters
Signal is down from several vantage points and accounts in Europe, I'd guess because of this dependence on Amazon overseas

We're having fun figuring out how to communicate amongst colleagues now! It's when it's gone when you realise your dependence

Twilio seems to be affected as well
Couple of years ago us-east was considered the least stable region here on HN due to its age. Is that still a thing?
Alexa devices are also down.
Can confirm. I was trying to send the newsletter (with SES) and it didn't work. I was thinking my local boto3 was old, but I figured I should check HN just in case.
I cannot create a support ticket with AWS as well.
Can't resolve any records for dynamodb.us-east-1.amazonaws.com

However, if you desperately need to access it you can force resolve it to 3.218.182.212. Seems to work for me. DNS through HN

curl -v --resolve "dynamodb.us-east-1.amazonaws.com:443:3.218.182.212" https://dynamodb.us-east-1.amazonaws.com/

There's also dynamodb-fips.us-east-1.amazonaws.com if the main endpoint is having trouble. I'm not sure if this record was affected the same way during this event.
Cant even login via the AWS access portal.
Can't even get STS tokens. RDS Proxy is down, SQS, Managed Kafka.
Amazon.ca is degraded, some product pages load but can't see prices. Amusing.
Did someone vibe code a DNS change
docker hub or github cache internal maybe is affected:

Booting builder /usr/bin/docker buildx inspect --bootstrap --builder builder-1c223ad9-e21b-41c7-a28e-69eea59c8dac #1 [internal] booting buildkit #1 pulling image moby/buildkit:buildx-stable-1 #1 pulling image moby/buildkit:buildx-stable-1 9.6s done #1 ERROR: received unexpected HTTP status: 500 Internal Server Error ------ > [internal] booting buildkit: ------ ERROR: received unexpected HTTP status: 500 Internal Server Error

So much for the peeps claiming amazing Cloud uptime ;)
(comment deleted)
Wasn't the point why AWS is so much premium that you will always get at least 6 nines if not more in availability?
Why after all these years is us-east-1 such a SPOF?