HN Explorer

Ask HN: Is AWS S3 down?

38 points by coltonv ↗ HN
Our personal webapp is having S3 problems, and I can't seem to upload files for slack. Is anyone else having these issues?

Generally getting 503 slowdown errors and more recently 500 internal server errors.

30 comments

[ 2.0 ms ] story [ 68.8 ms ] thread
coreymayo
I am getting a 503 Slow Down from S3 requests.
greenhathacker
We are seeing both "Slow Down" and "Please reduce your request rate."
ihaveajob
Tough luck if your requests are triggered by users.
coreyw
We are noticing the slow down and the rate reduce issue only on GET requests. Our PUT requests seem to be going in just fine.
actuator
Encountering Slow Down errors as well
actuator
The errors seem to be reducing now.
jpetrucc
Getting 503's and slowdowns for everything on s3 and cloudfront randomly
[deleted]
(comment deleted)
donPinOB
getting this as well, 500 & 503
topstriker515
having problems here as well. Slow down errors, 500 errors, and a random EC2 termination we suspect might've been related to EBS
sigil
Yup I'm seeing 503s in us-west-2 (EC2 instance talking to S3), but outside EC2 is okay.
kainosnoema
Also getting 503 Slow Down and 500 internal server errors from S3 in us-east-1
mmontagna9
Yes we are getting slowdowns and internal errors as well. Also noticed one of our EC2 instances shutdown due to failed health checks.
leevlad
Yup, getting elevated error rates uploading to S3 - writes failing as well as closing the uploads. Also seeing 503 slow down as reported by others.
bigjust
S3 in both US-East-1 and US-East-2 are seeing these errors
jwg2s
Yep, we're seeing that but recovering now it seems.
xfax
Yup, it looks that way. Getting this error when using the CLI:

An error occurred (SlowDown) when calling the ListObjects operation (reached max retries: 4): Please reduce your request rate.

cperciva
Issues between 18:39 and 18:58 UTC. First InternalErrors, then SlowDowns -- I'm guessing due to nodes being taken out of service when they failed.

Given how quickly this was fixed, I suspect they were deploying new code and the rollout was automatically aborted when the elevated error rates were detected.

ljosa
Still seeing SlowDowns as of 19:08Z.
cperciva
They had stopped for 10 minutes, but just started up again.
sigil
And 10+ minutes after this thread was started, the AWS status page finally acknowledges there's an issue. HN really is a best-in-class monitoring service. https://status.aws.amazon.com/
ihaveajob
Question to folks out here: How do you deal with this? Ostrich strategy, or do you actively manage via multi-site redundancy?
actuator
We have support for other S3 regions in which we duplicate the data but the switch is manual not automatic.
jwg2s
Replicate to additional regions, point other services to these replicated buckets until issue is resolved.
paulddraper
Multi-region redundancy.

But with things in "read only" mode, to make the replicating simpler.

paulddraper
For us, lots of errors, then retries, then rate limits because of the retries.

---

On our account dashboard

> 11:58 AM PDT We are investigating increased error rates for Amazon S3 requests in the US-EAST-1 Region.

On AWS's status page https://status.aws.amazon.com/

> No recent events.

Damn thing never works.

vhost-
Seeing issues with cloudformation as well.
xfax
Does anyone else have some of their buckets missing from the console list?
ajmarsh
11:58 AM PDT We are investigating increased error rates for Amazon S3 requests in the US-EAST-1 Region.

12:21 PM PDT We can confirm that some customers are receiving throttling errors accessing S3. We are currently investigating the root cause.