Seeing huge numbers of 503s from the S3 API in us-east-1. Anyone else having problems? I only found one other on Twitter: https://twitter.com/cperciva/status/630641484677558273
I'm seeing it as well - majority of connections are being dropped for us atm
The Amazon S3 team recently completed some maintenance
changes to Amazon S3’s DNS configuration for the US STANDARD region on
July 30th, 2015.
You are receiving this email because we noticed that your bucket
is still receiving requests on the IP addresses which were removed
from DNS rotation. These IP addresses will be disabled on August
10th at 11:00 am PDT, at which time any requests still using
those addresses will receive an HTTP 503 response status code.
Applications should use the published Amazon S3 DNS names for
US STANDARD: either s3.amazonaws.com or s3-external-2.amazonaws.com
with their associated time to live (TTL) values. Please refer to
our documentation at:
http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region
for more information on Amazon S3 DNS names.
Something to do with that perhaps? AWS sent us that last thursday
The default behavior in Java 7 & 8 is to cache forever when a security manager is installed, and to cache for an implementation-specific period of time when a security manager is not installed. Sans security manager, I see 30s in Java 7 -- reference the DEFAULT_POSITIVE constant in sun.net. InetAddressCachePolicy
same here. our s3 services are reporting similar 503's and network timeouts. a few of our partners are already down as well with their own 500s. another stormy night in the cloud.
I wonder what it would take for amazon to show one of the yellow icons on their status page? Has it ever happened? Would a datacenter have to fall in the ocean?
I'm sure "elevated error rates" is the first alarm which goes off. And once they've put that description onto the status page, they're probably more worried about getting it fixed than going back and changing the wording.
They should worry about that a lot. Amazon are notoriously bad at communicating during outages. They've gotten better, but they're big enough that it should have priority.
Yes and no. Sure, Amazon has administrative personnel. Sure, some of those administrative personnel would probably be happy to get paid extra to carry a pager and be summoned to work at 3AM to update a status page.
But the last thing you want to do is put inaccurate information onto a status page; so mere administrative personnel isn't enough -- you'd need people who understand enough about the system to be able to write about it without introducing errors.
I'm guessing that the intersection of "administrative personnel", "willing to carry pagers" and "understand the internals of AWS services" is a very small set.
This is a situation where "we pay you, now do as you're told" comes in handy.
Not every job can be full of self-directed aspirational spiritual awakenings. If that were the case, nobody would deliver my dinner on a bike when it's -20ºF outside.
>I'm guessing that the intersection of "administrative personnel", "willing to carry pagers" and "understand the internals of AWS services" is a very small set
Being a non-engineer doesn't mean they don't know anything about the technology. And they don't need to know the internals, just enough to convey information from the engineers managers to the public.
Plenty of other organizations manage resolving issues while transmitting information about the issue to other stakeholders.
Also, most administrative personnel have far less job opportunities than engineers. If they can get the engineers to carry pagers they can get a PR minion to carry one.
Except "willing to carry pagers" is currently the basis of employment at Amazon, and not just for AWS but for whole chunks of their technical business. It's one of the many reasons why they have a pretty dire reputation (see plenty of discussions on here from former Amazon employees).
They also claim to have "customer obsession" as a leadership principle, this whole thread is an excellent example of that being failed in a big way.
Whats frustrating is when you have customers who are also down because of the outage - but when you say Amazon is experiencing severe outages causing 50% of our requests to be dropped and there's not much we can do, it makes us look pretty bad when they they go to the amazon dashboard and only see "Elevated Error Rates."
Often times it's just that though. Just because many customers are experiencing something doesn't mean ALL customers are experiencing something. When I worked there what the media would describe as a major outage was really less than 1% of one region... this particular instance seems pretty odd though.
It's yellow now. I'm pretty sure the datacenter is still there. I think they go yellow/red after a certain time has passed or someone manually changes it (probably rarely)
Seems that "S3 offline" is the AWS equivalent of a datacenter falling into the ocean. Gotta wonder how many services are using S3 as a faux message queue?
Probably just fallout from the fact that EBS snapshots are stored in S3. If you can't create an EBS volume, you won't be able to launch an EC2 instance from it.
I can't help but be a little surprised that Heroku's entire build system is disabled by an S3 failure in one region. Now I'm unable to add a notice about the issues to my site's HTML...
Since S3 is the defacto artifact delivery system for most people that run on AWS, it's not much of a surprise. For the most part, very isolated incidents aside, S3 is rock solid. Even EC2 relies on S3 for launching non-EBS instances.
I'm also having issues connecting to buckets based in Ireland (eu-west-1). Just hangs at authentication stage. Tried from 3 different internet connections, all having the same problem.
Amazon Elastic Compute Cloud (N. Virginia)
Increased API Error Rates
12:51 AM PDT We are investigating increased error rates for the EC2 APIs and launch failures for new EC2 instances in the US-EAST-1 Region.
"1:08 AM PDT We believe we have identified the root cause of the elevated error rates and latencies for requests to the US-STANDARD Region and are working to resolve the issue."
Time and priorities. There is a difference between "I wish this thing existed, so I can use it/contribute to it" and "I need it so badly that I'm willing to spend a lot of time to make it production ready".
Depending on your use case, it may be slightly easier to accomplish this with s3 event notifications + AWS Lambda to write to a different region or service.
Importantly, make sure you CNAME your bucket under your own domain so that you can switch services.
Apache jclouds® is an open source multi-cloud toolkit for the Java platform that gives you the freedom to create applications that are portable across clouds while giving you full control to use cloud-specific features.
This is pretty much the case. Years of evangelising the idea that (a) everybody should be on Amazon and (b) everybody should be on the cheapest regions of Amazon mean that while the underlying datacentres are probably much better managed, individually speaking, than the tapestry of colos that made up the world a decade ago, an outage has much more wide-ranging effects than you'd get at that point.
141 comments
[ 3.1 ms ] story [ 104 ms ] threadhttp://javaeesupportpatterns.blogspot.ie/2011/03/java-dns-ca... has more detail.
Can't pull any images either.
12:36 AM PDT We are investigating elevated errors for requests made to Amazon S3 in the US-STANDARD Region.
Companies 1/1000th the size of Amazon can manage it.
But the last thing you want to do is put inaccurate information onto a status page; so mere administrative personnel isn't enough -- you'd need people who understand enough about the system to be able to write about it without introducing errors.
I'm guessing that the intersection of "administrative personnel", "willing to carry pagers" and "understand the internals of AWS services" is a very small set.
Not every job can be full of self-directed aspirational spiritual awakenings. If that were the case, nobody would deliver my dinner on a bike when it's -20ºF outside.
Being a non-engineer doesn't mean they don't know anything about the technology. And they don't need to know the internals, just enough to convey information from the engineers managers to the public.
Plenty of other organizations manage resolving issues while transmitting information about the issue to other stakeholders.
Also, most administrative personnel have far less job opportunities than engineers. If they can get the engineers to carry pagers they can get a PR minion to carry one.
They also claim to have "customer obsession" as a leadership principle, this whole thread is an excellent example of that being failed in a big way.
12:51 AM PDT We are investigating increased error rates for the EC2 APIs and launch failures for new EC2 instances in the US-EAST-1 Region.
edit; does not seem to affect all the buckets though. Only one of ours is experiencing this, others are fine.
> 12:51 AM PDT We are investigating increased error rates for the EC2 APIs and launch failures for new EC2 instances in the US-EAST-1 Region.
Amazon Elastic Compute Cloud (N. Virginia) Increased API Error Rates 12:51 AM PDT We are investigating increased error rates for the EC2 APIs and launch failures for new EC2 instances in the US-EAST-1 Region.
looks like the cavalry are coming
There are many use-case when paying 2x for storage is a reasonable tradeoff for higher availability and also be provider independent.
So far S3 seems to be reliable enough...
[1] https://libcloud.apache.org/
Importantly, make sure you CNAME your bucket under your own domain so that you can switch services.
edit: Much easier than AWS Lambda, actually: http://aws.amazon.com/about-aws/whats-new/2015/03/amazon-s3-...
Apache jclouds® is an open source multi-cloud toolkit for the Java platform that gives you the freedom to create applications that are portable across clouds while giving you full control to use cloud-specific features.
good luck to the on-call engineers at amazon!
Feels like in future... If cloud provider goes down... All internet will stop working :)