couldn't get that to work for me. I tried doing an incognito window, hitting the back button, etc. I thought it was something specific to me as I had just killed a shell session and deleted my only droplet when I noticed. It's up now though, glad I saw this post though thanks.
I have been in that same situation before so I built StatusGator.io. It basically alerts you when a service posts a status update so you aren't spinning your wheels trying to figure out what's wrong with your app.
EU-West is also experiencing problems. A lot of our instances are currently in connection_lost status.
EDIT: The console is working fine at the moment
EDIT1: Apparently OpsWorks is hosted and managed by the North Virginia data center which is why all our opsworks instances in EU-West are experiencing issues too.
hey fredonrails, what u can do is just login to console , when u see 500 error page . Just hit the back button you will be logged into the account. this seemed to be working for me please try
CAn confirm we are also experiencing massive issues, took AWS about an hour to update status page. Attempted to file support ticket and that timed out, when the request finished it had created 10 duplicated tickets and still waiting on the phone call from AWS RE ticket.
Some of my Portland neighbors are concerned with rising rents as well as Oregon's No-Fault evictions.
San Francisco too. While it has rent control there are such workarounds as remodeling an apartment.
They figure they can give all the hipsters the boot by cutting optical fiber. Good thing it's clearly marked with all those "Call Before You Dig" signs!
tl;dr: backhoes are cheap to rent and easy to operate.
AWS Boston failed the same time Portland and San Francisco did, but I don't know why yet. My concern when I posted the above was the Livermore cut, I figured Vladimir Putin had grown weary of America's craft beer export restrictions.
"The FBI is investigating" according to all the newspapers, but when I reported the Portland cut to their national crime tips hotline, the FBI agent I spoke too didn't even know there was such a thing as optical fiber.
There was a cut in San Jose in June 2010.
8-1-1 exists specifically to prevent fiber cuts but there is no way to report that a cut had actually happened. Santa Clara Vallet 9-1-1 hung up on me when I requested they connect me to a Network Operations Center.
6-1-1 is what I wanted. I called the FBI for the Portland cut because 6-1-1 was down as well.
Emergency Responders all use radio for many good reasons but that Portland cut disabled 9-1-1.
What actually made the news around here?
A local TV station's cable channel was down for a few hours.
Mom's Earthlink was down. "That's because of the cable cut, Mom. There is nothing that Earthlink Tech Spport can do."
"That's fine, so long as I can get my eMail!"
Mom pays for her dialup, therefore she expects it to work.
We've been having issues with SES N.Virginia, but have managed to get mail through. It's intermittent, but we are getting our batches out. It's not pretty.
Our e-mails are more or less mission-critical, so we try to send, then try to send again -- I suspect the additional failures are causing this to back up a bit. For concrete numbers, of 27 e-mails we tried to send, only 2 got out successfully.
Also of interest is that I only got a 454 ("Temporary service failure") three times -- the rest of the time, it just freezes/times out in ssl.
For some users of AWS this would be irony, its also schadenfreude for the Ops/Sysadmins who tell management types that just trusting everything to AWS is a bad idea.
Ok. If you're receiving errors, and you're NOT using opsworks, please respond. We're using opsworks too and have ~30 servers down. Maybe we all should be looking at the opsworks agent.
We are getting errors across multiple AWS API's. It's nothing to do with Opsworks itself, rather it appears like there is an internal networking issue.
Both SQS and SNS were erroring and now SQS has gone down completly with all requests timing out.
Looks like there's critical infrastructure in us-east-1 that's broken and causing a ripple effect across all of AWS
Our platform is entirely hosted in ap-southeast-2 but we've had our EC2 instances deregistered and OpsWorks reporting them terminated where EC2 is showing them active and they're still reachable via SSH
Yeah, we don't use OpsWorks and had SQS/SNS/SES trouble as well -- thankfully those are not used to serve production traffic. From the set of services affected, it looks like Amazon's internal Kafka-like pub/sub system went down.
Can confirm. We were also seeing SQS errors around that time. We had some Opsworks instances reboot as well, but only minor outages overall (us-east and us-west).
I lost my buildserver - uncontactable, but not terminated. Can't even 'force off'. It had nothing to do with any form of AWS provisioning (manual ansible; a job for Monday), and it's a relatively recent machine (couple of months, t2.medium). Got an email from AWS that the host had degraded, and I noticed the instance was having weird disk issues earlier.
"Description:
The instance is running on degraded hardware"
There are a lot of ways to integrate, but we use datadog and I generally find it excellent. Alerts to hipchat, pagerduty, etc, with AWS Cloudwatch integration. Of course, you have to assume the Cloudwatch API is working... :)
djbdns enables one to configure multiple IPs for just one http server.
Host with multiple, geographically distant data centers. If any one server drops offline, remove it from your tinydns "data" file. For that to yield joy you need a shorter timeout than is normally used for webserver DNS.
Each data center must be operated by a different company. The problem tonight appears to be internal to Amazon, so different AWS data centers won't help you.
I only recommend four. The people that run them know more about hosting than you or I could ever hope to:
Ivory Tower Group LLC is no longer in the hosting biz but I expect I can convince them to return to it, given tonight's market opportunity. If so I recommend them highly as well, and will post their hosting service URL in a reply.
PRQ is somewhat pricy, but:
Our boundless commitment to free speech has been
tested and proven over and over again. If it is
legal in Sweden, we will host it, and will keep it
up regardless of any pressure to take it down.
We have ZERO tolerance against SPAM and related services!
We defend your integrity to the end. With our
discreet customer relations policy we don't even
have to know who you are, and if we do, we will keep
that knowledge strictly confidential.
PRQ has its own in-house legal team to fight takedown demands. However they only rent physical servers, they don't do VPS.
Truth In Labeling: BudgetDedicated and prgmr.com host me for free, prgmr.com because I helped them with their marketing, BudgetDedicated because the server I use hosts an Open Source and a Free Software project.
Ok, I can't tell if this is functioning normally. I am trying to launch a new instance. Numerous services have "increased API error rates" one even said "Elevated error rates"
All services panel were green indicating: Service is operating normally
I can't put confidence in this. Anyone have any info on whether this is resolved. I can do some on digital ocean, but I really need a few things on AWS. Confirmed up? COnfirmed working? Info?
Edit: Successfully launched an AMI instance and it deployed successfully and is accessible from shell and ip.
We have 4 projects in Amazon OpsWorks, all of them in eu-west. Only one of them has been affected. ELB removed instances from balancing, and then instances were automatically shut down. Now when we try to create new instances they freeze with "requested" status.
What an awful way to start a day (in Europe).
Update: we just found another of our projects has the worker instance being shut down (by Amazon, we didn't touch it).
Update 2: our new instances are finally being created and running setup.
Update 3: we found that the Health Check for our Elastic Load Balancer changed its path to the default one (index.html). We had to edit it again.. This is weird, I don't like the idea of Amazon changing our Health Check path for no reason.
I'm in the same situation on US West on OpsWorks. Servers all began to terminate and will not start back up. I put too many eggs in the OpsWorks basket, I suppose.
All our production servers in OpsWorks had "Connection lost" status. Currently most of them is back "online" but from time time downtime (few minutes) comes back on some instances. AWS Support is not working (50x and 40x errors) - they have moved the support for production to amazon.com
all our opsworks servers have been unreachable from the internet since around 05:13Z when we see a mysterious 'configure' command with no further details in the opsworks logs
Fortunately the Opsworks service wasn't able to contact EC2 API so although it attempted to terminate the servers, the termination call failed so the instances are still alive and functioning but there is now a disconnect between Opsworks and EC2
Chiming in as I was completely unaffected... maybe this can help someone here.
I switched our applications off of OpsWorks a couple months back after losing faith in the OW team, in favor of a lower-level CI flow using auto-scaling groups and CodeDeploy.
Aside from some headaches in the very beginning (where CodeDeploy maintenance window would crash the daemon and health checks would cause an "initialize/destruct" loop) it precluded this issue entirely.
Two of our boxes were affected by the outage and disabled, however they were re-initialized within minutes by the ASG, meaning we experienced essentially no downtime.
Just be aware that if you use CodeDeploy, it's essentially just a low-level deployment hook which takes the packaged revision passed to S3 from your continuous integration setup, unpackages and runs any initialization scripts you require. You'll need to configure the security groups and scaling policies on your own which is something I know OpsWorks tries to make easier with their higher level app/layer constructs...
68 comments
[ 2.9 ms ] story [ 141 ms ] threadCurrently can't raise support tickets either, so left in the dark until they fix that...
EDIT: The console is working fine at the moment EDIT1: Apparently OpsWorks is hosted and managed by the North Virginia data center which is why all our opsworks instances in EU-West are experiencing issues too.
There goes the weekend!!
San Francisco too. While it has rent control there are such workarounds as remodeling an apartment.
They figure they can give all the hipsters the boot by cutting optical fiber. Good thing it's clearly marked with all those "Call Before You Dig" signs!
http://www.warplife.com/tips/defense/national/c3i/fiber-opti...tl;dr: backhoes are cheap to rent and easy to operate.
AWS Boston failed the same time Portland and San Francisco did, but I don't know why yet. My concern when I posted the above was the Livermore cut, I figured Vladimir Putin had grown weary of America's craft beer export restrictions.
"The FBI is investigating" according to all the newspapers, but when I reported the Portland cut to their national crime tips hotline, the FBI agent I spoke too didn't even know there was such a thing as optical fiber.
There was a cut in San Jose in June 2010.
8-1-1 exists specifically to prevent fiber cuts but there is no way to report that a cut had actually happened. Santa Clara Vallet 9-1-1 hung up on me when I requested they connect me to a Network Operations Center.
6-1-1 is what I wanted. I called the FBI for the Portland cut because 6-1-1 was down as well.
Emergency Responders all use radio for many good reasons but that Portland cut disabled 9-1-1.
What actually made the news around here?
A local TV station's cable channel was down for a few hours.
Mom's Earthlink was down. "That's because of the cable cut, Mom. There is nothing that Earthlink Tech Spport can do."
"That's fine, so long as I can get my eMail!"
Mom pays for her dialup, therefore she expects it to work.
Their tech support is excellent, it is run far far better than Amazon could ever hope to achieve.
If any prgmr.com server ever goes down, Luke's girl puts his testicles in a bench clamp until that server comes back up.
Also of interest is that I only got a 454 ("Temporary service failure") three times -- the rest of the time, it just freezes/times out in ssl.
Both SQS and SNS were erroring and now SQS has gone down completly with all requests timing out.
Our platform is entirely hosted in ap-southeast-2 but we've had our EC2 instances deregistered and OpsWorks reporting them terminated where EC2 is showing them active and they're still reachable via SSH
We also have a bunch of files on S3 which Cross-Region Replication hasn't yet replicated... I think that depends on SQS as well.
"Description: The instance is running on degraded hardware"
Whats the recommended way to monitor if instances have become detached or non responsive? Want to immediately alert slack or send an email, etc.
Host with multiple, geographically distant data centers. If any one server drops offline, remove it from your tinydns "data" file. For that to yield joy you need a shorter timeout than is normally used for webserver DNS.
Each data center must be operated by a different company. The problem tonight appears to be internal to Amazon, so different AWS data centers won't help you.
I only recommend four. The people that run them know more about hosting than you or I could ever hope to:
Ivory Tower Group LLC is no longer in the hosting biz but I expect I can convince them to return to it, given tonight's market opportunity. If so I recommend them highly as well, and will post their hosting service URL in a reply.PRQ is somewhat pricy, but:
PRQ has its own in-house legal team to fight takedown demands. However they only rent physical servers, they don't do VPS.Truth In Labeling: BudgetDedicated and prgmr.com host me for free, prgmr.com because I helped them with their marketing, BudgetDedicated because the server I use hosts an Open Source and a Free Software project.
Our instances are docker hosts, network seemed to lag/stop when proxying traffic to the internal container IP addresses.
Our ASG spun up other instances but health checks reported "Insufficient Data". The web console also seems buggy (API requests are failing).
All services panel were green indicating: Service is operating normally
I can't put confidence in this. Anyone have any info on whether this is resolved. I can do some on digital ocean, but I really need a few things on AWS. Confirmed up? COnfirmed working? Info?
Edit: Successfully launched an AMI instance and it deployed successfully and is accessible from shell and ip.
What an awful way to start a day (in Europe).
Update: we just found another of our projects has the worker instance being shut down (by Amazon, we didn't touch it).
Update 2: our new instances are finally being created and running setup.
Update 3: we found that the Health Check for our Elastic Load Balancer changed its path to the default one (index.html). We had to edit it again.. This is weird, I don't like the idea of Amazon changing our Health Check path for no reason.
Can you please post your OpsWorks instances ids on the AWS forums and just c/p what you wrote here?
https://forums.aws.amazon.com/forum.jspa?forumID=153
thanks.
EDIT: they are in EU-West zone btw
I switched our applications off of OpsWorks a couple months back after losing faith in the OW team, in favor of a lower-level CI flow using auto-scaling groups and CodeDeploy.
Aside from some headaches in the very beginning (where CodeDeploy maintenance window would crash the daemon and health checks would cause an "initialize/destruct" loop) it precluded this issue entirely.
Two of our boxes were affected by the outage and disabled, however they were re-initialized within minutes by the ASG, meaning we experienced essentially no downtime.
Just be aware that if you use CodeDeploy, it's essentially just a low-level deployment hook which takes the packaged revision passed to S3 from your continuous integration setup, unpackages and runs any initialization scripts you require. You'll need to configure the security groups and scaling policies on your own which is something I know OpsWorks tries to make easier with their higher level app/layer constructs...
Could you share more about your code-deploy setup and auto-scaling ?