Yup, my sites are down. I'm sure it's something new and complex, like a ferret or a leprechaun got into the 7th backup power switch which overloaded this and that.. as usual something impressively difficult to plan for, but come on! Virginia you're killing me...
For info, Coursera seems to randomize the assignments. Specifically, in the case of multiple choice tests it seems to pick a random subset of answers to each question, and randomize the order. This is obviously a defense against dumb copying.
What it does mean though is that you can't resubmit your own work. I've re-done the assignment twice this evening (the questions change enough that you have to think a lot, even when re-doing the same assignment), only to get 500 errors both times.
Anyhow, I'm sure it'll all work out and these temporary technical problems are but minor hiccups in what is a fantastic course and a fantastic learning system. It's very, very much appreciated.
> I've re-done the assignment twice this evening (the questions change enough that you have to think a lot, even when re-doing the same assignment), only to get 500 errors both times.
It's been about 8 years, so my memory may wrong, but I believe WebAssign used do have some sort of adaptive homework, where the questions would be selected based on your answers to previous questions (the same way the GMAT or GRE works).
I'm surprised that Coursera doesn't do something of the sort, both to avoid the problem that you're having (the 'state' of your homework would be updated each time you submit an answer to a question), and also to capitalize on some of the flexibility of digital learning that isn't possible on paper.
The choices can change between each time you initiate a quiz. However, if you complete part of a quiz, save, then return later, the choices will be the same, but the order will be different.
Not all of the courses are set up with multiple choices though. In addition to varying sets of correct responses, some answers are mathematical. For instance it is defined such that a question might ask ("What is %d plus %d", a, b), then have choices involving the quantities: ("%d", a+1), ("%d", a*b), ("%d", a+b), etc. with a different random quantity for a and b each time you start a quiz.
Interestingly, I saw one course where it had single-choice questions, with more differences than simply right or wrong. For instance one answer gave 100%, another 80%, and another 0% of the points available for that question.
Amazon, those masters of tactful understatement, say:
> 10:38 AM PDT -- We are currently investigating degraded performance for a small number of EBS volumes in a single Availability Zone in the US-EAST-1 Region.
Not only that - the "information" icon that they use makes it very hard to spot issues quickly. Even on a minor issue it would be useful to have some kind of different icon.
AWS cleverly designed a very informative status dashboard: green means "things might work", green with a little speck on it means "nothing is working, the entire datacenter is down".
About 33 minutes after they first acknowledged the problem, they've changed one of those indicators to as goldenrod-yellow color. Presumably this indicates that the world is on fire and the sky has gone black with carrion crows:
> 11:11 AM PDT -- We can confirm degraded performance for a small number of EBS volumes in a single Availability Zone in the US-EAST-1 Region. Instances using affected EBS volumes will also experience degraded performance.
Amazon clearly needs a change in management at the head of AWS. The engineers are either petrified of telling the truth (very, very bad) or incompetent (less bad but still bad). And it has been this way for an unreasonably long time.
This has been noted almost every time AWS has outages, and yet it still amazes me that they keep formulating it as if "you might get slow response time, but everything works".
While it is not solely AWS' fault (cloud !== no need to do proper engineering) reddit etc. is down, I do have my doubts that "degraded performance" makes all of the ec2 instances reddit runs on go haywire and shuts down the entire website, leaving a static "come back later" html page.
Incredibly frustrating that they understate the magnitude of the problem. We have many instances that rely on ebs and are currently unresponsive. We suspect there's a massive flood of ec2 customers trying to restart instances to fix the ebs problems, which is slowing us down in getting our service back up.
I checked out the AWS status page, it says EBS is busted in NoVA, yet the status is still green. I think someone needs to fix their UI problem statte...
All they had was a little information icon by the green circle with a hover message letting you know what the problem was. They need to make that circle red!
I took our Web sites offline (RadioReference.com / Broadcastify.com) so our back end processes (audio archiving, audio serving etc) don't crater.
Our Master MySQL server uses EBS (striped RAID across 4 EBS disks) and it is getting killed due to severely degraded EBS performance. There is definitely a major problem with EBS at this time.
We are running our sites across three AZs, all in US-EAST. Our three web servers (one in each AZ) are not even able to load static test pages from their EBS volumes at this time. We also have a Percona MySQL cluster across three AZs and we are seeing very degraded perfomance in the EBS volumes on all three servers as well. It seems like this is impacting more than a single AZ from what I'm seeing.
Well, we've gone from "degraded performance" to completely losing our EBS instance store for our master MySQL server. If we don't see resolution shortly we might have to look at promoting one of our slaves and moving forward from there.
We ended up promoting a slave to master, and moving forward from there. Our master eventually returned to service, and we'll promote him tomorrow after things settle down.
Interesting to note that we issued a reboot on our Master and it went away and didn't return for over 45 minutes - we thought for sure it would have to be terminated. API calls and console access was severely restricted, so even launching in new AZ's was problematic.
Or build your application with the characteristics of the underlying platform in mind. Nobody would run a DB server on a single spindle drive since occasionally they fail. Accordingly, nobody should deploy non-multi-az AWS applications because occasionally they fail.
as a matter of fact, our app is using multi-zones not just for instances but also for RDS. As of now, I can't reach anything including the loadbalancer.
Possibly but there are bigger companies using RDS in production – also, the multi-az feature had worked out better than my own custom setup. Unfortunately lack of resources prevent me from rolling out my own setup on bare metal.
Haha, I probably should. A friend of mine has already suggested I would be better off doing a PaaS startup because of my sysops skills. I always felt the market is currently too saturated. But looking at Heroku having trouble themselves, may be I should give it some serious thought.
I found it to be pretty non-sensical to be honest. Not sure what's wrong with having a simple table with title/description/timestamp columns. I'm going to that page to figure out why my site isn't working, not view some designer's portfolio.
The lesson is: don't build a system that requires or advertizes high avilability but that runs in a single datacenter. Load balance. Elect leaders. Batch, queue, and distribute. Fail-over. Fail fast. So long as we have earthquakes, lightning, human malice, and human error, single AZs will fail.
I gave up on Amazon (and ate the reserved instance fees) 2+ years ago over this. I was paying through the nose for multi-AZ RDS and when EBS failed in one zone, nothing failed over.
As an AWS user since 2009, I can only imagine the tremendous difficulties of running such a service but given the stability of my long-running instances in US West vs East, I really do sometimes wonder if Mr Bean is on staff in Virginia.
"11:03 AM PDT We are currently experiencing connectivity issues and degraded performance for a small number of RDS DB Instances in a single Availability Zone in the US-EAST-1 Region."
Given that my sites are deployed using Multi-AZ RDS instances and yet they're still down, this takes the cake a little.
And this is why my new venture will be going pinboard.in's route and hosting everything on our own machines. More work for me but vastly cheaper and hopefully a lot more reliable.
Color me ignorant(no really, I am), but really- cheaper? I thought part of why you used AWS was because short of the biggest properties, running your own servers was more expensive?
The only way AWS is theoretically cheaper (and even then I find the argument a bit of a stretch) is if you factor in "sysadmin costs". The argument goes that setting up and monitoring your own servers (especially if you go colo rather than renting) is more costly in terms of labour than point and clicking it with AWS.
If you strip the labour cost out of the equation, hosting it yourself can be up to an order of magnitude cheaper. Of course stripping the labour cost out is only possible if you can do it yourself :)
In my case, I can (as could maciej of pinboard.in fame).
Edit: also helps if you can find a well priced data centre or two. For me, I'm planning on going with hetzner.de and ovh.net
The value I assign to the time it takes me to setup the hosting myself is a lot less than the cost of running on AWS.
Admittedly, the calculation is made easier by the fact that I'm not building the next Netflix or Heroku, however, neither are 95% of the other start ups out there.
Running your own servers costs more for engineers/devops/sysadmins, but in terms of how much you pay to your provider, hosting your own is FAR cheaper if you have more than 1 or 2 instances. You can get a 10U GigE unmetered colo for $550/month from this place on WHT: http://www.webhostingtalk.com/showthread.php?t=1200231
I'm fairly sure that if you have 10 instances at EC2, you're paying FAR more than $550/month.
The other benefit EC2 gives you is quick/instant deployment of even really high-end hardware if you quickly out-pace your current capacity and need another box ASAP, whereas even if you were doing dedicated servers, it still takes a hour or two for them to put your new box in a rack and get back to you with the IP and such.
Quicker than if I was on AWS and needed to scale it up tonight :)
Seriously though, because the costs are so much cheaper you just over provision the main machines slightly and keep a spare box or two ready to go. If that isn't enough to smooth out the spike then I guess at least you've got a nice problem to have.
Even if your own infrastructure relies on bare-metal hosting, odds are very good that some external service is going to be based in at least part on AWS or a constituent service for any non-trivial service offering. Your objective of being completely fault-tolerant and independent of Amazon is going to be hard to attain.
Good point. In my case I think I'm in the clear. The only services I'll be outsourcing are DNS (via dyn.com), backup (via rsync.net) and CDN work via Rackspace. As far as I can tell, none of them have a connection to or rely on Amazon/AWS.
Fundamentally, Reddit is the exact same sort of website as Hacker News, but Hacker News seems to be operating fine. What are these unavoidable AWS-hosted services you're thinking of? I cannot think of any but I'm sure you know of several.
If you're running RightScale, you're relying on AWS. Parts of New Relic. An Amazon affiliate service.
More generally, other tools such as a CDN, DNS, email delivery services, Facebook authentication, Twitter feeds, OpenStreetMap. Google APIs, mapping, Analytics, and others, all rely on some third-party infrastructure, if not AWS itself. I'm not saying that all sites rely on all of these, but many use one or more.
The Web is becoming increasing interconnected in ways that increase fragility.
Been getting crap loads of alerts stemming from their North Virginia data centre. Got 9 of these yesterday ...
"You are receiving this email because your Amazon CloudWatch Alarm "awsec2-WM-Live1-i-8747ece1-High-CPU-Utilization" in the US - N. Virginia region has entered the ALARM state, because "Threshold Crossed: 1 datapoint (93.13) was greater than or equal to the threshold (85.0)." at ."
While it's more frustrating to have no control over the reliability, I suspect that most companies could not achieve the level of reliability from AWS by self-hosting, at anywhere close to the cost (inc. stafftime). Nothing is 100%, and getting as close to 100% as you can gets exponentially more expensive the closer you get.
Of course, this could change if AWS's reliability trends downward. But it's never going to be 100%, and neither is anything you do yourself.
I don't think this is true. US-East's incident history over the past 3 years is terrible. If you had a completely non-redundant set of rented servers sitting in Softlayer or Rackspace's data center over the same period, even factoring in expected hardware failures over the time period (replacing a hard drive or two, probably), you'd have been available more than even a multi-AZ deployment with failover in AWS US-East.
I've got heavily used servers at Softlayer that haven't had more than a few minutes of downtime in 4+ years, and have never been rebooted. During that same time, I ran a site out of US-East for two years (EC2 + EBS + Multi-AZ RDS + ELB), and had more downtime and spent more admin time working around significant issues with Amazon. Amazon neither saved me time nor money, nor did it provide better availability.
wonder if anyone's been keeping and publishing statistics on exactly how often AWS US East has been down. Perhaps it's more than I thought. Of course, most failures haven't been effecting _all_ of US East, so maybe what would be interesting to look at is just _some_ customer's data on exactly how much downtime they had in the past 3 years or whatever. Without knowing what the uptime/downtime was, hard to say how terrible it was, or how it compares to other options.
I have a server for a client in our colo, that has been up for 1753 days. (Client doesn't want to move to new hardware, etc. otherwise we would have moved them long ago).
During that time, there was maybe network outage (at 2AM mostly) of perhaps 2 hours.
Complexity reduces reliability.
Claiming AWS' uptime approaches that of fairly standard colocation in a decent facility, is laughable.
122 comments
[ 4.2 ms ] story [ 242 ms ] threadIt's rather annoying as the first assignments for the Alex Aiken's excellent Compilers course are due today.
For info, Coursera seems to randomize the assignments. Specifically, in the case of multiple choice tests it seems to pick a random subset of answers to each question, and randomize the order. This is obviously a defense against dumb copying.
What it does mean though is that you can't resubmit your own work. I've re-done the assignment twice this evening (the questions change enough that you have to think a lot, even when re-doing the same assignment), only to get 500 errors both times.
Anyhow, I'm sure it'll all work out and these temporary technical problems are but minor hiccups in what is a fantastic course and a fantastic learning system. It's very, very much appreciated.
It's been about 8 years, so my memory may wrong, but I believe WebAssign used do have some sort of adaptive homework, where the questions would be selected based on your answers to previous questions (the same way the GMAT or GRE works).
I'm surprised that Coursera doesn't do something of the sort, both to avoid the problem that you're having (the 'state' of your homework would be updated each time you submit an answer to a question), and also to capitalize on some of the flexibility of digital learning that isn't possible on paper.
Not all of the courses are set up with multiple choices though. In addition to varying sets of correct responses, some answers are mathematical. For instance it is defined such that a question might ask ("What is %d plus %d", a, b), then have choices involving the quantities: ("%d", a+1), ("%d", a*b), ("%d", a+b), etc. with a different random quantity for a and b each time you start a quiz.
Interestingly, I saw one course where it had single-choice questions, with more differences than simply right or wrong. For instance one answer gave 100%, another 80%, and another 0% of the points available for that question.
> 10:38 AM PDT -- We are currently investigating degraded performance for a small number of EBS volumes in a single Availability Zone in the US-EAST-1 Region.
http://status.aws.amazon.com/
Effectively, this is exactly how it is.
> 11:11 AM PDT -- We can confirm degraded performance for a small number of EBS volumes in a single Availability Zone in the US-EAST-1 Region. Instances using affected EBS volumes will also experience degraded performance.
http://status.aws.amazon.com/images/status1.gif
http://status.aws.amazon.com/images/status2.gif
http://status.aws.amazon.com/images/status3.gif
While it is not solely AWS' fault (cloud !== no need to do proper engineering) reddit etc. is down, I do have my doubts that "degraded performance" makes all of the ec2 instances reddit runs on go haywire and shuts down the entire website, leaving a static "come back later" html page.
Our Master MySQL server uses EBS (striped RAID across 4 EBS disks) and it is getting killed due to severely degraded EBS performance. There is definitely a major problem with EBS at this time.
Interesting to note that we issued a reboot on our Master and it went away and didn't return for over 45 minutes - we thought for sure it would have to be terminated. API calls and console access was severely restricted, so even launching in new AZ's was problematic.
Talk about having all your eggs in one basket.
Given that my sites are deployed using Multi-AZ RDS instances and yet they're still down, this takes the cake a little.
If you strip the labour cost out of the equation, hosting it yourself can be up to an order of magnitude cheaper. Of course stripping the labour cost out is only possible if you can do it yourself :)
In my case, I can (as could maciej of pinboard.in fame).
Edit: also helps if you can find a well priced data centre or two. For me, I'm planning on going with hetzner.de and ovh.net
Admittedly, the calculation is made easier by the fact that I'm not building the next Netflix or Heroku, however, neither are 95% of the other start ups out there.
I'm fairly sure that if you have 10 instances at EC2, you're paying FAR more than $550/month.
The other benefit EC2 gives you is quick/instant deployment of even really high-end hardware if you quickly out-pace your current capacity and need another box ASAP, whereas even if you were doing dedicated servers, it still takes a hour or two for them to put your new box in a rack and get back to you with the IP and such.
Seriously though, because the costs are so much cheaper you just over provision the main machines slightly and keep a spare box or two ready to go. If that isn't enough to smooth out the spike then I guess at least you've got a nice problem to have.
More generally, other tools such as a CDN, DNS, email delivery services, Facebook authentication, Twitter feeds, OpenStreetMap. Google APIs, mapping, Analytics, and others, all rely on some third-party infrastructure, if not AWS itself. I'm not saying that all sites rely on all of these, but many use one or more.
The Web is becoming increasing interconnected in ways that increase fragility.
"You are receiving this email because your Amazon CloudWatch Alarm "awsec2-WM-Live1-i-8747ece1-High-CPU-Utilization" in the US - N. Virginia region has entered the ALARM state, because "Threshold Crossed: 1 datapoint (93.13) was greater than or equal to the threshold (85.0)." at ."
Maybe their data centre was over run by mutants ?
Of course, this could change if AWS's reliability trends downward. But it's never going to be 100%, and neither is anything you do yourself.
I've got heavily used servers at Softlayer that haven't had more than a few minutes of downtime in 4+ years, and have never been rebooted. During that same time, I ran a site out of US-East for two years (EC2 + EBS + Multi-AZ RDS + ELB), and had more downtime and spent more admin time working around significant issues with Amazon. Amazon neither saved me time nor money, nor did it provide better availability.
During that time, there was maybe network outage (at 2AM mostly) of perhaps 2 hours.
Complexity reduces reliability.
Claiming AWS' uptime approaches that of fairly standard colocation in a decent facility, is laughable.