Ask HN: How has the Amazon AWS outage affected you?

31 points by dholowiski ↗ HN
Thanks to the Amazon AWS Outage today, I missed the chance to have a QR code generated by my app displayed in an ad in a major newspaper. My Heroku hosted app (I have several) is still down. My other app was experiencing a huge traffic (30X usual) right when the outage happened.

Both apps are just fun toys, but it hurts for them to be down at such a critical time.

How has the outage affected you?

37 comments

[ 9.6 ms ] story [ 83.4 ms ] thread
I wasn't able to view a Foursquare checking for a little while and I've had significant amounts of my morning news be related to AWS, the cloud, downtime, etc.
Reddit is down, so I got more work done today.
I didn't. I was too busy reading blog posts about the downfall of cloud computing.
I didn't! I kept taking naps and coming back to see if it was up.

Reddit actually helps me work more.

(comment deleted)
I woke up this morning in disbelief that it was still down. Its friday all I want to do is put off work and browse reddit.
My team's single sign on prototype was deployed to heroku so that we could develop our apps against it (Android, iPhone, WP7, jQuery Mobile). It would have stalled development, but with a bit of tweaking I was able to get the prototype deployed to cloudfoundry and everything continued as planned.
The company I work for offers a small hosted document management SaaS solution that uses Amazon as the hosted environment, so we've been having issues related to (surprise surprise) EBS.
Strangely enough, not at all. We have a few ec2 instances running in the us-east region, allegedly the region that was supposed to be affected. They have been up and available all day, without issues of any kind.
Our registration/login/download/stat servers for all our iOS apps (Free Books + Free Audiobooks + Classicly) are all hosted on Heroku, and have been completely down all day. Not good.
I run my main startup on Heroku, so OrangeSlyce.com has been completely down all day.
Might as well get some good out of it.... please describe what your service is.
Sadly, I almost replied "just check out the website." I made a site for graphic design students to find freelance gigs that are posted by local small businesses who need cheap, small amounts of design work done. Thankfully this hasn't been very big, and mostly just a side project recently, so not too many users are being affected.
That's a great idea, can't wait to check it out when it's back.
Actually, i was surprised to find i wasn't affected by it at all. Apparently everything i use either knows what they're doing or wasn't in the us-east-1 region.
Ditto, we're just in the process of moving to EC2, and were happy to find our staging environment (the only thing moved over so far) is happily healthy (of course it gets minimal load).
I manage four EC2s all with EBS and located on the east coast; luckily haven't had any issues.
I am running my phone processing/tracking app on EC2 so it's been up and down all morning. The fall back also has some dependencies on S3 so it has been a bad day.
I provide media streaming services via EC2 to a few customers - basically all of the non-live streams are unavailable.
Our database (RDS) is completely inaccessible -- so our website has been down since 2am or so, showing our maintenance page. It's backed up -- but we have no way to download the most recent backup and move to another server. We tried rebooting our db about 11 hours ago -- and it's been stuck in reboot mode ever since. We tried creating a copy from a recent backup -- but it's stuck in boot mode. We can't move the snapshot to the West Coast farm, so we're really pretty blocked until Amazon gets things in order.

A lot of people on the boards suggested Amazon make the automatic backups of RDS available for download for instances like these. Having a backup is great, but not if you can't access them in an emergency.

On the message boards, someone said that they had selected to have RDS automatically keep a copy in multiple availability zones -- and they said that worked for them. I'm not sure however, since a top post on HN is saying that all of the Virginia zones were effected -- so your mileage may vary.

"Offsite Backups" - Where Offsite means, not on Amazon's infrastructure. Seriously.
No effect at all that I've noticed. Tarsnap doesn't use EBS.
Nope. Our failovers are on self hosted servers, where the core of each app lives anyway.
Our test environments aren't available.
How has it affected me? My posting of the Wikipedia page on "Fallacies of Distributed Computing" made it to #2 on HN, and I got a big karma bump; and wasted more time than usual on HN today, discussing this stuff.

Also, since Quora was down, I didn't get my usual quota of Quora surfing in.

Otherwise, today has been "business as usual." :-)

We're temporarily down with no permanent damage. But we have no way to relay that info to our customers because everything, including our users' email addresses, is inaccessible. Big oversight that I will be fixing as soon as we're back up.

It feels absolutely terrible to only be able to assure customers that their data is safe _after_ they write a panicked email asking if their data is gone.

My website and e-mail have been down for 12 hours and counting. Alas, my only instance that has survived this is running a website that is still under development.
My site still works! (Thanks, prgmr ;))
triviatise.com is down, but we just ended our last giveaway to close out our alpha on the 15th so the impact is minimal. We are working to figure out what a good backup plan to heroku is. Im sure a lot of other people (including heroku ) are doing the same.
I was hosting a client's app on Heroku and they had a big pitch today during which they weren't able to show the site. Really, really horrible timing.
Yikes. That's how I feel... Yesterday I explained how my site, although very ugly, could handle all the traffic you could throw at it and more. Today, it can't handle any.
My leaderboards, user-created levels and other miscellaneous bits of Playtomic run on AWS via MongoHQ and I've spent 1/2 the day answering emails explaining the situation. On the plus side MongoHQ are planning to introduce the ability to mirror databases onto our own servers which is going to be frigging awesome.

It at least highlighted another issue - because users are trying to access databases that don't exist they're taking ages to time out which is bogging down the servers' other thousands of requests/second.

Startup's site has been down all day, probably lost about ~$500-$1000 in sales, more concerned about the longer term loss of goodwill and the potential damage to google seo rankings though.
I run an app that helps farms manage their weekly or monthly CSA programs. One farm had their monthly distribution yesterday - thankfully they had already downloaded the data they needed to pack and label the orders, but they weren't able to tell customers their final total or send reminders, so they had higher-than-average missed pickups and are receiving money several days later than usual. Farmers are already distrustful of digital/automated solutions, so I'm sad that this has added to that distrust.