HN Explorer

81 comments

[ 3.6 ms ] story [ 189 ms ] thread
rwg
At ex-work, SSL certificates expiring was probably #3 in the list of reasons for services being unavailable. It's a really easy thing to overlook in service monitoring.
spamizbad
When I was a contractor, this happened to one of my customers that was under contract (I set up the cert initially). I fixed the issue overnight without major impact, but boy was I embarrassed by the bush-league mistake.

Since then, in addition to service monitoring, I've taken to calendaring certificate expiration dates ~14 days before they expire. Low tech but I've never forgotten since.

yajoe
I thought this was a repost of the same story from last February related to expired certs:

http://www.wired.com/wiredenterprise/2012/02/azure_outage/

I'm pretty tolerant of mistakes (I make them all the time), but I have little patience for repeating mistakes.

smarx
In this case, an SSL certificate has expired. The leap year issue last year was a bug about date addition: 2/29/2012 + 1 year was calculated as 2/29/2013, which is a date that doesn't exist.

The only similarity is that both involved certificates. (The invalid date was used internally as an expiration date for a certificate.)

vyrotek
Glad to see you around these parts Smarx :) You maybe not remember me but I came to Redmond for an Azure bootcamp you hosted for a few companies a few years ago. (I'm sure you did lots of those) I was with IActionable. We've had a few 'encounters' on the MSDN forums too. Azure sure has changed since those days... Definitely for the better!

I see you've been busy since you left MS. Somehow I missed all the news about Site44!

varunsrin
The Azure Dashboard is here, for anyone looking for updates on this: https://www.windowsazure.com/en-us/support/service-dashboard...
photorized
I am not sure I trust that dashboard. Day before yesterday, when Azure SQL was down for the whole day, it took serval hours of 100% outages until the dashboard got updated with red "problem" icons. Before that, it was "information", "routine maintenance", etc.

Amazon likes to downplay their AWS issues as well (their console is always green), but at least they provide a very detailed postmortem, and issue credits proactively.

Have not seen that from MS yet, and it's been two days.

jspaur
Any info on Sql Azure being down a whole day? We didn't see anything. This issue on the other hand, a real bummer.
photorized
From what I saw, started as routine maintenance, and we began seeing SQL timeouts and errors on SQL East, and then all our DB servers on SQL East just vanished. Stayed gone for 5 hours at least.

I took a screenshot of the status console once it got updated with the outage info (several hours into the outage):

https://mobile.twitter.com/iTrendTV/status/30395990016434176...

Pretty unhappy about the whole experience. Luckily, we have our own colo in addition to Azure.

If you want to see people's reactions as this event unraveled:

http://social.msdn.microsoft.com/Forums/en-US/ssdsgetstarted...

jspaur
Ouch, we saw a similar thing in the South when things went sideways late last year. We've taken a similar approach to Sql Azure, use DataSync to replicate to offsite warm instances.
tarekayna
Yeah I didn't see anything wrong with my services 2 days ago, which rely heavily on Sql Azure.

Right now everything is down.

photorized
And we are seeng sporadic SQL connection errors today (between Azure VMs to Azure SQL), so whatever the issue was - doesn't appear to be fully resolved.
[deleted]
(comment deleted)
vyrotek
Everything is down for you? All our systems that only need the WebRole and SQL Azure are doing just fine for us.
tarekayna
Some individual services are up. But my system as a whole is down with Compute failing to start and Storage/Service Bus completely down.
vyrotek
Ah ok. Yeah, I'm unable to deploy anything new obviously since that goes to the Blob storage. That's actually how I first saw this issue. Apparently things that were already running are doing ok, but you can't publish anything new.
cek
This hit us too. The fact that it went down was enough to frustrate me but the fact the portal said everything was ok really pissed me off. We just launched our product the day before!
Element_
Just out of curiosity what were you hosting there? How have you found the azure experience overall aside from this latest issue?
photorized
Some data crunching servers, some VMs, some Azure SQL DBs.
photorized
Aside from the latest issue, been happy with the experience. Easy provisioning, scaling, plus Git deployment with easy rollback for web apps.
UnoriginalGuy
At least it is more honest than Amazon's version.
gabbo
If this outage is solely caused by an expired SSL certificate then this is really bad. It makes Azure look like amateur hour (even though it's definitely not). This may be acceptable for a smaller operation, but not Microsoft's cloud platform. There should be processes (manual and automated) to deal with this.

On the bright side, after this you know they're not going to make the same mistake twice.

niggler
"It makes Azure look like amateur hour (even though it's definitely not)"

Why do you think Azure is not?

gdc
Do you think that it is?
niggler
You can't deny that the current issue is certainly not something you would expect of a serious offering
gdc
Every major cloud offering on the market has had outages that were traced down to silly issues. I'm not sure if your dislike is targeted at just Azure, or the cloud in general.
niggler
I'm always apprehensive of outsourcing critical business infrastructure without a failover plan, and I find the excuse "amazon is down" or "azure is down" unsatisfying (especially when its a service customers pay for)
suresk
Having a "hot spare" cloud provider is going to be so complex and expensive that it probably would kill the idea of using Azure, Amazon, etc for most companies.

Unless you run your own DC, you are still outsourcing part of your infrastructure to a colo facility, which will not be immune to power and connectivity issues, and of course, issues with your setup.

Saying "<cloud provider> is down" certainly isn't very comforting, but if the aggregate downtime is less than if you were running your own setup, I'm not sure that being powerless in the event of downtime is reason enough to not use them.

breck
Why do SSL certificates expire? Seems like a suboptimal design decision.
hwatson
Take a look at me.com, for example. Before Apple bought it, it was owned by SnappVille.com. If SSL certificates didn't expire, SnappVille could have continued using their certificates for me.com.
paxswill
That still doesn't fully explain why they expire, as CRLs and OCSP allow certificates to be revoked. I can't quite explain why having an expiration date is safer, I just feel it's a good practice, to protect against possible key compromise.
CJefferson
But, they would have to get the private certificate? And if someone got that even while Apple still owned me.com, lots of nasty man in the middle attacks could have been made.
PeterisP
Sometimes you forget about the certificates you have (such as in this case). In a 'default-deny' security mindset, things you forget should expire so that you don't get bitten by a security compromise for a certificate that you issued in 1999 but still are "trusting".

Furthermore, private keys can be cracked, given enough time. Expiry dates reduce this risk with (a) less time to do it; (b) key is invalidated even if you don't know that it's been compromised, so every compromise ends sooner or later, and (c) renewed certificates can be made with longer keys as hardware gets more powerful.

moonboots
I've been using the following script in my build process to avoid certificate expiration surprises:

  currentDateEpoch=$(date +%s)
  expirationDate=$(openssl x509 -in $my_cert -enddate -noout | sed 's/notAfter=//' | date -f -)
  expirationDateEpoch=$(date -d "$expirationDate" +%s)
  diff=$((expirationDateEpoch - currentDateEpoch))
  
  oneMonthInSeconds=$((30 * 24 * 3600))
  oneWeekInSeconds=$((7 * 24 * 3600))
  
  if [ "$diff" -lt $oneWeekInSeconds ]; then
    printf "Certificate $my_cert needs to be renewed! Expires on $(date -d "$expirationDate" +"%B %_d, %Y")\n" >&2
    exit 3
  fi
  
  if [ "$diff" -lt $oneMonthInSeconds ]; then
    printf "Certificate $my_cert expires in less than a month! Expires on $(date -d "$expirationDate" +"%B %_d, %Y")\n" >&2
  else
    printf "Note: certificate $my_cert expires on $(date -d "$expirationDate" +"%B %_d, %Y")\n" >&2
  fi
smhinsey
If your connection strings are set in .cscfg config files, remember that they are editable through the Management Portal and you can swap them to HTTP.

Edit: Maybe not such a great idea. It looks like this relies on Storage under the hood as well. Now my instances are recycling.

Later: It appears doing so actually corrupted the instance and required a redeploy, which is of course not currently possible. Awesome.

rmason
Letting domains and SSL certs expire has been a consistent problem for Microsoft. A few years back they lost a key domain that took down Hotmail: http://news.cnet.com/Good-Samaritan-squashes-Hotmail-lapse/2...

Couldn't find it but I know they've had other embarrassing domain or certs expire over the years. You would think they'd fix it before they lose microsoft.com!

InvisibleCities
Looks like somebody forgot to set up a share-point reminder.
[deleted]
(comment deleted)
[deleted]
(comment deleted)
Gargol
Can't logon to management dashboard. Staging instances went crazy. Live environment looks OK so far. Should I be worried and stay up all night monitoring this stuff ?
niggler
Does anyone use Azure for a production service? I'm interested in hearing people's thoughts ...
jborden13
Yes overall the service has been good for us. We are mostly down, with a few services partially up but throwing errors and causing more confusion. Now we are just sitting on our hands getting pelted by clients...
niggler
"Now we are just sitting on our hands getting pelted by clients..."

You don't have a backup plan?

jborden13
We do for our apps and database, but not for our blob storage & caching.
jspaur
We use Azure pretty seriously and overall, we really like it. We went with the MSFT stack due to some of our low level code to deal with printing. That being said, it's had it's share of outages, but honestly no less than if we were racking and stacking our own machines. The most frustrating thing though (with any cloud provider really) is that if it goes down all you can do is sit and wait...atleast if its our fault we feel like we're in control.
niggler
" it's had it's share of outages"

Do you just sit on your hands when that happens? Do you have a failure plan?

jspaur
We do have a failure plan (and ironically enough were actively working on improving storage resiliency, this just hit us before we were able to deploy it)

That being said, there unfortunately is always a level 'sitting on hands' in events like this. If you're in dev you rely on your operations people (and vicea versa). You put your best people on it and continue to go on the best you can.

All that being said, a fired rill just for the sake of doing something may cause more harm than good. If a rollover plan takes 2 hours (switch DNS, migrate data, test, stabilize, etc.) a) often times that'll take longer than the original issue requires to be resolved and b) you'll run into new issues in your new location (maybe environmental, maybe not)

tarekayna
You seem to be asking more than one person about failure plans. What kind of a failure plan do you have in mind? For something as big as this (many azure services are completely down)
niggler
I generally worry about outsourcing a critical part of a business to one vendor without a plan for what happens if they go down. I'd ask the same question if people said they used AWS.
tarekayna
I agree to the general concept. I'm just curious about how you would handle it. Would you use AWS as backup to Azure and switch DNS when this happens?

Anyone else is welcome to pitch in here, I'm new to native cloud applications and find myself just waiting in this case.

photorized
We do, for some of our infrastructure at www.itrendcorporation.com. We use Azure Websites, Azure Storage, Azure SQL. Websites has been great. SQL East had a major (I mean, complete 100% outage for more than 5 hours) outage, so we are now making some changes internally.

I like Azure, but I don't think I am ready to use it in production for anything critical without another standby infrastructure in place.

niggler
"I like Azure, but I don't think I am ready to use it in production for anything critical without another standby infrastructure in place."

Seems like a really good general rule for any service :)

727374
We're using it for a bigish platform (10mm monthly uniques). PAAS Deployment is really nice. Also, knowing servers will be automatically patched in a timely manner is awesome. Support was very good but I've heard it's going downhill. SQL Azure is underpowered but highly durable and manages itself. If we really needed to scale up SQL the federations feature would work well. We've only had one brief SQL Azure outage in the last 2 yrs. We also have a DR site on AWS. AWS has a lot more bells and whistles and also had a nice head start. E.G. low-tech conveniences like sticky sessions on a load balancers or IP whitelisting on a variety of services are missing in Azure. Both services have had outages and we wouldn't feel safe using one exclusively.
niggler
How do you handle DNS (is it hosted on Azure/AWS or elsewhere)?
mikeocool
Added this tool to my suite of monitoring tools last time an outage caused by an expired cert popped up on hacker news: http://prefetch.net/articles/checkcertificate.html

Runs on a daily cron job and emails if any of the certs it's monitoring are within 30 days of expiration.

awa
is there something similar for windows?
dangrossman
For those of us using Trello to manage projects, it's worth adding a list of reminders for your certificate renewals. Set a due date on each card and Trello will highlight when they're coming up soon.
sriramk
I was involved with obtaining a previous version of the same certificate when I was at Windows Azure. There were several safeguards in place to stop exactly this scenario from happening. I'm wondering how they broke down.
magic_haze
Dev churn, probably. Could you expand a bit on the sort of safeguards that were in place? A couple of comments here mentioned trello, build scripts or just plain post-its, it'd be interesting to know how bigcorps do it...
sriramk
The most basic one was that really large sets of people that would get emailed a long time before expiry by the central crypto/cert management system. Microsoft has a very streamlined system internally for obtaining/managing cents since they do so much of it.
YokoZar
Perhaps that's precisely the problem -- a large set of people were notified, rather than anyone in particular
mpyne
Admiral Hyman Rickover (the "Father of the nuclear Navy") had as one of his basic principles that "if you can't point your finger at the person responsible for something, then no one is responsible for it".
runesoerensen
I'd like to invite you to try AppHarbor. We provide Windows application hosting on top of AWS, and strive to deliver a higher quality of service than Azure. We're standing by if you need help to migrate immediately.

We do a number of things to keep our platform and your apps more available. For instance we set up reminders before our certificates expire...

emp_
The thing that stops me starting to use apphb is the paid hostname, it is very backwards to feel like geocities all over again to host a simple Nancy blog. The service itself is amazing and congrats on the platform.
runesoerensen
Thanks for your input! I'm glad to hear you like the platform.

I think it's worth noticing that a bunch of services (including a worker) is included on the free plan. The $10/month to run a full app with custom hostnames is actually pretty cheap when you take that into consideration.

If your app/blog is focused on the .NET/Windows developer ecosystem you can apply for inclusion in the community application program (https://appharbor.com/page/community-application-program). If included we'll pay for your app.

fmax30
I have used Appharbor in my class to host a Web Pages project (Basically the prof made the whole class host it). My experience is , that while the customer support is good. The Service I used ( the free one) was a bit slow. My site took quite sometime to get loaded ( around 5-10 secs where my page weight was around 300-400 KB ) , the same page took less than a second on my local service. But all in all I really liked AppHarbor :)
runesoerensen
The free worker is allocated the same amount of resources as paid ones, so that shouldn't be the problem. It definitely sounds like there must have been an issue somewhere if you experienced that kind of performance difference.

I don't think it was related to the page weight (unless you're on a very slow connection), nor the platform for that matter. I hope we were able to figure out a solution or give you advice on how to resolve it :-)

LogicX
A few services to check your SSL certs:

  http://checkmyssl.com/
  https://sslcheck.globalsign.com/en_US
Email notifications:

  https://www.sslshopper.com/ssl-checker
  http://www.serverexec.com/ (does domain expiration too)
Script to run yourself, does email notification:

  http://www.prefetch.net/articles/checkcertificate.html
Any others to recommend?
pasbesoin
Someone over there might want to fire up Project and put in some hard deadlines.