64 comments

[ 2.9 ms ] story [ 115 ms ] thread
I can't even get to status.slack.com ... I really hope that isn't on production infrastructure!

GET https://slack.com/favicon.ico 503 Service Unavailable: Back-end server is at capacity

The fact that we can't get to it means that it is, surely? :)
| "Back-end server is at capacity"

DDoS?

(comment deleted)
503 = at capacity, default response an ELB returns when all the servers in the ELB are OutOfService (or missing).
Also AWS's error code for no healthy nodes in elb.
Wow, that is a fail hosting your status page in the same datacenter/environment as production. Honestly, a bit surprised, since Slack has always seemed on top of things.
If you can't get to the status page, it's probably a good indication the service is down :)
You're missing the point. It is for communication to your customers. Give them an outline of what is happening and an estimated eta to being back up.
IIRC this error message is throw when an AWS ELB is empty(in my experience). I'd imagine it could be thrown if all servers in the ELB are maxed out.
I can connect to my Slack team, but then "Apologies, troubles connecting to Slack" :/.
Ha. if you refresh it comes back up. Looks like only some of the nodes are down
(comment deleted)
(comment deleted)
Huh, Freenode is still working YEAH I WENT THERE.
And they laugh when you try to explain that decentralized services are good. :/
That is until the next netsplit. Decentralized servers mean partition events can and do happen.
I just see opportunities for netsplit parties.
I don't envy being their social media person right now.
I didn't notice an interruption on the desktop client and it's been logged in for about 7 hours now and slack.com looks fine. Interestingly enough https://status.slack.com is currently down while curling status.slack.com returns a page that says it's moved to https://status.slack.com
You mean the http redirect to https is up but the https site is down.
No I meant it returned an HTML page that said "this link as moved" with a link directing you to https://status.slack.com
Umm, yeah that is called a 302 redirect:

curl http://status.slack.com <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>302 Found</title> </head><body> <h1>Found</h1> <p>The document has moved <a href="https://status.slack.com">here</a>.</p> <hr> <address>Apache/2.4.7 (Ubuntu) Server at status.slack.com Port 80</address> </body></html>

The HTML parts are fallback for if your browser (or curl) in this case doesn't handle a 302

I'm getting SSL errors from httpie:

http: error: SSLError: hostname 'status.slack.com' doesn't match either of '2015-06-25.dev.slack.com', 'slack.com', 'slack-files.com', 'slack-redir.com', 'slack-redir.net', 'dev.slack.com', '.dev.slack.com', '.enterprise.dev.slack.com', 'dev.slack-files.com', 'dev.slack-redir.com', 'dev.slack-redir.net'

Just another reason for OSS projects to never trust closed (centralized!) protocols: no recourse when it (inevitably) hits the fan.
Except when you have to manage it yourself and something breaks or the infrastructure you have to manage needs managing.
See the comment above. If you're using Slack to avoid managing infrastructure, there are plenty of great (read: open source) alternatives.

Even if IRC isn't for you, there's XMPP, Mattermost [0], Let's Chat [1], and Zulip [2].

[0]: http://www.mattermost.org/

[1]: https://sdelements.github.io/lets-chat/

[2]: https://github.com/zulip/zulip

I feel like this is the same sort of argument that opponents of self-driving cars will use - "We shouldn't use self driving cars, because they have problems sometimes and a driver doesn't have recourse to correct it!"

While this might be true, we are still trading more problems for fewer problems. Yes, we can't fix slack when it breaks, but it still breaks a lot less than when we were running our own internal chat client.

Except, of course, that self-driving cars do have a recourse for the driver - manual overrides. I do see your point, however, and you're correct in that running an internal chat network probably isn't the best course of action for most groups/small business. Luckily, that's what IRC and XMPP networks are for.
This is why IRC is superior.
(comment deleted)
Apparently you've never experienced repeated netsplits under DDoS.
What's the real world analogy to Slack being down? Locking your entire team out of the offices?

Since the start, at our company we've used Slack. We have never experienced an outage like this. Definitely a strange and uncomfortable feeling not having Slack available.

My company is unaffected because we use MatterMost internally
Ironically, everyone at our company is heads down and working uninterruptedly. Productivity seems to have gone up without the interruptions!
Aside a jaunt over to HackerNews? ;)

I agree. As an aside, Slack has been getting so much usage internally (and everyone learned about @notifications) that it's turned into a massive source of interrupts. It's hard to idle in a number of channels without eventually getting @channel notifications. Perhaps it's a cultural thing here, but there's an expectation that questions are answered immediately unlike email.

I've taken to turning off Slack for periods every day to get work done.

We experienced connectivity issues until a few minutes ago. Everything appears to be back to normal on our company's Slack account now. YMMV
That's an interesting decimal truncation convention.

http://i.imgur.com/FiJ4G4i.png

"Let's only show precision that makes us look good."

For convenience, the numbers on their status page [1] are:

  May 2015        99.999%
  April 2015      99.9%
  March 2015      99.999%
  February 2015   99.995%
  January 2015    99.99%
  December 2014  100%
  November 2014   99.84%
  October 2014    99.9%
  September 2014  99.998%
1: https://status.slack.com/calendar
Or not. Is this really news?
At some point Group chat is crossing into tier1 messaging infrastructure and all the sexy things that go along with it, especially when you start charging money.