HN Explorer

130 comments

[ 1.9 ms ] story [ 182 ms ] thread
cadr
When the status page is returning a 500 error... not a good sign.
munk-a
It's an even worse sign when the page half-loads with some stylesheets missing.
cadr
Yeah, it alternates loading telling me everything is fine, and just giving me a nginx 500 error. Seems that the status page should be hosted differently so it can be up even when other things are having issues.
Florin_Andrei
Maybe they should not serve the static content with "Cache-Control: max-age=1". That's rarely a good idea.
[deleted]
(comment deleted)
traviscj
On the other hand, makes it sound more likely to be a routing/reverse proxy issue instead of (say) a database issue. Those sound easier to deal with via a rollback vs something like "oops we dropped a critical index on the `messages` table".
nyrikki
With the length of the outage it seems more like a DB issue. And if this is still correct there look to be some fragile dependencies.

https://aws.amazon.com/solutions/case-studies/slack/

The API isn't even returning valid error codes.

  logging error: {"subtype":"api_call_error","message":"{\"ok\":false,\"error\":\"_http_error\",\"status\":0,\"retry_after\":null}","stack":"Error\n
As you can hit their api servers I am betting some replication error in mysql but that is just a guess based on that case study.

I am betting they are saying 'connectivity' because that is the error the client logs.

nyrikki
It is back up for me, but now I am annoyed they can't follow specs at all.

For l

They ignore:

https://standards.freedesktop.org/basedir-spec/basedir-spec-...

And kitchen sink everything under the xdg config dir...

  ~$ ls .config/Slack/
  Cache/                Cookies-journal       
  dictionaries/         installation          Local 
  Storage/        
  Preferences           QuotaManager-journal  
  Cookies               databases/            GPUCache/             
  local-settings.json   logs/                 QuotaManager          
  storage/
Gibbon1
It's a sign that we need to appease the beer gods.
[deleted]
(comment deleted)
seanp2k2
Returned for me when I was looking but took ~30 seconds to do so. They should host their status page on a separate domain and use something like CloudFlare in front of it to help with sudden spikes in traffic. Another alternative is to use Twitter / Facebook as the status page and let them deal with the traffic spikes, or just serve static HTML.
msarchet
not anymore? - EDIT: the status page was fine and then it wasn't.
yellowapple
Seems like every other request is throwing a 500 error. Maybe one server in a load-balanced cluster is erroring out?
coleca
How can I tell my team that Slack is down? :-)
nategri
Yo, people who like to complain that Slack just re-implements IRC... This Is Your Moment
mpd
At least with IRC I could connect to another server.
Flammy
.
auscompgeek
> Anyone who doesn't know, Slack actually runs off the IRC protocol underneath the hood.

[citation needed]

d4l3k
Pretty sure they provide an IRC interface, but almost certainly don't use IRC internally. There's almost no way they could support any of their fancier features using IRC. Reactions etc would be horrible to implement.
helper
Yeah and the IRC interface has been getting worse recently. When they added the shared channels across teams they completely broke being able to '@' a user from the IRC gateway. Support said something along the lines of 'yup and we're not planning on fixing it'.

I'm expecting them to completely turn off the IRC gateway in the next year or two.

flukus
Embrace, extend, extinguish. It's the SaaS business model.
kiloreux
I just realized that being a remote team. I can't join any of my teammates. Neither see live changes to the infrastructure and the repositories.

The things we take for granted.

Scriptor
If you're a fully remote team it'd probably be wise to figure out some fallbacks.
Johnny555
I'm sure they have fallbacks, but when their ecosystem (apparently) evolved around Slack, the fallbacks are less effective. Polling Jenkins to see when your job is done is more time consuming than receiving a Slack message.
urda
If slack being down means you lose all insight into your build process and code management, you seriously need to introduce a secondary option immediately.
typomatic
OP didn't say "all insight". There's a difference between being unable to see a stream of change events and not being able to see the current state of the system. The latter is completely unacceptable, whereas the former is just annoying.
ytjohn
I'd like to message members of my team about this issue, but slack is down...
a12l
Team: if you're reading this, get out the radios.
BrentOzar
And of course today's the first day we're using Slack for audience Q&A at a conference. 360 folks in a room now have to...raise their hands! So barbaric.
eridius
Hasn't Slack learned yet that you're supposed to host your status page on a different infrastructure?
Johnny555
I think they did.

The slack.com IP's are owned by AWS, while status.slack.com resolves to some Digital Ocean IP's.

eridius
Then why did the Slack Status page have so many problems at the same time? Half the time loading it would give a 500 Internal Server Error, 45% of the time you'd get broken resources (images and/or CSS), and only 5% of loads would give you the full working page.
Johnny555
Maybe because it's under a lot more load during an outage and they haven't upsized the status page infrastructure to handle their ever increasing user base.
detaro
Maybe they underestimated how much resources their status page needs during an actual outage.
philfreo
Reminder why your status page should be hosted in a very different way than your regular infrastructure... so you're way less likely to end up with issues on both at the same time.

I like how statuspage.io even has metastatuspage.com in case their primary domain/DNS/TLD has issues.

theboywho
Reminder that you should check things first before commenting. Slack's status page is in a different infrastructure, somewhere using digital ocean while slack.com is using AWS.
[deleted]
(comment deleted)
cheerioty
That's why there's StatusPage.io :)
tooltalk
I think you meant to say statuspage.io, not statuspage.com
internalfx
My RocketChat server is running just fine ;P
5ilv3r
So is my jabber server!
internalfx
Is this getting flagged off of HN?
gtdawg
Looks like it. When I first loaded HN 2 minutes after this outage started, the story was #2. Then I refreshed after 3 minutes and it wasn't on the front page at all. Used the search tool to find it and then upvoted.
internalfx
Looks like somebody wants to keep this quiet. 65 points in 12 minutes is good enough to be #1.
minimaxir
It is infeasable to keep a global outage quiet.
grzm
High comment rates can also trigger the "overheated discussion detector", which will downweight a submission.
chris_wot
Perhaps the HN mods should do something about that then.
grzm
If you see something like this and you think it's in error, you can let them know and they'll likely be able to respond more quickly. There's a contact link in the footer.
chris_wot
You're joking, right?
grzm
No, I'm not. In my experience the mods are quite responsive, and have explained site behavior on more than a few occasions. They've also adjusted flags and weights of submissions if they identify an issue.
sctb
Outages without any information are not intellectually interesting, and typically neither are the discussions that follow.
minimaxir
Granted, there’s not much constructive discussion that can happen as an outage is happening.
internalfx
Discussion? No, I agree there. But it _is_ relevant "hacker" news right?
yellowapple
Probably just the ratio of comments v. votes. Too many comments relative to the number of votes will lower a post's ranking, IIRC.

I'm probably not making this any better by commenting, of course.

legohead
don't push code at 4pm! and on Halloween, oof..
[deleted]
(comment deleted)
[deleted]
(comment deleted)
akulbe
Slack DOWN! Productivity UP!

#jokingpeopleCOMEON

chris_wot
Causing chaos at my workplace. We have Slack integrated into our incident management solution... very, very unfortunate.
jnaulty 

  We are aware of connectivity issues and are actively lnvestigating.
  3:58 PM PDT・See in your timezone
They spelled investigating with a lower-case 'l' :\ Does that bug anyone other than me?
Waterluvian
It doesn't bug me. But I really want to understand how that happens.
bramen
Just a simple typo. The keys are pretty close together if your finger slips, and I imagine they have enough problems distracting them from proper spellchecking at the moment. :-)
fiatpandas
When your hands are shaking from adrenaline because your pre-IPO company is suffering from a global outage, you might hit an l instead of an i.
theossuary
No worse feeling than typing out a status update without any idea of what's going on.
alexasmyths
I worked for BlackBerry when outages started to become a thing.

When your business motto is 'always on' - it's really, really bad to be 'off' - it's a deep transgression of the brand promise.

BB was structured poorly for this - they didn't grasp the concept of multiple nodes of redundancy very well. (Easy in hindsight).

But - we should all be impressed at how highly available Google, FB and some other brands are. That's impressive.

orthoganol
How is there nothing on the front page of HN, slack being out for almost an hour now?

Given timeframe and upvotes, how is this not the top of HN?

SolaceQuantum
This is certainly scary for the Slack devs... Happy Halloween!