> [May 12, 4:53 PM PDT] Users have reported general performance issues such message sending failures and timeouts. We’re working to get things back to normal as quickly as possible and will provide an update shortly.
If you’re going to recommend something non-e2e for a business team, go with email that is already distributed to everyone’s contact lists, policy controlled, 2fa’d, et c - not some snake oil like Telegram.
Disagree. You will spread out the communication and cause problems. People will use their preferred client instead of the one that everybody should be using as the primary. You also have to manage accounts and permissions for both. Unless you like people sending information around to their personal accounts, which I'm sure a lot of places do.
E-Mail is good enough to let people know that something like Slack is down. It's close enough to real-time to be used as a backup for something like this, and everybody already has an e-mail address.
My company already has Slack, HipChat and Teams, and it's a communications nightmare. People are constantly confused and asking for which chat client one manager uses, while others don't know where to go for the support channels.
Maybe this sounds like old-school IT, but the new-school "set everything up ad-hoc using your personal shit" just sets you up for disaster down the road.
You couldn’t do this with Telegram, but if you were using Hipchat or Teams or Mattermost or Discord as your backup, you could just keep the instance locked down most of the time. Open up permissions when Slack is down.
You can do that with Telegram if you have a bunch of centrally-managed groups; simply mute them until you need them. Admins can unmute them as necessary.
That being said, I don’t think I’d want Telegram to be my failover. They have a tendency to experience regional issues with great frequency, and if you’re in a lot of active chats, things start to get buggy.
> My company already has Slack, HipChat and Teams, and it's a communications nightmare. People are constantly confused and asking for which chat client one manager uses, while others don't know where to go for the support channels.
I'm curious about how many people in the company are affected by these problems. Most developers and infrastructure people use the same communication tools, but perhaps support and sales may use their own tools. Given that, how often do developers and infrastructure need to communicate in real time with sales or support?
I would suspect that it's not often enough that email wouldn't work as a way to handle issues or open support tickets. Now managers have to deal with communications between their own team and other teams and parts of the org, but, in my opinion, having to deal with different chat platforms is part of their job and they should be the ones to deal with it rather than forcinng everyone to use the same platform, even if it's suboptimal for certain users.
hangouts/teams (if you already use gsuite/0365 ). More important problem is our own devops relies heavily on slack based monitoring and chat ops, all integrations don't port without work and never designed for DR
We have a Keybase group that gets used twice a year when Slack takes a dirt nap. We also use Zoom so I guess the buyout doesn't have security implications for us?
Perhaps I should have been less subtle than opening with a 'Heh', and made it clear the claim that only the messaging & connection parts were failing was tongue-in-cheek.
This page's title is "Slack is down" but Slack's status page at the time was modestly conceding that "Something's not quite right".
It's interesting because the sarcasm came through for me and only because you said "they do say" rather than "they say" - a subtle difference that is probably odd to convey to non-English speakers.
Oh, that is interesting. I understood the sarcasm there as well but it's difficult to say why it is present. I guess because "do" implies a second, unsaid category "do not". I wonder if there's a Language Log entry on this.
Those being down means I can't use the app at all, so for all intents and purposes, it's down. As Rachelbythebay would say, your nines are not my nines.
Page asserted an 'incident' only, not an outage, but for all intents and purposes a failure of the 'messaging' component for a messaging tool is effectively a full, major, total outage as far as your users are concerned.
I can't login on one computer. The computer that's already logged in, I've bounced the app a few times, and it seems to be unable to load content from most channels, messages I've sent aren't showing up, etc.
It just so happens that your friend here is only MOSTLY dead. There's a big difference between mostly dead and all dead. Mostly dead is slightly alive.
I hope they change the status page from "Incident" to "Outage". Too often lately have I seen status pages (mainly Github) reporting incidents when a feature or site is down completely. Seems like their SLAs are influencing the reporting of their uptime...
SLAs and maybe even company politics. Incidents are politicized at a lot of companies. Even if the official rhetoric says "when it comes to incidents we don't blame people and there are no politics" the reality is often the complete opposite.
Outages mean down, badly, and require RFOs. In my experience working at an ISP we only did any sort of "blame-game" if customers demanded an RFO. We still did post-mortems but usually in a more gentle, roundtable way.
Setting a goal of "Zero or Near" down times doesn't necessarily mean less down times, but it usually does mean politics, shell games and 'clever' bookkeeping by middle and lower management, all of which is clearly visible to the rank and file.
Before you had down times and customers wondering if they could trust you. After a goal of "Zero or Near", you still have "down times", and you still have customers wondering if they can trust you while they stare at a status page is clearly lying to them.
But now you have employees watching their managers lying to their bosses, and wondering why they shouldn't do the same. You have a fall in morale, because those pretty numbers usually come at the expense of the things on the "Our Values" page. But to the Suits the graphs look better, and that's the important bit.
This isn't how is always is or always goes, but I've seen it enough.
This is really hard to understand. The only way to prevent future incidents is to have a culture free of blame.
Your goal is to identify service impacts, do post mortems where no one is afraid to say something went wrong, and to improve processes to prevent future events. You can't do that if there is a culture of blame.
On the flip side, it looks like Slack is back up for me but their status page is also lagging at the tail-end. I highly doubt they would manipulate their uptime status for SLA purposes. Highly doubtful that companies that sign SLAs would depend on their status page to determine whether or not the contract was met.
I agree, I don't think it's malicious and Slack did the right thing here.
My comment was more of a dig at Github specifically. Recently things haven't been working for me (from my perspective an outage) and when you go to their status page it just says "Degraded performance". When your performance has degraded to the point of failure it's time to acknowledge that is in fact an outage...
What I've noticed Slack doing is that a couple weeks after an "Incident", it's completely gone from the history when you go back and look at past incidents on that day. Keep an eye on it.
503 Service Unavailable
No server is available to handle this request.
And thanks to the electron-ness of the app, on the mac you cannot even drag the window - the titlebar is drawn by javascript and without it, it cannot be dragged
I love how one horny kakapo became a legend, never change Internet. (His lovelorn antics also raised a bunch of donations to kakapo conservation when it went viral, ka pai, Sirocco)
Ideally your DR and emergency response systems should be not hosted on your infra and have circular dependencies . i.e.Amazon should not be hosting status page assets on S3 as they were.
At the very least they should be hosted isolated of your production service components at all layers, so both don't go down at the same time.
For slack my guess is their internal slack is not hosted on slack prod and hopefully also does not share components below L4 as well ( DNS, IP , domain, routing etc )
If the disaster involves your connectivity from within your infra to without, it could be advantageous to have backup communications hosted internally. For e.g. an internal IRC for panic situations. The advantages of this might vary depending on whether your response happens from the office, home, or wherever. Pandemic might have changed the equation.
Slack uses its own product for all coordination during incidents and outages, so what happens when it goes down? “To the best of our ability, we use Slack,” says Richard Crowley, Director of Operations at Slack, “[but] If we’re really having a bad day, we use Skype.”
At SendGrid, years ago, we suffered an outage and did not realize as fast as we should have because our alerting provider used us for notifying about outages :p. This situation can no longer happen, but it was eye opening.
That's a funny anecdote and a good step, but you guys still have a long way to go in regards to outage alerting at SendGrid.
I was annoyed when after a several hour long outage SendGrid still had not updated their status page to reflect an issue.
But I was even more annoyed when 5+ hours into the outage, a different support person told me they aren't aware of any outage because their status page does not reflect one.
If I sound bitter it's because I am, I was the on call person who got woken up in the wee hours of the morning to deal with this. I'm referring to the outage you guys had in December if you're wondering.
I had the exact same experience. I was on vacation with family, but was on call. There was a bug on SendGrid’s end that only affected a small percentage of emails; in particular, they had the wrong DKIM signature, if I remember correctly. I spent the entire vacation trying to convince SendGrid engineers that it wasn’t a configuration issue on our end.
They were rather rude and kept insisting that it wasn’t an issue on their end. Eventually we gave up and migrated away from SendGrid. If I remember correctly, they did fix it, at least partially. I think it had something to do with one of their prod servers running a dev config, resulting in DKIM signatures for a SendGrid-owned test domain—something along those lines. That may have been a different incident, though.
That was just the straw that broke the camel’s back. There were often issues that only affected one or two servers—we were often told that a server didn’t have an up-to-date configuration or something like that. Each time, it was the same process: convince them it wasn’t an issue on our end, escalate it repeatedly until it lands on an engineer who understands the issue, they fix it in an hour after wasting days of my time insisting that yes, I know what a DNS record is. And the whole time, some (small) percentage of emails are failing. I can only provide so many logs, DNS records, and email headers to demonstrate the issue; at some point, someone has to be able to look at it and go, “well, yes, that’s very obviously the wrong value, and nothing a customer sets on their end should ever have that effect.”
Sure, Slack dropped the ball here, but at least they updated their status page. Even for small issues, when I contact them, they either get it resolved or tell me it’s on their to-do list—no need to convince people that I understand basic concepts.
If you noticed how insanely Slack freaked out and handled the service outage. The application just got stuck, didn't know what to do, I see a faint 503 error message which looks like a webpage nginx error displayed right in the app. Clearing cache didn't help and the app just quit.
Electron apps are a nuisance and should be eradicated from this planet. We've come a full circle -> Bloat the web -> Create javascript frameworks -> Build SPA/PWA apps -> Invade the desktop land -> Gain traction because "its crossplatform" and now try to reimplement GUI elements from some fancy_new framework that is developed by people who are entirely unqualified for it. Throw away all the work that's done by OS developers and designers that are getting paid as professionals to develop guidelines like this: https://developer.apple.com/design/human-interface-guideline...
Edit: People talking about would native app handle this better? Sure. I argue that it wouldn't just get rid of all UI elements and drop the entire DOM on the page, the UI just crashed. And everytime I open slack it seems like its doing something else entirely. There is a massive difference in quality of native UI libraries and Bulma for example, and its not just the library, you gotta rely on React to redraw things and it depends on 4 billion npm dependencies. The web truly sucks. And it is now eating into desktop apps.
This is one of those stances that I see on HN all the time but never feel myself.
Like, nothing about this outage affected my opinion of Electron apps at all. And in general I like them, as a developer and as a user. They're polishable with so much less effort.
As an OS engineer, I hate them. They cause problems and inefficiencies behind the scene that are pretty significant. I can see how you wouldn't say so from another position, it's a bit like propelling a passenger aircraft with controlled nuclear explosions: "It flies so effortlessly!"
I dislike the web bloat as much as anyone, but how exactly is Electron to blame here? Yes, the 503 error message is frustratingly obscure, but how is that the "webs" fault?
If you had a native app here, it'd still be talking to an API returning some version of a 503.
I appreciated the honesty. Instead of looking at an infinite spinner, or a tiny toast error message, or whatever cute thing a designer might come up with... Plenty of "nice" error handling muddles whether or not the issue is my network or theirs. But I clicked my slack tab in Firefox it was nothing but a big ugly 503 page. No need to guess about what's wrong.
Could the desktop Slack app have a better failure state? Sure.
But if you look at their native iOS app it doesn't have a failure state _at all_. It sometimes shows that it's loading and sometimes doesn't. Never once though does it say there has been a problem.
The problem is bad software in general, not a specific technology.
I've been an native application developer for 10 years, but there is absolutely nothing about the issue you mentioned that can't be solved by just slightly better error handling and a nice error message.
It's pretty straight forward to build that into an Electron app.
There are things that native apps can do better, but this example isn't one of them.
I've been an application user for 10 years, and everytime I come across an app, if it is developed using Electron, I instantly can tell by how it operates. Everything just feels sluggish and god awful. It is like I am dropped into molasses and my legs are tied. Simply put, Electron apps are shit.
When I first got a 503, the status site was all green. I checked back for at least 4 minutes and it was still green. Finally it started to change. Is that sort of lag normal?
It is manually updated, by humans. Humans have high latency. It is not "agile" or "modern" or "hip" to have automated status testing/reporting. A human had to go and change that status page.
Yup, that is also why the status page now says things are "9/10 up" and for the first 20 minutes said "something is maybe slightly kinda wrong" nad not "outage". As if pretending nothing is wrong will make things right again.
Yep. I would be surprised to learn of non-infrastructure companies that wired their public outage notice directly to automated monitors with no human confirmation.
The primary value of a status page is SLA compliance and the financial implications of uptime. Near real time external alerting/reporting is probably not critical value prop for monitoring systems to sell, a 2 minute faster external notification is probably not going to make a new sale.
Also the lag and human intervention is by design as there will be a lot of false positives as some error rates in these systems are expected, a very small percent of API calls will result in errors always like unhealthy instances etc. Unless the issue is substantially large it not in user(alert fatigue) or the business interest to declare it down.
"We are currently experiencing an outage on our network. We are working to resolve this as quickly as possible. Please check http://status.cogentco.com for updates on this event."
People were getting 503 errors, which indicates it’s something within Slack’s own infrastructure. Granted, external network issues can trigger internal problems, but I wouldn’t be pinning this on Cogent just yet.
Since no one can connect there are no messages to deliver and therefor no message errors therefor everything but connections is up and working. /s (The Slack Status Page)
This is a big problem, for teams using Slack for their own DevOps- monitoring and ChatOps. DR plans have to start thinking how to get all the integrations working with multiple communication providers to handle this type of scenario
What a perfect example of the modern status page meme.
App is completely fucking not working and it's only showing a slight problem with "connections to slack". Something's not quite right. Is this valley casualspeak or actually an attempt to minimize the fact that their shit is down?
If anyone is curious about their deployment process https://slack.engineering/deploys-at-slack-cd0d28c61701?gi=e...
Seems hard for a new deployment to cause a global outage since they do a phased deploy. Could be some network glitch causing this issue. We never know until they come up with an update.
Everywhere I’ve worked has done phased deployments. Shit still goes wrong all the time. Also, I would wager that most outages are not due to code deployments, but live config changes.
138 comments
[ 3.1 ms ] story [ 146 ms ] threadSee https://status.slack.com/2020-05/87ad1d12e36fcf0c for incident details:
> [May 12, 4:53 PM PDT] Users have reported general performance issues such message sending failures and timeouts. We’re working to get things back to normal as quickly as possible and will provide an update shortly.
What is everyone using?
Getaether.net/pro
E-Mail is good enough to let people know that something like Slack is down. It's close enough to real-time to be used as a backup for something like this, and everybody already has an e-mail address.
My company already has Slack, HipChat and Teams, and it's a communications nightmare. People are constantly confused and asking for which chat client one manager uses, while others don't know where to go for the support channels.
Maybe this sounds like old-school IT, but the new-school "set everything up ad-hoc using your personal shit" just sets you up for disaster down the road.
That being said, I don’t think I’d want Telegram to be my failover. They have a tendency to experience regional issues with great frequency, and if you’re in a lot of active chats, things start to get buggy.
I'm curious about how many people in the company are affected by these problems. Most developers and infrastructure people use the same communication tools, but perhaps support and sales may use their own tools. Given that, how often do developers and infrastructure need to communicate in real time with sales or support?
I would suspect that it's not often enough that email wouldn't work as a way to handle issues or open support tickets. Now managers have to deal with communications between their own team and other teams and parts of the org, but, in my opinion, having to deal with different chat platforms is part of their job and they should be the ones to deal with it rather than forcinng everyone to use the same platform, even if it's suboptimal for certain users.
Messaging and Connections.
- Login/SSO
No issues
- Connections
Something's not quite right View details
- Messaging
No issues
- Link Previews
No issues
- Posts/Files
No issues
- Notifications
No issues
- Calls
No issues
- Search
No issues
- Apps/Integrations/APIs
No issues
- Workspace/Org Administration
No issues
Perhaps I should have been less subtle than opening with a 'Heh', and made it clear the claim that only the messaging & connection parts were failing was tongue-in-cheek.
This page's title is "Slack is down" but Slack's status page at the time was modestly conceding that "Something's not quite right".
Page asserted an 'incident' only, not an outage, but for all intents and purposes a failure of the 'messaging' component for a messaging tool is effectively a full, major, total outage as far as your users are concerned.
Outages mean down, badly, and require RFOs. In my experience working at an ISP we only did any sort of "blame-game" if customers demanded an RFO. We still did post-mortems but usually in a more gentle, roundtable way.
Setting a goal of "Zero or Near" down times doesn't necessarily mean less down times, but it usually does mean politics, shell games and 'clever' bookkeeping by middle and lower management, all of which is clearly visible to the rank and file.
Before you had down times and customers wondering if they could trust you. After a goal of "Zero or Near", you still have "down times", and you still have customers wondering if they can trust you while they stare at a status page is clearly lying to them.
But now you have employees watching their managers lying to their bosses, and wondering why they shouldn't do the same. You have a fall in morale, because those pretty numbers usually come at the expense of the things on the "Our Values" page. But to the Suits the graphs look better, and that's the important bit.
This isn't how is always is or always goes, but I've seen it enough.
Your goal is to identify service impacts, do post mortems where no one is afraid to say something went wrong, and to improve processes to prevent future events. You can't do that if there is a culture of blame.
My comment was more of a dig at Github specifically. Recently things haven't been working for me (from my perspective an outage) and when you go to their status page it just says "Degraded performance". When your performance has degraded to the point of failure it's time to acknowledge that is in fact an outage...
503 Service Unavailable No server is available to handle this request.
And thanks to the electron-ness of the app, on the mac you cannot even drag the window - the titlebar is drawn by javascript and without it, it cannot be dragged
https://cultofthepartyparrot.com/
https://en.wikipedia.org/wiki/Glitch_(video_game)
https://techcrunch.com/2008/04/02/game-neverending-rises-fro...
https://en.wikipedia.org/wiki/Ludicorp
At the very least they should be hosted isolated of your production service components at all layers, so both don't go down at the same time.
For slack my guess is their internal slack is not hosted on slack prod and hopefully also does not share components below L4 as well ( DNS, IP , domain, routing etc )
Slack uses its own product for all coordination during incidents and outages, so what happens when it goes down? “To the best of our ability, we use Slack,” says Richard Crowley, Director of Operations at Slack, “[but] If we’re really having a bad day, we use Skype.”
Fair, I guess.
I was annoyed when after a several hour long outage SendGrid still had not updated their status page to reflect an issue.
But I was even more annoyed when 5+ hours into the outage, a different support person told me they aren't aware of any outage because their status page does not reflect one.
If I sound bitter it's because I am, I was the on call person who got woken up in the wee hours of the morning to deal with this. I'm referring to the outage you guys had in December if you're wondering.
They were rather rude and kept insisting that it wasn’t an issue on their end. Eventually we gave up and migrated away from SendGrid. If I remember correctly, they did fix it, at least partially. I think it had something to do with one of their prod servers running a dev config, resulting in DKIM signatures for a SendGrid-owned test domain—something along those lines. That may have been a different incident, though.
That was just the straw that broke the camel’s back. There were often issues that only affected one or two servers—we were often told that a server didn’t have an up-to-date configuration or something like that. Each time, it was the same process: convince them it wasn’t an issue on our end, escalate it repeatedly until it lands on an engineer who understands the issue, they fix it in an hour after wasting days of my time insisting that yes, I know what a DNS record is. And the whole time, some (small) percentage of emails are failing. I can only provide so many logs, DNS records, and email headers to demonstrate the issue; at some point, someone has to be able to look at it and go, “well, yes, that’s very obviously the wrong value, and nothing a customer sets on their end should ever have that effect.”
Sure, Slack dropped the ball here, but at least they updated their status page. Even for small issues, when I contact them, they either get it resolved or tell me it’s on their to-do list—no need to convince people that I understand basic concepts.
Electron apps are a nuisance and should be eradicated from this planet. We've come a full circle -> Bloat the web -> Create javascript frameworks -> Build SPA/PWA apps -> Invade the desktop land -> Gain traction because "its crossplatform" and now try to reimplement GUI elements from some fancy_new framework that is developed by people who are entirely unqualified for it. Throw away all the work that's done by OS developers and designers that are getting paid as professionals to develop guidelines like this: https://developer.apple.com/design/human-interface-guideline...
The modern tech stack needs to be refactored just like they refactor silicon architecture every 10 years: https://www.youtube.com/watch?v=Nb2tebYAaOA
Edit: People talking about would native app handle this better? Sure. I argue that it wouldn't just get rid of all UI elements and drop the entire DOM on the page, the UI just crashed. And everytime I open slack it seems like its doing something else entirely. There is a massive difference in quality of native UI libraries and Bulma for example, and its not just the library, you gotta rely on React to redraw things and it depends on 4 billion npm dependencies. The web truly sucks. And it is now eating into desktop apps.
Like, nothing about this outage affected my opinion of Electron apps at all. And in general I like them, as a developer and as a user. They're polishable with so much less effort.
If you had a native app here, it'd still be talking to an API returning some version of a 503.
If an app gains too much traction with bad tech they can always go back and fix it later. Twitters fail whale comes to mind :-)
But if you look at their native iOS app it doesn't have a failure state _at all_. It sometimes shows that it's loading and sometimes doesn't. Never once though does it say there has been a problem.
The problem is bad software in general, not a specific technology.
It's pretty straight forward to build that into an Electron app.
There are things that native apps can do better, but this example isn't one of them.
Add it to the HackerNews bingo card
Their work is complex to use and provides limited benefit to the company. Any company dump Slack over its use of Electron? I doubt it.
Also the lag and human intervention is by design as there will be a lot of false positives as some error rates in these systems are expected, a very small percent of API calls will result in errors always like unhealthy instances etc. Unless the issue is substantially large it not in user(alert fatigue) or the business interest to declare it down.
"We are currently experiencing an outage on our network. We are working to resolve this as quickly as possible. Please check http://status.cogentco.com for updates on this event."
It's crazy how easy it is to depend on Slack. Email collaboration feels downright archaic. What a great product!
App is completely fucking not working and it's only showing a slight problem with "connections to slack". Something's not quite right. Is this valley casualspeak or actually an attempt to minimize the fact that their shit is down?
Ok bro, you're not fooling anyone though.