It also seems like all integrations that use Slack webhooks are working. But the ones which are like a dedicated app are failing with the same `fatal_error` message.
Minor thing that bugs the heck out of me: I can't login to Slack, it's a total outage for me, but on their status page [0] their copy says, "Something's not quite right."
In reality, there are probably folks freaking out internally at Slack with their hair on fire, and probably tens of thousands of people are locked out of their business comms. Saying "something is not quite right" trivializes and downplays the severity.
No! Something like, "We have a real problem" would be much more appropriate and not pretending like "oh it's no biggie, we'll take a look later."
Yeah, and the incident details indicate call it "degraded functionality" when it seems broken for everyone across the board. Desktop app, website and mobile app all non-functional.
Oddly, my mobile app still has me logged in and seems to work, but the desktop app switched to its stupid 'oopsy daisy, something not quite right' screen on its own.
I worked as a sysadmin during a major outage a few years ago. A bug completely destroyed a table in one of the customer databases; we had to restore/rebuild the table. It took hours. We reported it to our leadership; they handled the comms. When the message got translated and posted on the company's status page - well, it sounded like we were doing our customers a favor. Nothing said was technically untrue, but I was amazed at the spin from "We fucked up and we're working to fix it" to "We're doing this all for you; you're welcome". Ever since then I've been skeptical of status pages.
To be fair, their status page reported problems super early. It might even be automated. Usually you have to search Twitter for accurate information on outages.
It really didn't -- we (Slack customer and integration) had our first internal reports of issues at 09:59 ET and the status page was updated at 10:27. For this broad an impact, that's a long time.
At a previous company that focused on small-business finance, we had an error page that said something like "We couldn't process your request, but we're working on it! In the meantime, here's a video of a puppy." with a link to a Youtube video of a puppy.
PEOPLE HATED THE PUPPY.
"I can't process SALES, i'm losing MONEY and you want to give me a VIDEO of a PUPPY?"
Our company of >100 people is all remote and all our internal comms are through Slack.
The silence is deafening, surprisingly so, when Slack goes down like this. It happens so rarely (to the Slack team's credit), but when it does, it feels just like when the power unexpectedly goes out.
We totally forget how much we rely on it until it disappears.
Email is always there as a backup, but no one in the company uses it anymore, to the point where we have to remind people to check it incase an external vendor reaches out.
Slack has provided so much additional functionality that trying to cram day to day working back into email just doesn't work anymore.
I had to pull everyone I could into newly created Gchat "Spaces" (Did anyone else just discover that even existed? I didn't know about it) -- though it's hard to get the message out to go there when normally that's the kind of thing I'd tag @channel in #general.
And my other go-to communication method is grabbing people's phone numbers from their slack profile. D'oh!
> Our investigation is still in progress with regard to deprecated functionality for Slack features such as workflows, threads, sending messages and API-related features.
I'm not sure what their status page is talking about, is sending messages a deprecated slack feature?
Just kidding. But there was a sizable fraction of time where I really hadn't heard the word "deprecated" and read it mentally as "depreciated" like an asset.
Their status updates look clearly like AI-generated blurbs saying the same thing with varied phrasing, hour after hour. Thanks for the slop? I know writing status page updates is annoying, but “a human is paying attention to this” is the specific thing that status updates are trying to convey, so trying to get info from this page felt dispiriting today.
Edit: this would also explain why some details of the updates were nonsensical.
Everything seemed fine at around 9:00 AM PST (UTC-8). A few minutes later, I made the mistake of reloading Slack after a few messages began failing to send. Haven't been able to access my company's Slack workspace since. At the time of writing this post, it's now 11:00 AM PST.
Emailed my manager explaining the situation. Hopefully they can understand. Also plan to ask others about Slack alternative for situations like this. For now, we are relaying things through e-mail.
This is another remainder that crucial business communication should be based on self-hosted solutions to be independent on third-parties and for privacy reasons. There are good and even more lightweight open source Slack alternatives: Nextcloud Talk, Mattermost, Zulip, Element.io, Jami, to name a few. Relying on a third-party provider is asking for trouble. Plus giving away your private business conversations to feed AI and NSA is another thing.
Based on the status report, it seems like it was related to a database disaster and mention recovering shards. Publicly they say they are using vitess[0] so I wonder if there was some issues there.
I wonder if there will be a more detailed post-mortem, would be super interesting to learn more!
56 comments
[ 2.9 ms ] story [ 31.8 ms ] threadEdit: the mobile client loaded messages but also fails to send.
Good reminder that we need a backup real time messaging app.
I'm quite enjoying the quiet. There's still video or email in an emergency.
Check it out @ https://emdash.io
In reality, there are probably folks freaking out internally at Slack with their hair on fire, and probably tens of thousands of people are locked out of their business comms. Saying "something is not quite right" trivializes and downplays the severity.
No! Something like, "We have a real problem" would be much more appropriate and not pretending like "oh it's no biggie, we'll take a look later."
[0] https://slack-status.com
PEOPLE HATED THE PUPPY.
"I can't process SALES, i'm losing MONEY and you want to give me a VIDEO of a PUPPY?"
We removed the puppy.
The silence is deafening, surprisingly so, when Slack goes down like this. It happens so rarely (to the Slack team's credit), but when it does, it feels just like when the power unexpectedly goes out.
We totally forget how much we rely on it until it disappears.
Edit: Email can have latency, which is why I did not recommend it.
Slack has provided so much additional functionality that trying to cram day to day working back into email just doesn't work anymore.
And my other go-to communication method is grabbing people's phone numbers from their slack profile. D'oh!
I'm not sure what their status page is talking about, is sending messages a deprecated slack feature?
Just kidding. But there was a sizable fraction of time where I really hadn't heard the word "deprecated" and read it mentally as "depreciated" like an asset.
Edit: this would also explain why some details of the updates were nonsensical.
Emailed my manager explaining the situation. Hopefully they can understand. Also plan to ask others about Slack alternative for situations like this. For now, we are relaying things through e-mail.
Will be an interesting post mortem to read...
Good luck to SREs over there! (I know, they would say luck is not a strategy)
I wonder if there will be a more detailed post-mortem, would be super interesting to learn more!
[0] https://slack.engineering/scaling-datastores-at-slack-with-v...