110 comments

[ 2.6 ms ] story [ 176 ms ] thread
Slack has been struggling for my workspace the last 10 minutes. Can still get messages through (sometimes) but threads won't load at all.
As usual - status page is green, you can't use slack app or web.
Your message is showing as "6 minutes ago" for me, and when I load the status page, it's full of yellow warning icons now.
(comment deleted)
> Something’s gone awry, and we’re having trouble loading your workspace.

Same here - UE

No, their health page is the worst place to look for honest information. They don't want to lose another decimal on their uptime percentage.

Try down detector. https://downdetector.com/status/slack/

Their incident was reported a full 33 minutes after downdetector first started showing spikes in reports. That's pretty bad.
A sampling of Down Detector comments on https://downdetector.com/status/aws-amazon-web-services/ (which is also showing a spike right now, with no indication of an actual AWS outage I'm aware of):

> Was the internet and #AWS recently hacked? Seems like a lot of softwares are not working properly

> That’s so odd. Wi-Fi is out for me too. AWS outage?

> Another problem which definitely doesn’t make sense why server itself gone wild and not responsive to ssh. We will keep investigating this but it can be aws virtualization issues but interesting.

Slack has to do a little more due dilligence than "wifi's down, must be AWS" for their status page.

They need to do more due diligence on the cause before announcing the cause, but they don't need to do more than confirm users are having problems to update the status page that problems exist.

Even assuming they can't instantly see the problems in a dashboard and need to take time to understand user feedback to see which aspects specifically are affected to update the page in more detail, nothing is stopping them from very quickly putting a "Connectivity issues reported by some users, investigating the extent and cause currently" banner at the top of the page which should be ready to go at a moment's notice anyway.

I started having failed messages in channels and threads around 0605 PST... so about 20 minutes. Not too terrible considering the usual due diligence to escalate, determine scope of impact, and then push the buttons to update the status page with the approved messaging.
I completely disagree. It should go yellow the second they see that there is a spike in any abnormality. This type of issue would have been obvious seconds after it occurred. They don't need to report why but they should report that something is wrong. It's not fair to say "it's not me it's you" when they definitely know it's them.
> They don't want to lose another decimal on their uptime percentage.

I was curious and found it:

> September 2021 99.999% uptime

Loonies.

Entertaining to see them simultaneously claim 100% uptime in February and list outages both yesterday and today. If 'users can't send or receive messages' on a messaging platform, your service is down.
Can't fail those SLAs if you make up the numbers
"When a measure becomes a target, it ceases to be a good measure"
I think in this case it's more, you can't be held accountable without oversight. You can't be held accountable to measures you define, measure, and report yourself without any oversight from anyone else.
What's also interesting is their carefully crafted language. They have "Slack is not loading for some users" listed everywhere instead of "most users" or "all users" and also "Something's not quite right" instead of "Something's not working" or "Something's wrong".
If you look today (one day after the incident) you'll see that the status page now shows only 3 9s of uptime for the quarter.
time to start up a SaaS service status SaaS to serve SaaS status
I've definitely already seen that before

edit: Yeah it was Atlassian's Statuspage. I am pretty sure I saw them before they were acquired though and never realized

For most large companies, their health page is updated manually by project managers or directors. The reason is simple: accurate reports make them look bad. And make them liable for refunds due to broken SLAs.

Amazon's famous for this; it's no surprise that Slack does it too.

Wow, this might be more disruptive than the AWS outages. Remote companies should have some sort of backup plan for when this happens.
Its time for Teams to shine
The duration of constant freezes Teams has on every run outlast the total outage time Slack has per month.
Like … email?
Or your phone. Or your companies conferencing software of choice.
I am seeing a lot of errors with the following message: 'Message undefined (message missing text)'
...and cue the thread about how IRC is a better medium and SAAS has ruined communication.
I feel like that's a pretty reactionary painting of the issue, but wanting large platforms to provide access through open protocols that have been in use for decades and have many accessibility and usability benefits really seems pretty much a no brainer for anyone reading Hackernews.
should be

But how to you get that $1b valuation by making yet another client based on open standards? You have to corner the market, and 90% of HN readership want to create or work for a new unicorn, not just improve technology for the sake of it.

> But how to you get that $1b valuation by making yet another client based on open standards? You have to corner the market, and 90% of HN readership want to create or work for a new unicorn, not just improve technology for the sake of it.

Feel free to make a better Slack, that's more usable, more reliable, with more integrations, and run it on IRC. The implementation details mean little to users. Thinking Slack won because it was a closed standard and not that it's better than IRC for the majority of its users is quite the reach.

Your colleague from marketing:

- ... how do I get on it?

- How do I upload files to a channel?

- How do I search for something I wrote a month ago?

I just don't see why everyone doesn't just run their own always-online IRC bouncer and tweak their irssi config for hours to their liking to get an experience that's worse than Teams.
So while I often use irssi, I have become recently fixated on how to compartmentalize all my different chats and I keep coming back to weechat on a vps that bridges slack, discord etc, so weechat bridges are another alternative. If you have a good idempotent spinup method with your vps it can also mean less data hanging around to be grabbed because you can just make a fresh one anytime, in any datacenter.
I definitely prefer Slack over IRC for internal communication.

But I've also kind of come to hate live chat for internal communication entirely...but email isn't a great alternative either. I'm beginning to think there are just irreducible complexities to human communication that make every platform non-ideal, and the best thing to do is to just set and manage expectations properly as a team and as individuals.

Zulip is a great alternative (hi Austin!).
SAAS / outsourcing has ruined organisation self-responsibility, but the MBA said it was a good idea, and the ill-informed CEO just followed.
IRC has existed for over 30 years and was widely popular for the big part of that timespan. Yet no one ever made it into or extended it into a realistic competitor to the modern wall garden proprietary chat platforms.

Is there a fundamental issue with IRC that precludes that? Did no one just bother? Is it the specific mentality among its core users that "absence of certain features in IRC considered a feature" that lead to its demise?

Money.

You can't make as much money with open protocols.

Worst of all, it's very hard to pitch such a business to people who would fund you, so your business is outcompeted very early on.

What users want and what entrepreneurs and investors want are two, largely uncorrelated, things.

IRC was never widely popular outside of a gamer/geek niche. At its peak, QuakeNet never broke 250,000 "users", many of which were bots.
You are so incredibly wrong about this, it saddens me.
If you're gonna tell someone they're wrong in such a dramatic way you need to explain why.
Even if his numbers are wrong, I'll never believe you that IRC was more used than say, AIM or ICQ, let alone any of the popular modern instant messenger and chat apps.
Ok, fine. But can we at least reverse the gamer/geek label. It was always a geek tool - gaming was far secondary. Look at bash.org and tell me the majority of IRC was used for gaming vs. geeking.
(comment deleted)
There's no business model in IRC, and thus no marketing money.
looks like the guy who took over freenode is doing his best to create a walled garden. Also plenty of people still on IRC, including me :)

FWIW, I like irssi much better than the slack front end, but as font ends go, slack's is not terrible.

There's Pest [0], but I don't think it tries to be more usable, maybe even less so. I think it's hard to take old standards and put modern features on them; at some point it's more beneficial to just invent your own thing.

[0] http://www.loper-os.org/?p=4003#11-what-is-pest

Just to preserve the full message :)

"Something’s gone awry and we’re having trouble loading your workspace. Sorry we can’t be more specific - this is one of those cases where we don’t know what’s gone wrong either. A restart of Slack might help, and you can always contact us."

SAAS companies cannot be responsible for their own incident/status pages. This has been proven time and again.
Sounds like a SaaS opportunity.
I assume a joke here haha -- no way are any of these companies pay another to hold them ACTUALLY responsible for their outage %. They would MUCH rather just lie.
Who'd pay?

https://stop.lying.cloud/ exists for AWS.

I would consider paying a 3p a reasonable price for accurate/reliable visibility into status of cloud services, if my business depended on it.
I’ve thought about setting something like this up for the major clouds & SaaS.

The biggest problem is that I’d have to run some dummy loads on all their services in all regions - even at the lowest tiers I’d probably be looking at a few thousand a month just to keep the service up.

I've considered doing this but I think it would only be feasible as an enterprise offering. The costs of continuously making API calls to start machines, test network performance etc would be quite high.
Couldn't Catchpoint or 1000eyes do it with their existing infrastructure?
hmm didnt this happen once before after a long weekend?
2022 Checklist for Confirming a Service is Down:

[ ] Do a Twitter search [ ] Scan through HN [ ] Check with peers [ ] Consult downdetector.com

…in this order. Do not bother with official status pages.

In this case - having happened to have noticed this at the exact time it occurred, I observed:

downdetector had it first

twitter had it second

peers were unfortunately unreachable (slack)

HN lagged by a significant margin

that has also been my recent experience. works best for large services like discord and twitter which are recent examples for me.
> Do not bother with official status pages.

You should. If the status page is also down, you know it's going to take a while ;-)

> You should. If the status page is also down, you know it's going to take a while ;-)

Maybe. I've been shocked by some big companies co-hosting their status pages.

HN pointed me to the official status page though
Maddening. “Can’t send message. Tap to try again” - when Slack’s own status page shows green.
(For me at least...) The client.boot file is returning a 429 (too many requests).
another data point to encourage groups of a certain size to host their own instance of something like https://zulip.com/ / https://zulip.readthedocs.io/en/stable/production/install.ht...
This option is dead for most of us because of Slack Connect (which allows cross-tenant communication, such as from x.slack.com to y.slack.com).

I now have 5-10 other organizations connected to any one of my Slack tenants, and now we're just stuck in Slack's ecosystem.

cross-tenant communication as a method of a vendor locking in their customers, hah!
Slack being down is so detrimental… it’s crazy.

I’ve almost forgotten how to send my coworkers email. It blows my mind how much we are so reliant on a few tools to carry out so much work.

Russian cyber attacks starting?
I came here expecting to read lots of discussion about why this might or might not be the case. Oh, well.
You can use http://chat.google.com/ as a backup chat. I started using it with my team. It does not come with desktop app, but usable
There's a "progressive web app" - if you're in Chrome on chat.google.com, there's a new icon on the right hand side of the URL bar which permits you to download it. Not discoverable at all - this was mentioned in the help pages [https://support.google.com/chat/answer/9455386]
Just as I was searching my slack workspace and found info to help fix a problem, what are the chances
Slack status page started showing some outages. Is it a manual process, I guess?
It's likely that a member of their SRE or DevOps team has to confirm the outage and "deploy" updates to that status page, yes.
Finally, a chance for Zoom Chat to shine! /s
And google chat is working very slow and weird... They can't figure out IM still in 2022.
Github is also unreachable for quite a few people. But works through a VPN to another geography. Wonder if there is any connection.