Using our teams backup chatroom in a competing service. One of these days P2P Matrix will reach GA, then I plan to make a backup for my backups, Starfleet style.
GILORA: Starfleet code requires a second backup?
O'BRIEN: In case the first backup fails.
GILORA: What are the chances that both a primary system and its backup would fail at the same time?
O'BRIEN: It's very unlikely, but in a crunch I wouldn't like to be caught without a second backup.
Makes perfect sense for O'Brien, DS9 had serious backup issues in the first couple of years
The Forsaken (season 1 episode 17)
LOJAL: I've been reading the reports of your Chief of Operations, Doctor. They gave me the impression that he was a competent engineer.
BASHIR: Chief O'Brien? One of the best in Starfleet.
LOJAL: Then why aren't the backup systems functioning?
BASHIR: Well, you know, out here on the edge of the frontier, it's one adventure after another. Why don't I escort you back to your quarters where I'm sure we can all wait this out.
Rivals (season 2 episode 11)
KIRA: My terminal just self-destructed.
DAX: What?
KIRA: I lost an evaluation report I've been working on for weeks.
DAX: Even the backups?
KIRA: Even the backups.
There's a reason to have a backup to the backup by Destiny (season 3 episode 15)
We were just joking with the work mates -- SF bought Tableau 2 years ago and haven't ruined it yet, only because it takes them that long to do anything ;)
Not all - many workflows these days rely on Slack or its ilk. Benderbot, Jira/etc. connectors, calendar connectors, remote communication/standups, alerting…
If something went awry, and it caused more pain because Slack was down, how would you feel?
If you’re missing comms/observability then waiting to deploy seems prudent.
I have less of an excuse not to be more personally productive, but I can't help anyone else (easily) if my primary method of communication is down. Not only because it's harder to contact you, but also because it's impossible for you to just ask in a channel and have me notice you.
There's also this perverse incentive to Slack all the things. Lots of CI notifications are sent through it. Some org processes are implemented as workflows. There's been talk of how wonderful it would be to hook up tasking and work tracking to slash commands. I and others often use Slack instead of the 'official' tool to video call each other.
An outage like this is still really disruptive. It's not like everyone realizes what's going on immediately or at the same time; we have backup tools, but our turn radius is pretty wide. Some of us can't even communicate effectively without memes, too, and backup tools don't have a giphy integration.
EDIT: Do your CI integrations fail if Slack can't be contacted? Do those failures fail your pipeline? Whoops!
Particularly on a Monday morning after a holiday, there are tasks that I know I need to be working on but cannot because relevant details were never transposed from slack to our actual work scheduling tools like google docs, jira, etc. and I cannot access Slack history.
If you use slack primarily as a water cooler then yes.
However, I drive everything through slack - GitHub, linear, calendars, Notion, support emails, etc. I have notifications turned off for every service we use except for slack. This allows me to effectively ignore everything except for slack. These types of outages destroy that workflow for me.
Absolutely! Before the holiday shutdown, I Slacked myself a huge reminder list of things to jump on as soon as we started up again, so that I could hit the ground running in the new year. Oh, wait....
If you're asking genuinely then I can tell you my experience when I was part of a SaaS shop, though the times have changed a lot and "my metric is not necessarily your metric".
But it was roughly "one large impact a month, for six months", with large caveats that upper management for whatever company had to be working with the product during that month.
Large companies don't care if X service went out during the night and impacted someone not in their timezone.
If the CTO notices that he can't use something with the same regularity that he gets paid, then it doesn't take long for it to stick in their mind. But migrating everything is _so painful_ that the majority of large companies will do anything they can to avoid moving away.
This is a key point is the popularity amongst VCs in investing in B2B SaaS. I take their (and your) word for it. But honestly, I don't actually understand this.
Medium sized team on Slack. We'd need to move ~60 full time in-house employees, ~10 remote contractors who aren't on other comms channels, ~20 infrequent freelance contributors who may not check messages often, ~5 custom bots and apps, and ~15 3rd party integrations (of which some won't support any given choice of alternative).
This is not to mention the fact that half our staff aren't hugely technical, so have actively _learnt_ how to use Slack and it's features around notification control (things that may come "naturally" to the tech-savvy crowd on HN), @-things, bots, etc, and they would need to re-learn a new tool that is going to work in a different way.
This would be a substantial effort for us, and we're a small company. Are there ways to materially minimise this cost?
IRC is easy to migrate from since there is nothing to migrate other then chat history. IRC is also missing so many features that slack provides out the box. And a law like that would not work since you would need to write complicated transformation scripts to transform between services. Also not all services are a 1-1 mapping. I like IRC but it has its limitations. That is why slack succeeded where IRC did not.
Getting workflows re-established, any integrations you had developed or otherwise come to depend on may not work, you will probably lose history, etc.
Plus, it will just take a long time to get everyone on board and using the replacement system. My department is slowly plodding towards using Teams over Slack, but there are enough hold-outs (my sub-department being one of them) that it still doesn't have wide-spread adoption.
Training, integration with proprietary internal systems, sheer momentum in the employee base, justifying or even creating a metric to show cost savings of a migration effort, business processes that rely on a specific feature of existing infrastructure needing to be met, the uncertainty of new vs the certain and known instability of something you have....
If you had a small shop with a dozen tech-savvy people and Slack became a problem which was used exclusively for quick business chats, you could probably push a change to another chat platform the next day. You might struggle when you have thousands of employees, some that needed training to use Slack and still aren't that proficient.
I worked at a huge corp which employed many non-technical people and had global offices and all sorts of contractors / full-time employees at various levels.
But they invested in organizational / business software early (think SAP-type stuff) and so at any moment every employee / hired hand is accounted for, has a number, has a position in the org chart, has access to a wiki-type platform where they can be trained and informed of any changes to the workplace software suite and guided through any migrations.
I've seen the company migrate off Slack onto Microsoft Teams. I've seen the company migrate to MS Sharepoint from Box. I've seen them migrate everyone onto a platform called SuccessFactors (still don't really know what it does, it's for tracking your career progress I think).
There's work involved. Someone has to write a guide to get users to sign up with the new service (even with SSO linked to your corporate account it's not easy for most non-technical people). In the case of Slack, any hooks or bots created for any teams need to be turned off and people need to be informed well in advance of the shift and multiple times. Optional in-person trainings need to be provided. Some employees may have issues with the change (a missing feature in the new software that they rely on) and they need a forum (or even just someone to contact) where they can lay out these issues and get them resolved.
But that's not that bad honestly. If moving from a platform gives you serious gains in uptime, it's not that bad. I think Slack's downtime problems are not so bad that most people will move yet, but they may soon get there.
Many reasons, almost none of them technical. Off the top of my head, a few:
* Getting out of the Enterprise Contract, or waiting for the year to end.
* Training people on new software.
* Loss of productivity. (1) Learning a new UI, processes, workflows -- both individually and organizationally. A feature or concept in "Tool A" may exist in a completely different form in "Tool B". Or not exist, and then people need to adapt to and work around the missing feature. (2) Missing out on needed information due to the above. Ultimately, software exists to move and transform data, and when you change the software people have to adjust. Sometimes that doesn't go great. "Oh, I didn't realize I needed to check this checkbox".
Another way to say this is "organizational inertia", which is a fancy term that means "it's hard for people to adjust to change".
And you might think developers and other technical people would have an easier time of it. They (we) do, but not to the extent you may expect. I've been on the front lines of a handful of migrations that affected only the IT staff, and it was a long and arduous process each time.
Man it bothers me so much when applications change their UIs on updates for no apparent reason other than "it looks better".
IntelliJ changed the way build and debug buttons looked in some update and it took me days to get used to it and I could find them in a snap again. Slack did a couple of no-reason changes as well.
There are plenty of UX reasons (learning new interfaces, etc). The burden here is generally distributed and diffuse.
The really big one, for companies of a certain size / cash flow, is compliance. Companies spend a lot of time developing compliant work flows around a service like Slack.
Migrating to another service requires rewriting the compliance narrative. The current compliance people might not have the confidence or willpower to do that effectively, and can raise legal objections to any such migration indefinitely.
I say this every time Slack is down, but they just seem so shady to me. Nobody can connect right now, and their status site says "100% uptime in the last quarter". Maybe it's close to 100%, but it ain't 100%.
I think we should push for a metric where "up" means 100% of people that want to use the service are able to use the service. If 1% of users can't send messages, then that should count as a full-blown outage and should start counting against whatever SLA they advertise.
The underlying problem here is that apparently everyone lies about uptime, so if you don't, that looks bad to potential customers. I fear that we will have to push for some legal regulation if we want accurate data, and ... people will probably be opposed to that.
They should at least update the status site to reflect issues currently happening.
I was wondering why the link from a Jira wasn't opening in slack, the page eventually timed out and gave me a link to status.slack.com where it told me everything was peachy. Cue me wasting time trying it again because apparently there was no issue with slack..
You'll just end up with no SLA or pay a hefty amount to use services because that's an impossible standard to support for any service of a size like this.
Isn't this the problem? Companies like Slack set SLA's that they only meet by lying about their uptime. It's as good as having no SLA, except you're likely paying a premium based on the SLA they set.
I'm not demanding 100% uptime, I'm asking that they say "99.94% uptime" when there has been an outage.
Honestly, I could live with a 99.50% SLA, if that's what it really was. After today's probably full-day outage, they'd just have to be extra careful for the rest of the year (or pay me money). Kind of sucks when it's 1/4 that you blow your year's SLA budget though.
> I think we should push for a metric where "up" means 100% of people that want to use the service are able to use the service.
I mean, that’s nice to say, but how do you measure/prove it?
Certainly, having the SLAed party check themselves is silly. But what are the other options? If it was up to the customer, customers could make up faults to get free service. (Since it’d be up to the customer to prove, and customers are generally less technical than vendors, you’d have to expect/accept very non-technical — and thus non-evidentiary! — forms of “proof”, e.g. “I dunno, we weren’t able to reach it today.” Things that could have just as well been their own ISP, or even operator error on their side.)
IMHO, contractual SLAs should be based on the checks of some agreed-upon neutral-third-party auditor (e.g. any of the many status/uptime monitoring services.) If the third party says the service is up, it’s up in SLA terms; if the third party says the service is down, it’s down in SLA terms.
(And, of course, if the third party themselves go down, or experience connectivity issues that cause them to see false correlated failures among many services, that should be explicitly written into the SLA as a condition where the customer isn’t going to get a remedial award against the SLA, even if the SLAed service does go down during that time. If the Internet backbone falls over, that’s the equivalent of what insurance providers call an “act of God.”)
But in a neutral-third-party observer setup, you aren’t going to get 100% coverage for customer-seen problems. An uptime service isn’t going to see the service the way every single customer does. Only the way one particular customer would. So it’s not going to notice these spurious some-customers-see-it-some-don’t faults.
So, again: what kind of input would feed this hypothetical “100% of customers are being served successfully” metric?
ETA: maybe you could get closer to this ideal by ensuring that the monitoring service 1. is effectively running a full integration test suite, not just hitting trivial APIs; and 2. if gradual-rollout experiments ala “hash the user’s ID to land them in an experiment hash-ring position, and assign feature flags to sections of the hash ring” are in use by the SLAed service, then the monitoring service should be given N different “probe users” that together cover the complete hash-ring of possible generated-feature-flag combinations. Or given special keys that get randomly assigned a different combination of feature-flags every time they’re used.
Some companies do this, though probably not publishing data. Any customer downtime is treated the same - for one, for many, for all (in theory, ha ha). But they take it pretty seriously.
The idea is to define availability as "the probability that the site 'appeared' to be down for a random user, averaged over a time window of size w". You can choose a particular value of w and look at trends over time, or you can plot availability as a function of w to understand patterns of downtime.
Seems silly to worry about quarterly stats several hours into an outage. The most obvious explanation is quarterly stats aren't generated in real-time -- which isn't "shady" to me.
HN is where we all go when the Internet (or large portions of it) are down. It's more reliable than all the 'downforeveryoneorjustme' or 'downtime monitor' services.
They always have been, since they clearly don't fit the guidelines for what a good submission is and usually leave little for interesting discussions. (unlike postmortems of past outages, which often are good)
Agreed, when a major service goes down HN is the most accurate overview, often a useful sanity check when its AWS or Slack size orgs before I open an incident with whichever party.
These events seem to be happening almost on a monthly basis now. IRC was never this unreliable and at least with netsplits it was obvious what had happened because you'd see the clients disconnect.
IME messages just fail to send with Slack, then you can retry but they're not properly idempotent and you end up sending the messages twice.
I'm the opposite. Back when in my early teens, friends and I would attempt to hijack opposing groups' channels via takeovers during net-splits (and ofcourse having the same done to us). What a time to be alive.
In the early battle.net days competing clans would split and steal channels. It was tons of fun. Taught me lots about bots, proxies, simple scripting, in the process too.
I do miss them, terribly. Lightweight, fast, brutally simple. Even with splits, it was better, and ever since IRC bouncers exist, like ZNC, they are rock solid.
I'm still dreaming of a world where everyone uses IRC through an interface identical to Slack or Discord or whatever, and features like these are implemented.
You might like irccloud; it's a web client (similar to slack) and bouncer, with support for image uploads, has a decent app, preserving history and I think it supports search too.
Not really a fan of the Slack or Discord user interface myself, but there are modern looking web clients for IRC such as thelounge[0] or kiwiirc[1] that might be what you are after.
I agree in principle, but IRC is a poor way to do this. I love IRC for it's simplicity, but that makes it hard to do more advanced features. It's a text-only protocol (other than DCC), so if you want to do something like allow users to click phone numbers to dial them then you have to regex it and hope for the best. Any kind of link is the same way. If you want to show images inline, you'll have to search for links, then either do another regex to see if the link is an image or prefetch the page to see if it's an image. Most servers still implement user authentication as a secondary service (i.e. it isn't part of the IRC server itself) afaik. I think the newer IRC specs include those, but support for it is missing in many servers.
Really a huge part of IRC's difficulty and beauty is in not having a markup language, but most of that beauty is for the eyes of the developer, not the user.
I like the concept of Matrix. That's kind of what they're trying to do by creating an open protocol, but when I looked at implementing a client it was non-trivial. For IRC, you can usually send someone a telnet log of you joining an IRC server and they could implement a client. I don't get the impression that that's true for Matrix.
https://news.ycombinator.com/item?id=20948530 is my attempt to demonstrate that implementing a Matrix client is almost as trivial as telnetting to port 6667 on an IRC server, fwiw :)
Several IRC servers do have support for authentication and access control (and audit trails as well I suppose).
Only centralized history/logging and search would need to be bolted on if needed.
In the non-centralized case your IRC client takes care of all of that.
For business users, there are regulatory requirements. You need to keep information around for some period of time, but not forever. History and searching is useful for spreading tribal knowledge throughout an organization.
Does that actually extend to Slack/slack-like things though?
Since I would see Slack more of a replacement for phone calls or hallway discussions.
Neither of which typically has any logs or recordings (and I wouldn't want to work somewhere that did keep such logs).
In what areas would you find such requirements? And shouldn't the default position be that it is illegal to keep those logs? Especially those involving direct messages between employees.
Our prod systems seem to be working, but our lower environments seems to be not working. I don't know enough about where these things come from. I wonder if the real problem is regional. Some connections work and some don't.
I'm sure you know this already, but that status page isn't worth the cycles on your CPU, you would be better served asking the toaster if AWS is functioning properly than checking that status page.
I never knew this, but I think it makes sense. Is there any documentation that explains why this is the case? I suspect it is to distribute bias to the first option, but I'd love to read about it.
Our company uses Cliq. I wouldn't say that it's as good as Slack, but it's probably 80-90%, and even has a few unique features (integration into Zoho's suite, remote work checkin, integrated bot development environment, etc)
Its especially strange when you think about how unoriginal Slack's product domain is, and how comparable, and in some cases small, their userbase is.
* iMessage, which likely handles something in the range of 750M-1B monthly actives.
* WhatsApp, 2B users [1], though no clarity on "active" users.
* Telegram, 400M monthly actives [2]
* Discord, 100M monthly actives [3]
* Slack, 12M daily actives [4]
* Teams, which is certainly more popular than Slack, but I shudder to list it because its stability may actually be worse.
The old piece of wisdom that "real-time chat is hard" is something I've always taken at face-value as being true, because it is hard, but some of the most stable, highest scale services I've ever interfaced with are chat services. iMessage NEVER goes down. I have to conclude that Slack's unacceptable instability, even relative to more static services like Jira, is less the product of the difficulty of their product domain, and moreso something far deeper and more unfixable.
I would not assume that this will improve after they are fully integrated with Salesforce. If your company is on Slack, its time to investigate an alternative, and I'm fearful of the fact that there are very few strong ones in the enterprise world.
I didn't realize that Discord has way more active users than Slack. I'm glad, Discord is a fantastic service in my experience. It's a shame they got shoe horned into a mostly gaming oriented service. I've never had a class or worked somewhere where Discord was a considered solution instead of Slack, but I can't think of anything that Slack does better (in my experience). In general, I think Discord has the best audio and video service that I've used, especially kicking Zoom to the curb.
Discord is definitely in the same realm of scale as Slack, and probably bigger (they publish different metrics, so its hard to say for sure).
The really impressive thing about Discord's scale is the size of their subscriber pools in the pub-sub model. Discord is slightly different than Slack in the sense that every User on a Server receives every message from every Channel; you don't opt-in to Channels as in Slack, and you can't opt-out (though some channels can be restricted to only certain roles within the Server, this is the minority of Channels).
Some of the largest Discord servers have over 1 million ONLINE users actively receiving messages; this is mostly the official servers for major games, like Fortnite, Minecraft, and League of Legends.
In other words, while the MAU/DAU counts may be within the same order of magnitude, Discord's DAUs are more centralized into larger servers, and also tend to be members of more servers than an average Slack DAU. Its a far harder problem.
The chat rooms are oftentimes unusable, but most of these users only lurk. Nonetheless, think about that scale for a second; when a user sends a message, it is delivered (very quickly!) to a million people. That's insane. Then combine that with insanely good, low latency audio, and best-in-class stability; Discord is a very impressive product, possibly one of the most impressive, and does not get nearly enough credit for what they've accomplished.
For comparison; a "Team" in Microsoft Teams (roughly equivalent to a Discord Server or Slack Workspace) is still limited to 5,000 people.
>I didn't realize that Discord has way more active users than Slack
Keep in mind you're comparing daily active users vs monthly active users. I'd guess most slack users are online weekday for pretty much the entire day (because it's for work and your boss expects you to be online), whereas a good chunk of discord users are only logging in a few hours a week when they're gaming.
Minecraft official server: 190k online users. | Fortnite official server: 180k online users. | Valorant official server: 170k online users. | Jet's Dream World (community): 130k online users. | CallMeCarson server (YouTuber): 100k online users. | Call of Duty official server: 90k online users. | Rust (the game) official discord: 80k online users. | League of Legends official server: 60k online users. | Among Us official server: 50k online users.
Their scale is insane. Even with their usage spiking during after-hours gaming in major countries, their baseline usage at every hour of the day, globally, makes it one of the most used web services ever created.
Slack's DAU and MAU numbers are probably pretty close to one-another. Discord's MAU/DAU ratio is probably bigger than Slack's. That just means that Discord is, again, solving a harder problem; they have much bigger (and more unpredictable) spikes in usage than Slack. Yet, its a far more stable and pleasant product.
Well for the real time side, I can't tell you how big a boon it's been to build our platform on top of Elixir/BEAM. Hands down the best runtime / VM for the job - and a big big secret to our success. Where we couldn't get BEAM fast enough - we lean on rust and embed it into the VM via NIFs.
2021 is the year of rust - with the async ecosystem continuing to mature (tokio 1.0 release) we will be investing heavily in moving a lot of our workloads from Python to Rust - and using Rust in more places, for example, as backend data services that sit in front of our DBs. We have already piloted this last year for our messages data store and have implemented such things as concurrency throttles and query coalescing to keep the upstream data layer stable. It has helped tremendously but we still have a lot of work to do!
To help scale those super large servers, in 2020 we invested heavily in making sure our distributed system can handle the load.
Did you know that all those mega servers you listed run within our distribution on the same hardware and clusters as every other discord server - with no special tenancy within our distribution. The largest servers are scheduled amongst the smallest servers and don't get any special treatment. As a server grows - it of course is able to consume a larger share of resources within our distribution - and automatically transitions to a mode built for large servers (we call this "relays" internally.) At any hour, over a hundred million BEAM processes are concurrently scheduled within our distributed system. Each with specific jobs within their respective clusters. A process may run your presence, websocket connection, session on discord, voice chat server, go live stream, your 1:1/group DM call, etc. We schedule/reschedule/terminate processes at a rate of a few hundred thousand per minute. We are able to scale by adding more nodes to each cluster - and processes are live migrated to the new nodes. This is an operation we perform regularly - and actually is how we deploy updates to our real time system.
I was responsible for building and architecting much of these systems. It's been super cool to work on - and - it's cool to see people acknowledge the scale we now run at! Thank you!! It's been a wild ride haha.
As for scale, our last public number perhaps comparable to Slack is ~650 billion messages sent in 2020, and a few trillion minutes of voice/video chat activity. However given the crazy growth that has happened last year due to COVID - the daily message send volumes are well over the 2 billion/day average.
I really agree Discord is amazing and wish I could use it for work instead of Slack.
I think the big things that prevent it from being adopted more for professional use is the lack of a threading model (even though I hate it when people use threads in Slack) and the whole everyone in every channel except for role-based privacy settings. The second one especially is a big deal because you can't do things like team-only channels without a prohibitive amount of overhead.
That said (with zero knowledge of their architecture) I have to feel like both of those missing features aren't too terribly hard to build. Its very likely Discord is growing as a business fast enough on the gaming and community spaces they don't feel the added overhead of expanding into enterprise (read: support, SLAs, SOC, etc) makes sense and are waiting until they need a boost to play that card.
> I think the big things that prevent it from being adopted more for professional use is the lack of a threading model
They do have a threading model now (if you are talking about replying to a message in a channel and having your reply clearly show what you are responding to). If you are talking about 1-on-1 chats with other people in your same server then yes, that is still lacking IMHO in discord. The whole "you have to be friends" to start a chat (or maybe that's just for a on-the-fly group) is annoying.
Discord gives every user an identity that is persistent beyond the server; you have a Discord account, not a server account. Slack does the opposite. Enterprises would hate Discord's model, as they prefer to control the entire identity of every user in their systems, such that when they leave the company they can destroy any notion of that identity ever existing.
Absolutely agree. I like the 1 main discord account but I wish I could have 1 "identity" per-server as well. I don't love that I am in some discords that I don't want tied to my real name and others where I've known these people for over a decade and would see in person multiple times a week (before the pandemic). I know you can set your name per-server but you can't hide your discord username (or make it per-server) which sucks.
Agreed completely. Discord has always been much smoother for me than Slack, and the voice/video chat quality is literally the best I've ever seen anywhere.
If they made their branding a bit more professional and changed the permission model from the (accurate) garbage you described to something closer to Slack then I think Slack would be doomed.
It isn't just # of users, though - SlackOps is probably unique to Slack in that list (minus Teams, I guess) - so # of messages per month is a better metric. Not that I'm letting Slack off the hook, it still may be that their codebase and/or dev process is just nasty.
Slack and the others have different contractual guarantees and different regulatory environments. Comparing them is not really fair because the reality is that these other services probably just lose tons of messages and slack/teams can't do that! They have to have better guarantees.
That's kind of the definition of a service being up. :) I've experienced numerous "soft" outages which result in messages not sending and getting lost - and even more double sends, sometimes very distant from where the message was originally sent.
Just anecdotal, but as someone who has used Teams continuously for 1.5 years, I can say that it has never been down for me.
That being said, individual instances of the app are notoriously unstable causing random annoyances. But, I am on a very early build of Teams, which is buggy by definition.
I have stopped using Down Detector as an accurate measure because a lot of "outages" are just people having issues with a service unrelated to the service they are reporting as down. Ex: AT&T outage in Nashville caused people to report Xbox Live as down, when it wasn't actually down, etc.
I have issues reaching a lot of sites, especially american.
Both downdetector, hacker news and others loads extremely slow or not at all. Downdetector had a bunch of failed resources for me..
> Customers may have trouble loading channels or connecting to Slack at this time. Our team is investigating and we will follow up with more information as soon as we have it. We apologize for any disruption caused.
- Jan 4, 10:14 AM EST
The status for messaging and connection services has been marked as [incident]
> We're continuing to investigate connection issues for customers, and have upgraded the incident on our side to reflect an outage in service. All hands are on deck on our end to further investigate. We'll be back in a half hour to keep you posted.
- Jan 4, 11:20 AM EST
> There are no changes to report as of yet. We're still all hands on deck and continuing to dig in on our side. We'll continue to share updates every 30 minutes until the incident has been downgraded
I'm positive that they have internal monitoring, and probably knew about the issues well before they decided to manually update their status page to reflect the issue. Manually updating the status page does not equal no monitoring, after all.
A lot of organizations essentially took the last two weeks off from work, which is long enough for a 10-day autoscale window to spin down servers, and then get confronted by a load spike that wasn't pre-spun for.
What does this mean? What do cloud providers do when customers scale down their services? Do the providers literally power down servers? Do they sell the capacity to new customers?
Relax. GP is clearly referring to an increase in people signing on do to the holidays ending and everyone coming back from work.
Also, Slack has significantly more users in the US than in any other country[1], and it really isn't even close. So the offense you're taking is unwarranted anyway.
Slack makes ~61% of its revenue from US customers which only has 4 time zones compared to the remainder of their revenue being spread out across ~20 time zones. It's not an unreasonable hypothesis.
- Revenue is not same as users. Slack have tons of free users and some countries also has lower priced plans.
- Many companies like Amazon etc. probably is counted as US revenue for Slack but they have more than 30% of their employees outside the US. This should not be huge numbers but significant.
I'm a Carl. I'm also looking for a coworker who was trying to contact me. If it's about last saturday, I promise nothing really happened between me and her, but I'm sure she already told you.
I have tried to sell my organisation on a shared Google Chat doc for 90s style realtime ICQ chat in times like these, but there has been little uptake.
G Suite actually has an entire Slack clone, chat.google.com. I've been on G Suite (now annoyingly renamed to Google Workspace) for years and actually just recently found it existed from another comment on HN.
Yeah, this is what we actually use as a fallback, and I did push for this as an full time alternative given we'd get it free, but people dislike it for all sorts of frivolous reasons.
> We’re still investigating the ongoing connectivity issues with Slack. There's no additional information to share just yet, but we’ll follow up in 30 minutes. Thanks for bearing with us.
it's not based on only that. Slack costs a lot of money and moving off of it is something that has continually come up over the last year or two. We even had a Rocketchat server up and running for awhile.
When I was at Uber, we noticed that most incidents are directly caused by human actions that modify the state of the system. Therefore, a large "backlog" of human actions that modify the system state have a much higher chance of causing an incident.
My bet is that this incident is caused by a big release after a post-holiday "code freeze".
I would bet it's just the influx of traffic post holiday with systems that haven't been updated in so long maybe some annoying memory leaks have crept up and gone unnoticed or some other bad state that was exacerbated by return to work day for most NA folks. Code freezes were good at identifying bugs that only show up after long periods.
Doubt anyone releasing big changes Monday morning.
That might be true but when you take the global usage of Slack and their respective time zones, more than half the world would have signed into Slack this morning before SV had and I certainly didn't notice any downtime this morning in my time zone.
I haven't worked at Slack, so I can't speak with high confidence. A traffic spike is a possible reason, but I'm willing to bet that it's not the reason:
> Doubt anyone releasing big changes Monday morning.
This is definitely an engineering best practice, and by best practice, I mean something that Uber's, I mean Slack's SRE team strongly pushed for, and got politely overruled on. After a code freeze is lifted, it's quite common for lots of promotion-eager engineers to release big changes.
I'm not sure about that. I feel like I get more upvotes from sarcasm and jokes than from insight. In this instance, I think it's because when people hear something dumb said seriously in real life, they're not going to readily recognize online that it's a joke.
IMO it really doesn't have to be promotion-eager engineers or antsy product managers. I'm fairly satisfied with my role and comp and work type with where my career/life-stage is. I just did a code release first thing this morning, not because I am promotion-eager, but just because I'm picking back up where I left off, like any normal day. Granted I work at a much smaller company than Slack with orders of magnitude less traffic.
Why? I had a rewrite of some core logic the last day before Christmas that I didn'td deploy, as it wasn't time critical to get out and I didn't want to be disturbed during holidays. Today it was perfect to deploy, as I can watch it the whole week if needed.
Well, I think it probably depends on where you work. At my work, people just took 2-3 weeks of time off. It takes a moment to get your head back in the game.
Everywhere I've worked often has a massive backlog of things that get released after a moratorium or extended holiday week. Those are usually the worst weeks to be oncall since things are under so much churn.
Does Uber/Slack not release in CI/CD? At least in backend?
I don't see any need to deploy a big change at once in the software world today. At worst feature gate the thing you want to do and run it in a beta environment, but still push the actual code down the pipeline.
I'm actually more confused after reading that. I assumed that you meant that tested in production on purpose, but it sounds, at a skim, like they do non-prod testing environments - in fact, it looks like they've gone to having multiple beta environments of every service?
My understanding is that they have a "tenancy" variable in every service call which can take a different code path. They seem to only have one environment for everything and do tests/experiments at code level based on this variable.
Interesting, I've never worked anywhere where engineers decide when to release changes. That's a product decision, and there is a process of review and approval at both the code level and the functional/end-user-experience level that has to happen first.
Did you mean that literally? E.g. is it common at Uber that engineers can release changes to production on their own?
At Cisco (Webex team), the engineers decide when to release code, and most features are enabled by configs or feature flags independently of the deploys.
The engineering team is responsible for the mess caused by a bad deploy, so it's appropriate that those engineers should also choose the timing.
Our team typically deploys between 10am and 4ish, local time, since that's when we're at our desks and ready to click through the approvals and monitor the changes as they go through our pipelines.
The feature enablement happens through an EFT / beta process, and the final timing of GA enablement is a PM decision. But features are widely used by customers ahead of that time, as part of the rollout process.
Our team usually rolls out non-feature changes to services via dynamic configuration switches, so that we can get new bits in place, and then enable new behavior without a redeploy. This also enables us to roll back the dynamic config quickly if something unexpected happens.
(We generally don't do this for net new functionality; there's lower risk in adding a new REST endpoint etc. than in changing an existing query's behavior or implementation.)
What would make that strange? Where I work it is frowned upon to do releases on weekends and so bad changes due to buildups happen on Monday.
Although, we also don’t close the pipeline for just any holiday break. In fact low holiday traffic is a good time to keep pipelines open, since changes will impact less people.
To elaborate a bit more on this point, you have to think about it like any complex system failure - it's almost never one thing, but rather a combination of many different factors. The factors around post NYE releases:
- high risk changes that weren't released pre-holidays get released. Depending on the company, this could mean a 1-week to 1-month delay between implementation and release. The greater that interval, the higher the divergence world of production and the world of the new feature
- lots of new hires (new year = new hiring budget). New hires are missing some tribal knowledge about the system and make a production-breaking release.
I tried to think of other reasons, but these two overwhelmingly stand out as the two biggest reasons. Would love to hear from others.
Doubtful. It's not impossible a company the size of Slack would be reliant on a specific engineer logging on in the morning before a traffic spike so the service can handle the spike in load, but that's a misuse of modern distributed cloud-based computing.
Hate on the cloud all you want, but AWS has (several flavors of) load balancers and various ways to automatically scale up and down resources (and if you're conservative, you can disable the 'down' part). If you're operating a major SaaS company like Slack and not taking advantage of them, something's gone wrong.
It's easy to fall behind on bumping up the high watermark for your max autoscaling or for new traffic patterns to cause emergent instability. New code paths are taking unprecedented amounts of traffic all the time.
In 2021, how does one keep track of resource starvation at the process, container, os, service, pod, cluster, availability zone and region levels?
If new hires tends to break production, it's not in the first business day of the calendar year. December gets really quiet for recruiting, typically, as candidates get busy with their social lives, and scheduling interviews gets harder.
January is busy for recruiting, but given a week or two of interviewing and negotiating, two weeks notice, it's probably February before new employees are starting, and they're not making big, production-damaging deploys for a week or two after that.
You will also get a pause in new hires in late December for the same reason. I've certainly accepted an offer late in the year and then didn't start until the new year.
Probably not as big of a rush as the end of school year rush in summer though.
I also doubt that new people will be breaking production on day one. Even at a fast moving startup I'd expect it to take a bit to go through the onboarding paperwork, get a laptop and actually try pushing a change to production.
I think some big company (maybe Facebook) has this rule that you had to deploy something to production on your first day. They seemed pretty confident in their processes and devops teams. A company trying to imitate that policy without doing the work necessary to make it possible would probably have outages on days when lots of new people joined :-P
Could be Facebook as I think production releases are always rolled out in phases e.g. first to 10 users, then 100, then 1000 and so on. That means there's much less chance of even the worst mistake having a serious effect.
Wow, onboarding new hires here is going good, if they can access slack, O365, LDAP, VPN and clone the repo by the end of the first day. Tho we have the initiation ritual of installing the OS to your laptop.
I think you're right on the first bullet, but not the second. If it was mid-Feb, then maybe, but the next FY hasn't even started yet for a ton of companies, let alone onboarding newbies to production.
I would add here the potential scaling issue - holidays were a dry season - less meeting. So if they have some automation for scaling down to reduce cost, it may have bitten them in their arses now.
People came back to work, and most of them start around the same time (US wise at least).
Hence kids - a vital lesson for all of us - don't start the call at a full hour, give it 3-7 min to make your coworkers confused and give some time for the systems to auto-scale ;)
Yeah, makes sense. A system typically optimized for performance and real time delivery is suddenly asked to perform multiple batch retrievals in large chunks. Ouch!
It means that one is not sending a session cookie of any kind, thus should be sent to a 100% cached version. No "Are you XYZ and what to log into ABC's Slack again?" box.
An incognito browser would ignore all client-side cookies, so the Slack web client would not try to - say - resume a previous user's session or re-use any previously saved data.
Likewise, incognito mode will also ignore most cached web content, meaning all assets on the Slack web app will get loaded again from scratch. This "clean state" start could, theoretically, get around issues with old - potentially incorrect/outdated - assets being loaded, even though that really shouldn't happen under most circumstances.
I have definitely worked in places where the times right before and right after a change freeze were the most unstable, so that could be it. However, as others have mentioned, it's pretty early on the west coast of the US. Unless some engineer was up extra early (perhaps at the behest of an anxious project manager) it seems unlikely to be a release.
What it could be is some engineer somewhere coming in after the holiday, noticing a slightly flaky thing, and thinking, "I'll reboot/redeploy/refresh this thing so the flakiness doesn't get worse". Only it turns out the flaky thing was a signal of something else falling over. Or maybe the redeploy was the wrong version because of bad CI/CD, or maybe the person just fat-fingered it.
Very possible. I don't know what Slack's workforce distribution is. In places I've worked there have definitely been some incidents in US off-hours triggered by someone on the other side of the world.
It varies a lot by team... I think it's common to have a single click "start" button to press. It's a good sanity check that a release isn't going to happen during a fire drill, outage, or strike...
This is one of the original concepts why to go capital-A Agile. Make smaller releases more often, so at least if something breaks, it's (hopefully) something small, and least it's easier to trace.
(I'm not making a statement if that's good or bad or if it works or whatever. Please don't read an opinion into it.)
This. If you roll many changes into a single deployment, you don’t know which change broke what. But if you have two or three weeks of commits waiting, it’s hard to do otherwise.
That's why good regression tests and CI are so important; in an ideal world (which we were close to in one of my projects), every change is pending in a pull request; the CI rebases the change on top of its upstream (e.g. master/main), simulating the state the codebase will be in once merged, and runs the full suite of tests. The build is invalidated and has to be re-run if either the branch or upstream is changed.
Now, caveats etc, this was a collection of single applications in a big microservices architecture, and as the project grows it becomes more and more difficult to manage something like this, especially if you get more pull requests in the time it takes to do a build. But it is the way to go, I think.
Anyway, since tests and CI are not definitive, you also need a gradual rollout - 1%, 5%, etc - AND you need a similar process for any infrastructure change, which gets more and more tricky as you go down to the hardware level.
Another common cause is resource exhaustion as a result of poorly monitored resources (or bugged monitoring). For example Google's authentication was down because their system reported (wrongly) available quota of 0. The last two incidents at my company were also related to resource exhaustion.
This is my SignalR alternative with end-to-end encryption. Choose a password and the file and message will be encrypted in client side using that password.
839 comments
[ 3.1 ms ] story [ 333 ms ] threadhttps://www.youtube.com/watch?v=UaPkSU8DNfY
The Forsaken (season 1 episode 17)
Rivals (season 2 episode 11) There's a reason to have a backup to the backup by Destiny (season 3 episode 15)I guess everyone hopping back online over the course of a few hours for the new year is too much to handle!
I wonder if this is because I haven't used the phone app in a few days, so I was already logged out, but you and others were still logged in?
I imagine their infrastructure to send push notifications is decoupled from their infrastructure for chat services themselves.
It'd be interesting to know if they have a master switch to disable notifications in times like this where they aren't usable anwyay.
Depends on the timezone you're in, though one could theoretically cite a disparity between physical and mental/emotional/temporal time zones...
https://www.youtube.com/watch?v=BPCjC543llU
There's also this perverse incentive to Slack all the things. Lots of CI notifications are sent through it. Some org processes are implemented as workflows. There's been talk of how wonderful it would be to hook up tasking and work tracking to slash commands. I and others often use Slack instead of the 'official' tool to video call each other.
An outage like this is still really disruptive. It's not like everyone realizes what's going on immediately or at the same time; we have backup tools, but our turn radius is pretty wide. Some of us can't even communicate effectively without memes, too, and backup tools don't have a giphy integration.
EDIT: Do your CI integrations fail if Slack can't be contacted? Do those failures fail your pipeline? Whoops!
However, I drive everything through slack - GitHub, linear, calendars, Notion, support emails, etc. I have notifications turned off for every service we use except for slack. This allows me to effectively ignore everything except for slack. These types of outages destroy that workflow for me.
But it was roughly "one large impact a month, for six months", with large caveats that upper management for whatever company had to be working with the product during that month.
Large companies don't care if X service went out during the night and impacted someone not in their timezone.
If the CTO notices that he can't use something with the same regularity that he gets paid, then it doesn't take long for it to stick in their mind. But migrating everything is _so painful_ that the majority of large companies will do anything they can to avoid moving away.
This is a key point is the popularity amongst VCs in investing in B2B SaaS. I take their (and your) word for it. But honestly, I don't actually understand this.
Why is migration so hard?
This is not to mention the fact that half our staff aren't hugely technical, so have actively _learnt_ how to use Slack and it's features around notification control (things that may come "naturally" to the tech-savvy crowd on HN), @-things, bots, etc, and they would need to re-learn a new tool that is going to work in a different way.
This would be a substantial effort for us, and we're a small company. Are there ways to materially minimise this cost?
Why would anyone make it easy
Things that are easy to migrate from get replaced by things that are hard to migrate from, eventually.
IRC is incredibly easy to migrate from.
The parent meant a law as in "a law of physics", not a piece of legislation.
Plus, it will just take a long time to get everyone on board and using the replacement system. My department is slowly plodding towards using Teams over Slack, but there are enough hold-outs (my sub-department being one of them) that it still doesn't have wide-spread adoption.
If you had a small shop with a dozen tech-savvy people and Slack became a problem which was used exclusively for quick business chats, you could probably push a change to another chat platform the next day. You might struggle when you have thousands of employees, some that needed training to use Slack and still aren't that proficient.
I worked at a huge corp which employed many non-technical people and had global offices and all sorts of contractors / full-time employees at various levels.
But they invested in organizational / business software early (think SAP-type stuff) and so at any moment every employee / hired hand is accounted for, has a number, has a position in the org chart, has access to a wiki-type platform where they can be trained and informed of any changes to the workplace software suite and guided through any migrations.
I've seen the company migrate off Slack onto Microsoft Teams. I've seen the company migrate to MS Sharepoint from Box. I've seen them migrate everyone onto a platform called SuccessFactors (still don't really know what it does, it's for tracking your career progress I think).
There's work involved. Someone has to write a guide to get users to sign up with the new service (even with SSO linked to your corporate account it's not easy for most non-technical people). In the case of Slack, any hooks or bots created for any teams need to be turned off and people need to be informed well in advance of the shift and multiple times. Optional in-person trainings need to be provided. Some employees may have issues with the change (a missing feature in the new software that they rely on) and they need a forum (or even just someone to contact) where they can lay out these issues and get them resolved.
But that's not that bad honestly. If moving from a platform gives you serious gains in uptime, it's not that bad. I think Slack's downtime problems are not so bad that most people will move yet, but they may soon get there.
* Getting out of the Enterprise Contract, or waiting for the year to end. * Training people on new software. * Loss of productivity. (1) Learning a new UI, processes, workflows -- both individually and organizationally. A feature or concept in "Tool A" may exist in a completely different form in "Tool B". Or not exist, and then people need to adapt to and work around the missing feature. (2) Missing out on needed information due to the above. Ultimately, software exists to move and transform data, and when you change the software people have to adjust. Sometimes that doesn't go great. "Oh, I didn't realize I needed to check this checkbox".
Another way to say this is "organizational inertia", which is a fancy term that means "it's hard for people to adjust to change".
And you might think developers and other technical people would have an easier time of it. They (we) do, but not to the extent you may expect. I've been on the front lines of a handful of migrations that affected only the IT staff, and it was a long and arduous process each time.
Man it bothers me so much when applications change their UIs on updates for no apparent reason other than "it looks better".
IntelliJ changed the way build and debug buttons looked in some update and it took me days to get used to it and I could find them in a snap again. Slack did a couple of no-reason changes as well.
The really big one, for companies of a certain size / cash flow, is compliance. Companies spend a lot of time developing compliant work flows around a service like Slack.
Migrating to another service requires rewriting the compliance narrative. The current compliance people might not have the confidence or willpower to do that effectively, and can raise legal objections to any such migration indefinitely.
I think we should push for a metric where "up" means 100% of people that want to use the service are able to use the service. If 1% of users can't send messages, then that should count as a full-blown outage and should start counting against whatever SLA they advertise.
The underlying problem here is that apparently everyone lies about uptime, so if you don't, that looks bad to potential customers. I fear that we will have to push for some legal regulation if we want accurate data, and ... people will probably be opposed to that.
I was wondering why the link from a Jira wasn't opening in slack, the page eventually timed out and gave me a link to status.slack.com where it told me everything was peachy. Cue me wasting time trying it again because apparently there was no issue with slack..
Honestly, I could live with a 99.50% SLA, if that's what it really was. After today's probably full-day outage, they'd just have to be extra careful for the rest of the year (or pay me money). Kind of sucks when it's 1/4 that you blow your year's SLA budget though.
I mean, that’s nice to say, but how do you measure/prove it?
Certainly, having the SLAed party check themselves is silly. But what are the other options? If it was up to the customer, customers could make up faults to get free service. (Since it’d be up to the customer to prove, and customers are generally less technical than vendors, you’d have to expect/accept very non-technical — and thus non-evidentiary! — forms of “proof”, e.g. “I dunno, we weren’t able to reach it today.” Things that could have just as well been their own ISP, or even operator error on their side.)
IMHO, contractual SLAs should be based on the checks of some agreed-upon neutral-third-party auditor (e.g. any of the many status/uptime monitoring services.) If the third party says the service is up, it’s up in SLA terms; if the third party says the service is down, it’s down in SLA terms.
(And, of course, if the third party themselves go down, or experience connectivity issues that cause them to see false correlated failures among many services, that should be explicitly written into the SLA as a condition where the customer isn’t going to get a remedial award against the SLA, even if the SLAed service does go down during that time. If the Internet backbone falls over, that’s the equivalent of what insurance providers call an “act of God.”)
But in a neutral-third-party observer setup, you aren’t going to get 100% coverage for customer-seen problems. An uptime service isn’t going to see the service the way every single customer does. Only the way one particular customer would. So it’s not going to notice these spurious some-customers-see-it-some-don’t faults.
So, again: what kind of input would feed this hypothetical “100% of customers are being served successfully” metric?
ETA: maybe you could get closer to this ideal by ensuring that the monitoring service 1. is effectively running a full integration test suite, not just hitting trivial APIs; and 2. if gradual-rollout experiments ala “hash the user’s ID to land them in an experiment hash-ring position, and assign feature flags to sections of the hash ring” are in use by the SLAed service, then the monitoring service should be given N different “probe users” that together cover the complete hash-ring of possible generated-feature-flag combinations. Or given special keys that get randomly assigned a different combination of feature-flags every time they’re used.
Google published a paper last year describing this approach to measuring uptime: https://blog.acolyer.org/2020/02/26/meaningful-availability/
The idea is to define availability as "the probability that the site 'appeared' to be down for a random user, averaged over a time window of size w". You can choose a particular value of w and look at trends over time, or you can plot availability as a function of w to understand patterns of downtime.
If you look at the history page you can see its not 100% for every month: https://status.slack.com/calendar
I think HN is hiding these posts. Maybe status threads are discouraged now? But they're much more useful than status.slack.com etc.
They always have been, since they clearly don't fit the guidelines for what a good submission is and usually leave little for interesting discussions. (unlike postmortems of past outages, which often are good)
IME messages just fail to send with Slack, then you can retry but they're not properly idempotent and you end up sending the messages twice.
It's really poor.
https://battlepenguin.com/tech/matrix-one-chat-protocol-to-r...
It works fairly well.
[0] https://thelounge.chat/ [1] https://kiwiirc.com/
Really a huge part of IRC's difficulty and beauty is in not having a markup language, but most of that beauty is for the eyes of the developer, not the user.
I like the concept of Matrix. That's kind of what they're trying to do by creating an open protocol, but when I looked at implementing a client it was non-trivial. For IRC, you can usually send someone a telnet log of you joining an IRC server and they could implement a client. I don't get the impression that that's true for Matrix.
Only centralized history/logging and search would need to be bolted on if needed. In the non-centralized case your IRC client takes care of all of that.
Since I would see Slack more of a replacement for phone calls or hallway discussions. Neither of which typically has any logs or recordings (and I wouldn't want to work somewhere that did keep such logs).
I'm sure you know this already, but that status page isn't worth the cycles on your CPU, you would be better served asking the toaster if AWS is functioning properly than checking that status page.
[edit] Nevermind, I just needed the right combination of terms to find it: https://docs.aws.amazon.com/ram/latest/userguide/working-wit...
- https://downdetector.com/status/slack/
- https://downdetector.com/status/aws-amazon-web-services/
The dashboards are all green. (which doesn't mean that much ... I'm aware...)
* iMessage, which likely handles something in the range of 750M-1B monthly actives.
* WhatsApp, 2B users [1], though no clarity on "active" users.
* Telegram, 400M monthly actives [2]
* Discord, 100M monthly actives [3]
* Slack, 12M daily actives [4]
* Teams, which is certainly more popular than Slack, but I shudder to list it because its stability may actually be worse.
The old piece of wisdom that "real-time chat is hard" is something I've always taken at face-value as being true, because it is hard, but some of the most stable, highest scale services I've ever interfaced with are chat services. iMessage NEVER goes down. I have to conclude that Slack's unacceptable instability, even relative to more static services like Jira, is less the product of the difficulty of their product domain, and moreso something far deeper and more unfixable.
I would not assume that this will improve after they are fully integrated with Salesforce. If your company is on Slack, its time to investigate an alternative, and I'm fearful of the fact that there are very few strong ones in the enterprise world.
[1] https://blog.whatsapp.com/two-billion-users-connecting-the-w...
[2] https://techcrunch.com/2020/04/24/telegram-hits-400-million-...
[3] https://wersm.com/discord-reaches-100m-monthly-active-users-...
[4] https://www.cnbc.com/2019/10/10/slack-says-it-crossed-12-mil... (this was also announced on Slack's blog, but that's down).
The really impressive thing about Discord's scale is the size of their subscriber pools in the pub-sub model. Discord is slightly different than Slack in the sense that every User on a Server receives every message from every Channel; you don't opt-in to Channels as in Slack, and you can't opt-out (though some channels can be restricted to only certain roles within the Server, this is the minority of Channels).
Some of the largest Discord servers have over 1 million ONLINE users actively receiving messages; this is mostly the official servers for major games, like Fortnite, Minecraft, and League of Legends.
In other words, while the MAU/DAU counts may be within the same order of magnitude, Discord's DAUs are more centralized into larger servers, and also tend to be members of more servers than an average Slack DAU. Its a far harder problem.
The chat rooms are oftentimes unusable, but most of these users only lurk. Nonetheless, think about that scale for a second; when a user sends a message, it is delivered (very quickly!) to a million people. That's insane. Then combine that with insanely good, low latency audio, and best-in-class stability; Discord is a very impressive product, possibly one of the most impressive, and does not get nearly enough credit for what they've accomplished.
For comparison; a "Team" in Microsoft Teams (roughly equivalent to a Discord Server or Slack Workspace) is still limited to 5,000 people.
Keep in mind you're comparing daily active users vs monthly active users. I'd guess most slack users are online weekday for pretty much the entire day (because it's for work and your boss expects you to be online), whereas a good chunk of discord users are only logging in a few hours a week when they're gaming.
Minecraft official server: 190k online users. | Fortnite official server: 180k online users. | Valorant official server: 170k online users. | Jet's Dream World (community): 130k online users. | CallMeCarson server (YouTuber): 100k online users. | Call of Duty official server: 90k online users. | Rust (the game) official discord: 80k online users. | League of Legends official server: 60k online users. | Among Us official server: 50k online users.
Their scale is insane. Even with their usage spiking during after-hours gaming in major countries, their baseline usage at every hour of the day, globally, makes it one of the most used web services ever created.
Slack's DAU and MAU numbers are probably pretty close to one-another. Discord's MAU/DAU ratio is probably bigger than Slack's. That just means that Discord is, again, solving a harder problem; they have much bigger (and more unpredictable) spikes in usage than Slack. Yet, its a far more stable and pleasant product.
Well for the real time side, I can't tell you how big a boon it's been to build our platform on top of Elixir/BEAM. Hands down the best runtime / VM for the job - and a big big secret to our success. Where we couldn't get BEAM fast enough - we lean on rust and embed it into the VM via NIFs.
2021 is the year of rust - with the async ecosystem continuing to mature (tokio 1.0 release) we will be investing heavily in moving a lot of our workloads from Python to Rust - and using Rust in more places, for example, as backend data services that sit in front of our DBs. We have already piloted this last year for our messages data store and have implemented such things as concurrency throttles and query coalescing to keep the upstream data layer stable. It has helped tremendously but we still have a lot of work to do!
To help scale those super large servers, in 2020 we invested heavily in making sure our distributed system can handle the load.
Did you know that all those mega servers you listed run within our distribution on the same hardware and clusters as every other discord server - with no special tenancy within our distribution. The largest servers are scheduled amongst the smallest servers and don't get any special treatment. As a server grows - it of course is able to consume a larger share of resources within our distribution - and automatically transitions to a mode built for large servers (we call this "relays" internally.) At any hour, over a hundred million BEAM processes are concurrently scheduled within our distributed system. Each with specific jobs within their respective clusters. A process may run your presence, websocket connection, session on discord, voice chat server, go live stream, your 1:1/group DM call, etc. We schedule/reschedule/terminate processes at a rate of a few hundred thousand per minute. We are able to scale by adding more nodes to each cluster - and processes are live migrated to the new nodes. This is an operation we perform regularly - and actually is how we deploy updates to our real time system.
I was responsible for building and architecting much of these systems. It's been super cool to work on - and - it's cool to see people acknowledge the scale we now run at! Thank you!! It's been a wild ride haha.
As for scale, our last public number perhaps comparable to Slack is ~650 billion messages sent in 2020, and a few trillion minutes of voice/video chat activity. However given the crazy growth that has happened last year due to COVID - the daily message send volumes are well over the 2 billion/day average.
I think the big things that prevent it from being adopted more for professional use is the lack of a threading model (even though I hate it when people use threads in Slack) and the whole everyone in every channel except for role-based privacy settings. The second one especially is a big deal because you can't do things like team-only channels without a prohibitive amount of overhead.
That said (with zero knowledge of their architecture) I have to feel like both of those missing features aren't too terribly hard to build. Its very likely Discord is growing as a business fast enough on the gaming and community spaces they don't feel the added overhead of expanding into enterprise (read: support, SLAs, SOC, etc) makes sense and are waiting until they need a boost to play that card.
They do have a threading model now (if you are talking about replying to a message in a channel and having your reply clearly show what you are responding to). If you are talking about 1-on-1 chats with other people in your same server then yes, that is still lacking IMHO in discord. The whole "you have to be friends" to start a chat (or maybe that's just for a on-the-fly group) is annoying.
We have a few bots we've integrated with things (deployment, stats, etc).
We use it for all our voice/video calls.
Edit: We've got roles setup well for things like contractors, devs, marketing, etc, so it's easy to lock down different conversations in channels.
It's been fantastic.
The only thing I'm not a huge fan of is the (IMO) poor implementation of threaded discussions.
Edit: it definitely has issues with connectivity from time-to-time too, but not bad overall.
TBH, I'm not sure why companies use Slack (I use it for other organizations, so have experience with it too, but not extensive).
ITT: Anecdotes
That being said, individual instances of the app are notoriously unstable causing random annoyances. But, I am on a very early build of Teams, which is buggy by definition.
https://t.me/durov/142
I wonder if it's an AWS region issue
- Jan 4, 10:14 AM EST
The status for messaging and connection services has been marked as [incident]
https://status.slack.com/
> We're continuing to investigate connection issues for customers, and have upgraded the incident on our side to reflect an outage in service. All hands are on deck on our end to further investigate. We'll be back in a half hour to keep you posted.
- Jan 4, 11:20 AM EST
> There are no changes to report as of yet. We're still all hands on deck and continuing to dig in on our side. We'll continue to share updates every 30 minutes until the incident has been downgraded
- Jan 4, 11:52 AM EST
EDIT: And down goes Notion, too: https://news.ycombinator.com/item?id=25634159
What does this mean? What do cloud providers do when customers scale down their services? Do the providers literally power down servers? Do they sell the capacity to new customers?
I don't know if they power down some servers if usage stays low for a very long time.
See for example, Amazon Prime day:
https://www.cnbc.com/2018/07/19/amazon-internal-documents-wh...
Just a theory though.
Also, Slack has significantly more users in the US than in any other country[1], and it really isn't even close. So the offense you're taking is unwarranted anyway.
1: https://saasscout.com/statistics/slack-stats/
See page 12 of the document (which is page 14 of the PDF) https://d18rn0p25nwr6d.cloudfront.net/CIK-0001764925/70df834...
- Revenue is not same as users. Slack have tons of free users and some countries also has lower priced plans.
- Many companies like Amazon etc. probably is counted as US revenue for Slack but they have more than 30% of their employees outside the US. This should not be huge numbers but significant.
looks through Inbox of 850 new aws, batch job and logging messages
oh yea, that's right..
"Hey, our site has been down for 2 hours, why aren't you doing anything"
Looks at 850 unread messages in ops-notifications folder
ooh yeah, that's right..
In my organization it's spelt "deleted items"
Oh wait, how would you share the link...
I lost the login for our shared AWS account. Mind sending it to me here?
Seems to be working intermittently, however.
[0] - https://twitter.com/SlackHQ/status/1346132040249470979
But also, the status page still proudly proclaims that the "Uptime for the current quarter: 100%" — which is clearly false at this point.
My bet is that this incident is caused by a big release after a post-holiday "code freeze".
Doubt anyone releasing big changes Monday morning.
Guess it was Slack being Slack.
> Doubt anyone releasing big changes Monday morning.
This is definitely an engineering best practice, and by best practice, I mean something that Uber's, I mean Slack's SRE team strongly pushed for, and got politely overruled on. After a code freeze is lifted, it's quite common for lots of promotion-eager engineers to release big changes.
EDIT: Guys it was a joke, chill
Releasing 1 change a year with a 100% chance of working -- no promotion for 10 years
Releasing 10 changes a year each with a 10% chance of breaking something -- 1 in 3 chance of promotion in a year, and a 2 in 3 chance of downtime
Where did that assumption come from? Also are you claiming that it takes 10x more time to release a non-breaking change?
I don't see any need to deploy a big change at once in the software world today. At worst feature gate the thing you want to do and run it in a beta environment, but still push the actual code down the pipeline.
Every Uber/ex-Uber engineer is nervously chuckling at this comment right now
Did you mean that literally? E.g. is it common at Uber that engineers can release changes to production on their own?
The engineering team is responsible for the mess caused by a bad deploy, so it's appropriate that those engineers should also choose the timing.
Our team typically deploys between 10am and 4ish, local time, since that's when we're at our desks and ready to click through the approvals and monitor the changes as they go through our pipelines.
The feature enablement happens through an EFT / beta process, and the final timing of GA enablement is a PM decision. But features are widely used by customers ahead of that time, as part of the rollout process.
Our team usually rolls out non-feature changes to services via dynamic configuration switches, so that we can get new bits in place, and then enable new behavior without a redeploy. This also enables us to roll back the dynamic config quickly if something unexpected happens.
(We generally don't do this for net new functionality; there's lower risk in adding a new REST endpoint etc. than in changing an existing query's behavior or implementation.)
Although, we also don’t close the pipeline for just any holiday break. In fact low holiday traffic is a good time to keep pipelines open, since changes will impact less people.
That sounds pretty early to think somebody on the west coast did something, other than maybe acknowledge the pages and declare the incident.
- high risk changes that weren't released pre-holidays get released. Depending on the company, this could mean a 1-week to 1-month delay between implementation and release. The greater that interval, the higher the divergence world of production and the world of the new feature
- lots of new hires (new year = new hiring budget). New hires are missing some tribal knowledge about the system and make a production-breaking release.
I tried to think of other reasons, but these two overwhelmingly stand out as the two biggest reasons. Would love to hear from others.
Hate on the cloud all you want, but AWS has (several flavors of) load balancers and various ways to automatically scale up and down resources (and if you're conservative, you can disable the 'down' part). If you're operating a major SaaS company like Slack and not taking advantage of them, something's gone wrong.
In 2021, how does one keep track of resource starvation at the process, container, os, service, pod, cluster, availability zone and region levels?
January is busy for recruiting, but given a week or two of interviewing and negotiating, two weeks notice, it's probably February before new employees are starting, and they're not making big, production-damaging deploys for a week or two after that.
Probably not as big of a rush as the end of school year rush in summer though.
I also doubt that new people will be breaking production on day one. Even at a fast moving startup I'd expect it to take a bit to go through the onboarding paperwork, get a laptop and actually try pushing a change to production.
People came back to work, and most of them start around the same time (US wise at least).
Hence kids - a vital lesson for all of us - don't start the call at a full hour, give it 3-7 min to make your coworkers confused and give some time for the systems to auto-scale ;)
Likewise, incognito mode will also ignore most cached web content, meaning all assets on the Slack web app will get loaded again from scratch. This "clean state" start could, theoretically, get around issues with old - potentially incorrect/outdated - assets being loaded, even though that really shouldn't happen under most circumstances.
What it could be is some engineer somewhere coming in after the holiday, noticing a slightly flaky thing, and thinking, "I'll reboot/redeploy/refresh this thing so the flakiness doesn't get worse". Only it turns out the flaky thing was a signal of something else falling over. Or maybe the redeploy was the wrong version because of bad CI/CD, or maybe the person just fat-fingered it.
At least that how it worked at one FAANG
(I'm not making a statement if that's good or bad or if it works or whatever. Please don't read an opinion into it.)
Now, caveats etc, this was a collection of single applications in a big microservices architecture, and as the project grows it becomes more and more difficult to manage something like this, especially if you get more pull requests in the time it takes to do a build. But it is the way to go, I think.
Anyway, since tests and CI are not definitive, you also need a gradual rollout - 1%, 5%, etc - AND you need a similar process for any infrastructure change, which gets more and more tricky as you go down to the hardware level.
URL: https://symmetric-crypto-chat-room.herokuapp.com/
Repo: https://github.com/amir734jj/SymmetricCryptoChatRoom