I’ve recently taken an interest in data analytics and have been using daily volumes of interruptions I receive via slack (and other sources) as a data set to learn myself on and try to understand how I can be more productive individually.
Today’s disruption should provide an interesting data point all of its own.
Do you work for yourself or someone else? If the latter, why do you spend any effort maximizing your efficiency? This is a pattern that I see often on HN. People take an employer's problem (they are wasting the time they've purchased with distractions) and making it their personal problem.
Some people try to approach an employment relationship as one of good faith, rather than an adversarial one where you try to do the minimum amount of work without getting fired. Plus, investing in your craft has long term benefits for your career and skill set.
I don't think there's anything adversarial about simply doing what you're asked to the best of your ability. I might occasionally drop a tidbit of advice about how I could be used more effectively, but ultimately that's not my decision. Just because I perceive it to be wasteful doesn't mean it is to the person signing the checks.
Regarding long term investment. For most people at most companies long term (more than 2-3 years) investment means moving to a better job somewhere else. Investing in your future is figuring out how to get the next job, not maximizing one aspect of your current job that your employer doesn't even seem concerned about.
I'm seeing slow performance switching/joining channels, and it's not clear whether DMs or @mentions are working consistently. Messages I send are frequently coming back as 'could not send message'.
I'm not delighted by their use of might/may language and lack of detail in the status messages. Slack is pretty clearly not working well for a large number of users, and based on the above IDK what can be attributed to degradation vs. people being unresponsive.
Well, in their defence, you can't really know how much the uptime is going to be affected by until the incident is resolved.
If it was a live counter then you'd see the uptime go up as the month progresses!
I don't have a problem with it if they update the uptime stats afterwards, but why couldn't it be implemented "live"?
Basically time "down" / total time elapsed during $period. At the beginning of the month/quarter/period you'd be at 100% (if they are up at that moment). Or even simpler, just have it be a measurement of a rolling window X days in the past.
Unfortunately true. About a year ago, I had reported a pod-specific issue to Zendesk (I'm fuzzy on the details). They confirmed the issue but also acknowledged that it would not reflect on status.zendesk.com.
Slack is unusable, but it did get me to restart Docker which had ballooned to 16 GB memory and a bazillion processes. Modern software development is something else
I was checking with top, but it's the same value. A double restart of the docker engine + changing the memory settings seems to have it holding. It's like an upgrade just ignored the previous setting.
waiting the 10 seconds it takes to build a javascript frontend these days always leaves me reminiscing about my highschool days of editing my javascript in notepad and refreshing my browser to instantly see my mistakes.
I understand there are benefits and reasons for the state of affairs...
I don't even have to refresh the browser to see my changes! (I use clojurescript with shadow-cljs, but similar tools with incremental rebuild and live reload exist for javascript)
if its taking 10 seconds to rebuild then I doubt the bundler is the problem, I get near instant updates on changes with a fairly vanilla react/webpack setup
> but it did get me to restart Docker which had ballooned to 16 GB memory and a bazillion processes.
Hey, so in 2020 we have unlimited sandboxed environments that do not require virtualization and can be spun up, destroyed very quickly, and their construction is fully automated. And it's somehow a bad thing?
Get off my lawn! When Docker arrived it was a godsend, given that we were trying to do similar things directly with LXC.
(I'm fully expecting someone to chime in saying, 'you had LXC? All we had was ones and zeros, and we had ran out of zeros')
This backend outage reveals how horrible the Slack UI is implemented. You post a message.... you think it got sent. Then 1min later, it tells you it was not.
Broadly speaking maybe, but situationally not really. The likelihood that the issue I'm experiencing is a service wide issue impacting many others is usually low probability.
The message is in a light grey colour until it's confirmed. I'm not saying it's amazingly obvious but at least the information's there. In fact I'm mostly posting this comment so others know what to look for.
(I know this because I just spent 60 seconnds staring at my message trying to figure out why the formatting was off before it told me it wasn't sent. It sent immediately when I retried though.)
I had sequence of six messages that were not all delivered, and once they were delivered (because I clicked on each one of them) they were re-ordered, altering the my whole part of the conversation.
> The message is in a light grey colour until it's confirmed.
This morning alone, I've had multiple messages start off black, then turn grey after about 10 seconds. And that's not just true today, but other time Slack has been down as well.
"If it's black, it's confirmed delivered" is not an invariant.
You can tell it was built for Silicon Valley. The marginally lighter gray tone on unsent messages is only noticeable by design-aware people on fancy IPS monitors. Messages are reordered semi-randomly.
> slacks operational transform or CRDT implementation or whatever they use to get edited messages to converge
I'm pretty sure they have none of that, considering you can only edit your own messages. So what's likely happening is that two quick edits are reaching the server at the same time and overriding each other.
I could be wrong but i'm relatively sure that I reach message states that do not directly correspond to anything i typed into the editor -- which i wouldn't expect if the edited state was always sent in full rather than as a delta on a previous message state.
I suppose it's possible that the corruption happens on the client side tho --
1. initiate edit action and get editable view of snapshot of text
2. server acks some previous edit request causing some client state related to the message to update and promote a previous edit in some way
3. hit save after finishing editing -- and some delta operation on the client side updates a different message state than the one i initiated edit action from ...
This is just super unlikely as you describe it. There's no reason for deltas and fancy mechanisms when you can only edit your own messages. It would be a waste of resources to do it like this for short messages.
I don't know the specific implementation mechanism -- just guesses.
But that doesn't mean my observation of the behavior of the bug is unlikely.
And also -- there is not a hard requirement that only one editor exists -- the same user could use multiple devices at the same time. Consider that and you might find your assumptions to be invalid.
The RGBA value for the "unsent" text is #1d1c1db3. Normal "sent" (or "attempting to send but haven't yet decided sending will fail" is #1d1c1d. The RGBA value (alpha = 0.7) ends up being equivalent to #616161 when rendered over white background which the Slack default skin does.
Optimistic design is good, but you need to also tell the user when you’re being optimistic. iMessage, for example, “sends” a message, but it will also tell you when it’s been delivered so you can check to make sure.
Funny story—iMessage on one of my laptops used to very frequently tell me "This message could not be delivered" even though the person on the other end received it fine. (The problem went away after an OS reinstall.)
It's pretty necessary to inform users when messages haven't sent though, especially when these little failure indicators are hidden in the channels themselves - i.e you need to be looking at the message to know it failed.
If I post a message in a channel and go back to work, I don't want to find out later when I check for replies that my message didn't actually send - I need a notification!
I worked on a project that used this, I hated it. The excuse I was given repeatedly was "it's unlikely to fail" which I kept responding with "but when it does, the user is already 3 pages away and we have no good way to tell them so they're SOL"
The user might have navigated 3 pages away to a different website ... Or exited a mobile app. (But well yes I suppose notifications are better than nothing)
I somewhat often send messages just before I have to run to catch the subway — and when I think the message got sent, I type `sudo poweroff`
I didn’t know it was that controversial. I feel like that article is making a bit of strawman argument from the get go by talking about 1-3% failure rate from the backend. If I had a system that flaky, I’d be taking a good hard look at myself before blaming anything on my optimistic UI :-)
There are other things to consider too. Like for example network failures vs backend failures. In the case of network failures you can actually notify users before they even try to take an action. So now you’re down to some diminishing case where your code is effectively wrong (maybe the client is too old), and again there are more elegant ways of handling that. Also, I bet you have the response back really quickly to make a call on it.
Having a worked on unstable networks I can say that there are parts of slack’s implementation that suck - like if you try to change a message that hasn’t synced yet, they get into broken states with the edit. But that’s not the fault of optimistic UI, it’s other technical choices at play.
I know you haven’t named a horse in this race, but I feel people hating on Slack for having an optimistic UI are a bit off base. Considering that I’ve sent (apparently) 2.5k slack messages in the last 30 days and I don’t recall any going wrong, I think they’ve got it right.
> I didn’t know it was that controversial. I feel like that article is making a bit of strawman argument from the get go by talking about 1-3% failure rate from the backend. If I had a system that flaky, I’d be taking a good hard look at myself before blaming anything on my optimistic UI :-)
It's not a 1-3% failure rate for the backend, it's a 0.01-0.1% failure rate for each request between dozens of microservices each with their own bespoke error handling, each owned by a team with more political clout than the frontend guys/gals who keep reporting Sentry errors due to "UnknownNetworkError" or some other such nonsense, all wrapped in some spaghetti code frontend thats written like the whole thing is running locally.
On Slack it's only ever been an annoyance for me, because at least they visibly tell you when the message eventually fails to send. But on services like Twitter I feel gaslit when I've liked a tweet and then I scroll by it again later, un-liked. It undercuts my faith in the integrity of the entire interface. Any part of it could be lying to me in a way I may never find out about.
Overall I strongly dislike this UX pattern. Users should have immediate feedback about actions that haven't completed, but that feedback should only tell them the truth: that the operation has begun.
The problem with failures and optimistic UI isn't that failures happen often - it's that they're not uniformly distributed. I do not have a 1% or 0.0001% chance of experiencing failure any given message. I have 0.0001% chance of experiencing failure, unless I'm currently in a tunnel, or my ISP breaks something again, or [insert dozen other infrequent client-side failure modes], at which point my failure rate briefly hits 100%, and I'd very much appreciate if the UI didn't lie to me about it.
That’s what I was saying above, you can detect those scenarios a lot of the time and handle them reasonably. Oftentimes they’re temporary and will resolve without losing any data (in a well designed system). Like I say, you can even head those off before the user knows about them.
There are trade offs to consider in UX. You’re trying to make things as seamless as possible for the end user without distracting them with things they don’t care about.
I understand why they use this UI paradigm, but I don't understand why it's not reversed eg make it light grey until the API has confirmed it's been posted. That would maintain the nice 'you posted a message nice and quickly' responsive UI idea, but without making the user angry when it doesn't work.
Even if the idea is to fool the user in to thinking Slack is faster than it is, the UI could at least use the reversed 'light grey until confirmed' UI once it's detected that there's a problem. That would make it a bit less tiresome.
This morning I sent a message and it was greyed out until finally sending a minute later. So it would seem both cases are possible somehow. Different stages of data propagation?
Ripcord[1] does what you describe. My guess is that Slack doesn't do this because it makes Slack's latency extremely obvious. Even on a great connection, a decent fraction of messages will take multiple seconds to go from gray to black. Apparently even when Slack's API is operating normally, it can take a very long time to respond with, "Your message was received."
I was surprised that I couldn't even open up a DM conversation without it blocking on the network - the Ctrl-T/Ctrl-K quick-find just got stuck when I tried clicking on the person's name.
At which point there is no option to cancel and say "you know what... don't send it".
If you notice it 24 hours later, your choices are (1) be nagged about retrying it forever or (2) retry sending a message that may now be irrelevant.
When it comes to retries, the easiest thing for the user is for the software to be responsible for retrying failed actions. Choosing not to do that makes sense IF the goal is to give the user more control because they may not necessarily still want the action done.
But if you don't give them that control, it's the worst of both worlds. The retry happens slower, with more manual work, and without the benefit of choice.
I have a DM on mobile with a failed slack message in it that has clogged my UI for months; literally months.
On mobile it means that I can't tell if this user has responded to a message or is online at all. I tried telling it not to send ages ago when it actually failed but it hasn't worked, and frankly now it's so far back in history it's impossible for me to try again.
I've deleted slack and re-installed to no avail. It's really an awful problem.
The mismatch between expectation and clearly visible feedback is in the timing. It would be fine if there was a 'still sending' type message after 2 seconds. If this comes up too frequently then the connection is not as reliable as the UI is pretending it to be.
The concept is called 'optimistic updating' and the intent is to make the UI 'feel faster' by assuming a positive response from the back-end, and only reverting when things go awry. Given the vast majority of messaging attempts will complete successfully, it's easy to see why the interaction pattern is used.
The converse is pessimistic rendering: waiting for the server response before updating the UI (and possibly showing a loading spinner in the interim).
Facebook Messenger has the best UX paradigm for this I've seen, since it's most graceful in handling the edge case failures while still presenting as optimistic. Your message immediately appears as a conversation bubble, with an icon displaying if the server has received it, an 'X' if it fails (clicking the icon gives you options: re-send, delete, etc), and a read receipt for when the other party views the message. It's a fantastic way of handling the lifecycle.
Slack fails because it doesn't communicate that pending, the-server-hasn't-received-it-yet status, and the fallback isn't as graceful.
The new Slack on mobile is absolutely unusable. There are no way to know if the interface is up to date, when was the last update or to force an update.
The only way I found to know that your message has been received is to have an ack from your interlocutor.
My personal conspiracy theory is that Slack likes the way users "engage" with the mobile app when indicator badges don't clear right away. Often you will open the app because you see the badge, only to realize you've already read the message and the badge just hasn't cleared. They're hoping that you will then click something unrelated in the app, thus increasing "engagement".
To be wildly pedantic, pessimistic rendering would then be immediately showing an error, only to later clear the error away if the request is successful.
I got thrown recently by an update log (powerapps solutions) which seems to do exactly this - an in-flight install/update is shown as a failure until it completes. Eventually figured out I should ignore entries with a null completion time.
It depends on your frame of reference. If the message succeeds then there will be a window (all else equal and no funny business in the networking, around 1/2 round trip time) where the UI displays the message as spinning when it has in fact been successfully sent. If your frame of reference is not whether the message has been successfully sent but whether you know for certain that it has been successfully sent then I think you'd be right to call it "realistic rendering", but from the message's frame of reference "pessimistic" isn't a bad choice of words.
The terminology is occasionally used with a more technical meaning in algorithms, so perhaps that's how it's being used here.
"Optimistic" often means "assume X, perform or calculate something, change the result if X turns out to be false".
"Pessimistic" often means "assume not-X, perform or calculate something, maybe change the result if X turns out to be true".
This comes up in other domains.
For example in compilers (and reverse engineering decompilers ;-) there are optimistic and pessimistic analyses.
An optimistic analysis might initially assume a particular variable has a specific value at a particular location, or a branch is always taken. Then trace through the program paths, using that assumption (which limits those paths). If it loops back to the location with a different value in the variable or different value for the branch condition, broaden the assumption and update the trace.
A pessimistic analysis might initially assume the variable at a particular location could have any value because it hasn't yet traced everywhere to find out the possibilities, or initially assume both directions at the branch could happen and include them both in the trace. By the time it finishes tracing all paths and values, those assumptions may turn out pessimistic and it can start pruning away unused options.
These two strategies can yield different answers, and for some analyses the optimistic strategy gives more accurate results than the pessimistic strategy.
> Facebook Messenger has the best UX paradigm for this I've seen
FYI, Signal has used this paradigm for quite some time (I'm not sure which software predates the other on this feature), it's two checkmarks which remind me of the old "open Apple, closed Apple" keyboard keys - message bubble is shown with two open checkmarks right away, a single closed checkmark means the server has received, the second closed checkmark means the other end has received (assuming read receipts are enabled, Signal allows them to be turned off for privacy reasons).
> The concept is called 'optimistic updating' and the intent is to make the UI 'feel faster' by assuming a positive response from the back-end, and only reverting when things go awry. Given the vast majority of messaging attempts will complete successfully, it's easy to see why the interaction pattern is used.
It's easier than making the UI and server faster for sure, I'm just not sure why engineers don't insist of fixing things the right way.
>>It's easier than making the UI and server faster for sure
In majority of cases, I assume network latency will be the dominant factor, and that is outside the control of the engineers.
I happen to use slack from hardwired gigabit ethernet to a gigabit fiber internet connection. But I maintain some awareness that some others may use it from their older android phone with 2 bars of service, or on a laptop with 27 open blasting wifi spots, etc etc etc.
I will therefore put forward that assumption you can ensure instant communication, is the wrong wrong wrong way for engineers to write code...
Optimistic updating is when the message moves from the chat box into the conversion thread. It assumes that in most cases the message will be sent and it makes sense to update the UI. Usually though it’s accompanied by an indication of some kind to also show progress. If slack doesn’t indicate it in anyway that’s confusing.
I wonder if this is why slack for iOS (not sure about Android) will load the notification badge from the server before loading the actual messages, so when those get out of sync the UI is essentially saying "Someone is trying to contact you but that's all I can tell you".
> Facebook Messenger has the best UX paradigm for this I've seen
Viber had a similar thing long before FB's messenger: single gray check mark character if the message is sent, double gray check mark if it is received by the recipient and double check mark in purple if it is seen.
The only issue is with the purple color of the "seen" as it can sometimes blend into an image behind it (if the message was consisting of an image).
Could someone comment on why exactly this a poor design for their backend? Genuinely curious, I don't have any real world context on systems like this.
IMHO, at scale SQL will breakdown. Even with sharding like Slack is able to per organization. It's why we have great things like Cassandra and DynamoDB. They're designed to solve replication in an easier way than replicating RDBMS iff you know your data access patterns in advance and they're not ad-hoc (which SQL is great at). This is the case for Slack. The typical way to solve RDBMS bottlenecks is to put a queue and messaging system in front of them. This breaks down when your services have bugs (my guess at what's happening).
is pretty good on why some NoSQL approaches are a step forward (perhaps not MongoDB at scale if consistency is necessary https://jepsen.io/analyses). In particular though:
There could be other issues about why Slack is slow. But at Slack scale, you need to be extremely heightened in your database strategy or you should follow the industry and use Cassandra/DynamoDB's built in partition tolerance. Key value stores scale horizontally much easier. B-trees don't scale as easily horizontally past a certain point.
Essentially, good NoSQL DBs have abstracted scale for you (so you don't have to think about it as much). But you have to know the access patterns in advance (the types of queries and updates you'll be running for most use cases), since you need to design your table around these access patterns. RDBMS leaks scaling from the abstraction (you need to use message queues, etc.).
It’s proven itself for scaling. Mostly startups don’t see the issues with SQL and don’t need to worry about it. At planet/top-Alexa-ranked-website scale though, you either use Spanner at Google, or use Cassandra at Apple, DynamoDB at Amazon, Cassandra at Instagram, parts of Facebook, Netflix, Manhattan at Twitter etc.
You keep using MySQL at Github and Slack if you want periodic downtime/degradation in service tho.
The core of Facebook, Twitter, YouTube, LinkedIn, all of Microsoft and lots more all run on MySQL and other similar RDBMS servers. It's a myth that you cannot scale MySQL.
I'm not sure about the others (e.g., there's nothing recent for YouTube I could find.. likely they'd use Spanner though for things like comments?).
I think you need a really talented database infra team if you're trying to use RDBMS for something like a real-time messaging store at scale. I can't say more about Amazon. But I just don't think it makes sense for these use cases to use RDBMS where real-time messaging (pull requests, comments, etc. for Github - messages, posts for Slack) is the 90% use case.
UPDATE. Maybe I'm wrong can you can use: https://vitess.io/ for horizontally scaling MySQL. I don't know enough about the details of it. But getting the data store right is so important to the overall backend's stability (I think it's no coincidence that Twitter stability became "solved" when they moved to something like Manhattan). And I just don't see why you wouldn't rewrite things using something that logically makes a lot more sense instead of trying to push connection pooling, query rewriting, etc. to the limit. They don't fundamentally solve what consistent hashing solves.
Discord understands to use this too.[1] Not to say you can't have your user database in SQL like Facebook, etc. But for messaging? And really for anything with high throughput / low latency, where you know the access patterns, just doesn't make sense to not use something with consistent hashing.
My original point - why I raised this to begin with - is I briefly browsed that Slack CTO's video. At one point he mentioned using RDBMS "because that's what we're experienced with." That's never a logical reason. It may be a practical one. But with time... it just doesn't stand up to ideas that are better and have proven themselves (e.g., consistent hashing). But again, using MongoDB the wrong way or assuming the "document store" is the main reason for using NoSQL can confuse people (it's one nice benefit for ad-hoc data models! but the big innovation in NoSQL is consistent hashing for the low-latency / high throughput use cases). And SQL has its benefits for certain use cases. But there's a better solve for the messaging storage at scale. I post this because: (a) I'm interested in others' opinions and feedback about how they've made RDBMS work (thanks) (b) tired of Github and Slack being down periodically, and for each mature SaaS to go through this learni...
Don't have much to add, but want to say thanks for sharing those blog posts, they were interesting reads.
The slack CTO's comment about choosing RDBMS because 'familiarity' is interesting. IMO it's a gamble. I've seen it happen with my company when being a latecomer to containerization.
When it came to picking a container management tool, it was a tossup between k8s, Nomad, or just saying to hell with and running those containers ourselves on EC2 instances. Having run our stack on bare metal for year = we were really pretty good at it. There was a surprising amount of automation that could be ported over.
Eventually we picked k8s, and coincidentally, our usage grew more in 6 months than it had in the last 2ish years. So all in all, the gamble paid off.
... but I like to think there's another world where we picked the 'its familiar option' and things still worked out. If our traffic hadn't grown the way it did, we would never have felt the pain of having to manually scale out our systems - or basically write an in-house version of Kubernetes.
So in that sense, I'd guess that maybe some teams have the bad habit of playing the same side of the coin everytime. It may be prudent to stay conservative when picking a Datastore, maybe it's would be smart to pick a risky technology for your app servers? (and vice-versa)
Teams is a nightmare. Instead of just creating the simple chat that everyone was just asking for, they went full PhD and created a mix between a chat and a forum.
A Teams "channel" has counter intuitive behavior, the messages you send are more like forum threads, users can reply to them, and when that happens it reorders the whole messages to put that one at the bottom. It makes it impossible to "catch up" on a channel since people will happily reply to any messages so the ordering is completely random.
There is also a completely counter intuitive distinction between "groups chats" and "channels".
Teams is also the only chat service I know where there is an actual limit on the number of non-public channels you can have. After 30 channels have been created, you're doomed.
Want to remove an existing channel to make room for a new one? Nope, you cannot remove a channel, only "archive" them. Archived channels are removed after 30 days and there is no way to force their removal early. So basically if you reach 30 channels you have to wait minimum 30 days to create a new one.
Did I mention the UI is slow as hell, and half the screen space is used by unnecessary large padding between messages? I am rarely able to have more than 3/4 messages displayed on a 23" screen.
The only positive point of Teams is integration with outlook 365 and meetings. It works really well, quality is good, reliable, nice UI, présentation mode, screen share, etc
> The video chat is really good from experience though.
I have had a few people on calls with poor connection speeds. It degrades horribly by stuttering and distorting audio. It performs worse than a phone call. The best fix would be for Australia to fix its woeful internet.
When you load the web interface it identifies each person with their initials, not full name. This isn’t helpful when you have never met the person before.
Their status page is annoying me more and more with these recent outages.
Recalling the incidents in September, it's a bit cheeky they call that "99.93%" uptime (30 mins of downtime).
Having remembered that there were at _least_ 2 seperate 1 hour+ incidents where I was struggling to send messages in Sept, I went through to look at the history, and they seem to have purposefully hidden these incidents on the calendar. There's no way of listing incidents in a month.
The information and accessibility of the status page is obviously politically motivated by marketing.
If companies were honest, they would multiply downtime by the number of users at the time. Because of course, great uptime doesn't matter if it's only uptime when no one is using it in the middle of the night.
It's like when trains report 95% on time, except that the times when the service is delayed is when the trains are full. All the delays that actual people experience (person-hours) happen in that 5%.
Its interesting, I looked at that page earlier (~1145 EST) and it was actually tagged as an "Outage" rather than the "Incident" it is right now.
For me it has only been slow performance all morning. Its not good, but not what I would call "down" either. At least from my perspective. I could have a bunch of other folks who are completely unable to send messages and I just can't tell.
I have intermittently faced each of these issues over the last hour across several European and a US Slack:
* Messages are shown as sent (not grey), then, after several minutes, a "Try to send again" message appears. Or the message turns grey. Or both.
* The webapp randomly reloads. Sometimes already typed messages are preserved, sometimes they disappear. ( I assume a reload happens when a new version is deployed with some "force reload" flag )
* Threads and channels just show a loading spinner forever.
* After switching channels, all fonts disappear and just the menu background remains. No remedy except reload.
Outages happen. They get resolved. That's live, and it's fine.
What is not fine is how ill-equipped the frontend is to deal with such failures.
I used enterprise slack ~4 years ago for a job and I remember having these issues every so often. ~2 years ago I used free gitter for a university project and I had the same issues a lot, and we had to switch -- funnily enough, we switched to discord and it was better than all of them.
My only real complaint with Discord is how it treats audio (and maybe video?) links when you close and open the application. Don't make the mistake of shutting off your computer while connected to a voice chat as it will automatically resume the connection when it starts back up. Mute status being the same as it was when shut down if I recall.
> The webapp randomly reloads. Sometimes already typed messages are preserved, sometimes they disappear. ( I assume a reload happens when a new version is deployed with some "force reload" flag )
I have been seeing some version of this on my laptop for months. Don't seem to have the same issue on my work computer. Maybe it will finally get resolved.
Any of the below or all of the below
(Not my own opinions, its a collection)
1. Another maintenance burden
2. You have some other company to scold or blame who is responsible for that.
3. Running it at scale.
4. Have to devote time for building features in market vs existing solutions.
5. Their UI/UX is not polished enough
6. sales bribed the manager
7. XMPP sucks for their use case.
It is a legitimate improvement to see everything that happened when you were offline. At that point you're not supporting XMPP; you're supporting "XMPP with some pixie dust to save that and let it be viewed, which is in a totally different place than your live chats".
Slack is a not-great implementation of a good idea, including a use case that IRC and XMPP don't meet. Tulip is a better implementation of that good idea, plus yet another good idea.
Example: We pay for looker. One day I thought to myself I'll spin up Metabase and try it out for myself against our prod database. It worked very well, fast and easy to query my data unlike Looker.
I haven't used it in about a month and today I try to access it but the app just doesn't load. Now I need to spend time to look into why this app isn't loading. Nevermind that I have zero Java experience or devops experience. I just ran a Docker file per their instructions.
If everyone is bought-in on using them, they can be a really nice way to keep single topics in one place rather than having interleaved conversations in a single channel. If people don't use it 100% of the time, though, it's honestly worse because now you're not always sure where to read/continue a conversation.
Wow really? I'm genuinely surprised. I couldn't imagine using Slack without threads, how do you keep discussions on track for certain topics? At any given time I have 30-40 threads going, which means I can easily track conversations. Without threads I'm just endlessly scrolling, or trying to find something, or god forbid relying on Slack's utterly broken search.
I would say they help to scope a part of a discussion that clarifies some detail or maybe when giving a link elsewhere, without derailing the mainline of the discussion.
It's useful when people arrive in the discussion at different times and you don't want to necessarily let the whole channel to be notified about your specific remark, and don't want to write a private message either -- one reason being it would lose the context.
In our project we have multiple channels that have actively 3-5 persons and work at different times. The things we chat about are lightweight changes or news, or maybe someone has a quick clarifying question about implementation detail. Something that wouldn't be posted in an issue for one reason or other.
FWIW, I was also a threads nay-sayer but I've since seen the light
Remember the irc support channels where conversations and problem/solution dialogs were constantly muxed with other conversations / dialogs?
Imagine each question being a root message, and all of the dialog / suggestions / answers being in a thread.
Also makes it super convenient for search.
In addition to that I find that beyond a certain channel / DM count it's a lot easier to follow up on things with threads
Give it a shot, try to get your whole team on board. You can always revert back to not using them later :)
yes! I actually love that format! it's how I grew up!
I'll recognize that I'm a bit of an oddball in that respect nowadays, but it can work, at least with a small enough room. I love having all the conversations cross over one another and just being able to read back and follow them.
Yeah, you definitely are, especially for larger companies. Just from one example: you can tell when a specific issue requires attention based on how many comments the thread has.
So let's say I wake up in the AM and there's a thread that says "how do I use the API for companyservice.example-api.com" and the thread has 50 comments.
Let's say it's a service I support. First, I know there's an issue because a customer required attention to begin with. This is then confirmed by the fact that the thread has 50 comments in it so there are probably documentation and usability issues with this service.
The company I work for and the services I am responsible for deals with internal customers and we use these two absurdly simple metrics to optimize support regularly.
When we first started with certain services that my team is responsible for, we used Slack metrics A LOT. As in, we had spreadsheets with this data and we'd go over them every week. Nowadays, our services are almost 100% self-service and we rarely, if ever, get pinged on Slack for those services because the documentation is optimized and the usability has been specifically designed so that our engineers don't get in the way of our customer's needs.
If Discord wades into the business communication space, I would move without a second thought. The voice/call reliability seems better, anecdotally. I think the voice lounge concept could translate well into the enterprise space as well, if done right.
If you're question is it a viable alternative to Slack for business purposes, the answer is no. And it's not bc of anything related to the application itself but because of the insane TOS.
> By uploading, distributing, transmitting or otherwise using Your Content with the Service, you grant to us a perpetual, nonexclusive, transferable, royalty-free, sublicensable, and worldwide license to use, host, reproduce, modify, adapt, publish, translate, create derivative works from, distribute, perform, and display Your Content in connection with operating and providing the Service.
we have fallen back to discord for work today. Obviously the terms/security controls are not a great permanent solution, but it's a great fallback for real-time chat across all devices
We have Discord as a backup for our engineering team. And Slack's UI needs to be fixed where if a message appears in the stream, it means the server persisted it.
224 comments
[ 3.0 ms ] story [ 220 ms ] threadToday’s disruption should provide an interesting data point all of its own.
Regarding long term investment. For most people at most companies long term (more than 2-3 years) investment means moving to a better job somewhere else. Investing in your future is figuring out how to get the next job, not maximizing one aspect of your current job that your employer doesn't even seem concerned about.
I'm not delighted by their use of might/may language and lack of detail in the status messages. Slack is pretty clearly not working well for a large number of users, and based on the above IDK what can be attributed to degradation vs. people being unresponsive.
Also check their status page in a month or so, and you'll see "October 5, 2020" with some message like "no issues".
Basically time "down" / total time elapsed during $period. At the beginning of the month/quarter/period you'd be at 100% (if they are up at that moment). Or even simpler, just have it be a measurement of a rolling window X days in the past.
I've questioned status pages since.
I understand there are benefits and reasons for the state of affairs...
Between that and using vscode remote SSH onto my server environment, it’s almost like the old days of save-refresh, but with auto refresh!
We have build times in the dozens of minutes thanks to a decision in the past to use Dart.
Hey, so in 2020 we have unlimited sandboxed environments that do not require virtualization and can be spun up, destroyed very quickly, and their construction is fully automated. And it's somehow a bad thing?
Get off my lawn! When Docker arrived it was a godsend, given that we were trying to do similar things directly with LXC.
(I'm fully expecting someone to chime in saying, 'you had LXC? All we had was ones and zeros, and we had ran out of zeros')
(I know this because I just spent 60 seconnds staring at my message trying to figure out why the formatting was off before it told me it wasn't sent. It sent immediately when I retried though.)
This morning alone, I've had multiple messages start off black, then turn grey after about 10 seconds. And that's not just true today, but other time Slack has been down as well.
"If it's black, it's confirmed delivered" is not an invariant.
Recently i've realized that this actually introduces more typos -- where a large number of phrases are duplicated or corrected typos reapplied.
There's something wrong with slacks operational transform or CRDT implementation or whatever they use to get edited messages to converge
I'm pretty sure they have none of that, considering you can only edit your own messages. So what's likely happening is that two quick edits are reaching the server at the same time and overriding each other.
I suppose it's possible that the corruption happens on the client side tho --
1. initiate edit action and get editable view of snapshot of text
2. server acks some previous edit request causing some client state related to the message to update and promote a previous edit in some way
3. hit save after finishing editing -- and some delta operation on the client side updates a different message state than the one i initiated edit action from ...
But that doesn't mean my observation of the behavior of the bug is unlikely.
And also -- there is not a hard requirement that only one editor exists -- the same user could use multiple devices at the same time. Consider that and you might find your assumptions to be invalid.
(Where a message sent doesn't match what was typed sometimes.)
I don't mean to disparage at all, but IPS monitors are far from universal.
The RGBA value for the "unsent" text is #1d1c1db3. Normal "sent" (or "attempting to send but haven't yet decided sending will fail" is #1d1c1d. The RGBA value (alpha = 0.7) ends up being equivalent to #616161 when rendered over white background which the Slack default skin does.
Demo for reference: https://jsfiddle.net/9smqe8y7/
If I post a message in a channel and go back to work, I don't want to find out later when I check for replies that my message didn't actually send - I need a notification!
I somewhat often send messages just before I have to run to catch the subway — and when I think the message got sent, I type `sudo poweroff`
There are other things to consider too. Like for example network failures vs backend failures. In the case of network failures you can actually notify users before they even try to take an action. So now you’re down to some diminishing case where your code is effectively wrong (maybe the client is too old), and again there are more elegant ways of handling that. Also, I bet you have the response back really quickly to make a call on it.
Having a worked on unstable networks I can say that there are parts of slack’s implementation that suck - like if you try to change a message that hasn’t synced yet, they get into broken states with the edit. But that’s not the fault of optimistic UI, it’s other technical choices at play.
I know you haven’t named a horse in this race, but I feel people hating on Slack for having an optimistic UI are a bit off base. Considering that I’ve sent (apparently) 2.5k slack messages in the last 30 days and I don’t recall any going wrong, I think they’ve got it right.
It's not a 1-3% failure rate for the backend, it's a 0.01-0.1% failure rate for each request between dozens of microservices each with their own bespoke error handling, each owned by a team with more political clout than the frontend guys/gals who keep reporting Sentry errors due to "UnknownNetworkError" or some other such nonsense, all wrapped in some spaghetti code frontend thats written like the whole thing is running locally.
Overall I strongly dislike this UX pattern. Users should have immediate feedback about actions that haven't completed, but that feedback should only tell them the truth: that the operation has begun.
There are trade offs to consider in UX. You’re trying to make things as seamless as possible for the end user without distracting them with things they don’t care about.
Even if the idea is to fool the user in to thinking Slack is faster than it is, the UI could at least use the reversed 'light grey until confirmed' UI once it's detected that there's a problem. That would make it a bit less tiresome.
1. https://cancel.fm/ripcord/
Just... how!?
At which point there is no option to cancel and say "you know what... don't send it".
If you notice it 24 hours later, your choices are (1) be nagged about retrying it forever or (2) retry sending a message that may now be irrelevant.
When it comes to retries, the easiest thing for the user is for the software to be responsible for retrying failed actions. Choosing not to do that makes sense IF the goal is to give the user more control because they may not necessarily still want the action done.
But if you don't give them that control, it's the worst of both worlds. The retry happens slower, with more manual work, and without the benefit of choice.
On mobile it means that I can't tell if this user has responded to a message or is online at all. I tried telling it not to send ages ago when it actually failed but it hasn't worked, and frankly now it's so far back in history it's impossible for me to try again.
I've deleted slack and re-installed to no avail. It's really an awful problem.
The converse is pessimistic rendering: waiting for the server response before updating the UI (and possibly showing a loading spinner in the interim).
Facebook Messenger has the best UX paradigm for this I've seen, since it's most graceful in handling the edge case failures while still presenting as optimistic. Your message immediately appears as a conversation bubble, with an icon displaying if the server has received it, an 'X' if it fails (clicking the icon gives you options: re-send, delete, etc), and a read receipt for when the other party views the message. It's a fantastic way of handling the lifecycle.
Slack fails because it doesn't communicate that pending, the-server-hasn't-received-it-yet status, and the fallback isn't as graceful.
You post a message, it looks good, you move on.
Then later on you come back to it and it was never sent.
The only way I found to know that your message has been received is to have an ack from your interlocutor.
I hate the phrase "I'm a realist not a pessimist" as much as the next person, but isn't this actually "realistic rendering"?
"Optimistic" often means "assume X, perform or calculate something, change the result if X turns out to be false".
"Pessimistic" often means "assume not-X, perform or calculate something, maybe change the result if X turns out to be true".
This comes up in other domains.
For example in compilers (and reverse engineering decompilers ;-) there are optimistic and pessimistic analyses.
An optimistic analysis might initially assume a particular variable has a specific value at a particular location, or a branch is always taken. Then trace through the program paths, using that assumption (which limits those paths). If it loops back to the location with a different value in the variable or different value for the branch condition, broaden the assumption and update the trace.
A pessimistic analysis might initially assume the variable at a particular location could have any value because it hasn't yet traced everywhere to find out the possibilities, or initially assume both directions at the branch could happen and include them both in the trace. By the time it finishes tracing all paths and values, those assumptions may turn out pessimistic and it can start pruning away unused options.
These two strategies can yield different answers, and for some analyses the optimistic strategy gives more accurate results than the pessimistic strategy.
So which one is more "real"? :-)
FYI, Signal has used this paradigm for quite some time (I'm not sure which software predates the other on this feature), it's two checkmarks which remind me of the old "open Apple, closed Apple" keyboard keys - message bubble is shown with two open checkmarks right away, a single closed checkmark means the server has received, the second closed checkmark means the other end has received (assuming read receipts are enabled, Signal allows them to be turned off for privacy reasons).
It's easier than making the UI and server faster for sure, I'm just not sure why engineers don't insist of fixing things the right way.
That's like saying "just fix global warming", it's more complicated than that.
Einstein figured out why in 1907, and most of us have just been going along with the assumption that superluminal communication just isn't possible.
https://en.wikipedia.org/wiki/Tachyonic_antitelephone
In majority of cases, I assume network latency will be the dominant factor, and that is outside the control of the engineers.
I happen to use slack from hardwired gigabit ethernet to a gigabit fiber internet connection. But I maintain some awareness that some others may use it from their older android phone with 2 bars of service, or on a laptop with 27 open blasting wifi spots, etc etc etc.
I will therefore put forward that assumption you can ensure instant communication, is the wrong wrong wrong way for engineers to write code...
Viber had a similar thing long before FB's messenger: single gray check mark character if the message is sent, double gray check mark if it is received by the recipient and double check mark in purple if it is seen.
The only issue is with the purple color of the "seen" as it can sometimes blend into an image behind it (if the message was consisting of an image).
https://youtu.be/WE9c9AZe-DY
https://www.youtube.com/results?search_query=aws+rick
is pretty good on why some NoSQL approaches are a step forward (perhaps not MongoDB at scale if consistency is necessary https://jepsen.io/analyses). In particular though:
https://youtu.be/hwnNbLXN4vA?t=992
There could be other issues about why Slack is slow. But at Slack scale, you need to be extremely heightened in your database strategy or you should follow the industry and use Cassandra/DynamoDB's built in partition tolerance. Key value stores scale horizontally much easier. B-trees don't scale as easily horizontally past a certain point.
Essentially, good NoSQL DBs have abstracted scale for you (so you don't have to think about it as much). But you have to know the access patterns in advance (the types of queries and updates you'll be running for most use cases), since you need to design your table around these access patterns. RDBMS leaks scaling from the abstraction (you need to use message queues, etc.).
You keep using MySQL at Github and Slack if you want periodic downtime/degradation in service tho.
I think you're mistaken about Twitter:
"Manhattan(the backend for Tweets, Direct Messages, Twitter accounts, and more)"
https://blog.twitter.com/engineering/en_us/topics/infrastruc...
I'm not sure about the others (e.g., there's nothing recent for YouTube I could find.. likely they'd use Spanner though for things like comments?).
I think you need a really talented database infra team if you're trying to use RDBMS for something like a real-time messaging store at scale. I can't say more about Amazon. But I just don't think it makes sense for these use cases to use RDBMS where real-time messaging (pull requests, comments, etc. for Github - messages, posts for Slack) is the 90% use case.
UPDATE. Maybe I'm wrong can you can use: https://vitess.io/ for horizontally scaling MySQL. I don't know enough about the details of it. But getting the data store right is so important to the overall backend's stability (I think it's no coincidence that Twitter stability became "solved" when they moved to something like Manhattan). And I just don't see why you wouldn't rewrite things using something that logically makes a lot more sense instead of trying to push connection pooling, query rewriting, etc. to the limit. They don't fundamentally solve what consistent hashing solves.
EDIT. Also, wrong about LinkedIn re: messaging:
https://engineering.linkedin.com/blog/2020/bootstrapping-our...
https://en.wikipedia.org/wiki/Voldemort_(distributed_data_st...
https://en.wikipedia.org/wiki/Consistent_hashing
Discord understands to use this too.[1] Not to say you can't have your user database in SQL like Facebook, etc. But for messaging? And really for anything with high throughput / low latency, where you know the access patterns, just doesn't make sense to not use something with consistent hashing.
But as mentioned in Rick's AWS videos about https://en.wikipedia.org/wiki/Technology_adoption_life_cycle - there's a lot of late majority, laggards, etc.
My original point - why I raised this to begin with - is I briefly browsed that Slack CTO's video. At one point he mentioned using RDBMS "because that's what we're experienced with." That's never a logical reason. It may be a practical one. But with time... it just doesn't stand up to ideas that are better and have proven themselves (e.g., consistent hashing). But again, using MongoDB the wrong way or assuming the "document store" is the main reason for using NoSQL can confuse people (it's one nice benefit for ad-hoc data models! but the big innovation in NoSQL is consistent hashing for the low-latency / high throughput use cases). And SQL has its benefits for certain use cases. But there's a better solve for the messaging storage at scale. I post this because: (a) I'm interested in others' opinions and feedback about how they've made RDBMS work (thanks) (b) tired of Github and Slack being down periodically, and for each mature SaaS to go through this learni...
The slack CTO's comment about choosing RDBMS because 'familiarity' is interesting. IMO it's a gamble. I've seen it happen with my company when being a latecomer to containerization.
When it came to picking a container management tool, it was a tossup between k8s, Nomad, or just saying to hell with and running those containers ourselves on EC2 instances. Having run our stack on bare metal for year = we were really pretty good at it. There was a surprising amount of automation that could be ported over.
Eventually we picked k8s, and coincidentally, our usage grew more in 6 months than it had in the last 2ish years. So all in all, the gamble paid off.
... but I like to think there's another world where we picked the 'its familiar option' and things still worked out. If our traffic hadn't grown the way it did, we would never have felt the pain of having to manually scale out our systems - or basically write an in-house version of Kubernetes.
So in that sense, I'd guess that maybe some teams have the bad habit of playing the same side of the coin everytime. It may be prudent to stay conservative when picking a Datastore, maybe it's would be smart to pick a risky technology for your app servers? (and vice-versa)
This is more like how their infrastructure currently looks like I guess as it’s somewhat newer and follows up as an update to your video.
They still use MySQL though.
Be honest to your customers, or I will make sure our team switches to Microsoft Teams.
>Some users may be unable to connect to Slack, while others are still experiencing general performance issues.
A Teams "channel" has counter intuitive behavior, the messages you send are more like forum threads, users can reply to them, and when that happens it reorders the whole messages to put that one at the bottom. It makes it impossible to "catch up" on a channel since people will happily reply to any messages so the ordering is completely random.
There is also a completely counter intuitive distinction between "groups chats" and "channels".
Teams is also the only chat service I know where there is an actual limit on the number of non-public channels you can have. After 30 channels have been created, you're doomed.
Want to remove an existing channel to make room for a new one? Nope, you cannot remove a channel, only "archive" them. Archived channels are removed after 30 days and there is no way to force their removal early. So basically if you reach 30 channels you have to wait minimum 30 days to create a new one.
Did I mention the UI is slow as hell, and half the screen space is used by unnecessary large padding between messages? I am rarely able to have more than 3/4 messages displayed on a 23" screen.
The only positive point of Teams is integration with outlook 365 and meetings. It works really well, quality is good, reliable, nice UI, présentation mode, screen share, etc
I've used teams in the past and it's not that bad. It's a valid alternative to slack imho.
I know I sound like a Zulip shill, but try that for a bit and you'll see the difference, I love it.
Everywhere I've worked Slack's basically a free for all, can add custom emojis and create private channels etc.
Teams was always more locked down. One place even disabled the GIF feature!
I know you can lock down Slack too but nowhere I've been has bothered to do so.
I have had a few people on calls with poor connection speeds. It degrades horribly by stuttering and distorting audio. It performs worse than a phone call. The best fix would be for Australia to fix its woeful internet.
When you load the web interface it identifies each person with their initials, not full name. This isn’t helpful when you have never met the person before.
Recalling the incidents in September, it's a bit cheeky they call that "99.93%" uptime (30 mins of downtime).
Having remembered that there were at _least_ 2 seperate 1 hour+ incidents where I was struggling to send messages in Sept, I went through to look at the history, and they seem to have purposefully hidden these incidents on the calendar. There's no way of listing incidents in a month.
The information and accessibility of the status page is obviously politically motivated by marketing.
It's like when trains report 95% on time, except that the times when the service is delayed is when the trains are full. All the delays that actual people experience (person-hours) happen in that 5%.
For me it has only been slow performance all morning. Its not good, but not what I would call "down" either. At least from my perspective. I could have a bunch of other folks who are completely unable to send messages and I just can't tell.
* Messages are shown as sent (not grey), then, after several minutes, a "Try to send again" message appears. Or the message turns grey. Or both.
* The webapp randomly reloads. Sometimes already typed messages are preserved, sometimes they disappear. ( I assume a reload happens when a new version is deployed with some "force reload" flag )
* Threads and channels just show a loading spinner forever.
* After switching channels, all fonts disappear and just the menu background remains. No remedy except reload.
Outages happen. They get resolved. That's live, and it's fine.
What is not fine is how ill-equipped the frontend is to deal with such failures.
I have been seeing some version of this on my laptop for months. Don't seem to have the same issue on my work computer. Maybe it will finally get resolved.
Slack is a not-great implementation of a good idea, including a use case that IRC and XMPP don't meet. Tulip is a better implementation of that good idea, plus yet another good idea.
Example: We pay for looker. One day I thought to myself I'll spin up Metabase and try it out for myself against our prod database. It worked very well, fast and easy to query my data unlike Looker.
I haven't used it in about a month and today I try to access it but the app just doesn't load. Now I need to spend time to look into why this app isn't loading. Nevermind that I have zero Java experience or devops experience. I just ran a Docker file per their instructions.
It's often cheaper to pay for a service.
It's useful when people arrive in the discussion at different times and you don't want to necessarily let the whole channel to be notified about your specific remark, and don't want to write a private message either -- one reason being it would lose the context.
In our project we have multiple channels that have actively 3-5 persons and work at different times. The things we chat about are lightweight changes or news, or maybe someone has a quick clarifying question about implementation detail. Something that wouldn't be posted in an issue for one reason or other.
Remember the irc support channels where conversations and problem/solution dialogs were constantly muxed with other conversations / dialogs?
Imagine each question being a root message, and all of the dialog / suggestions / answers being in a thread. Also makes it super convenient for search.
In addition to that I find that beyond a certain channel / DM count it's a lot easier to follow up on things with threads
Give it a shot, try to get your whole team on board. You can always revert back to not using them later :)
I'll recognize that I'm a bit of an oddball in that respect nowadays, but it can work, at least with a small enough room. I love having all the conversations cross over one another and just being able to read back and follow them.
So let's say I wake up in the AM and there's a thread that says "how do I use the API for companyservice.example-api.com" and the thread has 50 comments.
Let's say it's a service I support. First, I know there's an issue because a customer required attention to begin with. This is then confirmed by the fact that the thread has 50 comments in it so there are probably documentation and usability issues with this service.
The company I work for and the services I am responsible for deals with internal customers and we use these two absurdly simple metrics to optimize support regularly.
When we first started with certain services that my team is responsible for, we used Slack metrics A LOT. As in, we had spreadsheets with this data and we'd go over them every week. Nowadays, our services are almost 100% self-service and we rarely, if ever, get pinged on Slack for those services because the documentation is optimized and the usability has been specifically designed so that our engineers don't get in the way of our customer's needs.
I assume Discord's gamer branding (which they only dropped recently) might be the reason Slack is more widely used for teams.
> By uploading, distributing, transmitting or otherwise using Your Content with the Service, you grant to us a perpetual, nonexclusive, transferable, royalty-free, sublicensable, and worldwide license to use, host, reproduce, modify, adapt, publish, translate, create derivative works from, distribute, perform, and display Your Content in connection with operating and providing the Service.
https://discord.com/terms
Those absolutely seem like overreach in terms of what someone would consider reasonable from a user perspective.
I'm not a lawyer, but that seems like a key limiting clause in that statement.
No new updates for the moment as we're continuing to investigate. Customers may notice that other functionality, such as Search, is also affected.
Oct 5, 12:40 PM EDT