the slack app was telling me messages weren't sent, asking for a retry. perhaps people retry and succeed twice. but i've seen messages as sent twice in my window, but they disappeared after reload of the app/page.
Best thing is that some messages go through despite returning 500s. This leads to our integrations spamming our companies channels with redundant messages while retrying. :/
Ouch, that one hurts. I don't know slack API but the contract expectation in the call should absolutely be "500 means the message failed to send"
Tells me there's some subsystem throwing the error, not the actual messaging (or, if intermittent, something wrong with the distribution.) What do you want to bet it's some kind of new analytics?
It's not possible to always ensure that you report whether an operation succeeded. Consider a timeout - you waited 5 seconds for the backend to respond and heard nothing - was it, or will it be successful? You don't know.
Typically a 500 should be considered as a "may or may not have succeeded", and steps taken to deduplicate operations if necessary.
Http 500 is like when the server drops connection before returning a response. Something happened, but you have no idea what happened. You can't safely retry unless your request is idempotent enough.
Has the CTO and key technical staff retired after the Slack IPO ? :-) In the UK there is no real outage, although Slack struggles sending some messages and this can result in some messages appearing twice. This problem seems to be worse when communicating with offsite staff, onsite messaging seems to be fine.
Or not. I'm remote and I cannot ask questions nor coordinate action to solve live production problems due to this outage. Slack is becoming a SPOF for many organizations, especially distributed.
I don't want to sound snarky by questioning what's probably an org mandated tool usage, but I can't quite believe companies where there's only one accepted way to get in contact.
Ideal would be a live conference call (phone, Skype, Hangouts etc...), but you've got SMS, email, heck, I once solved a pressing issue with my team in a Sharepoint Word document where all our comments were just new paragraphs being edited live (which actually had the handy benefit of being savable without needing to pay extra for the privilege...).
Does this change anything though? The internet has many of these SPOF services. And even more AS's who through malice or fluke can take trigger a failure of those SPOFs + more.
At the end of the day, that sounds like a massive pile of SPOFs that just barely keep up..
Interestingly that you say this; we need an additional layer called layer 8.
As app is 7 but we have grown beyond the seven into meta layers of service types which have become the favric of how we operate... such as email, txt, chat etc to keep all other aspects of infra up and running.
the OSI model is not a truthful representation of the different layer of networking. Things get messy between layer 2 and 3 (ARP for instance?).
Also, everything above layer 4 is a enormous mess, considering that stuff is usually happening at the application itself anyways. There are network protocols which have proper OSI seperation, but they are rarely used. (IS-IS in combination with CLNS for instance).
Knowledge atrophy is a serious thing... i used to be deeply skilled in all of that but attention is where the heart is... so ive literally forgotten so much abt networking over the years.
A failure of mine was to not keep a project portfolio runbook which i could look back upon.
Imagine a system where all your status reports were logged and searchable - that would be move valuable than linkedin...
Not really? The internet is literally an internet, i.e. a network that connects networks. You can nearly always find another connection that still works, be it a coffee shop or cellular.
Sure, having a single connection is a SPOF, but anyone living in an urban area and anyone with a cellphone (and not using the same cellular carrier for home internet) has more than one connection.
I'm on Google Fi, so theoretically I have redundancy on my phone as well. I have used my phone as a hotspot for work, and I've also gone to other locations to get work done (library, coffee shop, a neighboring company's lobby, etc).
A lot has to go wrong for me to not get work done. If Slack goes down, I use text, email, or Jira comments (in roughly that order of priority). Relying on only one communications method is foolhardy.
This is part of the problem - we all need to ask too many questions. We should be working on pre-defined tasks most of the time, where it doesn't matter if it's waterfall pre-defined or scrum time allocated.
I slipped over the "live" word, but I still stand by what I said. I most probably work in a similar environment, with tonnes of live production issues, and I believe it's a result of not being allowed to proactively plan for situations - most of the issues are a result of cost savings, systems driven to their absolute maximum capacity, etc. Live production issues are a byproduct of moving too fast, breaking too many things.
I'd style my way out of that argument by saying out reliance on tools like Slack rather than other pre-planned work builds a reliance on that one tool for all tasks being done "in the moment", which then means it's relied on as the medium in live production issues.
If there was a lower reliance on IM for all tasks, other solutions for crisis management would be utilised (potentially).
What an incredibly arrogant generalization to make. You know there are also production issues created by moving too slow and failing to address pending scalability issues, right?
Well, there are other means of communication but they either:
* do not fit well for a quick message interchange like you can have during an outage (email)
* don't have the whole company (or at least the whole tech team) on it (whatsapp, telegram, you name it)
* make compliated to share big chunks of text/logs (phone call)
It probably makes sense creating another secondary - usually dormant - communication hub like a WhatsApp group for emergencies but people tend to misuse it (like using it to ping for some minor incident when all the other more suited communication tools are working perfectly).
Anyway it wasn't a big outage on our systems, just a minor hiccup, but it's a lesson learnt nonetheless.
we use a mumble server as backup and parallel mean to discuss in the operations department... Sometimes talking all together using voice as if we were in the same office is much nicer than writing!
Mumble / murmur [1] is great. I set one up for a family member in minutes. Super easy to deploy. People should use their RC releases. They are solid, sound quality is great and it scales really well. A small to medium VM can handle thousands of people.
At one stage when there was an outage about a year ago, folks at work were scrambling around trying to get HipChat resurrected (which we'd been using prior to moving to Slack).
I had an ircd of some description installed, configured and ready for use in under 3 minutes, and started to get my team using it as a stop gap measure. Had that rolling long before either the outage was resolved, or the HipChat server was resurrected. People forget just how stupidly easy it is to set up IRC.
Do you have a recommended server / client? The last time I tried to use IRC (joining an existing chat) I actually had quite a bit of trouble, and ended up just giving up.
Inspircd is pretty easy to set up (as a server). Hexchat is a great client, Weechat is great for a commandline client. If you have server access you can also set up Kiwi which is a webclient, making it easy for those with less tech prowess to connect.
For ages, I ran unrealircd [1] and anope [2] services. One could put TheLounge [3] (web front end with redis) in front of it for persistence. Not as easy as inspire to set up, but rock solid and feature rich.
I had a similar outage at another job for a day. Pre slack days, but instant messaging and email was down.
I needed to do some things so I just stopped checking with people. So did other people.
The result was I found was that we were double checking, coordinating and doing a lot of verification that honestly wasn't needed. The sky didn't fall, everyone still saw the changes and were ok with it.
After that folks stopped doing a lot of the traditional coordination that proved to be superfluous and maybe never accomplished anything. The handful of mistakes that happened, were also easily caught / fixed.
Granted, this requires people to make good choices.
Yeah, i understand. hosting your own mattermost (or other slack clone) is not enough for you? You can also have 2 self-hosted solution on different servers (one on our own, one on any CSP), in case your server fails.
According to some prospective studies, some people would (and please stay with me on that) actually use slack to communicate about work-related matters with co-workers.
More study are needed to clarify if this work-related communication might actually be required for work to get done. (Cal Newport is definitely preparing a book on that any time soon.)
Unfortunately, the researchers are too busy maintaining their IRC server to actual do it - but at least they're using IRC.
Whereas, my productivity definitely goes up when HN is down ;)
I'm not saying this is "unbearable", and plenty of organisations have people whose job description would probably correspond to doing those tasks.
But I can definitely understand why you would want to skip them entirely and have someone host your chat - which is basically the job slack is paid for.
"Mission critical, but not paid by your customer" is always tricky to staff for, isn't it ?
Odds are that you’ll have updates worth installing once every couple of years.
>* update the version of the os of the machine running the IRC server
Almost never unless there’s a remotely exploitable code execution vulnerability. Local bugs wont matter unless you run multiple services on the same box.
>* repair a broken disk / fan / overflowed disk on the machine
Depends on your hosting setup. With a cloud setup perhaps never.
I’m certainly not trying to suggest that anyone should use IRC over slack in any situation, just that it’s not a horrible time sink requiring significant maintenance.
I think what may have triggered people here were the absolute terms (can’t remember ever having to do “maintaining” after the initial setup).
In this forum you'll be hard pressed to find someone who thinks it's a good idea to leave a cloud-based (or any internet-facing, really) server completely without maintenance for any significant period of time.
If you had said that usually it's as simple as checking in every now and again to reboot / make sure security updates are installed / check functionality and that it's a manageable burden which is worth the effort you might have gotten a more favourable response.
Edit: or if you had automated it to an acceptable level, maybe some more information about that.
That is a remarkably good point that I can't believe I hadn't noticed. What a strange thing to name a product that one would try to sell as Enterprise productivity software.
Is it just me, or do those outages seem far more common lately? Or maybe this is happening just as often as it used to, but now we're growing more dependent on cloud-based software so this is reported more prominently?
I understand that post mortem's are interesting learning lessons, but if you're curious whether S3 or whatever is down or not right at this moment, then use the subsequent status pages of those services.
Personally, I enjoy the speculation and discussion during an outage like this. Also, occasionally someone from the mentioned company pops up in these threads
Agreed. It's fascinating to see speculation and it's almost a game when the real reason is found - we see who got it right!
It's also important - many here use Slack, and an outage may be confusing - most of us don't immediately look at status pages, so we have somewhere to go and discuss the happening.
If we could trust status pages, sure. But I've lost count of the number of times I've tried to use a service, failed, the status page says "all OK!" but Twitter and HN are full of outage reports.
Besides, the discussion here is useful. Someone has already pointed out that some messages are getting through despite returning 500s, so integrations are spamming duplicate messages everywhere. That's useful to me if I'm running an integration.
(besides x2, it's one thread with a very self explanatory title. Maybe just don't read it?)
I get most of my outage notifications from HN. If it's a blip, it probably won't make the front page through my standard refresh cycle. If it persists through two refresh cycles, there's a TITSUP somewhere and it's time to take a look.
This community is awesome. Always someone with their finger on the pulse :)
Why? These events are often great opportunities to learn about system failures and their impact on other systems in real time. Status pages are often not even genuinely accurate.
We use downdetector.com because status pages tend to take up to an hour or so to update, if they ever do.
reply
jgrahamc 4 days ago [-]
1042 UTC First alert of global traffic problem 1057 UTC Internal group chat room up and running 1102 UTC Status page updated
So, first alert to status page was 20 minutes.”””
In those 20m we had repeatedly checked your status page, realised it was our issue and started pulling engineers to deal with it as per procedure. People are on call, it's highly disruptive.
Surely you knew within those 20m that something was up?
In the end we realised it wasn't our issue because we checked Twitter.
Yep. I tend to agree that we could have gone faster. The difficulty is that getting clear information out fast when you are dealing with a difficult situation is hard.
But, I guess we could have put some status up quicker.
I'm irritated, however, that you picked on Cloudflare as a bad actor here when we strive to be transparent and quick to get out full information whereas others (e.g. Amazon) are slow as mud.
But I get that being on the other side of this is difficult when you don't have information.
Amazon, Google, seem far far worse. I wasn't intending for my post to me representative, just that the CF outage is recent and affects me more personally than the others atm. Also, I had zero expectation my comment would actually be read, this is quite a surprise.
If it's any incentive, if the status page had any inkling that something was wrong, I'd be first on here posting "omg their status page is real".
Fair enough. Thanks for making the comment. Made me think about how fast we act in terms of public status pages. And sorry that route leak affected you.
There are reasons why email is async and supposed to be decentralized, with priority based fallback options. (The priority in mx records). Most email servers try to deliver to fallbacks, and even if that fails, will try to deliver for _days_.
For all people who think slack can replace email, think about these safeguards.
In general mail servers will reply back with a failure notice within an hour or so. This isn't a standardized behavior and timeouts and retry semantics differ.
If it's a temporary outage, eg. network loss of a datacenter, email is not lost. It will be delivered with a delay, tried over and over again, until the timeout limit is reached. Due to the async nature the delay is not a problem.
reminds me of that one time a genius at IBM created an `ibm-global-announcements` channel and force-invited 200'000 people in it, and then some guy `@channel`d and all ibm's slack workspaces were down for 30 minutes
also that channel made using slack impossible for mac book air users, they had around 80% cpu usage for slack. so basically entire marketing and PM part was unable to work that day. developes machines were wasting around 20% on slack.
after people started complaining in that channel, posting in it was limited to admins only, but they didn't lock commenting. so, for approx 6 hours all of IBM was posting memes in ＴＨＥＴＨＲＥＡＤ as we dubbed it, and @mentioning the genius who created that channel. next day the channel was nuked, not even an archive preserved.
some guy calculated that the entire affair, considering electricity prices, a 20% decrease in developers productivity, and so on resulted in IBM loosing several millions with that stunt
fun times
p.s. shout out to Martinj for that spicy jeff-coffee-mug meme
There should be a way to turn off @here and @channel. It should be a user-level setting, a channel-level setting, and an organization-level setting. Even if my org or channel doesn't opt out, I should be able to.
I hate that it can't be controlled per channel. There are many channels where it's useful and desirable, so our org doesn't disable it globally. This creates situations where new people @here and @channel not knowing any better, which leads to hurt feelings all around.
Some of this you can already do. Users can opt out of @here and @channel for any given channel. I know that orgs can restrict usage of @everyone, I assume they can do the same for @here/@channel but I'm not sure.
I know you can in Discord, I'd be shocked if Slack doesn't let you do that. checks Settings
Yes, you can mute a channel and suppress @everyone and @here if you want. Still friggin nuts that a chat service can bring a modern machine to it's knees.
Ah I knew you could mute it but muting a channel does not suppress @here. I did more exploring and found it's a multi-step process to suppress @here vs muting, but it is indeed possible.
It would be interesting to have a karma function whereby it was based on your use of @channel or @everyone - or something similar to how HN used to require 500 karma to be able to downvote...
My work has a 1000+ person channel for ad-hoc support questions that aren't critical and could be answered by anyone. If it's critical, our support team has an SLA, but Slack doesn't.
There was one specific person who would ask a question, expect an immediate response, and if they didn't get the response they would @here. They were instructed not to do that because it pings over 1000 people, so the next time they asked twice in an hour and when they didn't get a response they said "I hate to do this but I really need a response @here". They were kicked from the channel. Really unfortunate since it's a very useful channel to be in, but @here and @channel is just so disruptive in big channels.
It should just be turned off for large channels. Pop an error message and direct users who need it to a support article, if they follow up real support people then see how many people complain.
i stand by my opinion that "lets create one chat room and invite hundreds of thousands of people into it" is a bad idea regardless of what technology implements the chat room
I respect your opinion, because even if an application could handle 200k users at once was able to be run on a MBP without serious issue, it's still gauche to forcibly invite 200k people in your company to a chat.
Slack is designed for business chat, and many businesses have thousands or more employees. Slack should be able to handle any features it provides, and @everyone is a feature. I don't know how they've implemented things or what the specific issue is, but it should at least have blocks in place that prevent stupid things. Maybe channel sizes should be limited to 1000 people or whatever their testing shows it can handle. Idk.
Software designed for large numbers of people shouldn't go down because one of those people does something stupid.
If you want to tell everyone in your company something, you could email them I guess. Is it the fact that it's a "chat room" which means you expect it not to work? What's best, here? SMS? Email? Putting something on the website and making everyone check the website once a day at noon? Tell one in ten people and get them to "pass it on"?
Yes, and he's making the point that it manages to work. And that perhaps your expectation should be the same for a chat client as for an email client. Which may or may not be a valid point.
And without proper mailing-list-moderation, that too can cause email storms. No one blames "the genius who decided to put everyone in one mailing list" when that happens.
The real fail here isn't putting everyone in one channel. The real fail is not using/providing proper admin tools
I'm going to guess you could also break IBM's phone system by attempting to start a conference call with 200K people on it, and destroy their auditorium by putting too many people on the stage.
Although usually only a limited number of people can post to an "announce-list" that goes to everyone in the company. (And if there are company-wide lists that anyone can post to, most users filter those to a folder. I suspect such lists don't scale past a certain point either.)
I love that companies will make it hard to expense a $50 meal but they will gladly let you book an hour meeting with 20 high-paid developers or even let you directly ping 200,000 people which must cost the company many thousands of dollars.
I still hate that anyone can add anyone to a channel - it's not an "invite", it's just global "add to room" privileges.
In general I find Slack not very concerned with your own ability to control your experience (no ignore user feature, can be added to rooms, can be added to rooms that then get pinged with @channel and your notifications start blowing up, etc).
This may be 'expected' in a small company where of course everyone should talk to everyone, but for larger companies and the plethora of various open-source or interest-based communities now using Slack, it's not so good.
Even worse than the channel mechanic is the thread mechanic - anyone can turn your comment to a thread, and you cannot mute threads by default. Even worse, updates to threads suppress channel unread notifications, causing you to miss updates.
Manually having to follow threads and unfollow threads is a nightmare compared to the one time configure experience of channels, especially since you cannot just mute all threads, and is easily the worst part of Slack to me.
This is not old news. There is still an IBM slack channel called #general with 30k+ people in it. And people regularly arrive in it to say "hi". No useful information is shared.
Can someone explain to me why any company should outsource something as critical as internal communication to a company that is 5 years old? The video messaging left aside, is it really that better than time-tested solutions like email or just running an IRC server?
Because they don’t trust their IT or operations groups to manage IRC or any other open source communication system; even as they trust those same people to run the software that makes them money.
It's a reasonably nice client, and available on every platform.
It has easy integrations with basically everything, and non-developers can set them up on their own.
It has bouncer functionality built in - You don't miss messages. It can also archive them, and handle them in a corporate-compliant way without needing to speed a bunch of time setting that up manually.
Because it's a central company and not IRC push-notifications are easy.
Emoji are shared corporate wide, anyone can add them.
It's a really nice out of the box experience.
You can absolutely replicate it with enough setup and tweaking, but the free version of Slack will go a long way.
"just running an IRC server" is... a lot. Or can be. Who is running it? Does it scale well? How is login synced with corporate logins? What about mobile devices?
It's not "better" than email, it's different. Real time. It has a million and one service integrations, which IRC doesn't.
Point is, it's really not all that crazy to just pay a third party to manage all this stuff for you. You might imagine the alternative is a perfect IRC server with 100% uptime, but I doubt that would be the reality. If you're saying a service like this is too important to be run by a third party then you're going to have to invest a lot internally to come up with an alternative that's more reliable.
Obviously you can create IRC integrations for anything. But how many are installable with one click, like Slack integrations are? Having to write custom implementations is yet another extra task and point of failure compared to just paying someone to manage it all for you.
IRC does have limits. Some. One is that to make it "modern" it needs a bouncer - that said znc is spectacularly simple to set up with it's web interface.
Integrations were never one of those limits. Just google IRC bots.
How do you make it possible for people to view the IRC channel history when they were disconnected? I know you can have a bot that saves a log, but can it be made seamless?
I think it's really sad how the world has migrated away from an open protocol to a closed one, but I remember IRC having some very real annoyances, well beyond stuff like integrations.
That's my entire point. It's death by a thousand cuts. Now I'm responsible for an IRC server, a web frontend, a mobile app and a bouncer.
The OP said "just run an IRC server". There is no "Slack or just an IRC server". There's "Slack or a number of connected, interdependent services you need to maintain". I'm not saying it can't be done, or that it isn't the right answer for some people, but it's also a lot more work (and thus, expense) than just paying for Slack. It absolutely makes sense that people do it.
With that many employees you're signing up for Slack's enterprise tier, and you can bet they price it so that they come out the right side of this cost/benefit calculation for a lot of companies.
But managing an IRC server cluster for 10.000+ users is no simple task.
Paying experts in their field (chat tools in this case) is most of the time the most reliable and cheapest variant. You must also include that Slack has plenty of features that are not available in IRC and mean the productivity is not as good.
Tbh., if you need two full-time sysadmins to run your IRC server + bouncer + log backup + web frontend, you're doing something really wrong. It's more like an after-school job for a bored CS student.
Features is not the only thing, does it have well funded sales team and strong corporate support ? Slack itself did support IRC till recently, the product is not just a sum of its features
Yes at that scale it may be useful to roll out a custom system. However I'd count into account that large customers probably have different pricing and that maintaining custom systems is always bothersome because you are always the largest customer.
What happens when your sysadmins say "Maintaining IRC is a waste of time and gets us 1% of the features Slack does, let's just get Slack?" (At my last company, it was the sysadmin team, including myself, who decided to switch to Slack and killed our IRC server.)
Come on, in the 90s and early 00s, volunteers ran and managed IRC networks scaling up to hundreds of thousands of users. This was back when hardware and bandwidth were much weaker and 100x more expensive. It's really not that hard.
Yes, volunteers ran these things. How many? Because you're going to need to have paid employees rather than volunteers running your servers. Is it going to be more reliable and cost effective than just paying a third party to run a service for you?
For something like IRC? Yes. The service is dead simple and one server can handle thousands of users with modest hardware and without breaking a sweat. A completely unfamiliar sysadmin could set it up in an hour or less. Since it's probably internal, there's little need for public services (nick/channel registration, etc).
So what about the mobile device issue? Corporate logins?
If you want to match what Slack offers you aren't just running an IRC server, you're going to have a small fleet of dependent services that, added up, just about match what Slack does.
Is it really that surprising that people just opt to pay for someone to deal with all of this for them?
At which point you're not looking at IRC as a service anymore. There are different niches served - but the question was about the reasonableness of setting up IRC alone, which is still dead easy.
Most of how I use Slack, and how everyone I know uses Slack, involves 1. sending messages to people who are “away” to get their attention/loop them into a conversation; and 2. people reading back up in history for context when they come online.
Now, modern IRCds might have features I’ve never experienced on raw IRC... but if they don’t, I don’t suppose you’re suggesting that this server also run thousands of bouncer bots in order to provide this contextual history feature?
If you're a big enough company (more than 60 people if computer are your means of production, around a thousand if not), you should have your own server and a fallback on a CSP. Just pay some guy in India/South Africa to manage all your internal services and architecture. If they are sensible and you need to trust the guy, just pay him well and hire locally.
I can set you up a resillient architecture and 3 services (communication, documentation, filesharing) in less than two weeks (let's say two weeks and a half because security concerns/limitations and all that stuff). Add one week and you will have mail server, VOIP communication, log conservation and monitoring, backup images on CEPH, vnc service for "easy" maintainance and a small documentation explaining the few action needed to maintain the monster for two years. Initial cost for a company: 15 to 25k$, + 1 hour and ~100$ a month.
They somehow got Clevels on the bandwagon. Those who actually understand the implications don't have a word.
Truth is I see many capable people buying into these as well because of the convenience - read: lazyness - factor. The "ain' nobody got time for that" effects everyone and the willingness to pay for services had skyrocketed in the past decade. This applies to blogging, to home servers, backup, anything.
We're all guilty at some level.
IRC is pretty much useless since you don't have history. Comparing Slack and IRC are like comparing TCP and HTML5.
Why IRC is pretty bad in enterprise:
- no audio / video
- no history
- no SSO / AD integration
- no rich features like links, pictures, upload docs ect ...
- easy bot integration with APIs, I can have Datadog pushing things to Slack ( alerts, graph... )
- no archiving channels on the go, Slack is very useful when you have an incident and you create a temp channel and then archive it when the incident is closed.
Does the opensource matrix server scale? I may be wrong but I got the impression it was more an example implementation and was not used by Matrix in production?
There is no closed source Matrix server implementation that I'm aware of. Synapse is the server used in production for matrix.org, the French government deployments, and tons of other servers including my own. It is somewhat resource intensive but it can scale using workers.
There is of course several other server implementations in progress but none are fully ready yet.
There are products that are far better for audio and video communication than Slack. At my company, we use Zoom.
> no history
Can be done with a bouncer if you need it.
> no rich features like links, pictures, upload docs ect
Is there a reason why an external service can't be used? I can't upload a image here on Hacker News, but I can always include a link to it.
> easy bot integration with APIs
So I have to use the HTTP protocol to manage a bot rather than checking for certain key words in the message itself.
> no archiving channels on the go, Slack is very useful when you have an incident and you create a temp channel and then archive it when the incident is closed.
That depends on the settings. One thing that happens every so often at work is that people get trapped as the last remaining person in a channel because only an administrator can archive a channel. In IRC, that's not an issue.
"The Wheel of Time turns, and Ages come and pass, leaving memories that become legend. Legend fades to myth, and even myth is long forgotten when the Age that gave it birth comes again."
I certainly prefer it, but I've both heard many other millennials strongly oppose slack, and Gen X:ers talk highly about other "no-setup" systems. I don't think not a generational thing.
* copy / pasting images that just work (the'll never admit it, but animated gif is their killer feature)
* stable-enough infrastructure for something that costs you 15€/month/user not taken from your "salary" budget (often a different and substantially less taxed part)
* and their outages are funny (receiving messages twice is arguably better than not receiving message)
Yes, it's because it "just kinda works for what it costs, and you can forget about it most of the time and do your job."
This seems to be characterized as "laziness" by some people, a point I honestly don't understand (I though lazyness was a virtue in our circles.)
That being said, there are legitimate reasons to not use it:
* software licence
* data property
* technical implementation of the native clients
* general distrust of centralized solutions
* cost of having IM at all (see Newport's whole publishing career)
At least, it means there's an opportunity to create a company that handles decentralized user-friendly open-source standard-based data-respectful attention-preserving uber-reliable chat system.
HN users would be a market.
For some definitions of "just" and "working". Even ignoring the business limitation, the UX of Slack search is meh. That said, I'm yet to see any of the modern web&mobile-oriented IMs that would have a decent search UX.
But yeah, your list is solid. They offer plenty of value, just mixed with plenty of negatives.
A big source of the complaints against Slack is usually the network effect - in context of both work and OSS communities, it's usually somebody else that makes the decision and imposes it on you, forcing you to run face-first into all the "legitimate reasons to not use it" you listed. Half of the pain would be gone if they opened up their service to third-party clients; I'd say the reason they didn't have as much opposition from people initially is because they run the bait-and-switch with IRC gateway.
Interesting point about : work environment vs communities.
For work, the decision of which IM tool is most likely going to be a corporate one, without much choice of solution.
I guess the problem here is the same as for other corporate tools. (And people have strong feelings about corporate tools, but it seems there are less strong feelings about "using jira vs bugzilla" as there are "using slack vs irc". Or maybe there are as much strong feelings.)
On the other end, I get the troubles with the "network" effect for out-of-work communities (open source projects, clubs, etc...)
Such communities can at least have "some" level of debate about which solution to use, and balance the pros and cons as they see fit.
It's completely acceptable for an OSS community to discourage using such platforms - as any other non-free, centralized platforms. We're still not forced to use anything.
Then, there is the "club of non-power user" situation. I would be hard pressed to ask someone from my drama club to setup and maintain an IRC server (neither do I want to do it.)
Thanks for adding this dimension.
Follow-up: is there a "nice" provider of some slack-like solution (mattermost, etc...) that hosts servers for communities of non-techies for not too much money ?
- Scrollback/history available in the main client interface people who newly join a channel (extraordinarily important during incident response)
- Ability to link to a message in history
- Ability to handle multi-line messages in a vaguely competent matter
- Image uploads, snippets, etc.
- Users of a GUI tool / web interface are first-class, users of a command-line client are second-class; this is unlike IRC, and important when the goal is a single chat system for the entire company, not just engineering (and usually a subset of engineering at that)
- Client/UI expectation that you do not catch up on every message unless you want to (like IRC, but unlike email)
- Quality integrations that have already been written with the third-party apps that you're already using
IRC and email have both been time-tested, and have failed the test.
I have seen enough corporate mail servers going down, irc and jabber servers breaking etc.
Using those won't magically give you 24/7 with more than five nines uptime.
Key difference is: If slack goes down many companies are affected and it goes on the internet. If a company's mail setup goes down employees are a bit annoyed, but after a while it works again and nobody outside notices and it's no news on HN (unless it's a real major thing bringing a notable company completely down)
Our company switched from self-hosted RocketChat to Slack just this week. Personally I didn't agree with it, but it's not my call. The reason was mostly that Op guys were too busy with other things and didn't want to care about yet another critical company-wide service. They though outsourcing it would solve the downtimes and bugs. This outage couldn't come at a better time.
I’ve heard from countless of friends how many times Outlook goes down for their company, whether it’s a midsize company or a large company. I don’t think this is unique to Slack or company age.
We used to run an IRC server at work, and it was pretty exclusionary to non-engineers; setting up something that'd log backscroll required maintaining a vm somewhere, searching through history required another service, etc. If all slack does is chat (and video communication by way of an acquisiton), I'm happy letting them be 5 years old. I'd scrutinize a company that:
1. has no fallback mechanism of Slack is mission-critical and goes offline
2. isn't careful about including non-technical stakeholders in communications.
Edit: clarifying that I'm talking about my workplace, not a company I run.
Why don't people use free software instead, e.g. a Matrix homeserver run via https://github.com/matrix-org/synapse/ ? IRC without a BNC doesn't really work for a corporation due to disconnects.
One problem with Slack we have in the finance world is that _literally everything_ employees say on company assets needs to be recorded and readily accessible for audit (No this is not an invasion of privacy, you're on company time on company systems, we don't peer into your personal phone or anything).
Unfortunately, Slack does not do this very well. I'd really like to know if there's another service that _does_ do this well.
My guess is most of the major chat players handle this pretty easily -- it's a non-starter for many potential users if you don't. Zulip stores all data indefinitely (including message edits, etc) in its default configuration.
Genuine question - when I use slack - I'm able to see messages going back to when the channel was created for my organization as well as all files shared by date. I'm not sure how slack doesn't do this well?
Slack records everything you do in an immuteable audit log. It's used by lots of finance companies as well as government institutions ( who have just as strict if not stricter rules ). Also they have lots of features to help with e-discovery, so not only is everything recorded, but getting your needed data out is quite easy.
We used (and I assume they continue to use, hi COFers!) Slack at Capital One. They are super careful about compliance so I'm sure everything was above board with the regulators.
Mattermost does, and just raised about $70m to do it well. ING is on their website. They have e-discovery tools and it's all hosted on your own cloud/servers.
Just yesterday I was musing that if I were King of the (World|Company) I'd want an open-source Slack-alike that I could just drop into the Cloud of my choice and operate entirely within my private network, subject to my own access control just like other internal services, and with full access to all message histories in whatever database-like thing it uses in its Cloud. Sure, I'd still have a SPOF but it's game over anyway if my Cloud goes dark.
Is there such a project, and if so does it have any traction in the real world?
(Oh and ditto for every damn thing Atlassian owns. Yikes.)
Mattermost API being a superset of Slack's seems to be mostly true, but looking at the features our Slack app uses I don't see block kit (https://api.slack.com/block-kit) or message scheduling. If we were to integrate with Mattermost it doesn't look like it'd be a ton of work though
Have to say Mattermost's API docs are so much easier to read than Slack's.
Honestly, the same. It's tiring that people just has to suggest self-hosting IRC or any other software all the time when Slack is mentioned. It's a good way to communicate and it's easy. Hope they resolve it, overall a bad situation.
245 comments
[ 3.1 ms ] story [ 237 ms ] threadThat's the symptom of the outage as far as I can tell.
https://twitter.com/slackstatus/status/1144577107759996928?s...
It's odd because the message said it failed to edit, but my client still shows the message as edited.
Tells me there's some subsystem throwing the error, not the actual messaging (or, if intermittent, something wrong with the distribution.) What do you want to bet it's some kind of new analytics?
Typically a 500 should be considered as a "may or may not have succeeded", and steps taken to deduplicate operations if necessary.
If Slack is your only forum of communication with your team it’s time to rethink your support structure and DR plans.
Edit: It’s very unhelpful if I point out issues and don’t provide solutions. PagerDuty has a great article about Incident Response[1].
[1]https://response.pagerduty.com/
Ideal would be a live conference call (phone, Skype, Hangouts etc...), but you've got SMS, email, heck, I once solved a pressing issue with my team in a Sharepoint Word document where all our comments were just new paragraphs being edited live (which actually had the handy benefit of being savable without needing to pay extra for the privilege...).
However, when major backbones go down it takes it a while to recover.
[1]: https://blog.cloudflare.com/how-verizon-and-a-bgp-optimizer-...
We build services on top of that that don't cope with failures.
At the end of the day, that sounds like a massive pile of SPOFs that just barely keep up..
As app is 7 but we have grown beyond the seven into meta layers of service types which have become the favric of how we operate... such as email, txt, chat etc to keep all other aspects of infra up and running.
Also, everything above layer 4 is a enormous mess, considering that stuff is usually happening at the application itself anyways. There are network protocols which have proper OSI seperation, but they are rarely used. (IS-IS in combination with CLNS for instance).
A failure of mine was to not keep a project portfolio runbook which i could look back upon.
Imagine a system where all your status reports were logged and searchable - that would be move valuable than linkedin...
Sure, having a single connection is a SPOF, but anyone living in an urban area and anyone with a cellphone (and not using the same cellular carrier for home internet) has more than one connection.
A lot has to go wrong for me to not get work done. If Slack goes down, I use text, email, or Jira comments (in roughly that order of priority). Relying on only one communications method is foolhardy.
If there was a lower reliance on IM for all tasks, other solutions for crisis management would be utilised (potentially).
> failing to address pending scalability issues
How is that not related to not getting enough funding and resources?
And yes, having scalability issues means moving too fast - the _business_ moving too fast.
* do not fit well for a quick message interchange like you can have during an outage (email)
* don't have the whole company (or at least the whole tech team) on it (whatsapp, telegram, you name it)
* make compliated to share big chunks of text/logs (phone call)
It probably makes sense creating another secondary - usually dormant - communication hub like a WhatsApp group for emergencies but people tend to misuse it (like using it to ping for some minor incident when all the other more suited communication tools are working perfectly).
Anyway it wasn't a big outage on our systems, just a minor hiccup, but it's a lesson learnt nonetheless.
You won't have access to the things slack integrates with but at least you can talk.
I know some people use IRC as a low-dependency fallback channel.
[1] - https://wiki.mumble.info/wiki/Main_Page
I had an ircd of some description installed, configured and ready for use in under 3 minutes, and started to get my team using it as a stop gap measure. Had that rolling long before either the outage was resolved, or the HipChat server was resurrected. People forget just how stupidly easy it is to set up IRC.
Freenode runs this one for a server.
https://github.com/freenode/ircd-seven
[1] - https://www.unrealircd.org/
[2] - https://www.anope.org/
[3] - https://thelounge.chat/
I had a similar outage at another job for a day. Pre slack days, but instant messaging and email was down.
I needed to do some things so I just stopped checking with people. So did other people.
The result was I found was that we were double checking, coordinating and doing a lot of verification that honestly wasn't needed. The sky didn't fall, everyone still saw the changes and were ok with it.
After that folks stopped doing a lot of the traditional coordination that proved to be superfluous and maybe never accomplished anything. The handful of mistakes that happened, were also easily caught / fixed.
Granted, this requires people to make good choices.
More study are needed to clarify if this work-related communication might actually be required for work to get done. (Cal Newport is definitely preparing a book on that any time soon.)
Unfortunately, the researchers are too busy maintaining their IRC server to actual do it - but at least they're using IRC.
Whereas, my productivity definitely goes up when HN is down ;)
* update the version of the IRC server
* update the version of the os of the machine running the IRC server
* repair a broken disk / fan / overflowed disk on the machine
* add / remove / reset password / change weird settings of users
I'm not saying this is "unbearable", and plenty of organisations have people whose job description would probably correspond to doing those tasks.
But I can definitely understand why you would want to skip them entirely and have someone host your chat - which is basically the job slack is paid for.
"Mission critical, but not paid by your customer" is always tricky to staff for, isn't it ?
Odds are that you’ll have updates worth installing once every couple of years.
>* update the version of the os of the machine running the IRC server
Almost never unless there’s a remotely exploitable code execution vulnerability. Local bugs wont matter unless you run multiple services on the same box.
>* repair a broken disk / fan / overflowed disk on the machine
Depends on your hosting setup. With a cloud setup perhaps never.
I’m certainly not trying to suggest that anyone should use IRC over slack in any situation, just that it’s not a horrible time sink requiring significant maintenance.
In this forum you'll be hard pressed to find someone who thinks it's a good idea to leave a cloud-based (or any internet-facing, really) server completely without maintenance for any significant period of time.
If you had said that usually it's as simple as checking in every now and again to reboot / make sure security updates are installed / check functionality and that it's a manageable burden which is worth the effort you might have gotten a more favourable response.
Edit: or if you had automated it to an acceptable level, maybe some more information about that.
I understand that post mortem's are interesting learning lessons, but if you're curious whether S3 or whatever is down or not right at this moment, then use the subsequent status pages of those services.
It's also important - many here use Slack, and an outage may be confusing - most of us don't immediately look at status pages, so we have somewhere to go and discuss the happening.
Besides, the discussion here is useful. Someone has already pointed out that some messages are getting through despite returning 500s, so integrations are spamming duplicate messages everywhere. That's useful to me if I'm running an integration.
(besides x2, it's one thread with a very self explanatory title. Maybe just don't read it?)
If you don’t think the article is appropriate flag and move on.
This community is awesome. Always someone with their finger on the pulse :)
Also don't get me wrong, you do a great service. It's just a pet peeve that it seems invariably status pages are a lie.
”"”sauldcosta 4 days ago [-]
We use downdetector.com because status pages tend to take up to an hour or so to update, if they ever do. reply
jgrahamc 4 days ago [-]
1042 UTC First alert of global traffic problem 1057 UTC Internal group chat room up and running 1102 UTC Status page updated So, first alert to status page was 20 minutes.”””
In those 20m we had repeatedly checked your status page, realised it was our issue and started pulling engineers to deal with it as per procedure. People are on call, it's highly disruptive.
Surely you knew within those 20m that something was up?
In the end we realised it wasn't our issue because we checked Twitter.
Edit - added quotes
But, I guess we could have put some status up quicker.
I'm irritated, however, that you picked on Cloudflare as a bad actor here when we strive to be transparent and quick to get out full information whereas others (e.g. Amazon) are slow as mud.
But I get that being on the other side of this is difficult when you don't have information.
If it's any incentive, if the status page had any inkling that something was wrong, I'd be first on here posting "omg their status page is real".
For all people who think slack can replace email, think about these safeguards.
- https://www.brianmcadam.com/5-reasons-replace-email-text-im-...
- https://tech.co/news/slack-replacing-email-workplace-2018-08
[...]
Also, how many IT service desk ticket systems just lit up with people asking them to fix Slack?
https://status.slack.com/ibm/2018-03/f01d4c22cd953dd7
https://i.imgur.com/Rk6Kdgp.png
EDIT:
also that channel made using slack impossible for mac book air users, they had around 80% cpu usage for slack. so basically entire marketing and PM part was unable to work that day. developes machines were wasting around 20% on slack.
after people started complaining in that channel, posting in it was limited to admins only, but they didn't lock commenting. so, for approx 6 hours all of IBM was posting memes in ＴＨＥＴＨＲＥＡＤ as we dubbed it, and @mentioning the genius who created that channel. next day the channel was nuked, not even an archive preserved.
some guy calculated that the entire affair, considering electricity prices, a 20% decrease in developers productivity, and so on resulted in IBM loosing several millions with that stunt
fun times
p.s. shout out to Martinj for that spicy jeff-coffee-mug meme
Yes, you can mute a channel and suppress @everyone and @here if you want. Still friggin nuts that a chat service can bring a modern machine to it's knees.
Learn something new every day.
There was one specific person who would ask a question, expect an immediate response, and if they didn't get the response they would @here. They were instructed not to do that because it pings over 1000 people, so the next time they asked twice in an hour and when they didn't get a response they said "I hate to do this but I really need a response @here". They were kicked from the channel. Really unfortunate since it's a very useful channel to be in, but @here and @channel is just so disruptive in big channels.
Are you saying it is the user's fault where it is clearly Slack's inadequate implementation that is the problem?
Software designed for large numbers of people shouldn't go down because one of those people does something stupid.
The real fail here isn't putting everyone in one channel. The real fail is not using/providing proper admin tools
In general I find Slack not very concerned with your own ability to control your experience (no ignore user feature, can be added to rooms, can be added to rooms that then get pinged with @channel and your notifications start blowing up, etc).
This may be 'expected' in a small company where of course everyone should talk to everyone, but for larger companies and the plethora of various open-source or interest-based communities now using Slack, it's not so good.
Manually having to follow threads and unfollow threads is a nightmare compared to the one time configure experience of channels, especially since you cannot just mute all threads, and is easily the worst part of Slack to me.
> 200'000
Interesting. Why an apostrophe?
using an apostrophe `U+0027` appears to be the least wrong use of possible glyphs accessible on my keyboard layout
also c++14 does it this way
https://finance.uonbi.ac.ke/node/114148
/shrug
It has easy integrations with basically everything, and non-developers can set them up on their own.
It has bouncer functionality built in - You don't miss messages. It can also archive them, and handle them in a corporate-compliant way without needing to speed a bunch of time setting that up manually.
Because it's a central company and not IRC push-notifications are easy.
Emoji are shared corporate wide, anyone can add them.
It's a really nice out of the box experience.
You can absolutely replicate it with enough setup and tweaking, but the free version of Slack will go a long way.
It's not "better" than email, it's different. Real time. It has a million and one service integrations, which IRC doesn't.
Point is, it's really not all that crazy to just pay a third party to manage all this stuff for you. You might imagine the alternative is a perfect IRC server with 100% uptime, but I doubt that would be the reality. If you're saying a service like this is too important to be run by a third party then you're going to have to invest a lot internally to come up with an alternative that's more reliable.
It's IRC. On modern hardware. It's practically infinite.
> it has a million and one service integrations, which IRC doesn't.
There and were are things way before Slack, eg https://loqi.me/
First "but it doesn't scale". Now "but integrations".
There will always be another "but", and there will always be a link on the internet showing that it's been done already.
"but shiny web gui" - https://thelounge.chat/
"but mobile app" - https://github.com/MCMrARM/revolution-irc
"but it's complicated" - https://www.mirc.com/ since 1995
IRC does have limits. Some. One is that to make it "modern" it needs a bouncer - that said znc is spectacularly simple to set up with it's web interface.
Integrations were never one of those limits. Just google IRC bots.
I think it's really sad how the world has migrated away from an open protocol to a closed one, but I remember IRC having some very real annoyances, well beyond stuff like integrations.
That's my entire point. It's death by a thousand cuts. Now I'm responsible for an IRC server, a web frontend, a mobile app and a bouncer.
The OP said "just run an IRC server". There is no "Slack or just an IRC server". There's "Slack or a number of connected, interdependent services you need to maintain". I'm not saying it can't be done, or that it isn't the right answer for some people, but it's also a lot more work (and thus, expense) than just paying for Slack. It absolutely makes sense that people do it.
When did this madness start?
Paying experts in their field (chat tools in this case) is most of the time the most reliable and cheapest variant. You must also include that Slack has plenty of features that are not available in IRC and mean the productivity is not as good.
30 years ago, efnet was (apparently) 35k users.
Get 2 sysadmins. 2 because of redundancy. Pay them to maintain your service.
"well funded sales team and corporate support". F that. Trust your employees.
Yes. Someone above have mentioned ibm using slack though. I think that's huge enough for the argument.
Do you stop trusting them then?
If you want to match what Slack offers you aren't just running an IRC server, you're going to have a small fleet of dependent services that, added up, just about match what Slack does.
Is it really that surprising that people just opt to pay for someone to deal with all of this for them?
[1] - https://thelounge.chat/
[2] - https://thelounge.chat/docs/configuration#ldap-support
Now, modern IRCds might have features I’ve never experienced on raw IRC... but if they don’t, I don’t suppose you’re suggesting that this server also run thousands of bouncer bots in order to provide this contextual history feature?
(Also, whence push notifications?)
I can set you up a resillient architecture and 3 services (communication, documentation, filesharing) in less than two weeks (let's say two weeks and a half because security concerns/limitations and all that stuff). Add one week and you will have mail server, VOIP communication, log conservation and monitoring, backup images on CEPH, vnc service for "easy" maintainance and a small documentation explaining the few action needed to maintain the monster for two years. Initial cost for a company: 15 to 25k$, + 1 hour and ~100$ a month.
Truth is I see many capable people buying into these as well because of the convenience - read: lazyness - factor. The "ain' nobody got time for that" effects everyone and the willingness to pay for services had skyrocketed in the past decade. This applies to blogging, to home servers, backup, anything. We're all guilty at some level.
- no audio / video
- no history
- no SSO / AD integration
- no rich features like links, pictures, upload docs ect ...
- easy bot integration with APIs, I can have Datadog pushing things to Slack ( alerts, graph... )
- no archiving channels on the go, Slack is very useful when you have an incident and you create a temp channel and then archive it when the incident is closed.
There is of course several other server implementations in progress but none are fully ready yet.
The features you're listing is server/client dependent.
There are products that are far better for audio and video communication than Slack. At my company, we use Zoom.
> no history
Can be done with a bouncer if you need it.
> no rich features like links, pictures, upload docs ect
Is there a reason why an external service can't be used? I can't upload a image here on Hacker News, but I can always include a link to it.
> easy bot integration with APIs
So I have to use the HTTP protocol to manage a bot rather than checking for certain key words in the message itself.
> no archiving channels on the go, Slack is very useful when you have an incident and you create a temp channel and then archive it when the incident is closed.
That depends on the settings. One thing that happens every so often at work is that people get trapped as the last remaining person in a channel because only an administrator can archive a channel. In IRC, that's not an issue.
"The Wheel of Time turns, and Ages come and pass, leaving memories that become legend. Legend fades to myth, and even myth is long forgotten when the Age that gave it birth comes again."
* push-notifications that just work
* client available for enough people
* copy / pasting images that just work (the'll never admit it, but animated gif is their killer feature)
* stable-enough infrastructure for something that costs you 15€/month/user not taken from your "salary" budget (often a different and substantially less taxed part)
* and their outages are funny (receiving messages twice is arguably better than not receiving message)
Yes, it's because it "just kinda works for what it costs, and you can forget about it most of the time and do your job."
This seems to be characterized as "laziness" by some people, a point I honestly don't understand (I though lazyness was a virtue in our circles.)
That being said, there are legitimate reasons to not use it:
* software licence
* data property
* technical implementation of the native clients
* general distrust of centralized solutions
* cost of having IM at all (see Newport's whole publishing career)
At least, it means there's an opportunity to create a company that handles decentralized user-friendly open-source standard-based data-respectful attention-preserving uber-reliable chat system. HN users would be a market.
For some definitions of "just" and "working". Even ignoring the business limitation, the UX of Slack search is meh. That said, I'm yet to see any of the modern web&mobile-oriented IMs that would have a decent search UX.
But yeah, your list is solid. They offer plenty of value, just mixed with plenty of negatives.
A big source of the complaints against Slack is usually the network effect - in context of both work and OSS communities, it's usually somebody else that makes the decision and imposes it on you, forcing you to run face-first into all the "legitimate reasons to not use it" you listed. Half of the pain would be gone if they opened up their service to third-party clients; I'd say the reason they didn't have as much opposition from people initially is because they run the bait-and-switch with IRC gateway.
For work, the decision of which IM tool is most likely going to be a corporate one, without much choice of solution. I guess the problem here is the same as for other corporate tools. (And people have strong feelings about corporate tools, but it seems there are less strong feelings about "using jira vs bugzilla" as there are "using slack vs irc". Or maybe there are as much strong feelings.)
On the other end, I get the troubles with the "network" effect for out-of-work communities (open source projects, clubs, etc...) Such communities can at least have "some" level of debate about which solution to use, and balance the pros and cons as they see fit. It's completely acceptable for an OSS community to discourage using such platforms - as any other non-free, centralized platforms. We're still not forced to use anything.
Then, there is the "club of non-power user" situation. I would be hard pressed to ask someone from my drama club to setup and maintain an IRC server (neither do I want to do it.)
Thanks for adding this dimension.
Follow-up: is there a "nice" provider of some slack-like solution (mattermost, etc...) that hosts servers for communities of non-techies for not too much money ?
- Search
- Scrollback/history available in the main client interface people who newly join a channel (extraordinarily important during incident response)
- Ability to link to a message in history
- Ability to handle multi-line messages in a vaguely competent matter
- Image uploads, snippets, etc.
- Users of a GUI tool / web interface are first-class, users of a command-line client are second-class; this is unlike IRC, and important when the goal is a single chat system for the entire company, not just engineering (and usually a subset of engineering at that)
- Client/UI expectation that you do not catch up on every message unless you want to (like IRC, but unlike email)
- Quality integrations that have already been written with the third-party apps that you're already using
IRC and email have both been time-tested, and have failed the test.
I have seen enough corporate mail servers going down, irc and jabber servers breaking etc.
Using those won't magically give you 24/7 with more than five nines uptime.
Key difference is: If slack goes down many companies are affected and it goes on the internet. If a company's mail setup goes down employees are a bit annoyed, but after a while it works again and nobody outside notices and it's no news on HN (unless it's a real major thing bringing a notable company completely down)
Edit: clarifying that I'm talking about my workplace, not a company I run.
Unfortunately, Slack does not do this very well. I'd really like to know if there's another service that _does_ do this well.
url:https://symphony.com/en-US
https://docs.mattermost.com/administration/compliance.html
Is there such a project, and if so does it have any traction in the real world?
(Oh and ditto for every damn thing Atlassian owns. Yikes.)
Have to say Mattermost's API docs are so much easier to read than Slack's.
https://zulipchat.com
https://github.com/zulip/zulip
Best wishes to all of the folks at Slack and those who depend on it.