It doesn't make sense for large projects to move to other systems. Control is lost of the history, which automatically disqualifies any platform. Using services others than e-mail will also entail a revamp and move to a different service every few years.
Email is a far lower cost and more reliable medium most other methods.
I've always wondered why newsgroups aren't used instead of email. They pretty much have the same advantages of email and they're far better in terms of searching archives compared to email. Web interfaces for searching email list archives are painful to use. If it wasn't for services like gmane (and later public-inbox), it would be very difficult to view archives of those lists.
Newsgroups could be an exception to what I wrote. That's a good point. But, newsgroups don't use the special protocols that I meant, which may not be long-lived enough.
One of the best things about email is that, with the right client, you can view discussions as hierarchical trees. That means you can have multiple simultaneous discussions about a topic asynchronously and not lose track of anything.
GitHub and the like only work that well when there's maybe one or two things being discussed at once, because there's just a single linear stream of comments.
On the other hand, the unthreaded approach can help keep the discussion focused and tight. There are few things worse than setting out to get some work done and having to wade through a bunch of comments to find the meat. In the age of Bugzilla, if you wanted to discuss something related, you'd open up a new bug. You'd string things together with a tracking (meta) bug if you liked. This Bugzilla-style workflow, however, doesn't accommodate the stuff that you wouldn't feel you should open a bug for. That's a feature.
Like so many other things, though, this only works if you enforce it. It's pretty horrifying what the GitHub generation tends to think passes for acceptable on bugtrackers, for example.
> On the other hand, the unthreaded approach can help keep the discussion focused and tight.
Or that earlier posts get lost because no one actually sees them. With threaded discussion, if one subthread has gone off-topic, you can always collapse or kill that subthread.
1. You shouldn't expect that it would "always hold true". I really don't know why you would, because I did not write that it would always hold true. In fact, what I did write is the opposite: a tacit acknowledgement that it's not a hard and fast guarantee. I mean, I specifically made comments like, "Like so many other things, this only works if you enforce it". It's right there. You can go back and read it. Thing is, I even edited my comment (before you read it and responded) from the original; I edited in the words "can help" so that it says "the unthreaded approach can help keep the discussion focused". You want to know something perverse? While doing this, I stopped and asked myself whether I was hedging too much—writing too defensively. Apparently the answer to that is a big nope; I was still not defensive enough.
2. Regardless of everything above, the page you linked to is supposed to prove what? It's a misdirected pull request, and the very first comment is a message to the requestor telling them so. I don't know what you expect past that; even after the message explaining that GitHub is not the correct place to make a pull request and that it will not be moving forward, you seem to be trying to score the discussion against a rubric of how effective the discussion was at staying focused. Do you not see what's wrong with that?
3. I made an a priori remark about how lousy the GitHub community is in my very first comment. I mean, my exact words were calling out what's found on GitHub as "pretty horrifying". If there's some way to be clearer regarding how I feel about GitHub, I don't know what it is. GitHub is terrible. And yet, despite this, despite everything above, your choice example is to link to a GitHub thread where this exact thing is on display, in an attempt to prove me wrong. Or something. I don't get it.
This'll be my last comment here. I think I'm gonna go away for a while.
My reply wasn't meant to be disparaging in any way. It was to point out that your assertion that unthreaded discussions tend to be more focused and tight compared to threaded discussions isn't necessarily true.
Regarding the example I cited. it's an unthreaded discussion that has been going on for several years and, if you bother to read through it, it's a mixture of troll comments along with some informative comments/discussion. Had it been a properly threaded discussion, it would have been much easier to follow the informative thread branches and filter out the uninformative ones (where the classification of what falls under one category or the other is up to the person reading it).
But, because it's not threaded, you're effectively forced to wade through all the troll posts to find the informative comments (or miss the informative comments entirely if you're not willing to do that). The lack of threading did not change the "social aspect" of the discussion. It would have been the same either way given the topic, but one way of rendering it would have made it much easier to follow (and possibly spawned more informative discusson about the topic compared to what actually took place).
Getting back to your points:
> the very first comment is a message to the requestor telling them so. [...] after the message explaining that GitHub is not the correct place to make a pull request and that it will not be moving forward
If your assertion about unthreaded discussions was true, then it should have ended soon after that comment (maybe after a couple of comments from the person who submitted the pull request after Linus' comment). That obviously wasn't the case and nothing kept it in check.
> the unthreaded approach can help keep the discussion focused
Based on what I've read online over the years (both threaded and non-threaded discussions), it doesn't seem to be the case. The only way I think this could be settled would be to come up with a representative sample of discussions under both formats and how well they stay on the original topic compared to each other. It would be interesting to see the results of such an analysis.
This is one of the things that web-based forums have yet to get right. Email (and NNTP news) clients from 20 or 30 years ago are far superior in this respect, because they can intelligently deal with threading and folding. These features alone makes large conversations much easier to deal with than on web-forums.
To add to that, email (and NNTP news) clients even from 20 or 30 years ago have other powerful features that web forums have yet to catch up on:
- kill files[1] (which you can use to filter out unwanted articles/mails based on content or metadata such as subject, user, etc)
- scoring
- user-configurable anti-spam filtering or other "intelligent" filtering (such as bayesian filtering not just for spam/ham, but for interesting/unintersting content)
- tagging not just on a site-wide level but at the client level so each user can tag messages/articles the way they make sense to them
- other advanced filtering and scripting based on any of the above
Web-based forums are just incredibly primitive compared to this many-decade-old technology.
Exactly. Gnus is a marvelous client because it has really good NNTP and threading support (silencing and automatic scoring) that allows you to cope with gigantic volumes of email.
It's a bit sad the code is messy and it's a tad slow compared to other options.
Notmuch for Emacs has so much potential to replace Gnus. The interface is incredibly fast and friendly (a classic tree view and a threaded view that IMHO surpasses Gmail). Plus super quick search.
I think it could become a killer application for Emacs (the other two now are org-mode and Magit IMHO).
However, there's no good tooling to keep Notmuch in two-way sync with maildirs right now. Everything works based on tags, which is incredibly flexible, but there should be some tooling to implement logic that allows moving mails around based on their tags, so that the remote IMAP store gets tidy.
People usually rely on scripts for simple use cases, which is fine albeit a bit hacky. Some DSL that resembles Sieve, but implemented in ELisp would be cool.
I have spent a decade running mutt, a year running Gnus and a few months running mu4e before switching to notmuch on Emacs.
Mu4e is fine, but I found the interface slightly slow and awkward compared to notmuch. Notmuch's interface is really really good and quick.
Mu4e doesn't ignore underlying folders, which is cool if you have a very simple workflow. It's also superior to notmuch in the sense tags are stored in message headers, so they can easily be sync'ed across clients.
Notmuch has potential, but also a bunch of limitations and issues compared to Gnus.
Notmuch doesn't have scoring, it doesn't have BBDB integration, it has no way to get arbitrary message headers from elisp, it only indexes a handful of headers.
I've heard it is possible to use notmuch as a backend for Gnus, to get the best of both worlds.
As for two-way sync, some people have come up ways of doing that, using something like Sieve.
For me two-way sync is not super critical, as I've come to view the webmail interface of my mail provider as just a backup to use in case my laptop dies or something. In all other cases, I'll just use notmuch, for which local tags are enough.
I ran Gnus with notmuch backend (nnir-notmuch) for many months. It's not very fast as Gnus is not very quick on Maildirs.
Scoring could be quite easy to implement with auto-tagging. Tags are a unifying concept in email. You can implement any custom user logic with tags. What notmuch is missing is a bit of tooling to do this. It'd be cool if this tooling had access to any arbitrary header, as you have pointed out.
I feel I don't need BBDB integration when I have autocompletion based on the email database. That said, there's bbdb-notmuch.el, and I assume other options as message-mode is really a Gnus thing.
I think basic two-way sync is almost always needed. Otherwise you end up with a huge inbox and even if you don't care at all about the remote IMAP store, mbsync calls will start slowing down.
Autocompletion from the notmuch database is fine when you don't have many addresses in there. But when you are subscribed to a bunch of large mailing lists and you try to search your notmuch database for "John" you're going to get a thousand completion candidates.
If instead you search just your BBDB database, which only has the contacts you actually correspond with, you're going to get just the useful and relevant ones... and you'll always have the notmuch database to search on as a backup, in case the contact you want to to correspond with is not in BBDB yet.
> This is one of the things that web-based forums have yet to get right. Email (and NNTP news) clients from 20 or 30 years ago are far superior in this respect, because they can intelligently deal with threading and folding.
If you buy Reddit gold which makes Reddit to highlight posts that are new since you last time viewed the discussion, then Reddit becomes almost as good as NNTP clients were 20 years ago.
The problem is we don't have the users from 20-30 years ago either.
Now, even though I and my email client still use threading, much of my incoming corpus does not. Instead, I see a mixture of users who break every thread (no In-Reply-To headers, ever), users who cannot comprehend subject metadata versus first line of message, and users who use the most recent few screens of activity as an address book, always replying to the end of a thread with a completely unrelated and new topic.
i'm implementing a webmail app as we speak (from the bottom up). my immediate thought was "wtf are 'conversations', gmail? of course i want threads!".
then i started actually handling the raw email messages. i got threads working pretty well but noticed frequent rendering issues and digging into the cause. my conclusion is, email threading is mostly a pipe dream; it can only be done properly if you have a view of the entire conversation (as is the case with centralized forums/message stores like HN).
the 'References:' header only represents the message ids of any particular branch up to the root. if you try turning a number of messages with the same root into proper threads, you quickly discover missing branches, partial branches/holes, missing leaf nodes, etc. If your knowledge of a thread starts with a message forwarded to you, no thread is possible (even though one clearly exists as indicated by the References chain in the header!). side-conversations that happen (selective replies, cc/bcc), additional fwd/reply chains are all invisible to you. sadly, an incomplete thread view is an extremely common thing (at least in my inbox/sent of ~15k messages).
the client can only thread the actual raw messages it has; those simply quoted in any replies or forwards (which established the Reference chains) don't count. the only commonality you have is knowing the root message id and the timestamp for ORDER BY. everything in between, you may or may not have to reconstruct a partial view of the thread...and each user will have a different view of that thread based on which parts they've been party to.
the threading that i did get working was nice, but not as useful as a bulletproof solution would have been. unfortunately, i don't believe that's possible with email.
you can prune/flatten these containers but then the deeper messages that you do have are not direct replies to their ancestors in the flattened/pruned version of the tree.
i didnt say that threading did not work. it does. but it is imperfect at best and can be misleading at worst.
in my case i was constructing an MPTT structure (Nested Set) for optimal thread fetching and simply leaving in the dummy message placeholders that i didnt have a copy of.
i plan to experiment further, but there arent really any earth-shaking revelations in that post. it's all fairly obvious.
GitLab has discussions, which thread topics within an issue/MR/commit. In the context of a MR, individual messages or whole threads can be marked as resolved. This is an invaluable tool to get out of linear quote hell.
Actually good craftsman knows how to use the tools available to him to do the job in the most effective way. Choice is a luxury that is not always present.
Just because you say that doesn't make it true, though. Sometimes a tool just is worse, even if you become skilled at it.
I can try to work around Microsoft Word's UI by learning the keyboard shortcuts, but I can't do anything against Internet Explorer taking its sweet time loading webpages. No skill in the world will make that faster.
Well, that's the point. If you can get your job done with IE (I can only guess what kind of job is that), you just do it without pointless clicking, pressing F1 button or anything else that does not belong to the path of success. Talks about "it's worse", "omg it's so slow", "why not firefox/chrome/whatever" are meaningless when you are in strict corporate environment with only IE allowed or even possible. True professionals do not pay much attention to the tools. They pick the best when they have choice or find the way to use what's available, because craftsmanship is not about the tools.
For 99% of folks outside the HN echo chamber, email works fine. Spam is a problem, but I've never heard or read of anyone who thinks email needs to go the way of the dodo except on this site.
I have no idea why you're downvoted. Email is great especially because it's an open standard. I don't understand why people expect Linux kernel development to adopt some kind of proprietary platform for communications. I think it's a terrible idea.
btw "echo chamber" is generally perceived as disparaging, so it's not weird people wouldn't be impressed by that. and feigning victimhood after antagonizing people is pretty gauche.
It was intended to be disparaging: one of my major beefs with this community is the massive culture of groupthink and aggressive self reinforcement/adulation.
Yes. Because it wasn't trashtalk.
HN in generel seems (from what I observe) like a place that uses fancy apps for communication instead of emails. So if someone claims that HN doesn't like emails and get downvoted for that, they got triggered.
Sometimes one needs to adapt to the outer world :) I use Whatsapp even though I hate instant messaging and phone calls (I like communicating at my own pace when not face-to-face, and I dislike any sorts of smalltalk on the phone or messaging), because communication is a two-way street, and sometimes conforming is easier. I disable last-online and blue ticks in Whatsapp so it's more e-mail like, definitely improves the experience.
Eh. If you were to measure the percentage of personal communication that users of this forum conduct over email as opposed to other mediums (Whatsapp, FB Messenger, iMessage, etc.), I bet you'd find it indexes way higher than the general population.
Without notifications, which I, of course, do not use, email is the least intrusive and least disrupting way someone can contact me. I simply check my email when I feel like it, which is about two times daily, morning and sometime late evening.
Then, even when I look at my email, I don't have to respond. I don't have to do anything. And if there's an email I want to reply to, I won't necessarily do so right now. Maybe tomorrow. Maybe never.
The best part is the record of all communications and the fact that you have a searchable archive. Man, I wish I could search all my other conversations with people. In fact, I think I should just start recording everything, every day, all the time, and convert the audio to text.
It is also the pinnacle of interoperability. Nowadays, two apps in the appstore can't send messages between each other, because reasons.
With email you have compatibility between so many vendors, devices, programs, programming languages ... everything is compatible. It is THE standard.
For one to one or one to many types of communication, I agree. Email, in my opinion, breaks down when there's a discussion amongst multiple people. For instance, you have to make sure that everyone in the discussion addresses are added by your MUA before clicking the reply button (or using the equivalent keyboard command). The other problem is that if you need to include a new person in an existing discussion, they won't have any of the previous messages to refer to (other than what happens to be quoted in the email they receive).
Newsgroups, on the other hand, address the issues I mentioned and are just as good as email in communication IMO.
"In fact, I think I should just start recording everything, every day, all the time, and convert the audio to text."
About a decade or perhaps 15 years ago in Birmingham UK, the Ikon gallery had a multi-artist exhibition of a conceptual nature. one of the artists had used a micro-cassette recorder to record every word he uttered over a week and then had the tapes transcribed by audio-typists. The results where exhibited as an impressively high stack of A4 sheets. Unfortunately, they were, apart from a few sample sheets on the wall, piled in a glass case thus preventing viewers from sampling the text. I was taking bets on the sheets that were hidden from view being blank. The artist also exhibited the micro-cassette recorder and tapes used.
Google has nothing to say about the exhibit, but it has to be said my memory is a bit short on key words.
I've been working on a side project to bring the best of both worlds. It aims to provide git repository hosting, issue tracking, continuous integration, and mailing lists to organize efforts. But I'll also be providing web tools for submitting and reviewing patches that provides a workflow familiar to GitHub users, but backed by mailing lists.
If anyone is interested in getting involved in the closed alpha for this software, my email's in my profile. Hit me up!
I find Discourse absolutely horrendous. Discourse mailing-list mode is a tack-on at best, and the formatting you get in mail notifications basically require you to be online to visit the link. And the gamification is also a huge put-off.
Web forums made us forgot how far we came with NNTP years ago. Even the shittiest NNTP could do subthread splitting, hiding and scoring. Handling hundreds of messages per day was easy. The convenience of doing-it-all under one powerful client was massive.
You can still do that with mailing lists, as long as it's not as badly managed as forums-with-a-wanna-be-mailing-list-mode addon as Discourse is.
>, kernel developers still use email because it is faster than any of the alternatives. Over the course of the last year, the project accepted about eight changes per hour — every hour — from over 4,000 developers sponsored by over 400 companies
Compare with Google's ~1875-per-hour from 2015[1]:
>, and Google’s 25,000 engineers make about 45,000 commits (changes) to the repository each day.
Obviously, we can't extrapolate that Google should switch to an internal SMTP project management system layered on top of Perforce because "email is faster than any of the alternatives". Same for other companies with high commit rates to their source repos like Microsoft and Facebook.
Since Linus & Linux community never did an A-vs-B experiment of email vs open-source-JIRA-clone as a project workflow, I think we can only safely say that email works best for a disparate group of volunteers with different employers. Email is the lowest common denominator that everyone could agree to use.
You couldn't get 4000 opinionated developers to agree to use an open-source JIRA-clone or MS-Team-Foundation-clone or some other ALM[2]-in-the-cloud.
Therefore, listing advantages of email clients such as "threaded tree views" etc is actually hiding the underlying reason email-as-workflow works for Linux. Email isn't about "throughput"; it's "acceptance".
Its not comparing the same thing I think, commits to a repo and diffs sent over email, and even if it was, its not really the same thing anyway...
Kernel development seems to progress fine, I'm pretty sure if there was a problem, Linus would have made some effort to change it, just like the whole source control thing that ended up becoming Git.
I'm not claiming that there's a bottleneck in kernel commits. (I'd guess human code reviews checking quality is the ultimate bottleneck.)
I'm saying that using emails as "building blocks to reconstruct an pseudo ALM" instead of using an actual special-purpose ALM -- is the only realistic workflow that would be accepted by the disparate Linux community who don't work for a single employer. (If you're a single company employing all 4000 of Linux kernel programmers, you could impose an ALM tool be mandatory for code reviews, assigning work, tagging bug reports, etc.)
Discussing positive bullet points of email ... even though each bullet point is true actually obscures the underlying reason the devs won't use an ALM.
>; what about developers who refuse to use the email-based process?
I guess I don't understand your question. In Linux, the email-as-the-workflow[1] has been agreed upon from Linux Torvald and his lieutenants all the way down.
I suppose that if a one refused to use email-based-workflow, they'd have to find a Linux kernel maintainer to be a champion for their proposed changes and submit them on their behalf. (Is this a common scenario? I don't know.)
If a rebel Linux kernel programmer insisted on staging changes for review in Gerritt/JIRA/github/etc, he's going to be isolated in an island because he's not part of the agreed-upon workflow: emails.
If Linus declared that everyone must use, say, Gitlab from now on, some Linux developers might leave. But developers who don't use email may never start working on Linux. That's why it cuts both ways: either option alienates some developers.
I got in this same discussion with gregkh on Reddit, and Torvalds in person. Linux development is much slower than many large software engineering firms, because the gatekeeper model has many more integrators and reviewers as a proportion of the members than the "pull request"/ team code review model
Acceptance is certainly part of it, but 8 changes per hour is 8-10 email threads per hour, which is just not that hard to track as an individual, thus no need to change anything. Right now at work on a large (200ish) team, I get cc'd on about the same amount of patches.
isn't google's situation a monorepo composed of many distinct projects? there doesn't seem much utility in the comparison.
> Email isn't about "throughput"; it's "acceptance".
this seems to be the similar, in the other direction. using (or not) an alm in an organization isn't about "throughput", it's about "accepting" a company's policy assertions.
>using (or not) an alm in an organization isn't about "throughput", it's about "accepting" a company's policy assertions.
I don't think that equating it from the reverse direction works. Here's why:
Using the article's count of ~4000 developers, let's consider all software companies that employ 4000+ developers as part of a "natural experiment"[1]. This includes companies and organizations like MS, Facebook, NASA, SAP, Oracle, etc.
Each company "votes" in the marketplace of ideas to coordinate software development for a complex project. One company might use JIRA. Another uses MS Team Foundation. Another uses FogCreek. Maybe another employer might use SMTP emails similar to Linux kernel development.
Since companies have every incentive to develop software with cost-efficiency and velocity, they will want to use the best workflow possible.
Since no single software company I know of uses SMTP emails as their ALM workflow, it either means that (1) companies are wasting money & losing iteration velocity by avoiding email -- or -- (2) the Linux kernel community has special circumstances that makes using a true unified ALM unrealistic. I believe that special circumstance is: 4000 programmers work for different employers instead of a single employer.
> Since companies have every incentive to develop software with cost-efficiency and velocity, they will want to use the best workflow possible.
in the abstract these are generally viewed as or presumed to be positive, but the pressures and objectives of a business can be rather complex and other things will often compete for finite resources.
having worked in a number of large organizations, the waste and inefficiencies one can find is absolutely staggering.
outside the sales pitches, it's also not clear how an alm interacts with notions of 'cost-efficienty' and 'velocity'. introducing process and bureaucracy isn't always a win.
perhaps the presumed predilection of large organizations for alms can be found in alms addressing (or seeming to address) other needs, pressures or requirements.
>having worked in a number of large organizations, the waste and inefficiencies one can find is absolutely staggering.
Sure, I agree this is true in a general sense (Dilbert cartoons, Peter Principle, etc). However, for things like choosing technology to help with market superiority, companies (especially software engineering companies) will eventually gravitate to something better.
We could argue that companies are "inefficient" for not embracing remote work-from-home instead of open offices, etc. (general inefficiencies). But, even badly run companies don't insist on using inferior floppy drives instead of USB drives or wireless network file transfers. Yes, a law firm might limp along with an 15-year old version of Microsoft Word 2000, but a software engineering company will not. (technology inefficiencies)
> introducing process and bureaucracy isn't always a win.
The Linux dev community also has process & bureaucracy.
See my previous link for the strict and precise formatting rules to submit kernel patches in emails. That's process. Just because it's not in a web form with fields doesn't mean it's not "process". They're just using emails as the workflow for it. (Linus has famously chastised contributors for not following "the rules".) Process & bureaucracy is not bad thing -- its effort just has to match the scope of work being delivered. (E.g. using JIRA for a "hello world" toy program is overkill.) The Linux kernel community seems to have converged on the level of bureaucracy that works for them.
It's interesting that even young YCombinator companies with much less than 4000 employees also don't stay with SMTP email-as-ALM as they get bigger. If email-as-ALM was a better software dev philosophy, there would be a huge arbitrage opportunity to outcompete everybody. Instead, the email-as-ALM fits Linux kernel devs' specific circumstances.
having worked in large and small orgs, what I can say is, sure, a large org has many more inefficient workflows and latency than a smaller one. However it also has much more drive than the N independent smaller orgs it needs to accomplish the same scope. As a whole, a bunch of 10 person startups is generally more inefficient to accomplish something a big enterprise can do. (and of course, to accomplish what a startup can do, a big org won't be able to do for ages. Formula one vs huge train.) My point being, dont compare a big organization with a small team. Compare a small team within this org with an independant team. The indeêndent team wins, but not by far. With several competing small startup, the thing is, you cannot see the inefficiencies.
>Or, alternatively, ALMs provide [...] iteration velocity [...] that Linux developers don't care about.
I agree and that's why the author quoting Greg Kroah-Hartman about commit velocity (throughput of "8 changes per hour") in his article is actually a distraction from answering the actual question as stated in the article's title. The real reason for kernel devs use of emails isn't the throughput.
Let than sink in. It's not only 4000 opinionated devs. It's, potentially, 400 completely different organizations; 400 management layers on top; 400 different priority sets; devs in every time zone; etc. Google is just 1 company (a dozen, if you want to break down Alphabet).
It's amazing that they manage to get 8 changes an hour, and still be one of the most successful open source, and software project in general.
14000 developers from 1300 companies have contributed, since 2005.
From 3.19 to 4.7, an average of 4600 lines of code was added daily.
Over 80% of all kernel development is done by developers paid for their work.
A small number of companies are responsible for a large portion of the total changes. These being, from biggest to lowest contributor: Intel (12.9%), Red Hat (8,0%), Linaro (4%), Samsung, SUSE, IBM, Google (2%), AMD, Texas Instruments, ARM, Oracle.
Contributions from unpaid developers have been in decline for many years. 14.6% in 2012, 7.7% in 2016.
I split my time between Gerrit-based and email-based projects. While I appreciate Gerrit for its benefits (like the excellent CI integration), I instinctively tend to gravitate towards my email-based workflow that is blazing fast, allows me to work completely offline (once OfflineIMAP fetches email in the morning), and lets me process tons of email / patches.
The said offline workflow (setup with: Mutt, OfflineIMAP, Postfix for local mail queueing, and Notmuch for fast indexing / searching / tagging) gives me unalloyed joy compared to the clickety-clicky, RSI-inducing pain that is Gerrit.
The problems listed in the below email, by Dan Berrangé, about Gerrit are still a bloody pain in the neck for me:
[I learnt 2 days ago (can't find the URL to it; typing from my phone) that Gerrit now allows you to respond via email (strangely the announcement only mentioned the dreaded HTML email). I'm a bit cautious to say "sounds promising" without trying it yet.]
Doubt they'd prefer any of those. None of those are open source, every single one is a GUI app (which has a history of limiting access to the plumbing that can be provided), and some of those don't even run on Linux.
I think a crucial point is missed: email does not have a single point of failure.
If you go through a website, a server downtime halts the development. If the mailing list crashes, patches can still be sent and some discussions kind of continue until it is resolved.
Individual email servers may fail, but it is easy to work around it. Many people have several email and other channels through which to be reached.
Also, a very important point that Linus explained was at the core of some design decisions for git: it works offline. You have your email archive offline so if you take a plan, are at a conference in a place with no internet (does still exist, sadly), are on the road, you can still go through your archives and the patches you have to examine are still available.
96 comments
[ 35.3 ms ] story [ 493 ms ] threadEmail is a far lower cost and more reliable medium most other methods.
GitHub and the like only work that well when there's maybe one or two things being discussed at once, because there's just a single linear stream of comments.
Like so many other things, though, this only works if you enforce it. It's pretty horrifying what the GitHub generation tends to think passes for acceptable on bugtrackers, for example.
Or that earlier posts get lost because no one actually sees them. With threaded discussion, if one subthread has gone off-topic, you can always collapse or kill that subthread.
And your point doesn't always hold true. For example https://github.com/torvalds/linux/pull/17
This is a social problem. I deliberately added a disclaimer about the tendency for things to turn to crap if you tolerate it.
I don't know what this is supposed to mean:
> And your point doesn't always hold true. For example https://github.com/torvalds/linux/pull/17
> I don't know what this is supposed to mean
What I linked to is an example of a discussion that didn't stay focused or tight despite the fact that it's not threaded.
1. You shouldn't expect that it would "always hold true". I really don't know why you would, because I did not write that it would always hold true. In fact, what I did write is the opposite: a tacit acknowledgement that it's not a hard and fast guarantee. I mean, I specifically made comments like, "Like so many other things, this only works if you enforce it". It's right there. You can go back and read it. Thing is, I even edited my comment (before you read it and responded) from the original; I edited in the words "can help" so that it says "the unthreaded approach can help keep the discussion focused". You want to know something perverse? While doing this, I stopped and asked myself whether I was hedging too much—writing too defensively. Apparently the answer to that is a big nope; I was still not defensive enough.
2. Regardless of everything above, the page you linked to is supposed to prove what? It's a misdirected pull request, and the very first comment is a message to the requestor telling them so. I don't know what you expect past that; even after the message explaining that GitHub is not the correct place to make a pull request and that it will not be moving forward, you seem to be trying to score the discussion against a rubric of how effective the discussion was at staying focused. Do you not see what's wrong with that?
3. I made an a priori remark about how lousy the GitHub community is in my very first comment. I mean, my exact words were calling out what's found on GitHub as "pretty horrifying". If there's some way to be clearer regarding how I feel about GitHub, I don't know what it is. GitHub is terrible. And yet, despite this, despite everything above, your choice example is to link to a GitHub thread where this exact thing is on display, in an attempt to prove me wrong. Or something. I don't get it.
This'll be my last comment here. I think I'm gonna go away for a while.
Regarding the example I cited. it's an unthreaded discussion that has been going on for several years and, if you bother to read through it, it's a mixture of troll comments along with some informative comments/discussion. Had it been a properly threaded discussion, it would have been much easier to follow the informative thread branches and filter out the uninformative ones (where the classification of what falls under one category or the other is up to the person reading it).
But, because it's not threaded, you're effectively forced to wade through all the troll posts to find the informative comments (or miss the informative comments entirely if you're not willing to do that). The lack of threading did not change the "social aspect" of the discussion. It would have been the same either way given the topic, but one way of rendering it would have made it much easier to follow (and possibly spawned more informative discusson about the topic compared to what actually took place).
Getting back to your points:
> the very first comment is a message to the requestor telling them so. [...] after the message explaining that GitHub is not the correct place to make a pull request and that it will not be moving forward
If your assertion about unthreaded discussions was true, then it should have ended soon after that comment (maybe after a couple of comments from the person who submitted the pull request after Linus' comment). That obviously wasn't the case and nothing kept it in check.
> the unthreaded approach can help keep the discussion focused
Based on what I've read online over the years (both threaded and non-threaded discussions), it doesn't seem to be the case. The only way I think this could be settled would be to come up with a representative sample of discussions under both formats and how well they stay on the original topic compared to each other. It would be interesting to see the results of such an analysis.
To add to that, email (and NNTP news) clients even from 20 or 30 years ago have other powerful features that web forums have yet to catch up on:
- kill files[1] (which you can use to filter out unwanted articles/mails based on content or metadata such as subject, user, etc)
- scoring
- user-configurable anti-spam filtering or other "intelligent" filtering (such as bayesian filtering not just for spam/ham, but for interesting/unintersting content)
- tagging not just on a site-wide level but at the client level so each user can tag messages/articles the way they make sense to them
- other advanced filtering and scripting based on any of the above
Web-based forums are just incredibly primitive compared to this many-decade-old technology.
[1] - https://wikivisually.com/wiki/Kill_file
[0]: https://eul.im
It's a bit sad the code is messy and it's a tad slow compared to other options.
Notmuch for Emacs has so much potential to replace Gnus. The interface is incredibly fast and friendly (a classic tree view and a threaded view that IMHO surpasses Gmail). Plus super quick search.
I think it could become a killer application for Emacs (the other two now are org-mode and Magit IMHO).
However, there's no good tooling to keep Notmuch in two-way sync with maildirs right now. Everything works based on tags, which is incredibly flexible, but there should be some tooling to implement logic that allows moving mails around based on their tags, so that the remote IMAP store gets tidy.
People usually rely on scripts for simple use cases, which is fine albeit a bit hacky. Some DSL that resembles Sieve, but implemented in ELisp would be cool.
Mu4e is fine, but I found the interface slightly slow and awkward compared to notmuch. Notmuch's interface is really really good and quick.
Mu4e doesn't ignore underlying folders, which is cool if you have a very simple workflow. It's also superior to notmuch in the sense tags are stored in message headers, so they can easily be sync'ed across clients.
Notmuch doesn't have scoring, it doesn't have BBDB integration, it has no way to get arbitrary message headers from elisp, it only indexes a handful of headers.
I've heard it is possible to use notmuch as a backend for Gnus, to get the best of both worlds.
As for two-way sync, some people have come up ways of doing that, using something like Sieve.
For me two-way sync is not super critical, as I've come to view the webmail interface of my mail provider as just a backup to use in case my laptop dies or something. In all other cases, I'll just use notmuch, for which local tags are enough.
Scoring could be quite easy to implement with auto-tagging. Tags are a unifying concept in email. You can implement any custom user logic with tags. What notmuch is missing is a bit of tooling to do this. It'd be cool if this tooling had access to any arbitrary header, as you have pointed out.
I feel I don't need BBDB integration when I have autocompletion based on the email database. That said, there's bbdb-notmuch.el, and I assume other options as message-mode is really a Gnus thing.
I think basic two-way sync is almost always needed. Otherwise you end up with a huge inbox and even if you don't care at all about the remote IMAP store, mbsync calls will start slowing down.
If instead you search just your BBDB database, which only has the contacts you actually correspond with, you're going to get just the useful and relevant ones... and you'll always have the notmuch database to search on as a backup, in case the contact you want to to correspond with is not in BBDB yet.
Very sorry to hear that Gnus with a notmuch backend is slow, as that's what I was considering trying next.
I really like Notmuch's interface. I think limitations are perhaps not that hard to patch on Elisp unless you have a complicated setup.
I'm trying to implement a DSL to coordinate tag <-> maildir sync on top of a few macros.
If you buy Reddit gold which makes Reddit to highlight posts that are new since you last time viewed the discussion, then Reddit becomes almost as good as NNTP clients were 20 years ago.
Now, even though I and my email client still use threading, much of my incoming corpus does not. Instead, I see a mixture of users who break every thread (no In-Reply-To headers, ever), users who cannot comprehend subject metadata versus first line of message, and users who use the most recent few screens of activity as an address book, always replying to the end of a thread with a completely unrelated and new topic.
Or maybe give up and have some sort of manual threading overlay where I could rearrange and add my subject tags for messages..
then i started actually handling the raw email messages. i got threads working pretty well but noticed frequent rendering issues and digging into the cause. my conclusion is, email threading is mostly a pipe dream; it can only be done properly if you have a view of the entire conversation (as is the case with centralized forums/message stores like HN).
the 'References:' header only represents the message ids of any particular branch up to the root. if you try turning a number of messages with the same root into proper threads, you quickly discover missing branches, partial branches/holes, missing leaf nodes, etc. If your knowledge of a thread starts with a message forwarded to you, no thread is possible (even though one clearly exists as indicated by the References chain in the header!). side-conversations that happen (selective replies, cc/bcc), additional fwd/reply chains are all invisible to you. sadly, an incomplete thread view is an extremely common thing (at least in my inbox/sent of ~15k messages).
the client can only thread the actual raw messages it has; those simply quoted in any replies or forwards (which established the Reference chains) don't count. the only commonality you have is knowing the root message id and the timestamp for ORDER BY. everything in between, you may or may not have to reconstruct a partial view of the thread...and each user will have a different view of that thread based on which parts they've been party to.
the threading that i did get working was nice, but not as useful as a bulletproof solution would have been. unfortunately, i don't believe that's possible with email.
i now understand why gmail does flat convos.
Them why has it been working so well for me since I started using email?
Maybe take a look at this for inspiration: https://fkref.com/?RbC_-NEE
the problem is with this step:
4. Prune empty containers.
you can prune/flatten these containers but then the deeper messages that you do have are not direct replies to their ancestors in the flattened/pruned version of the tree.
i didnt say that threading did not work. it does. but it is imperfect at best and can be misleading at worst.
in my case i was constructing an MPTT structure (Nested Set) for optimal thread fetching and simply leaving in the dummy message placeholders that i didnt have a copy of.
i plan to experiment further, but there arent really any earth-shaking revelations in that post. it's all fairly obvious.
> A poor craftsman famously complains about his tools, Greg said, but a good craftsman knows how to choose excellent tools.
Unless their company forces tools on them. :(
I can try to work around Microsoft Word's UI by learning the keyboard shortcuts, but I can't do anything against Internet Explorer taking its sweet time loading webpages. No skill in the world will make that faster.
and thus your thesis of 'HN doesn't like to be reminded' falls to the simpler explanation of 'trashtalk usually gets downvoted'.
I prefer to see it as the down votes reaffirming that I am correct.
Never mind that "text messaging" in say Japan has been email based since virtually forever, since the SMS standard had issues with Japanese script.
Without notifications, which I, of course, do not use, email is the least intrusive and least disrupting way someone can contact me. I simply check my email when I feel like it, which is about two times daily, morning and sometime late evening.
Then, even when I look at my email, I don't have to respond. I don't have to do anything. And if there's an email I want to reply to, I won't necessarily do so right now. Maybe tomorrow. Maybe never.
The best part is the record of all communications and the fact that you have a searchable archive. Man, I wish I could search all my other conversations with people. In fact, I think I should just start recording everything, every day, all the time, and convert the audio to text.
For one to one or one to many types of communication, I agree. Email, in my opinion, breaks down when there's a discussion amongst multiple people. For instance, you have to make sure that everyone in the discussion addresses are added by your MUA before clicking the reply button (or using the equivalent keyboard command). The other problem is that if you need to include a new person in an existing discussion, they won't have any of the previous messages to refer to (other than what happens to be quoted in the email they receive).
Newsgroups, on the other hand, address the issues I mentioned and are just as good as email in communication IMO.
About a decade or perhaps 15 years ago in Birmingham UK, the Ikon gallery had a multi-artist exhibition of a conceptual nature. one of the artists had used a micro-cassette recorder to record every word he uttered over a week and then had the tapes transcribed by audio-typists. The results where exhibited as an impressively high stack of A4 sheets. Unfortunately, they were, apart from a few sample sheets on the wall, piled in a glass case thus preventing viewers from sampling the text. I was taking bets on the sheets that were hidden from view being blank. The artist also exhibited the micro-cassette recorder and tapes used.
Google has nothing to say about the exhibit, but it has to be said my memory is a bit short on key words.
A sibling post mentioned Ted Nelson.
If anyone is interested in getting involved in the closed alpha for this software, my email's in my profile. Hit me up!
[0]: https://github.com/google/git-appraise
Web forums made us forgot how far we came with NNTP years ago. Even the shittiest NNTP could do subthread splitting, hiding and scoring. Handling hundreds of messages per day was easy. The convenience of doing-it-all under one powerful client was massive.
You can still do that with mailing lists, as long as it's not as badly managed as forums-with-a-wanna-be-mailing-list-mode addon as Discourse is.
Compare with Google's ~1875-per-hour from 2015[1]:
>, and Google’s 25,000 engineers make about 45,000 commits (changes) to the repository each day.
Obviously, we can't extrapolate that Google should switch to an internal SMTP project management system layered on top of Perforce because "email is faster than any of the alternatives". Same for other companies with high commit rates to their source repos like Microsoft and Facebook.
Since Linus & Linux community never did an A-vs-B experiment of email vs open-source-JIRA-clone as a project workflow, I think we can only safely say that email works best for a disparate group of volunteers with different employers. Email is the lowest common denominator that everyone could agree to use.
You couldn't get 4000 opinionated developers to agree to use an open-source JIRA-clone or MS-Team-Foundation-clone or some other ALM[2]-in-the-cloud.
Therefore, listing advantages of email clients such as "threaded tree views" etc is actually hiding the underlying reason email-as-workflow works for Linux. Email isn't about "throughput"; it's "acceptance".
[1] https://www.wired.com/2015/09/google-2-billion-lines-codeand...
[2] https://en.wikipedia.org/wiki/Application_lifecycle_manageme...
Kernel development seems to progress fine, I'm pretty sure if there was a problem, Linus would have made some effort to change it, just like the whole source control thing that ended up becoming Git.
I'm not claiming that there's a bottleneck in kernel commits. (I'd guess human code reviews checking quality is the ultimate bottleneck.)
I'm saying that using emails as "building blocks to reconstruct an pseudo ALM" instead of using an actual special-purpose ALM -- is the only realistic workflow that would be accepted by the disparate Linux community who don't work for a single employer. (If you're a single company employing all 4000 of Linux kernel programmers, you could impose an ALM tool be mandatory for code reviews, assigning work, tagging bug reports, etc.)
Discussing positive bullet points of email ... even though each bullet point is true actually obscures the underlying reason the devs won't use an ALM.
I guess I don't understand your question. In Linux, the email-as-the-workflow[1] has been agreed upon from Linux Torvald and his lieutenants all the way down.
I suppose that if a one refused to use email-based-workflow, they'd have to find a Linux kernel maintainer to be a champion for their proposed changes and submit them on their behalf. (Is this a common scenario? I don't know.)
If a rebel Linux kernel programmer insisted on staging changes for review in Gerritt/JIRA/github/etc, he's going to be isolated in an island because he's not part of the agreed-upon workflow: emails.
[1] https://github.com/torvalds/linux/blob/master/Documentation/...
Acceptance is certainly part of it, but 8 changes per hour is 8-10 email threads per hour, which is just not that hard to track as an individual, thus no need to change anything. Right now at work on a large (200ish) team, I get cc'd on about the same amount of patches.
isn't google's situation a monorepo composed of many distinct projects? there doesn't seem much utility in the comparison.
> Email isn't about "throughput"; it's "acceptance".
this seems to be the similar, in the other direction. using (or not) an alm in an organization isn't about "throughput", it's about "accepting" a company's policy assertions.
I don't think that equating it from the reverse direction works. Here's why:
Using the article's count of ~4000 developers, let's consider all software companies that employ 4000+ developers as part of a "natural experiment"[1]. This includes companies and organizations like MS, Facebook, NASA, SAP, Oracle, etc.
Each company "votes" in the marketplace of ideas to coordinate software development for a complex project. One company might use JIRA. Another uses MS Team Foundation. Another uses FogCreek. Maybe another employer might use SMTP emails similar to Linux kernel development.
Since companies have every incentive to develop software with cost-efficiency and velocity, they will want to use the best workflow possible.
Since no single software company I know of uses SMTP emails as their ALM workflow, it either means that (1) companies are wasting money & losing iteration velocity by avoiding email -- or -- (2) the Linux kernel community has special circumstances that makes using a true unified ALM unrealistic. I believe that special circumstance is: 4000 programmers work for different employers instead of a single employer.
[1] https://en.wikipedia.org/wiki/Natural_experiment
in the abstract these are generally viewed as or presumed to be positive, but the pressures and objectives of a business can be rather complex and other things will often compete for finite resources.
having worked in a number of large organizations, the waste and inefficiencies one can find is absolutely staggering.
outside the sales pitches, it's also not clear how an alm interacts with notions of 'cost-efficienty' and 'velocity'. introducing process and bureaucracy isn't always a win.
perhaps the presumed predilection of large organizations for alms can be found in alms addressing (or seeming to address) other needs, pressures or requirements.
Sure, I agree this is true in a general sense (Dilbert cartoons, Peter Principle, etc). However, for things like choosing technology to help with market superiority, companies (especially software engineering companies) will eventually gravitate to something better.
We could argue that companies are "inefficient" for not embracing remote work-from-home instead of open offices, etc. (general inefficiencies). But, even badly run companies don't insist on using inferior floppy drives instead of USB drives or wireless network file transfers. Yes, a law firm might limp along with an 15-year old version of Microsoft Word 2000, but a software engineering company will not. (technology inefficiencies)
> introducing process and bureaucracy isn't always a win.
The Linux dev community also has process & bureaucracy. See my previous link for the strict and precise formatting rules to submit kernel patches in emails. That's process. Just because it's not in a web form with fields doesn't mean it's not "process". They're just using emails as the workflow for it. (Linus has famously chastised contributors for not following "the rules".) Process & bureaucracy is not bad thing -- its effort just has to match the scope of work being delivered. (E.g. using JIRA for a "hello world" toy program is overkill.) The Linux kernel community seems to have converged on the level of bureaucracy that works for them.
It's interesting that even young YCombinator companies with much less than 4000 employees also don't stay with SMTP email-as-ALM as they get bigger. If email-as-ALM was a better software dev philosophy, there would be a huge arbitrage opportunity to outcompete everybody. Instead, the email-as-ALM fits Linux kernel devs' specific circumstances.
I agree and that's why the author quoting Greg Kroah-Hartman about commit velocity (throughput of "8 changes per hour") in his article is actually a distraction from answering the actual question as stated in the article's title. The real reason for kernel devs use of emails isn't the throughput.
Let than sink in. It's not only 4000 opinionated devs. It's, potentially, 400 completely different organizations; 400 management layers on top; 400 different priority sets; devs in every time zone; etc. Google is just 1 company (a dozen, if you want to break down Alphabet).
It's amazing that they manage to get 8 changes an hour, and still be one of the most successful open source, and software project in general.
Why changing the system?
From 3.19 to 4.7, an average of 4600 lines of code was added daily.
Over 80% of all kernel development is done by developers paid for their work.
A small number of companies are responsible for a large portion of the total changes. These being, from biggest to lowest contributor: Intel (12.9%), Red Hat (8,0%), Linaro (4%), Samsung, SUSE, IBM, Google (2%), AMD, Texas Instruments, ARM, Oracle.
Contributions from unpaid developers have been in decline for many years. 14.6% in 2012, 7.7% in 2016.
The problem with a ALM is it is a global lock. It is enormously disruptive if it is unavailable or even slow.
The said offline workflow (setup with: Mutt, OfflineIMAP, Postfix for local mail queueing, and Notmuch for fast indexing / searching / tagging) gives me unalloyed joy compared to the clickety-clicky, RSI-inducing pain that is Gerrit.
The problems listed in the below email, by Dan Berrangé, about Gerrit are still a bloody pain in the neck for me:
https://lists.nongnu.org/archive/html/qemu-devel/2014-02/msg...
[I learnt 2 days ago (can't find the URL to it; typing from my phone) that Gerrit now allows you to respond via email (strangely the announcement only mentioned the dreaded HTML email). I'm a bit cautious to say "sounds promising" without trying it yet.]
If you go through a website, a server downtime halts the development. If the mailing list crashes, patches can still be sent and some discussions kind of continue until it is resolved.
Individual email servers may fail, but it is easy to work around it. Many people have several email and other channels through which to be reached.
Also, a very important point that Linus explained was at the core of some design decisions for git: it works offline. You have your email archive offline so if you take a plan, are at a conference in a place with no internet (does still exist, sadly), are on the road, you can still go through your archives and the patches you have to examine are still available.