I work at Nylas (Gmail/MS/IMAP email & calendar connectivity APIs) on the product side and I really wish something like this received adoption.
Between the insufficiencies of IMAP for server-side use and the disparities between the "winners" of the space, the lowest common denominator of what's supported for email is abysmally small.
I don’t know what I would do without sieve, I use it to manage my mailing list subscriptions, eg move to folders, mark emails from specific chatty users as read, etc.
It automates a lot of manual cruft and I’m endlessly thankful for it.
I do wish it was more widely supported, and I try to do my part to encourage adoption (I maintain several sieve related packages for Arch Linux).
In case people are curious on using Sieve, the official(?) site[0] offers a list of libraries implementing the language in C, Java, Ruby, etc. There is also a list of e-mail clients[1] that seem to support Sieve as a filtering language. (I didn't know Roundcube supported this!)
Has anyone written a tool to scan your email headers, and maybe show UI to help craft Sieve rules? Sort by top senders or mailing-lists, and create/extend rules to file them appropriately.
I'll have to look into that; right now I use isync and run maildrop on everything in my inbox; upside is it works for all my imap accounts; downside is that there is a delay, causing other clients to sometimes see unfiltered mail.
Except Sieve rules in Fastmail either work or do not, and it is non-deterministic. A few notoriously persistent problems I have been facing with them for years:
– A notorious spammer that has been spamming me for over a decade that the Fastmail spam filter does not trap[*]. They are easy to screen out via Sieve rules using the "From" header. Except, for 1 out 5 spam emails from them the Sieve rules do not fire, and the spam gets into the inbox.
– Simple email filing into different folders using From/Subject headers. It mostly works except for just a handful senders whose emails Fastmail Sieve do not fire upon no matter how I modify them. They just. do. not. work. I am talking about the most basic From/Subject filing rules.
[*] Spam filtering in Fastmail has been another pet peeve of mine. It is average at best, and mediocre on average. Some spam emails that slip through need to be sent into the spam folder (that has the training enabled) for more than 10x times before*
(I work for Fastmail). That sounds very surprising. Please email support@fastmail.com with the details and we'll be happy to look into it for you. I don't believe we've ever had an instance of Sieve not working correctly. (We have had plenty of reports that boiled down to people making errors in their Sieve scripts; we recommend most people use our UI instead to make their rules, to allow you to preview results and help ensure the syntax is correct.)
I am Fastmail customer so I have access to their Sieve functionality but decided against it. For me the fundamental limitation is that Sieve looks at email only once, when email arrives. I wanted to do the things like: archive email after 3 days being in my inbox, or older than my most recent read email (which would mean I scanned the inbox and the summary was enough) etc. I ended up implementing a Python script that manages my inbox connecting to IMAP server on cron schedule. It also sends me telegram notifications from follow ups on starred threads.
Yeah, "keep sales newsletters for two weeks, then delete forever" has been a thing I've wanted for a long time. Nothing seems to make that easy.
I've been leaning towards local processing on an mbox folder... that would also let me build up multiple Bayesian filters to try categorizing my other mail - why does almost everyone only auto-categorize spam? Google kinda broke out of that with their "social" and "important" semi-categories, but it'd be pretty easy to build in local clients, yet I've never seen it.
Interesting indeed. What kind of classification would you have in mind? Separate out different types of senders, e.g. friend groups, or professional clients? Or more the personal email, transactional email, newsletter?
For bayesian junk filtering to work, you train with messages marked either as junk or nonjunk. I suppose you could train based on positive tag, and assume the message has the opposite signal if it didn't get that tag. My historic email is only classified for (non)junk. If I would start more classifications, I would have to ignore the existing messages.
Fyi, I'm working on https://github.com/mjl-/mox, which also includes a bayesian filter. This sounds implementable.
Gmail's categories are a pretty reasonable starting place: primary, social, updates, forums, promotions.
All reasonably identifiable by word choices, and often at extremely different urgency / priority levels.
But I would personally also have an "interesting" category for newsletters and whatnot that are more likely to get a full read (newsblur has a weak version of this for my RSS, it's nice). And an "urgent" category that notifies more visibly than others. "Recruiters" is also periodically useful - a bunch of cold emails filled with buzzwords, should be identifiable a mile away.
But underneath it all it's mostly that it confuses me that there's so little ability to experiment here. Shared always-updating spam filters makes sense, but there's a lot of similar things people could try unique to their needs that are just totally unsupported. I don't want better global categorization (gmail), I want personalization. Email is weirdly non-personal in the vast majority of systems, and I suspect it's part of why people burn out on it and switch to a never-ending churn of apps that are a better fit for one piece or another.
I think there is an IMAP extension that runs Sieve scripts on IMAP actions (like when you move a message to a mailbox). Probably does not help with the use cases you mention though...
this was part of cyrus imap from cmu, which iirc was the first big imap server to support server side indexes. (i think they were bdbs, iirc)
believe it or not, but many hugely popular applications from 20+ years ago didn't even implement indexing. in this case if you had too much mail your imap client would just time out.
well through the mid-2000s you even had to use a third party plugin for outlook called "lookout" if you wanted searches to not take minutes.
that sounds right. and yes, very good stuff! i remember thinking at the time "hah. this is built with real computer science!". and well, at the time, it was the only thing that worked for large mail spools.
i also think it was the first support for sasl(?) encryption upgrades for legacy text/tcp mail protocols
also, fun sidebar: indices vs. indexes, both are apparently valid english... but it seems computer people have adopted the latter almost exclusively. never noticed it before...
SASL came out as a generalization of the IMAP AUTHENTICATE mechanism. CMU wanted Kerberos to work and it had been done as something of a one-off in telnet, and initially in IMAP. There were a couple companion protocols proposed for IMAP (stuff like contacts) that the same group was working on, and they wanted to leverage the same mechanism. From there, might as well do the same thing for POP and SMTP, etc. So they started working on a library (which Cyrus IMAP didn’t use for at least a while, sorry my fault).
Kind of funny that SASL is the most durable piece of the effort.
(I doubt this is entirely accurate. I wasn’t there for a lot of it.)
I always want to have a programmable mailserver, but not just for filtering. I hope it can also do things like calling API then react according to the respond etc. It's like a eBPF program but for mailservers.
I'm not sure Sieve is a language that enables such usage. I know maybe it's a domain-specific language, but since it's a language capable of programming the server, I would rather expect more from it than "it's just yet another way to configure your mail filter".
Sieve seems not powerful enough. I had this though experiment a few days ago: What if you could run code in a sandbox on the server (javascript, webassembly?) with api's (provided by the server) to access mailboxes/messages and to take actions (move to folder, change message flags), and perhaps some networking (e.g. dns blocklist)?
I use procmail, fetchmail and nmh via mh-e. Never used imap and the only email service account I have is with lavabit. Each email is one file so the only filtering is which folder the file goes to. Procmail does that and mairix finds them. I just have to write and read, maybe answer or delete.
24 comments
[ 0.22 ms ] story [ 868 ms ] threadBetween the insufficiencies of IMAP for server-side use and the disparities between the "winners" of the space, the lowest common denominator of what's supported for email is abysmally small.
It automates a lot of manual cruft and I’m endlessly thankful for it.
I do wish it was more widely supported, and I try to do my part to encourage adoption (I maintain several sieve related packages for Arch Linux).
---
[0] http://sieve.info/libraries
[1] http://sieve.info/clients
https://www.fastmail.help/hc/en-us/articles/360060591373-How...
– A notorious spammer that has been spamming me for over a decade that the Fastmail spam filter does not trap[*]. They are easy to screen out via Sieve rules using the "From" header. Except, for 1 out 5 spam emails from them the Sieve rules do not fire, and the spam gets into the inbox.
– Simple email filing into different folders using From/Subject headers. It mostly works except for just a handful senders whose emails Fastmail Sieve do not fire upon no matter how I modify them. They just. do. not. work. I am talking about the most basic From/Subject filing rules.
[*] Spam filtering in Fastmail has been another pet peeve of mine. It is average at best, and mediocre on average. Some spam emails that slip through need to be sent into the spam folder (that has the training enabled) for more than 10x times before*
I've been leaning towards local processing on an mbox folder... that would also let me build up multiple Bayesian filters to try categorizing my other mail - why does almost everyone only auto-categorize spam? Google kinda broke out of that with their "social" and "important" semi-categories, but it'd be pretty easy to build in local clients, yet I've never seen it.
For bayesian junk filtering to work, you train with messages marked either as junk or nonjunk. I suppose you could train based on positive tag, and assume the message has the opposite signal if it didn't get that tag. My historic email is only classified for (non)junk. If I would start more classifications, I would have to ignore the existing messages.
Fyi, I'm working on https://github.com/mjl-/mox, which also includes a bayesian filter. This sounds implementable.
All reasonably identifiable by word choices, and often at extremely different urgency / priority levels.
But I would personally also have an "interesting" category for newsletters and whatnot that are more likely to get a full read (newsblur has a weak version of this for my RSS, it's nice). And an "urgent" category that notifies more visibly than others. "Recruiters" is also periodically useful - a bunch of cold emails filled with buzzwords, should be identifiable a mile away.
But underneath it all it's mostly that it confuses me that there's so little ability to experiment here. Shared always-updating spam filters makes sense, but there's a lot of similar things people could try unique to their needs that are just totally unsupported. I don't want better global categorization (gmail), I want personalization. Email is weirdly non-personal in the vast majority of systems, and I suspect it's part of why people burn out on it and switch to a never-ending churn of apps that are a better fit for one piece or another.
believe it or not, but many hugely popular applications from 20+ years ago didn't even implement indexing. in this case if you had too much mail your imap client would just time out.
well through the mid-2000s you even had to use a third party plugin for outlook called "lookout" if you wanted searches to not take minutes.
Folders (mailboxes in proper IMAP lingo) had hand-built indexes. Good stuff. Credit to jgm, the original author.
i also think it was the first support for sasl(?) encryption upgrades for legacy text/tcp mail protocols
also, fun sidebar: indices vs. indexes, both are apparently valid english... but it seems computer people have adopted the latter almost exclusively. never noticed it before...
Kind of funny that SASL is the most durable piece of the effort.
(I doubt this is entirely accurate. I wasn’t there for a lot of it.)
I'm not sure Sieve is a language that enables such usage. I know maybe it's a domain-specific language, but since it's a language capable of programming the server, I would rather expect more from it than "it's just yet another way to configure your mail filter".
What you can do though is edit your rule as a seive filter and just add multiple rules in there and it works fine.