I don't know how many of these Outlook (any version ever that I've touched) violates, but I do know that it AT LEAST violates #9 by default.
It irks me to no end, and un-checking that accursed options box in the IMAP folders dialog is something no user should ever need to do to find their "hidden folders".
Obviously there's some significant benefits to Outlook from an IT administration side (or it turns into a completely different beast when used with Exchange instead of IMAP) because as far as I can tell, it's otherwise awful on any number of axes.
Outlook is a good email client only when used against Exchange. It should never be used as a stand alone client because while it CAN work as a POP3 and IMAP client, its terrible at it.
Its not even a good email client when used against Exchange but its 'good enough'
Does #1 mean that there are IMAP servers where my phone and my desktop and my laptop (etc.) can't all read my mail simultaneously?
#7 makes me sad. Evolution supports a 'virtual trash folder' perfectly, but since I also delete mails in other clients, my mailstore is polluted by a malingering selection of folders with names like Trash, Wastebasket, Deleted Items and so on. The same remark applies to a 'virtual junk folder' and malingering Spam, Junk folders.
>Does #1 mean that there are IMAP servers where my phone and my desktop and my laptop (etc.) can't all read my mail simultaneously?
I think they are talking about one particular client instance. In other words, you should have some data structure where you can look up a connection by mailbox and use that existing connection rather than opening a new one.
> Does #1 mean that there are IMAP servers where my phone and my desktop and my laptop (etc.) can't all read my mail simultaneously?
[edited]
I believe that's the case. The ancient and venerable UW implementation of the ancient, venerable, and awful Unix "mbox" format from the UW server has a lot of these sorts of problems.
The UW server did not have this problem with some of the other formats. It suffered a lot from supporting a lot of different formats in borderline hostile conditions (i.e., users were assumed to have filesystem access to the server so no improvement to locking or caching was practical).
None of the current commercial implementations suffer from this, but I suppose I can say that the lowest common denominator is still pretty low.
I may be overstating this because what I'm describing would also outlaw message delivery, which I don't think is the case. I think one of the formats did have problems with trivial concurrent access but I was never a UW-imapd expert. I am sure that the mbox format has problems with EXPUNGE (removing messages flagged deleted) as well as many other issues.
I guess it's also worth reiterating that STATUSing a SELECTed mailbox (one that's open in another connection) is a pretty big waste of server resources, as someone else noted.
> Does #1 mean that there are IMAP servers where my phone and my desktop and my laptop (etc.) can't all read my mail simultaneously?
The short answer is that you can access from multiple device. The long answer is, it depends on the server datastore implementation. Namely, any server using mbox will probably lock the mbox file to the first connection, and hence allow one connection per mailbox (folder).
Most large scale servers won't use mbox, though. The problem is described, in the original article, from writing an IMAP client perspective. You can't assume _all_ servers will allow multiple connections, even if most do.
So I'm working on an app that uses IMAP. I need to know which emails I've already processed. Has anyone ever used the message ID header to identify an email as unique/seen? Or do you hash the email in its entirety? Or is there some other way to do it?
I guess what I'm still confused about is determining how to handle a UID validity change. So I'm afraid to use UIDs.
I wrote a console-based mail-client (think "mutt", but with Lua used for all scripting and user-interface work).
MessageID headers are often missing, even though that is "invalid". I've also got thousands of examples of messages with identical MessageID values. (Largely from automated systems that use some trivial/broken template to generate the emails.)
Message-ID is not unique per message, or per folder or user. The trivial counterexample is a message that goes through a mailing list may exist from the list and in a Sent folder with the same Message-ID but possibly different content, among other (less convincing) reasons.
A UIDVALIDITY change should blow your whole cache. In practice they are rare.
Thank you for saving me from the heartbreak of finding this out later. Seriously you just saved me a ton of time.
>A UIDVALIDITY change should blow your whole cache. In practice they are rare.
So even in a huge pile of mail, everything would have to be reprocessed. Since it's so rare, I'm definitely going to set up metrics to monitor when it does happen.
Do you have any insight on how people store the mail? Do they actually use the UID to reference it or is it used as more of a cursor?
Most clients want to use UID, and it’s a good key for the same reason a SQL row number is, and inconvenient for servers for the same reason consistent hashing schemes don’t have it. I’m not sure what you mean about a cursor. I’m sure clients use UIDs as cursors. They’re good for that.
How about adding a flag to the emails that are processed? It's fairly well supported.
I needed this when i was working on my engineering project, and added 'processed' flags for already processed items. This way i could query all unprocessed items with "UNSEEN UNKEYWORD <flag>".
This is a good plan, probably a notch better than using UIDs. You can use UIDs as a cursor to the server as a hint to speed up the operation if that’s what you meant in the other thread, and that’s a great thing to do.
UIDS are more than simple 'message numbers'; they are part of a tuple (Folder, UIDVALIDITY, UID). It may be easy to think of UIDVALIDITY as a kind of 'generation number': for a given folder, as long as the UIDVALIDITY has not changed/incremented, the UID-to-messege has to be the same per the spec.
Though recently the OBJECTID extension has been published:
IMAP was an okay protocol for its time, but in my experience it's basically mindbogglingly bad by current standards. The root of the reason why is the whole concept of "message sequence numbers", which as far as I know are extremely difficult to implement in a scalable and performant way.
Basically, all messages in a mailbox are numbered from 1 to N, and you can reference a message by number, and when a message is deleted all message numbers above it shift down by one. But since this could confuse clients that are sending inflight requests, you can't start translating from the new shifted-down-by-one numbers until you've told your clients about it, which could take a theoretically unbounded amount of time.
One implementation of this I've seen is to get a list of every UID in a mailbox at time of selection, and then update that as clients are informed of new or deleted mail. Needless to say, that's a bit problematic with short-lived or multiple connections.
So if your mailbox gets too big, too bad. This ends up complicating the client: probably influencing rules 1, 3, part of 5, and 8 listed here. (1) You can't have concurrent mailbox access because it might mess up MSNs reified to disk, (3) you can't have connections live too long because they passively eat up RAM with queued updates and a stored MSN list, (5) there are perfectly good UIDs to reference messages with that you have to learn alongside message sequence numbers, and (8) because opening a new session can be heavy.
It was bad even 20 years ago, I remember a discussion when we were trying to implement something coining the phrase "University of Washington brain damage". It seems to have been extremely tightly tied to the UW-IMAP implementations.
This phrase made me laugh, because UW imapd was largely written by Mark Crispin, who invented the IMAP protocol!
It seems that in about 2007, Crispin forked UW IMAP to create his own IMAP server, "Panda IMAP" [1], which at least one website claims to be "fully compliant" with the IMAP RFCs while UW IMAP is not. [2]
> Although I see their point about non-REST, I think there's still a case for having a REST flavour available even if it may not be the most efficient.
I was pretty disappointed when I found out how ugly are JMAP requests. I'd also like to see REST variant as messages and their parts should be easily mappable to resources. Too bad SMTP was not also included in JMAP. ActivityPub comes really close to SMTP over HTTP/JSON in my opinion.
> Basically, all messages in a mailbox are numbered from 1 to N, and you can reference a message by number, and when a message is deleted all message numbers above it shift down by one.
There is no reason technical why this shifting has to happen. If there are ten messages on session creation, and message UID 3 is deleted, you then have UIDs 1-2,4-10.
If the server wishes to shift messages over it can, but it must bump UIDVALIDITY, which tells clients that there is a 'new generation' of UIDs for the given folder. But as long as the server's index of UID-to-message is not lost, there's no reason why renumbering has to occur.
It's no different than having an AUTO_INCREMENT int column in MySQL: you just keep going up without bothering to fill in any 'holes'.
That said, folders and messages can now be assigned UUIDs:
GP is talking about message sequence numbers, not UIDs. Message sequence numbers are literally from 1 to N, and should you delete (expunge) a message, the sequence number of everything subsequent is decremented by 1 to keep 1 to N.
It is pretty much impossible to write a modern mail client relying on message sequence numbers, since there are too many ways the bookkeeping go wrong; so everyone uses UIDs (or UUIDs) nowadays wherever possible.
> Basically, all messages in a mailbox are numbered from 1 to N, and you can reference a message by number, and when a message is deleted all message numbers above it shift down by one. But since this could confuse clients that are sending inflight requests, you can't start translating from the new shifted-down-by-one numbers until you've told your clients about it, which could take a theoretically unbounded amount of time.
Message ids are monotonically increasing, but not necessarily consecutive. You may need to renumber when you hit the 32-bit ceiling, but that is much less frequent than what you describe. Worst case scenario, you can disconnect unresponsive clients when renumbering, clients behave well on disconnects.
This invalidates your conclusions.
> (1) You can't have concurrent mailbox access because it might mess up MSNs reified to disk
You can. Renumbering is not frequent.
> (3) you can't have connections live too long because they passively eat up RAM with queued updates and a stored MSN list
My dovecot IMAP processes use an average of 5MB reserved memory, of which 4MB is shared memory, so a server with 100GB memory can hold close to 100k connections.
> (8) because opening a new session can be heavy.
Monitoring tells me that on loaded production servers, session open + select + list + logout occurs comfortably in less than 100ms. That's not heavy.
Dovecot starts to fail on mailboxes with about a million messages. It's a limit I can live with. Users have learned to archive messages by date.
IMAP has two ways of numbering messages. It is important to be clear whether you are discussing message sequence numbers (which are dense, and compacted on EXPUNGE) or unique IDs (which are not compacted). GP was complaining about MSNs, your reply is about UIDs.
As fanf2 said, MSNs != UIDs. Also, I didn't mean to imply that MSNs are an insurmountable limitation for performance, they just complicate things a lot.
Basically, building a mail index for UID<->MSN translation that can serialize to disk, which would explain how it doesn't necessarily need absurd amounts of RAM and can select fast, probably at heavy cost when EXPUNGeing old messages. I'm sure there were worse implementations that did just maintain it all in RAM way back in the day, and still some that do (like the one I ran into).
The cost of deletions is heavy, and I wouldn't have done it this way, but note that where it really matters is if you have to do this repeatedly in a loop. But you don't have to do it repeatedly -- you can just delete a ton of messages at once, at the same cost as deleting a single once. As long as user perceptions go, the network latency is probably going to dominate shifting down even a million integers (< 2 ms on my machine), which I suspect most people don't even have.
> Basically, building a mail index for UID<->MSN translation that can serialize to disk, which would explain how it doesn't necessarily need absurd amounts of RAM and can select fast
That does explain why indexes can get corrupted and need to be cleared out, and why load balancing IMAP can get tricky.
This sounds like exactly the situation where you would want to use a persistent tree. For a log(n) overhead, you can keep the ordering of all the messages in your mailbox at any given point in time. Then you don't need to inform clients unless a new message is added that they might want to reference. Otherwise you just keep track of the state that client is operating on and use the old state to let you apply their actions to the new state.
I've dealt a lot with email along the way, trying to use it as an async transport layer for higher level messaging etc.
At one point I had wrappers around Python's IMAP/SMTP implementations that I excuted from Go, since at least Python had 20 years to make theirs work properly.
Any other implementation I tried failed somewhere along the way, especially as soon as you start moving messages around in folders or anything else beyond simply fetching messages.
SMTP is simpler from my experience, but then so is the work it performs.
I even played around with the idea of cooking my own to get out of this loop, but reading the specifications soon killed that idea. It would take a life time, probably be deprecated by the time, and still wouldn't work properly.
The new JSON-stuff honestly doesn't look much better from what I've seen. It seems to still be doing IMAP, only using JSON instead, which wasn't the problem. I'm not holding my breath about it becoming standard enough to depend on.
It's a mess, so much that I'm thinking it would probably be better to start over entirely.
JMAP fixes the big problems with IMAP: no MSN bookkeeping that no one uses anymore; no UIDVALIDITY issues; you can listen to events from multiple folders; etc.
Essentially, the single biggest issue with IMAP is that most clients nowadays maintain an offline shadow copy of the database, and so what they need to do is use IMAP to synchronize these two databases. And IMAP is a lousy synchronization protocol--most of your IDs are all ephemeral. Extensions to IMAP make it better, but you can't rely on servers implementing them (or clients using them!). Incidentally, that's also why most library implementations of IMAP suck: you can't build a good interface that works around server deficiencies without baking in the integration into your database model, and that heavy opinionation makes it hard to use the IMAP code outside of the full email application.
"Hello IMAP server. I'd like to deal with the message with this UID."
"The UIDVALIDITY has changed since then."
"Can you tell me the new UID from my old UID perhaps?"
"No."
(Okay, I last worked with IMAP back in the 90s. Has modern IMAP since added a persistent ID that will stay with a message?)
Though if your mail server loses the UID-to-message mapping index and has to rev the UIDVALIDITY, that's a more a problem with the implementation IMHO.
I'm an IMAP client and my user has clicked the delete-message button. Accordingly, I've sent a command to make the change and I'm waiting for the response from the server.
Oh no! Before the response came back, my connection to the server died. I don't know if my request arrived and was accepted.
Time passed and I've just now managed to reconnect. I need to know if the delete happened or not.
45 comments
[ 3.9 ms ] story [ 99.3 ms ] threadIt irks me to no end, and un-checking that accursed options box in the IMAP folders dialog is something no user should ever need to do to find their "hidden folders".
Its not even a good email client when used against Exchange but its 'good enough'
#7 makes me sad. Evolution supports a 'virtual trash folder' perfectly, but since I also delete mails in other clients, my mailstore is polluted by a malingering selection of folders with names like Trash, Wastebasket, Deleted Items and so on. The same remark applies to a 'virtual junk folder' and malingering Spam, Junk folders.
The 'All About Calendars' file is a wealth of interesting tidbits about time: https://www.washington.edu/imap/documentation/calendar.txt.h...
I think they are talking about one particular client instance. In other words, you should have some data structure where you can look up a connection by mailbox and use that existing connection rather than opening a new one.
[edited]
I believe that's the case. The ancient and venerable UW implementation of the ancient, venerable, and awful Unix "mbox" format from the UW server has a lot of these sorts of problems.
The UW server did not have this problem with some of the other formats. It suffered a lot from supporting a lot of different formats in borderline hostile conditions (i.e., users were assumed to have filesystem access to the server so no improvement to locking or caching was practical).
None of the current commercial implementations suffer from this, but I suppose I can say that the lowest common denominator is still pretty low.
I may be overstating this because what I'm describing would also outlaw message delivery, which I don't think is the case. I think one of the formats did have problems with trivial concurrent access but I was never a UW-imapd expert. I am sure that the mbox format has problems with EXPUNGE (removing messages flagged deleted) as well as many other issues.
I guess it's also worth reiterating that STATUSing a SELECTed mailbox (one that's open in another connection) is a pretty big waste of server resources, as someone else noted.
The short answer is that you can access from multiple device. The long answer is, it depends on the server datastore implementation. Namely, any server using mbox will probably lock the mbox file to the first connection, and hence allow one connection per mailbox (folder).
Most large scale servers won't use mbox, though. The problem is described, in the original article, from writing an IMAP client perspective. You can't assume _all_ servers will allow multiple connections, even if most do.
So I'm working on an app that uses IMAP. I need to know which emails I've already processed. Has anyone ever used the message ID header to identify an email as unique/seen? Or do you hash the email in its entirety? Or is there some other way to do it?
I guess what I'm still confused about is determining how to handle a UID validity change. So I'm afraid to use UIDs.
MessageID headers are often missing, even though that is "invalid". I've also got thousands of examples of messages with identical MessageID values. (Largely from automated systems that use some trivial/broken template to generate the emails.)
A UIDVALIDITY change should blow your whole cache. In practice they are rare.
Thank you for saving me from the heartbreak of finding this out later. Seriously you just saved me a ton of time.
>A UIDVALIDITY change should blow your whole cache. In practice they are rare.
So even in a huge pile of mail, everything would have to be reprocessed. Since it's so rare, I'm definitely going to set up metrics to monitor when it does happen. Do you have any insight on how people store the mail? Do they actually use the UID to reference it or is it used as more of a cursor?
I needed this when i was working on my engineering project, and added 'processed' flags for already processed items. This way i could query all unprocessed items with "UNSEEN UNKEYWORD <flag>".
* https://tools.ietf.org/html/rfc3501#section-2.3.2
UIDS are more than simple 'message numbers'; they are part of a tuple (Folder, UIDVALIDITY, UID). It may be easy to think of UIDVALIDITY as a kind of 'generation number': for a given folder, as long as the UIDVALIDITY has not changed/incremented, the UID-to-messege has to be the same per the spec.
Though recently the OBJECTID extension has been published:
* https://tools.ietf.org/html/rfc8474
Basically, all messages in a mailbox are numbered from 1 to N, and you can reference a message by number, and when a message is deleted all message numbers above it shift down by one. But since this could confuse clients that are sending inflight requests, you can't start translating from the new shifted-down-by-one numbers until you've told your clients about it, which could take a theoretically unbounded amount of time.
One implementation of this I've seen is to get a list of every UID in a mailbox at time of selection, and then update that as clients are informed of new or deleted mail. Needless to say, that's a bit problematic with short-lived or multiple connections.
So if your mailbox gets too big, too bad. This ends up complicating the client: probably influencing rules 1, 3, part of 5, and 8 listed here. (1) You can't have concurrent mailbox access because it might mess up MSNs reified to disk, (3) you can't have connections live too long because they passively eat up RAM with queued updates and a stored MSN list, (5) there are perfectly good UIDs to reference messages with that you have to learn alongside message sequence numbers, and (8) because opening a new session can be heavy.
The need for a better protocol is why JMAP exists: https://jmap.io/spec-core.html
Although I see their point about non-REST, I think there's still a case for having a REST flavour available even if it may not be the most efficient.
I wonder if anyone's tried mail delivery over git repo yet.
This phrase made me laugh, because UW imapd was largely written by Mark Crispin, who invented the IMAP protocol!
It seems that in about 2007, Crispin forked UW IMAP to create his own IMAP server, "Panda IMAP" [1], which at least one website claims to be "fully compliant" with the IMAP RFCs while UW IMAP is not. [2]
[1] https://github.com/jonabbey/panda-imap
[2] https://imapwiki.org/ImapTest/ServerStatus
I was pretty disappointed when I found out how ugly are JMAP requests. I'd also like to see REST variant as messages and their parts should be easily mappable to resources. Too bad SMTP was not also included in JMAP. ActivityPub comes really close to SMTP over HTTP/JSON in my opinion.
There is no reason technical why this shifting has to happen. If there are ten messages on session creation, and message UID 3 is deleted, you then have UIDs 1-2,4-10.
If the server wishes to shift messages over it can, but it must bump UIDVALIDITY, which tells clients that there is a 'new generation' of UIDs for the given folder. But as long as the server's index of UID-to-message is not lost, there's no reason why renumbering has to occur.
It's no different than having an AUTO_INCREMENT int column in MySQL: you just keep going up without bothering to fill in any 'holes'.
That said, folders and messages can now be assigned UUIDs:
* https://tools.ietf.org/html/rfc8474
It is pretty much impossible to write a modern mail client relying on message sequence numbers, since there are too many ways the bookkeeping go wrong; so everyone uses UIDs (or UUIDs) nowadays wherever possible.
Message ids are monotonically increasing, but not necessarily consecutive. You may need to renumber when you hit the 32-bit ceiling, but that is much less frequent than what you describe. Worst case scenario, you can disconnect unresponsive clients when renumbering, clients behave well on disconnects.
This invalidates your conclusions.
> (1) You can't have concurrent mailbox access because it might mess up MSNs reified to disk
You can. Renumbering is not frequent.
> (3) you can't have connections live too long because they passively eat up RAM with queued updates and a stored MSN list
My dovecot IMAP processes use an average of 5MB reserved memory, of which 4MB is shared memory, so a server with 100GB memory can hold close to 100k connections.
> (8) because opening a new session can be heavy.
Monitoring tells me that on loaded production servers, session open + select + list + logout occurs comfortably in less than 100ms. That's not heavy.
Dovecot starts to fail on mailboxes with about a million messages. It's a limit I can live with. Users have learned to archive messages by date.
I'm not familiar with Dovecot's code or very good at reading C, but I think part of how they handle it is here: https://github.com/dovecot/core/blob/81b5b188c478ec36bea8bda...
Basically, building a mail index for UID<->MSN translation that can serialize to disk, which would explain how it doesn't necessarily need absurd amounts of RAM and can select fast, probably at heavy cost when EXPUNGeing old messages. I'm sure there were worse implementations that did just maintain it all in RAM way back in the day, and still some that do (like the one I ran into).
That does explain why indexes can get corrupted and need to be cleared out, and why load balancing IMAP can get tricky.
They're somewhat obscure though.
At one point I had wrappers around Python's IMAP/SMTP implementations that I excuted from Go, since at least Python had 20 years to make theirs work properly.
Any other implementation I tried failed somewhere along the way, especially as soon as you start moving messages around in folders or anything else beyond simply fetching messages.
SMTP is simpler from my experience, but then so is the work it performs.
I even played around with the idea of cooking my own to get out of this loop, but reading the specifications soon killed that idea. It would take a life time, probably be deprecated by the time, and still wouldn't work properly.
The new JSON-stuff honestly doesn't look much better from what I've seen. It seems to still be doing IMAP, only using JSON instead, which wasn't the problem. I'm not holding my breath about it becoming standard enough to depend on.
It's a mess, so much that I'm thinking it would probably be better to start over entirely.
regarding JMAP that should not be the case. From their FAQ the intent is to make it easier to use already common workflows.
Essentially, the single biggest issue with IMAP is that most clients nowadays maintain an offline shadow copy of the database, and so what they need to do is use IMAP to synchronize these two databases. And IMAP is a lousy synchronization protocol--most of your IDs are all ephemeral. Extensions to IMAP make it better, but you can't rely on servers implementing them (or clients using them!). Incidentally, that's also why most library implementations of IMAP suck: you can't build a good interface that works around server deficiencies without baking in the integration into your database model, and that heavy opinionation makes it hard to use the IMAP code outside of the full email application.
* https://tools.ietf.org/html/rfc8474
Though if your mail server loses the UID-to-message mapping index and has to rev the UIDVALIDITY, that's a more a problem with the implementation IMHO.
Oh no! Before the response came back, my connection to the server died. I don't know if my request arrived and was accepted.
Time passed and I've just now managed to reconnect. I need to know if the delete happened or not.