HN Explorer

Ask HN: Best solution for local mail storage in a modern email client?

4 points by kitsunesoba ↗ HN
I am planning to write a mail client as a hobby project. Such a project is very large in scope and requires some premeditation for each part. Retrieval of mail and interaction with mail servers is largely taken care of; there are plenty of libraries for this, but a piece that is appears more unclear is local mail storage.

At first, something universally supported like mbox and to a lesser extent, maildir seems like the obvious answer, but these both seem to come with serious tradeoffs. Using mbox means putting all of your eggs in one basket (single file per mailbox), making it easy to corrupt, and neither is particularly friendly to advanced filtering and search functions. Both also get gradually less usable with every message stored, and maildir strikes me as unnecessarily I/O heavy. Both can be made searchable/filterable by keeping a separate, manually curated index, but that adds in another moving part to maintain and keep from breaking and is potentially a huge source of trouble.

One potential option would be to keep a local relational database (SQLite? Postgres?) with indexed metadata and references to individual messages on disk, with each message stored exactly as it would be as part of an mbox file, except all by itself. This should allow for fast searching/filtering (up to millions of messages) while also making it easy to export into standard mbox for use with other clients (just concat each message file into the resulting mbox file). Does this sound or am I completely off base? Is there a better option I should consider first?

3 comments

[ 3.8 ms ] story [ 19.6 ms ] thread
stevekemp
I wrote a mail client, console-based, with integrated Lua-based scripting.

I had to resolve the same problem you mention here. Me? I chose to support Maildir. Exclusively. With nothing else. It kills the client for some users, who want IMAP, etc, but for me it works well.

http://lumail.org/

I think with the right abstractions you can go far, but using Maildir to get started isn't a bad choice. SQLite will suffer if you have many many many messages. I know that having handled large email volumes in the past.

Someone1234
If I was starting an email client today I would store all of the messages in boring ZIP archives, with all of the metadata stored in an SQLite database (which can actually grow pretty huge with version 3 and above). Then use something like Lucene and SQL to retrieve, sort, filter, and search records.

Postgres would be great, since it has a full text engine built within, but I don't know much about running a Postgres instance within a single piece of desktop software (rather than actually having to launch a Postgres process, and then connect to that).

atmosx
PostGress is too much and SQLite3 might be to little. You might wanna check out other options.

What is the problem with Maildir/ ? I/O heavy?? I don't think that modern computers & filesystems will struggle with maildir.

Please add GnuPG support. Also you could add mail folder encryption (e.g. having the emails encrypted in the system, key required to unencrypt them when mail client starts... key could be saved in the keychain for lazy users).