Gmail takeouts come in an arbitrarily-ordered mbox file; I wanted something a bit more backup friendly so I created a small tool for that purpose and wrote about it.
I switched to Gmail in 2007 or so. I used to have a gzipped mbox of my previous emails, dating back to maybe 1996 or 1997 when I got my first email account. This file was lost at some point, and I'm really sad about it. In some ways, it's like losing years and years of a journal, conversations I had with people, how I thought about the world at that age, etc. It's a huge loss to me.
About OP's tool, I also back up my Google account to an external disk periodically. Gmail is ~8 GB so it's manageable. But Google Photos is a pain. They recently removed most of the useful APIs, so AFAIK the only way to backup is via Takeout. It's terrible. Pictures in multiple albums are included as copies every time, so I had to make a script to find duplicates and replace them with symlinks. Just downloading the whole thing is a PITA (multiple 50 GB zip files). I get that Google has little incentive to make this better, in fact they might have an incentive to make it as inconvenient as possible, but I really wish they made it easier.
"Five years ago" was 2020. What you're asking is, "Have you, at some point in 2025, needed an email from 2019, 2018, etc. (i.e. from some time before that)?"
The answer: Yes, of course. (And I don't understand why anyone other than, say, a university undergrad or someone younger should find that answer surprising.)
Despite your down votes I'm completely in agreement. Emails and chat are transient and expire. Unless I need to keep it for, like you say, a good reason, then it's deleted.
> if you want to back this file up regularly with something like restic, then you will quickly end up in a world of pain: since new mails are not even appended to the end of the file, each cycle of takeout-then-backup essentially produces a new giant file.
As I'm sure the author is aware, Restic will do hash-based chunking so that similar files can be efficiently be backed up.
How similar are two successive Takeout mboxes?
If the order of messages within an mbox is stable, and new emails are inserted somewhere, the delta update might be tiny.
Even if the order of the mbox's messages are ~random, Restic's delta updates will forego large attachments.
It would be great to see empirical figures here: how large is the incremental backup after after a month's emails. How does that compare for each backup strategy?
The pro of sticking with restic is simplicity, and also avoiding the risk of your tool managing to screw up the data.
This risk isn't so bad if it's a mature tool that canonicalises mboxes (e.g. order them by time), but seems risky for something handrolled.
For emails, here is my current simple backup setup. Of course, I’m also looking to do this without having to open Thunderbird, or I might have an old laptop running it. So, work-in-progress.
For the email accounts I want a backup, I set it to spew out POP3 without doing anything (don’t mark read or delete). I set up Thunderbird with that POP3. It has a backup copy of all the emails. I’ve had searchable emails since like 2004/2005, and I’ve occasionally replied to people and gotten back in touch with very old friends from the Internet.
I saw an open-source tool sometime back (I think, here on Hacker News) that backs up your IMAP mails with a nicely done interface. That would be nice to have.
Edit: Perhaps Bichon,[1] mentioned somewhere in the other comment threads[2] was the one.
`zpaq add archive.zpaq new.mbox -fragment 0 -method 3` is great for this. It splits the input into fragments averaging 1024 bytes in size [0], which catches up to ~90% of redundancy. The remaining ~10% is packed and compressed into 64MB (max) blocks that are added to the .zpaq.
The resulting artifact is a single .zpaq file on disk. This file is only ever appended to, never overwritten, so it plays nice with Restic's own chunked deduplication. Plus it won't flood the filesystem with inodes and it suffers less small files overhead than TFA's solution.
Granted I suspect TFA splitting on the e-mail headers may be chunking more efficiently. Though, unless I skimmed the linked GitHub too fast, it looks like TFA's solution also doesn't use any solid compression to exploit redundancy across chunks. And I trust zpaq as a general purpose tool more than a one-off just for a single use case. The code does look clean, though, nice work.
[0] Average fragment size is 1024*2^N. If the most of the data is attachments that don't change, you can probably use a higher `-fragment N` to have less overhead keeping track of hashes. `-method 3` is a good middle ground for backups. `-m5` gets crazy high compression ratios, but also crazy slow speed. Old versions of ingested files are shadowed by default; use `-all` when you want to list/extract them.
Have you looked into using a full MIME/mbox parser library, e.g. GMime [0] or MimeKit [1]? Both support parsing mbox files directly, and they should be able to handle the intricacies of parsing any messages/attachments you throw at them. Then you could write out the MIME representation of each message (including any attachments) into its own file and then check for new messages. That way you can be sure each “chunk” represents a single message in its entirety. Not sure if this is any better since your solution seems to work pretty well.
You can just unpack it to file-per-email (format used by most sane email clients). There are dozen of programs to do it, one is included in Debian (and so most other distros), called mb2md
I have the same requirement and I solved it as follows:
Apply a label to emails dated after the last backup, using an “after:YYYY-MM-DD” search. Takeout then offers the option to export only that label. I do an annual backup so the amount of manual effort here is acceptable.
What about using the Gmail API and listening for recent changes? I suppose it wouldn't be in a mailbox format that could be easily exported to another provider, though?
This is what I do! The API itself is not particularly amazing (the way it handles batch requests as a MIME multipart formatted POST body where each part is itself a HTTP request is particularly obscene).
The underlying data model is kind of OK though: messages are immutable, they get a long lived unique ID that survives changes to labels, etc. There is a history of changes you can grab incrementally. You can download the full message body that you can then parse as mail, and I save each one into a separate file and then index them in SQLite.
That imprecise chunking (with MD5 sums, and distributed across an FS tree, and using hash IDs, then separately storing the assembly information) seems like it would be a headache to restore, or to use with other mail programs.
An alternative idea is to properly parse it and store in smaller mbox files, such as one file month, with the idea that any month in the past usually will not change. (And if it changes because they are storing frequently changing attributes in a faux header, like an atime, then maybe strip that header.) Then your incremental `restic` backups work fine, and you can also use it easily with a variety of mail programs (MUAs, impromptu IMAP servers for migration, quick text editor, etc.).
I haven't used it for a while, but imapsync[0] still supports Gmail. With an approach like this you can regularly sync and get your messages in a standard format. Plus you don't have to wrangle those 50GB takeout dumps.
I realized my inbox takes a lot of memory, even after a manual cleanup, it was still taking 5GB, despite regularly removing automated things and others.
I tried using takeout to have a more accurate listing. I thought I could open it with thunderbird, I failed, I then tried to open it with some python lib, also failed.
Extracting the attachment will break signatures. This likely won’t be a problem for 99% or the people out there, but it’s important to be aware of the caveat.
If at some point in the future you need to prove that you received a given message, having the signatures (eg: DKIM) intact makes all the difference.
Not sure if I'm missing something when I look at the discussion here, but there's Got Your Back [0], which backs up all emails in a Gmail account as .eml files.
This seems to be the easiest and most straightforward way for me.
29 comments
[ 2.3 ms ] story [ 60.1 ms ] threadhttps://github.com/rustmailer/bichon
I only save financial statements and contact information. Everything else gets deleted as soon as possible.
About OP's tool, I also back up my Google account to an external disk periodically. Gmail is ~8 GB so it's manageable. But Google Photos is a pain. They recently removed most of the useful APIs, so AFAIK the only way to backup is via Takeout. It's terrible. Pictures in multiple albums are included as copies every time, so I had to make a script to find duplicates and replace them with symlinks. Just downloading the whole thing is a PITA (multiple 50 GB zip files). I get that Google has little incentive to make this better, in fact they might have an incentive to make it as inconvenient as possible, but I really wish they made it easier.
The answer: Yes, of course. (And I don't understand why anyone other than, say, a university undergrad or someone younger should find that answer surprising.)
As I'm sure the author is aware, Restic will do hash-based chunking so that similar files can be efficiently be backed up.
How similar are two successive Takeout mboxes?
If the order of messages within an mbox is stable, and new emails are inserted somewhere, the delta update might be tiny.
Even if the order of the mbox's messages are ~random, Restic's delta updates will forego large attachments.
It would be great to see empirical figures here: how large is the incremental backup after after a month's emails. How does that compare for each backup strategy?
The pro of sticking with restic is simplicity, and also avoiding the risk of your tool managing to screw up the data.
This risk isn't so bad if it's a mature tool that canonicalises mboxes (e.g. order them by time), but seems risky for something handrolled.
For the email accounts I want a backup, I set it to spew out POP3 without doing anything (don’t mark read or delete). I set up Thunderbird with that POP3. It has a backup copy of all the emails. I’ve had searchable emails since like 2004/2005, and I’ve occasionally replied to people and gotten back in touch with very old friends from the Internet.
I saw an open-source tool sometime back (I think, here on Hacker News) that backs up your IMAP mails with a nicely done interface. That would be nice to have.
Edit: Perhaps Bichon,[1] mentioned somewhere in the other comment threads[2] was the one.
1. https://github.com/rustmailer/bichon
2. https://news.ycombinator.com/item?id=46429250
The resulting artifact is a single .zpaq file on disk. This file is only ever appended to, never overwritten, so it plays nice with Restic's own chunked deduplication. Plus it won't flood the filesystem with inodes and it suffers less small files overhead than TFA's solution.
Granted I suspect TFA splitting on the e-mail headers may be chunking more efficiently. Though, unless I skimmed the linked GitHub too fast, it looks like TFA's solution also doesn't use any solid compression to exploit redundancy across chunks. And I trust zpaq as a general purpose tool more than a one-off just for a single use case. The code does look clean, though, nice work.
[0] Average fragment size is 1024*2^N. If the most of the data is attachments that don't change, you can probably use a higher `-fragment N` to have less overhead keeping track of hashes. `-method 3` is a good middle ground for backups. `-m5` gets crazy high compression ratios, but also crazy slow speed. Old versions of ingested files are shadowed by default; use `-all` when you want to list/extract them.
[0] https://github.com/jstedfast/gmime
[1] https://github.com/jstedfast/MimeKit
Does takeout include any metadata not accessible via IMAP? Does it even include labels?
Apply a label to emails dated after the last backup, using an “after:YYYY-MM-DD” search. Takeout then offers the option to export only that label. I do an annual backup so the amount of manual effort here is acceptable.
The underlying data model is kind of OK though: messages are immutable, they get a long lived unique ID that survives changes to labels, etc. There is a history of changes you can grab incrementally. You can download the full message body that you can then parse as mail, and I save each one into a separate file and then index them in SQLite.
An alternative idea is to properly parse it and store in smaller mbox files, such as one file month, with the idea that any month in the past usually will not change. (And if it changes because they are storing frequently changing attributes in a faux header, like an atime, then maybe strip that header.) Then your incremental `restic` backups work fine, and you can also use it easily with a variety of mail programs (MUAs, impromptu IMAP servers for migration, quick text editor, etc.).
0: https://imapsync.lamiral.info/FAQ.d/FAQ.Gmail.txt
I tried using takeout to have a more accurate listing. I thought I could open it with thunderbird, I failed, I then tried to open it with some python lib, also failed.
https://github.com/gaubert/gmvault
There's no reason to go through Takeout when IMAP exists.
If at some point in the future you need to prove that you received a given message, having the signatures (eg: DKIM) intact makes all the difference.
This seems to be the easiest and most straightforward way for me.
[0] https://github.com/GAM-team/got-your-back