Given how easy and carelessly Google can close your account and ruin your digital life, I guess periodic backups of your cloud accounts will soon be considered a good practice.
In this case, it's not as good as a normal backup. To my knowledge, Google Docs/Sheets/etc. is exported in other file formats (MS Word format, PDF format, etc), not their internal file format. So it's not a real backup, just an export.
It uses the same IMAP protocol that your mail clients do. I have been using it for years and not a single issue with google closing down any of the 5-6 accounts I have archived.
Yep! And much more. But gmvault does also restore emails to an account, interestingly.
I am (slowly) working on a project to pull some statistics from Takeout's mbox file for Mail. Also want to play around with the Location History, Chrome data and Hangouts exports.
Off-topic. What are some of the reasons you use desktop clients? I have tried using thunderbird several times but the habit never stuck. Anything I might be missing out on (apart from local IMAP backups)?
Works offline, has its own OS-level windows (so I don't have to hunt around for the tab, can close the browser but keep e-mail open or vice vers), multiple e-mail accounts under one application while keeping them entirely separated (I don't want to give one provider access to my mailboxes at another provider, and I don't want to have to use multiple different web UIs). Also a matter of habit.
(1) Desktop clients used to be faster and more efficient than using a web interface, and probably still are in most cases. However, Gmail is impressively snappy so I don't think it applies in that case.
(2) You can do things with a desktop client that you can't do with a web interface. That includes sort by subject and sort by size.
(3) Desktop clients can have real folders, which Gmail doesn't.
(4) Works off-line. This is still useful, though not as useful as it used to be.
I find it incredibly easier to configure and run periodically in a cron job. Plus I've never had even the slightest stability issue with gmvault whereas Thunderbird (which I use anyway to read emails at work) has bad days from time to time.
Additionally, the output format (gzipped plain email + metadata) looks very convenient for indexing / analysis; something I'm dreaming of for a long time.
Since Gmail supports imap and pop3, one could simply use any proven email client to backup the emails. I don't think a special tool that or may not work is needed for this.
Which "proven email client" do you recommend? There's an Import/Export extension for Thunderbird, one of the last standing desktop mail apps, but it's not good at handling huge, multi-thousand message exports.
gmvault is purpose-built and quite simple. I've used it before with success. It backs up your whole mailbox in one command. Why should we fiddle with a desktop mail client?
Thunderbird works fine for me to handle several GB of emails (the mailbox archives that it generates are plain text and standard so there's no need to export them). For the command line there are also tools like `getmail`[0], with the added benefit that they work with any email provider.
+1 on offlineimap. Those of us that may be a little strange in the head and prefer using mail clients like mutt will often use it to handle background imap syncing.
I've successfully used getmail to back-up my gmail accounts for over ten years now. It's a classic case of a functional, if a bit crufty, solution blinding one to a potentially better solution, so i appreciate this (gmvault) heads-up. thank you.
To my knowledge you can't run Thunderbird as a background service so you'll have to routinely run it. This can be a job scheduled to run in the background.
I am actively trying to get off of dropbox. I really want a similar native application experience that syncs to-and-from S3. Not a cron job using the s3 cli, not something that only works on osx/windows/etc... So far owncloud enterprise is the only polished looking solution I've found, but thats a bit overkill...
I've been using syncthing and I'm reasonably pleased with it. I don't use it heavily and only over my local network, though, so I'm not sure how well it handles archiving changes and file conflicts.
You may want to have a look at odrive (https://www.odrive.com). Use it to sync a bunch of different storage accounts via this single app. Works alright for what it does, and they have an S3 option.
"The core team of Syncany is on hiatus for an indefinite amount of time. Feel free to do with the code what the license allows and encourages, but please don't expect any maintenance"
Not feasible to use this for critical infrastructure if there is no maintenance happening.
How have you found syncthing perf? When I tried it a while back the perf was horrible. It ate CPU and was pretty slow, even though I was just testing with 2 machines on my local network.
Thanks, Syncthing looks like exactly what I was looking for! I'm guessing it needs some external server for coordination? It seems a bit unnecessary to restart after each config chsnge, is there some technical reason for it? Nice job otherwise, will try it out.
The community also hosts relay servers, so if your two devices can't communicate with eachother directly, it will work anyway.
Relay servers take bandwidth. Anyone can run a relay server, and it will automatically join the relay pool and be available to Syncthing users. This is documented here:
Please don't use OwnCloud. It eats your files and cost us loads of time and effort, in addition to sowing FUD among my office coworkers, who thought someone was deleting files from the shared/sync drive. Plus it doesn't support delta sync [0], so if (for example) you're syncing large files like (for example) True/VeraCrypt volumes, you're going to be pushing a lot of data around. This is especially awful since you're not doing this on a LAN but to S3, which means your raw cost in dollars for operating this software will be much much larger than with another solution like SyncThing or Seafile which does support delta sync.
ownCloud, or nowadays NextCloud [1], is still the best solution I have, hosted on a private virtual server. It's not a drop-in replacement for sure -- I'm basically prepared to do a full clean reinstall each time I want to upgrade -- but they have desktop and Android clients, and it's working just fine. Of course, I also have a separate backup of all the files.
I use amazon cloud drive unlimited $60/yr and arq backup to have client side encrypted backups. I also arq backup to my local NAS.
If you want to use an open source tool you can use borg backup & rsync.net as your external backup site. Borg doesn't have good S3 integration, and using fuse & s3 doesn't work that well either. It works best when the borg daemon is on the reciever box too to help with indexing and such.
There's a big item on one of my whiteboards: "put gmvault into the environment" ... the idea being that you could run 'gmvault', over SSH, on rsync.net:
ssh user@rsync.net gmvault ... blah blah ...
I've been meaning to do this forever ... it would be great if rsync.net customers could not install anything, but just run gmvault as an ssh command.
The only reason it takes time is that we do not have a python interpreter in our environment - we try to keep things as simple and locked down as possible - which means we have to "freeze" gmvault as a binary executable in order to put it into place ...
I did an extensive lookup of dropbox alternatives last year, and ended up self-hosting seafile.
Multiple family members use it and it works great, plus encryption built-in from day one (me not being able to read their stuff is critical to me).
There has been no outage, and upgrades are easy.
The 'restore' feature looks nice. But would it be of any value if Google decides to close your account? That is, can I take my backup emails from Gmail and 'restore' it to myemail@self-hosted-domain.com in order to migrate all those emails over?
I have opted to have 3 gmail accounts with various names; 1 outlook, 1 yahoo account. Every email sent to my gmail is forwarded > outlook > yahoo. I use 2 other gmail accounts for newsletters and etc.
I am sure at least 1 company will remain free to access data! BTW thanks for the thunderbird tip.
I have used this in the past to backup email accounts of resigned employees so we continue to stay well within the maximum number of active accounts for our free Google Apps.
It generally works fine and it allows you to restore the emails to a different account name. (I sometimes temporarily restore an account to search for old emails). It seems to have some issues with restoring accounts with a lot of large emails (large or multiple attachments) especially those that have reached the 15GB quota.
Since yesterday I have a few Google Takeout zip files in my backup ( https://takeout.google.com/settings/takeout ). I've used gmvault in the past, but this looks superior from the outside. Haven't delved into the data I'll admit.
I am working on digging around this data right now, actually. Some details on my website (see profile) and I will post on HN about it at some point when I get things further along (also want to look at Location History and Chrome data exports).
Ironically using some of these buggy tools can be an effective and swift way to get your account locked out or at least rate limited by anti-abuse systems. Read the code and make sure they are using official APIs and are coded sanely.
I was trying to push some people off my Google Apps accounts since google won't allow you to "split" some users off the account. Wish I would have seen this then!
Offtopic: I have Gsuite (free) for my company (<50), is it possible to backup all the emails of my users without knowing their passwords (with XOauth tokens?).
I have been using this for years and it's great software. One tip, store each different email account in it's own "database" my crontab looks like this:
I sync it all locally to my house then back it up to Dropbox as well. The reason to store them in different datebases is you cannot "filter" them out when restoring so if they all go to the same DB if you restore you are restoring ALL your email across all accounts to one new account.
Chiming in to second this comment for anyone who is skeptical of using gmvault. I too have used it for years with great success. Thanks to the author for creating it!
I believe it is all flat-file. When I say "Database" I am just referencing what they call it:
-d DB_DIR, --db-dir DB_DIR
But from what I see the structure looks like this:
db/
YYYY-MM/
1234554543262346.eml.gz - I assume the meat and potatoes of the email along with attachments, not sure
1234554543262346.meta - JSON file with msg_id, thread_id, flags, labels, subject, etc
if you want to try something that downloads Gmail via imap and indexes it into an sqlite3 db (with FTS5 fulltext of from/to/cc/bcc/subject/body fields) and extracts attachments to filesystem, take a look at a recent project of mine:
It also saves the raw .eml files to disk. No support for labels (yet), but it does properly link up threads in the db using `References` from the parsed headers (setting both MPTT and adjacency-list fields)
FWIW regarding speed, i was able to download, index & extract my entire INBOX + Sent Items (14k emails, 3.5GB total) in < 10min on a fast connection. the limiting factor by far was connection/imap speed.
I'm interested to know what the pros and cons are of this utility vs using the Google takeout functionality? I like the idea of this project but I don't know what it would gain me over Google's native export? Is it the restoration that's missing from Google's service?
Exactly what you get from Takeout varies from service to service; for email you get an MBOX-format mailbox file (that you can then import into a desktop email client of your choice).
It is actually mbox. I should have provided more detailed numbers - in my Takeout file, for example, there are 91,360 chat messages and only 23,407 email messages .
My coworker and I both tried this yesterday, and today we both received an email with this error message: "Sorry, we encountered a problem when creating your Google data archive."
Not as full featured (can't restore), but it's just a 77-line Python script. You could audit it yourself to make sure it doesn't upload your creds to another server.
142 comments
[ 2.7 ms ] story [ 113 ms ] threadIt has been good practice for a couple of decades. Of course, getting people to follow good practices is another matter....
[0] https://takeout.google.com/settings/takeout
I am (slowly) working on a project to pull some statistics from Takeout's mbox file for Mail. Also want to play around with the Location History, Chrome data and Hangouts exports.
- XOauth
- Sync to another gmail account (incrementally)
Edit: Also supports only backing up emails with a specific gmail tag.
(2) You can do things with a desktop client that you can't do with a web interface. That includes sort by subject and sort by size.
(3) Desktop clients can have real folders, which Gmail doesn't.
(4) Works off-line. This is still useful, though not as useful as it used to be.
(5) Easy to backup. Easy to automate backup.
Additionally, the output format (gzipped plain email + metadata) looks very convenient for indexing / analysis; something I'm dreaming of for a long time.
gmvault is purpose-built and quite simple. I've used it before with success. It backs up your whole mailbox in one command. Why should we fiddle with a desktop mail client?
[0] http://pyropus.ca/software/getmail/
How about clawsmail ? It's a pretty good client too, and still updated.
I'm using Gmvault for Gmail (emails and chats), pretty effortless and with enough features for me, especially the export to maildir.
And rclone[2] for Google Drive, in "create a local mirror" mode (sync).
Any other tools to backup your Google life?
[1] - https://news.ycombinator.com/item?id=12972554 [2] - http://rclone.org/
But really, you may not need S3 at all and just sync between your devices with Syncthing: https://syncthing.net/
Shameless plug: Syncthing is in official Debian repositories! https://tracker.debian.org/pkg/syncthing
Not feasible to use this for critical infrastructure if there is no maintenance happening.
- https://github.com/syncthing/syncthing/releases/tag/v0.14.7
Also, It will improve further with File System Notifications (from what I understand?):
- https://github.com/syncthing/syncthing/pull/2807
By default, Syncthing is pre-configured with community-hosted discovery servers:
- https://github.com/syncthing/syncthing/blob/master/lib/confi...
The community also hosts relay servers, so if your two devices can't communicate with eachother directly, it will work anyway.
Relay servers take bandwidth. Anyone can run a relay server, and it will automatically join the relay pool and be available to Syncthing users. This is documented here:
- https://docs.syncthing.net/users/strelaysrv.html#strelaysrv
It would also be possible to host your own private relay and discovery pools, if you need that for some reason.
[0] https://owncloud.org/faq/#partialsyncing
[1] https://nextcloud.com
If you want to use an open source tool you can use borg backup & rsync.net as your external backup site. Borg doesn't have good S3 integration, and using fuse & s3 doesn't work that well either. It works best when the borg daemon is on the reciever box too to help with indexing and such.
http://www.rsync.net/products/attic.html
There's a big item on one of my whiteboards: "put gmvault into the environment" ... the idea being that you could run 'gmvault', over SSH, on rsync.net:
ssh user@rsync.net gmvault ... blah blah ...
I've been meaning to do this forever ... it would be great if rsync.net customers could not install anything, but just run gmvault as an ssh command.
The only reason it takes time is that we do not have a python interpreter in our environment - we try to keep things as simple and locked down as possible - which means we have to "freeze" gmvault as a binary executable in order to put it into place ...
So ... folks want this ?
Go to the website click on documentation then indepth then restore.
You can also export to other formats ex. mbox
It generally works fine and it allows you to restore the emails to a different account name. (I sometimes temporarily restore an account to search for old emails). It seems to have some issues with restoring accounts with a lot of large emails (large or multiple attachments) especially those that have reached the 15GB quota.
What's the database format it uses, sqlite3? (I tried looking on the repo but couldn't any obvious reference)
https://github.com/good-3g-apps/email/tree/master/imap-impor...
It also saves the raw .eml files to disk. No support for labels (yet), but it does properly link up threads in the db using `References` from the parsed headers (setting both MPTT and adjacency-list fields)
It's WIP, so contributions welcome :)
https://github.com/Good-3G/email
https://takeout.google.com/settings/takeout
I wonder how to use them once I have the backup data?
(2) easy to corrupt
(Note that these aspects may also make the tool a little more dangerous to use.)
btw, the data is pretty exhaustive, even for non major Google applications or more recently added features (Maps Location History).
Not as full featured (can't restore), but it's just a 77-line Python script. You could audit it yourself to make sure it doesn't upload your creds to another server.