No cloud? Indeed, borg does not integrate cloud functionality.
You'll either need a directory or a server with ssh and borg to store your backups into. Some people use rclone to clone their borg repos to the cloud.
Wow, Kopia looks pretty interesting.
I first thought that it's a fork of Restic, but it appears to be independent. It has all the features that are key to me: encrypted, deduplicated, works on object storage, can mount the backup as a filesystem.
On one hand, on may think that three programs with very similar approach and features is a waste of resource. On the other hand, this is what refinement of the idea looks like: each project improves over previous attempts.
I had to setup a quick backup to s3 storage to replace an aging rsnapshot setup. I looked at Borg, but Duplicity (https://duplicity.us/) was easier to configure and connect to S3.
For syncing S3 storage across providers, I went with rclone (https://rclone.org/). Note that using rclone to sync across providers (e.g. from Amazon to Wasabi) does require the files to be downloaded to the client machine and then uploaded again. Not ideal, but if you have extra bandwidth it is a convenient setup.
deduplicating backup tools use a symmetric approach: all archives needing some chunk (piece of data) reference it via its chunkid (the keyed hash over the chunk plaintext content).
so you can delete ANY backup archive without influencing any other backup archive. a chunk will be only deleted if nothing is referencing it any more.
also, each backup is logically a FULL backup (it has ALL files, references ALL content data). it is just made in a clever way, avoiding to re-transfer data that already is present in the backup repository, thus it FEELS like incremental (considering speed, amount of CPU and I/O used).
OTOH, full/incremental style backup tools build a chain of incremental archives depending on the previous incremental and the full backup, which gets more fragile the longer the chain gets.
because of that and also because you might want to delete older backups at some time, you are forced to create new full backups regularly (causing lots of CPU and I/O load).
Yeah the process is to run a full backup weekly, and keep four weeks of backups. This mimics the old rsnapshot system, but sends the data directly to S3. In this particular case, the total amount of data is only less than 50GB, so the cost of a weekly backup is not great. The goal was to eliminate the dedicated backup server which ran pull backups via rsync/rsnapshot. Having each host back itself up to S3 directly is cheaper than paying for a backup host (including the time spent maintaining the backup host itself). As far as I understand, borgbackup requires a dedicated backup server.
Borg works great. I used it at a shop that had lots of servers, but didn't have any real backups-other than when someone would remember to go swap external USB drives and hope they actually ran.
Set it up on a bunch of servers with a simple cron job, the initial backups went quickly, and the incrementals were really fast. Made great use of an old Dell server that wasn't doing anything else and had lots of slow disks in it.
Borg is surprisingly fast and memory-efficient, even when compared to Restic, which is written in Go. Recently did a benchmark to test the upcoming Borg v2 and this surprised me the most:
I've been using Borg for years. It's great! The deduplication feature allows me to take a "full" backup of my work station hourly. Taking frequent backups like this has already saved my bacon a number times in cases where I accidentally mangled/deleted a file I didn't mean to touch.
Probably. My benchmark was mostly to compare Borg v1.2 and v2 and some network optimizations. Restic was a stretch goal really.
For Kopia, I do try it once a year, but I still find the docs and CLI args confusing. Running the server part behind a reverse proxy needs 2x HTTPS and searching the forum to get it somewhat working. For a webdav target, the progress display doesn't really work and it's not possible to cancel a backup run. So for now I'm observing and will retry next year.
Too bad they couldn't target S3 endpoints or anything other than SSH for remote target on this breaking change or else it would've been the best of the bunch.
Theoretically you can mount rclone to a local directory, but realistically it doesn't work because in the absence of serverside borg it is very heavy on network IO and slows to a crawl, even for sshfs over a short physical distance. To use it remotely, you really need to have borg installed on a server. I currently do as you describe.
The GitHub ticket hints that support for cloud storage backends (s3) might come in a future release:
as you see above, there is already a huge scope of what should be done.
to not grow the scope even further, some stuff shall not be done (now):
* no new locking system
* no public key cryptography (neither by gpg nor by reinventing gpg for borg)
* not adding cloud storage backends (like s3)
What’s good for Windows 10 (NTFS) drives? I’m using the Veeam Agent free version [0] for years and no problems whatsoever but curious what are some good options
As far as I can tell (as a years-long user of this), this is a generic "here's a server that supports SCP, go nuts" discount. You don't have to use it with Borg or Restic, they're just supported.
What are the practicalities of the '680 GB Minimum Order'? I would have expected you pay by how much you use (hence the price per GB), but the minimal order (the submission form however won't let you submit less than 100GB, so it's a bit confusing there's a mis-match?) seems slightly at odds with that.
I recall trying to use restic instead of borg a couple years ago, and some major feature was unavailable. I don't recall what is was, I think it was compression, which made archives quite large, and required larger instances for backup.
A major factor why I wouldn't want to use Kopia (I looked into it) ist that it is opinionated with regards to how your system is set up (old-school unix FS layouts in a "pets, not cattle" way). It assumes the location of config files and does not allow you to change the backup path that's stored in a snapshot's metadata.
That's bad if you want to use it
1) on NixOS (I don't want backup configs laying around in `~/.config`). As Indy famously said: "That belongs in a Nix expression!"
Edit: On a second note: Why have mandatory config files at all? I don't have an issue with having the option or it being the default, but for my use case being able to specify the whole repository config via arguments sounds considerably more sane.
2) with ZFS snapshots (yes, I'm backing up `/path/to/dataset/.zfs/snapshot/<timestamp>/foo/bar`, but that should not be its path in the metadata!)
OTOH, it seems to have the upside that you can apparently alter snapshots after the fact more easily (e.g. if you find out you shouldn't have backed up that gigantic VM image you just moved somewhere temporarily). I leave the decision on whether this is a footgun or not to you.
And to be clear: The ZFS snaphot thing is also a pain with Restic, too. You can hack around it somewhat better with something like systemd-nspawn, but it really shouldn't be that hard.
ZFS snapshot thing was also a huge pain with bup back in the day. I remember doing some work to add in a path graft system for it but it never got merged.
Backup tool authors never seem to support this use case.
FYI removing data from backup repositories (e.g. deleting files from a backup) is currently in the master branch of restic and will likely be in the next release.
That's a reasonable point but the solution - as with all things backup - is diversification. Otherwise if everyone followed your logic then no new backup software or storage service could ever become established no matter how good it actually was.
Given that all of the options being discussed here look technically better and in some cases more than an order of magnitude cheaper than other popular backup services and software discussed on HN in the past you could afford to run full redundant backups with multiple combinations of software and backing storage and still have more options for a much lower price than a few years ago.
Kopia is pretty new, but looking through the documentation I felt it had some pretty good features that suggest it will be a good option in the future.
A few highlights (for me at least):
- They have implemented support for object locking and versioning to prevent a compromised set of S3 keys being used to erase backups. There seems to be a pull request outstanding to dynamically shift forward the compliance hold each time an object file is "deemed still used", but very few s3 bucket based backup tools seem to really think about resisting deletion and documenting best practice. I like to use minimum privilege and deny DeleteObject permission, but using versioning and object locking policies seems to be an interesting potential solution.
- Inclusion of (optional) Reed Solomon error correction codes in backup objects.
- a CLI verification option that will test a given % of your backup objects (at random) every time it's run, to give you opportunistic random verification to try find issues.
- reasonably well documented CLI options for recovering data were anything to go wrong - as the developers mention, there are a wide range of ways to recover data. Interestingly, they appear to have actually implemented these, rather than leave them as theoretical. For example, CLI commands to interact with indexes, manifests, blobs, snapshots, etc. You don't need them normally, but having these available lets you look around a simple backup and understand the format and gain confidence in it, and that the tools work.
The only feature it seems to me to be lacking is asymmetric encryption of backups, so you can keep the key needed for recovery away from the host you are backing up.
> - a CLI verification option that will test a given % of your backup objects (at random) every time it's run, to give you opportunistic random verification to try find issues.
Also Restic doesn't have any problems with buckets with deletions disallowed as long as you allow them for just the `locks` directory. A policy I used for one of my targets looked like:
Big thumbs up for Kopia and its very simple GUI / strategy. Have been using it for a couple of years now to remote backup hard drives and working folders on a bunch of family macs to B2. Restored twice now - logic board & corrupt hd. Chose it after trialing both Borg and Restic for ease of use and storage cost. My monthly backblaze bill hovers around $1.40.
I tried Restic first due to its support of rclone. But it created many small files which is difficult to sync over my high-latency cloud service. I also saw an issue on Github about data loss, so I switched to using borg. Very satisfied with borg. Side note: they offer single compiled binaries for borg too.
I tested restic a few years ago. Wanted to like it, since they directly support OpenBSD. I backuped roughly 4TB of multi media data. It took a week. After that I did a re-run. No new data, so it only took a few minutes. Than I added a tiny txt file and let it run again. It took a week again!
So I settled on borg. I use it for offsite backup. I did restores of individual files as well of whole snapshots after broken hard drives. There even was a time I used it with WSL to backup my parents data.
These backup software are all incremental, work similarly and take similar times (difference is in details). If you lost locally cached metadata files keeping information on what has changed, they all have to scan the whole source again.
Except that I tested them and they didn't. I hadn't "lost locally cached metadata". And it didn't just scan all the source files again, it re-transferred them over the network. All 4TB of it! That's why it took so long. (At that time the sever was still local, because I wanted to make the initial run not over the Internet. So there wasn't even added latency.)
I'm sure it was a bug, maybe even in combination with OpenBSD, which moste likely is fixed now. As I said it was years ago. I compared and was more happy with borg. Which stood the test of time and saved my ass multiple times.
This probably wasn't a bug. I experienced the same last week but it was because I ran the backup command from a different directory. I've created a note saying:
first `cd` to the parent directory of the `source_dir` to avoid too many nested directories in the repository and conflicts in the mounting point path (so that the file change detection of restic works when mounting the drive to be backed up at another point)
Does Restic do encryption well now? When I tried it years ago you had to type the password on the command line, and there was no interest in adding GnuPG support.
Don't underestimate the value of a self contained binary. I used to use Borg and Attic. Then I needed to recover my backup only to find that all the python dependencies had gone out of sync on Arch Linux. After that I switched to Restic. Then I had a power failure during upgrade. After that I switched to Guix.
You could have just used the self-contained borg binary (download from github releases page).
It contains everything needed (python, python libs, other libs) except glibc and related libs which must match the OS / kernel and therefore are intentionally not included.
I read [1] that Restic, in contrast to Borg, supports deduplication between snapshots of different machines - attractive for me, in case I have copies of large files on several machines.
Maybe I'm misunderstanding, but the deduplication of Borg happens at a repository level and after the client-side encryption. If several hosts are pushing archives containing the same files (or rather, file contents) into the same repository, they are deduplicated.
We rely on this in some places in fact. We have a couple of redundant hosts all serving the same file set. This deduplication allows us to push backups from all systems without coordination and only the first backup of the same file set requires storage beyond metadata.
Yes, all hosts accessing the borg repository need the encryption keys or the passphrase of the repository, since it relies on the homomorphic encryption of the repo.
And yeah, we tend to partition our borg repos along functionality and needs of restore. For example, database backups and file store backups are split into different repos, but several file store hosts in the same cluster all write to the same borg repo. After all, if something allowed to compromise one file store host, it will most likely allow compromise of all identically setup file store hosts in that cluster.
And on the other hand, in this way, we don't have to think about the backups if one of the file store hosts goes offline - the other two will just continue writing backups. This would be something that's really easy to forget and could bite very badly.
> Another difference between the two programs is related to deduplication. Borg is designed with the assumption that each machine being backed up will use its own repository. Letting multiple machines backup to the same repository can impact performance, and simultaneous backups from different machines to the same respository are not supported.
So I should have said "Borg has limitations" instead of saying the support is absent for deduplication across machines.
When you say "without coordination", does it mean simultaneous backups from several machines to the same repo are possible?
> When you say "without coordination", does it mean simultaneous backups from several machines to the same repo are possible?
It is, with a bit of (other) coordination. Basically, only one process can actively write to a borg repository. Once something is writing to the borg repository, the repo is locked and nothing else can write to that repository.
However, you can easily stagger backups from several hosts into the same repository - node 1 writes at 09:00, node 2 writes at 10:00 and node 3 writes at 11:00. This works without problems, it just needs some monitoring for quickly growing backup sets in case your timing goes awry. You can also configure a wait timeout, how long a borg process will wait to lock the repository, but that will require some tinkering with SSH heartbeats to avoid connection timeouts, as borg won't talk over the wire while waiting.
As the documentation correctly states, this slows down the backup writing process a bit, because each node has to synchronize its local chunk cache at the start of a backup or a prune. This wouldn't be necessary if each node had their own dedicated backup repository. This can take 5ish minutes on our large repos (4TB+) and usually takes less than a minute for our smaller repos (<1TB). It's not really a big deal imo.
However - and we made that mistake earlier - something like `borg check` and other commands become really slow if you have some 15TB - 20TB+ borg repository and borg repos that large become really messy to manage. It works, but some things work at a glacial speed on a good day - while locking out all other backups. That's why we tend to group our borg repos by dataset and storage type ("All databases supporting app X backup into the app-database-X repository"). This way, all databases are in that repository even after failovers or switches to geo-standbys and it's easy to setup a restore procedure. Hoqever the overall size of the repository is somewhat limited and manageable.
Of course there are some cifs issues in the borg tracker, too, but most of them seem to be based on unreliable network connections - while the restic problem seems to be related to golang details.
With backups I would rather not trust a tool that has basic problems like accessing files on a network drive, but this is just me.
Maybe a bit off topic, but what is a good utility for "imaging" a linux system. I have a task to reprovision a system but we want to keep a complete backup of the current system so that it's possible to restore completely as if it were never touched.
This is more than just data backup as we would need need to recover disk partitions/LVM metadata, boot records, etc. as well as all the data itself.
Take a look at Clonezilla. Used it many times for the exact same purpose. You could run it either on a running system or use the live iso they provide.
Clonezilla is awesome, fast, stable, flexible, and reliable, from the Taiwan Supercomputing Centre. I used it lots over a decade ago to manage Mac, Windows, and Linux workstations and servers.
For this specific need, just once. Disk is 1TB but only about 350GB used.
'dd' would have been my thought as well, I've heard of Clonezilla also but never used it and not sure it's really doing anything appreciably different.
I like the idea of 'dd' because I have a very clear mental picture of what it does. Just wasn't sure there was something else I might want to look at.
If you need multiple version of such a disk image, a tool like restic (or I guess borg too, not sure?) can also compress what's provided to it via stdin. So you'd dd directly into restic and it will delta compress to earlier backups.
Yea; this was the heart of my 1/10/1000 question. Once? I'd probably just use dd and call it a day. 10 times? Probably download clonezilla. 1000 times? Probably automate something w/ restic and some kind of object storage layer so I don't just have a directory full of giant images/deltas somewhere.
> we would need need to recover disk partitions/LVM metadata, boot records, etc. as well as all the data itself.
While other suggested, image-based solutions better fit your bit-for-bit requirement, you might also be interested in ReaR: https://relax-and-recover.org/
ReaR generates a bootable image which performs all that basic partitioning and is able to trigger an actual system/application data file restore using a variety of tools (including borg).
Can someone suggest an approach to backup container environment? E.g. running inside Kubernetes.
As I see it: I write some kind of configuration.
someproject-db is a deployment which runs a postgres db. Tool should connect to this DB, issue some kind of pg_backup command, capture output, retrieve some metadata about previous backup from S3, compute difference with previous run, compress that difference and store it to S3.
anotherproject is a deployment which runs an sqlite db. Tool should do the same but with sqlite-specific commands.
yetanotherproject-data is a pvc which has attached pv. Tool should find pod which mounted this volume, exec into that pod and retrieve pv data, again find different and store it to S3.
Of course things should be configurable. Like store difference every 15 minutes, store complete backup every week and so on.
I'm fine with manual recover and with manual configuration (I just don't want to write and test all the scripts myself).
What I don't want is some kind of magic tool which will backup the entire cluster, etcd and my grandparents automatically in some magic way only for $50k/cpu core.
Doesn't zfs kind of solve all the backup problems alone? Technically, no other backup tools can beat it as being the filesystem itself, it knows more than any external tools can know, like instantly know what file got changed over time without scanning the entire tree.
I use Borg as backup of backup (zfs snapshots), so I'll be having multiple implementations of backups (also both are on different remote location) just to be on the safe side.
I don't use any other fancier ones as I don't like risking data on less reliable tools.
How does it matter if anyone else is using zfs? You either use a service that supports zfs target or run your own Linux instance which is just installing a single package for Ubuntu.
I'm using Velero to do this in my toy kubernetes clusters. It uses Restic under the hood and can store things into S3. By default it will take a filesystem-level copy of whatever is on a pv. It looks like it supports hooks, e.g. to run pg_backup like you mentioned, but I haven't used them.
Borgmatic will beautifully deal with DB dumps and there is a popular container image to run it. As for the cache ("retrieve some metadata about previous backup from S3"), you don't need to keep it locally. It can be restored from the backup repository.
Have been using Borg for many years now, it saved me several times already when I accidentally deleted stuff I realized I still needed later on. What's great is that you can just mount your backup repository as a FUSE filesystem, Borg then gives you a directory structure containing all your backups over time. Personally I use dates to name my backups, e.g. 2022-10-11, so when I need to restore something from a specific date I just go to the appropriate folder and extract it.
Another nice thing with fuse is that I can let my users (devs) mount the backups with Borg, so that they can restore specific files that they need without asking me.
And the mount can restrict which directories to expose, so they only see the files they need.
(I wrote a small wrapper which can be run with sudo)
I am the author of bupstash - https://github.com/andrewchambers/bupstash which has many advantages over borg in my biased opinion (like air gapped decryption keys and better performance). Feel free to check it out.
To second this, there are some threat models that are often overlooked with backups. Separating the encryption and restore keys makes sense, and is a design pattern few of the modern new backup tools offer - bupstash and duplicacy are the two that spring to mind which support asymmetrically encrypted backups.
The recent lastpass issue shows the importance of keeping the backup decryption keys offline and inaccessible. A headless server with backup script can upload backups, but if someone can access the script and backups, they have the golden goose (users' vaults).
Most other backup tools don't give you the ability to separate out creating and retrieving backups.
Separate moan - very few of the backup tools that use s3 style buckets give a proper description of the IAM security model you should use - I like to understand what object prefixes are used, and what the absolute minimum I require is, as I like to start from minimum privilege - PutObject, GetObject on an index or config file, and nothing else if I can avoid it!
Hopefully we see more focus on backup security after lastpass.
I was about to mention this. It’s hard for me to take any symmetric or password-based backup tools seriously. User chosen passwords are a disaster for offline encryption. Most tools also use related key input for (non-)authenticated encryption and decryption. A server shouldn’t be able to decrypt backups and authentically encrypt new backups.
Very interesting, I like its design better. Could you comment on whether bupstash supports ACLs (POSIX, xattr, etc.)? According to issue #25 it does, but I couldn't find documentation on how to exclude it when restoring, viewing it, etc.
The default is to exclude them, but commands like get/restore/put take options to enable xattr support. The reason for ignoring them by default is simply pragmatic - fewer syscalls while walking the filesystem.
does bupstash has 'append-only' mode, or to put it differently: how can I achieve sth similar (ability to do backups, but not to meddle with previous ones) with bupstash features (roles/diff keys etc) ?
Just started using the append only feature to prevent a potential hacker from wiping out backups that live on a remote ssh server. Combined with restricted ssh access this can be made quite secure.
I also tested writing backups to my Android phone (as a backup target) using Termux and Wireguard, worked flawlessly with a bit of tuning (keeping the vpn alive)
I've been very happy with it. I switched away from Tarsnap because it was far too slow on restores. BorgBackup has slightly different security semantics, but similarly supports the notion of append-only credentials.
And borgbase.com has been a good place to host my backups. Really nice onboarding flow.
Borg has an append-only mode, but no append-only credentials as far as I could find (by necessity given the symmetric encryption I believe). That means the key still needs to be protected, which makes secure automated backups a bit of a pickle.
If you use a remote Borg server over SSH, you can set up your authorized_keys file to enforce append-only mode on the server by defining the command to be run at login, and reject Borg requests by that SSH key which try to purge or remove data.
I believe this is roughly what borgbase does to implement their own backup protection features.
Borg is really solid and I had to unfortunately use it to restore data a bit ago and it went very smoothly. I’m bad with testing backups so I was a bit nervous.
I’ve been using Borg for the last three years to replicate on-site ZFS snapshots as Borg archives on rsync.net. My local ZFS is plaintext on LUKS volumes and the remote Borg archives are encrypted with borg.
Would I be better off using ZFS native encryption and
replicating the encrypted stream without giving a key to the rsync.net side? I worry that one false move would mean I would have to do a full backup again. I think I kind of feel more comfortable having borg as a second technology in the system, too, so that my restore can’t be broken by a ZoL bug. Doing test restores with Borg is also nice because I don’t need a ZFS system to do the restore — restoring to macOS works just fine for example.
Funnily enough, I’ve never had to prune / ruler function any Borg archives. I just don’t have enough churn in my data to need to do it. Everything change is there in full.
It works so well. Thank you everyone involved for donating your time and ideas to providing these tools and services.
Thanks for responding and for providing such a great service. I meant dropping borg altogether and using zfs send/receive to sync my data to rsync.net.
As I understood it, if my dataset is encrypted then rsync.net only sees the name of the snapshot and the encrypted data.
I’m not sure this would be a great solution for my most common restore use case though: fetching a single file back with borg while I’m at a third location that is remote to both rsync.net and my ZFS host. The fact that I can do that with just Borg and rsync.net is incredibly useful (yes, I travel with my keys!)
"As I understood it, if my (zfs) dataset is encrypted then rsync.net only sees the name of the snapshot and the encrypted data."
Prior to encrypted datasets in ZFS we did, in fact, have a stopgap measure where we would create your zpool on top of GELI devices such that it was, in fact, encrypted - but we held the keys. This was for HIPAA compliance and such things.
Currently, we run the latest stable/release ZoL codebase which includes bona fide encrypted ZFS datasets where you hold the key and we see nothing.
That being said, we do have a 4TB minimum[1] for these zfs-send enabled accounts where you get your own zpool ... so a smaller account with the (excellent) borg utility is a better bet for most individual users.
It was pretty fast already and recently got multithread support. It has been the only thing usable for backing up a few TB in a raspberry for performance reasons.
Keep in mind it's relatively new and the author does not yet recommend to use in production as the only backup solution.
Can anyone recommend a (Linux CLI) backup tool for incremental encrypted backups via SCP that allows a verification run against the current filesystem state while excluding a configured list of paths from the verification (for some frequently changing files)? I’m currently using duplicity, which is missing that last feature (without post-processing its verification output).
Not much. rsnapshot is based on rsync and hardlinks. So, you'll get some whole-file deduplication for unmodified files. No compression, no encryption/authentication, no way to verify if a backup is still ok.
Also, if you rename a file or folder, that hardlink-based dedup will not work, because it is based on files having the same path. Also, if you change just 1 bit in a huge file, it will add the whole new file because the dedup only works on whole files.
borg does variable size chunk based deduplication. if a big file changed just a bit, your backup repo will also just grow a bit when backing up that file.
also you get compression, encryption, authentication and the ability to check backups if they are ok still.
also, borg won't create a gazillion of files/hardlinks in the destination filesystem, but way fewer files (each about 500MiB big by default).
181 comments
[ 0.22 ms ] story [ 268 ms ] threadNo cloud? Indeed, borg does not integrate cloud functionality.
You'll either need a directory or a server with ssh and borg to store your backups into. Some people use rclone to clone their borg repos to the cloud.
Been much more happy with my tries with https://kopia.io which also includes an optional cross-platform GUI, in addition to the CLI.
I wouldn’t trust my backups to Kopia (unless for experimentation).
On one hand, on may think that three programs with very similar approach and features is a waste of resource. On the other hand, this is what refinement of the idea looks like: each project improves over previous attempts.
For syncing S3 storage across providers, I went with rclone (https://rclone.org/). Note that using rclone to sync across providers (e.g. from Amazon to Wasabi) does require the files to be downloaded to the client machine and then uploaded again. Not ideal, but if you have extra bandwidth it is a convenient setup.
And that is one of the main reasons why chunk-deduplicating backup tools (like borg, restic, ...) are better than full/incremental style ones.
so you can delete ANY backup archive without influencing any other backup archive. a chunk will be only deleted if nothing is referencing it any more.
also, each backup is logically a FULL backup (it has ALL files, references ALL content data). it is just made in a clever way, avoiding to re-transfer data that already is present in the backup repository, thus it FEELS like incremental (considering speed, amount of CPU and I/O used).
OTOH, full/incremental style backup tools build a chain of incremental archives depending on the previous incremental and the full backup, which gets more fragile the longer the chain gets.
because of that and also because you might want to delete older backups at some time, you are forced to create new full backups regularly (causing lots of CPU and I/O load).
Set it up on a bunch of servers with a simple cron job, the initial backups went quickly, and the incrementals were really fast. Made great use of an old Dell server that wasn't doing anything else and had lots of slow disks in it.
https://github.com/borgbase/benchmarks
I recently stumbled upon the release notes for the (WIP) v2: https://www.borgbackup.org/releases/borg-2.0.html. Seems to address quite a few of the pain points of v1.
https://github.com/borgbase/benchmarks
Eager to take another look at Borg and Kopia, etc.
For Kopia, I do try it once a year, but I still find the docs and CLI args confusing. Running the server part behind a reverse proxy needs 2x HTTPS and searching the forum to get it somewhat working. For a webdav target, the progress display doesn't really work and it's not possible to cancel a backup run. So for now I'm observing and will retry next year.
Feature list is only half of what it is.
Can you explain in more detail?
Duplicacy has been one of the best pieces of software I've ever used.
Will my automated hourly backups (via the Borgmatic wrapper) just start throwing errors and stop working?
The deduplication allowed me to move from Windows to Linux, while keeping a consistent backup. It's great stuff!
0, https://www.veeam.com/agent-for-windows-community-edition.ht...
We're looking to replace our self-written borg backup scripts with https://torsion.org/borgmatic/ which is a wrapper around borg.
(The FAQ doesn't seem to answer that)
ETA: found the 680GB minimum on the normal pricing page. So it’s 680 if you want a normal account, 100 for the “I know what I’m doing” account.
Anyone knows when 2.0 will be out of beta, and stable?
Update: yet another take on basically the same approach, also as a self-contained binary: https://github.com/kopia/kopia
That's bad if you want to use it
1) on NixOS (I don't want backup configs laying around in `~/.config`). As Indy famously said: "That belongs in a Nix expression!"
Edit: On a second note: Why have mandatory config files at all? I don't have an issue with having the option or it being the default, but for my use case being able to specify the whole repository config via arguments sounds considerably more sane.
2) with ZFS snapshots (yes, I'm backing up `/path/to/dataset/.zfs/snapshot/<timestamp>/foo/bar`, but that should not be its path in the metadata!)
OTOH, it seems to have the upside that you can apparently alter snapshots after the fact more easily (e.g. if you find out you shouldn't have backed up that gigantic VM image you just moved somewhere temporarily). I leave the decision on whether this is a footgun or not to you.
And to be clear: The ZFS snaphot thing is also a pain with Restic, too. You can hack around it somewhat better with something like systemd-nspawn, but it really shouldn't be that hard.
Backup tool authors never seem to support this use case.
Instead of working out how to teach my backup tools about snapshots, I just mount them in a subtree and use that as a chroot env.
Looks like compression only added in the latest release of restic.
No mounts on windows.
No GUI?
Lack of compression was likely the reason I went with Kopia.
Kopia is too new for that state.
Given that all of the options being discussed here look technically better and in some cases more than an order of magnitude cheaper than other popular backup services and software discussed on HN in the past you could afford to run full redundant backups with multiple combinations of software and backing storage and still have more options for a much lower price than a few years ago.
A few highlights (for me at least):
- They have implemented support for object locking and versioning to prevent a compromised set of S3 keys being used to erase backups. There seems to be a pull request outstanding to dynamically shift forward the compliance hold each time an object file is "deemed still used", but very few s3 bucket based backup tools seem to really think about resisting deletion and documenting best practice. I like to use minimum privilege and deny DeleteObject permission, but using versioning and object locking policies seems to be an interesting potential solution.
- Inclusion of (optional) Reed Solomon error correction codes in backup objects.
- a CLI verification option that will test a given % of your backup objects (at random) every time it's run, to give you opportunistic random verification to try find issues.
- reasonably well documented CLI options for recovering data were anything to go wrong - as the developers mention, there are a wide range of ways to recover data. Interestingly, they appear to have actually implemented these, rather than leave them as theoretical. For example, CLI commands to interact with indexes, manifests, blobs, snapshots, etc. You don't need them normally, but having these available lets you look around a simple backup and understand the format and gain confidence in it, and that the tools work.
The only feature it seems to me to be lacking is asymmetric encryption of backups, so you can keep the key needed for recovery away from the host you are backing up.
> - a CLI verification option that will test a given % of your backup objects (at random) every time it's run, to give you opportunistic random verification to try find issues.
FWIW Restic also supports this - see restic check --read-data-subset (https://restic.readthedocs.io/en/stable/045_working_with_rep...)
Also Restic doesn't have any problems with buckets with deletions disallowed as long as you allow them for just the `locks` directory. A policy I used for one of my targets looked like:
And it even works with buckets with object locking enabled since >= 0.13.Then pruning could be done once every few months. The egress fees could be high though.
Borg can can run in WSL but has seen limited testing under such, per their own docs.
https://help.backblaze.com/hc/en-us/articles/217667478-Under...
So I settled on borg. I use it for offsite backup. I did restores of individual files as well of whole snapshots after broken hard drives. There even was a time I used it with WSL to backup my parents data.
I suppose the latency of the remote server was too high.
Except that I tested them and they didn't. I hadn't "lost locally cached metadata". And it didn't just scan all the source files again, it re-transferred them over the network. All 4TB of it! That's why it took so long. (At that time the sever was still local, because I wanted to make the initial run not over the Internet. So there wasn't even added latency.)
I'm sure it was a bug, maybe even in combination with OpenBSD, which moste likely is fixed now. As I said it was years ago. I compared and was more happy with borg. Which stood the test of time and saved my ass multiple times.
`restic backup -v -r <location>/<repo_name> <source_dir>`
first `cd` to the parent directory of the `source_dir` to avoid too many nested directories in the repository and conflicts in the mounting point path (so that the file change detection of restic works when mounting the drive to be backed up at another point)
The performance issues you experienced may have been resolved.
It contains everything needed (python, python libs, other libs) except glibc and related libs which must match the OS / kernel and therefore are intentionally not included.
Is that also true for Kopia?
1 - https://stickleback.dk/borg-or-restic/
We rely on this in some places in fact. We have a couple of redundant hosts all serving the same file set. This deduplication allows us to push backups from all systems without coordination and only the first backup of the same file set requires storage beyond metadata.
This may make sense, depending on your threat model.
And yeah, we tend to partition our borg repos along functionality and needs of restore. For example, database backups and file store backups are split into different repos, but several file store hosts in the same cluster all write to the same borg repo. After all, if something allowed to compromise one file store host, it will most likely allow compromise of all identically setup file store hosts in that cluster.
And on the other hand, in this way, we don't have to think about the backups if one of the file store hosts goes offline - the other two will just continue writing backups. This would be something that's really easy to forget and could bite very badly.
> Another difference between the two programs is related to deduplication. Borg is designed with the assumption that each machine being backed up will use its own repository. Letting multiple machines backup to the same repository can impact performance, and simultaneous backups from different machines to the same respository are not supported.
So I should have said "Borg has limitations" instead of saying the support is absent for deduplication across machines.
When you say "without coordination", does it mean simultaneous backups from several machines to the same repo are possible?
It is, with a bit of (other) coordination. Basically, only one process can actively write to a borg repository. Once something is writing to the borg repository, the repo is locked and nothing else can write to that repository.
However, you can easily stagger backups from several hosts into the same repository - node 1 writes at 09:00, node 2 writes at 10:00 and node 3 writes at 11:00. This works without problems, it just needs some monitoring for quickly growing backup sets in case your timing goes awry. You can also configure a wait timeout, how long a borg process will wait to lock the repository, but that will require some tinkering with SSH heartbeats to avoid connection timeouts, as borg won't talk over the wire while waiting.
As the documentation correctly states, this slows down the backup writing process a bit, because each node has to synchronize its local chunk cache at the start of a backup or a prune. This wouldn't be necessary if each node had their own dedicated backup repository. This can take 5ish minutes on our large repos (4TB+) and usually takes less than a minute for our smaller repos (<1TB). It's not really a big deal imo.
However - and we made that mistake earlier - something like `borg check` and other commands become really slow if you have some 15TB - 20TB+ borg repository and borg repos that large become really messy to manage. It works, but some things work at a glacial speed on a good day - while locking out all other backups. That's why we tend to group our borg repos by dataset and storage type ("All databases supporting app X backup into the app-database-X repository"). This way, all databases are in that repository even after failovers or switches to geo-standbys and it's easy to setup a restore procedure. Hoqever the overall size of the repository is somewhat limited and manageable.
restic fails on repo mounted via CIFS/samba on Linux using go 1.14 build https://github.com/restic/restic/issues/2659
Of course there are some cifs issues in the borg tracker, too, but most of them seem to be based on unreliable network connections - while the restic problem seems to be related to golang details.
With backups I would rather not trust a tool that has basic problems like accessing files on a network drive, but this is just me.
This is more than just data backup as we would need need to recover disk partitions/LVM metadata, boot records, etc. as well as all the data itself.
https://clonezilla.org/
The most boring answer is "connect the disks to something else and use `dd` to copy the full blocks from start to finish into a file".
'dd' would have been my thought as well, I've heard of Clonezilla also but never used it and not sure it's really doing anything appreciably different.
I like the idea of 'dd' because I have a very clear mental picture of what it does. Just wasn't sure there was something else I might want to look at.
Slightly smarter approach is dd, then zero unused sectors and compress.
Both will produce an image which could me restored with DD (or mounted offline). Second will be smaller.
They should be run with unmounted partitions.
While other suggested, image-based solutions better fit your bit-for-bit requirement, you might also be interested in ReaR: https://relax-and-recover.org/
ReaR generates a bootable image which performs all that basic partitioning and is able to trigger an actual system/application data file restore using a variety of tools (including borg).
As I see it: I write some kind of configuration.
someproject-db is a deployment which runs a postgres db. Tool should connect to this DB, issue some kind of pg_backup command, capture output, retrieve some metadata about previous backup from S3, compute difference with previous run, compress that difference and store it to S3.
anotherproject is a deployment which runs an sqlite db. Tool should do the same but with sqlite-specific commands.
yetanotherproject-data is a pvc which has attached pv. Tool should find pod which mounted this volume, exec into that pod and retrieve pv data, again find different and store it to S3.
Of course things should be configurable. Like store difference every 15 minutes, store complete backup every week and so on.
I'm fine with manual recover and with manual configuration (I just don't want to write and test all the scripts myself).
What I don't want is some kind of magic tool which will backup the entire cluster, etcd and my grandparents automatically in some magic way only for $50k/cpu core.
My suggestion is using ZFS.
I use Borg as backup of backup (zfs snapshots), so I'll be having multiple implementations of backups (also both are on different remote location) just to be on the safe side.
I don't use any other fancier ones as I don't like risking data on less reliable tools.
How does it matter if anyone else is using zfs? You either use a service that supports zfs target or run your own Linux instance which is just installing a single package for Ubuntu.
Even for my own use cases, not every server and system I maintain could use ZFS right away.
Still good to know about the encrypted volume feature. Will be sure to test this next year.
https://github.com/vmware-tanzu/velero
Borgmatic will beautifully deal with DB dumps and there is a popular container image to run it. As for the cache ("retrieve some metadata about previous backup from S3"), you don't need to keep it locally. It can be restored from the backup repository.
Hope some of this applies to your K8s setup.
Deduplicating Archiver with Compression and Encryption - https://news.ycombinator.com/item?id=27939412 - July 2021 (71 comments)
BorgBackup: Deduplicating Archiver - https://news.ycombinator.com/item?id=21642364 - Nov 2019 (103 comments)
And the mount can restrict which directories to expose, so they only see the files they need.
(I wrote a small wrapper which can be run with sudo)
To second this, there are some threat models that are often overlooked with backups. Separating the encryption and restore keys makes sense, and is a design pattern few of the modern new backup tools offer - bupstash and duplicacy are the two that spring to mind which support asymmetrically encrypted backups.
The recent lastpass issue shows the importance of keeping the backup decryption keys offline and inaccessible. A headless server with backup script can upload backups, but if someone can access the script and backups, they have the golden goose (users' vaults).
Most other backup tools don't give you the ability to separate out creating and retrieving backups.
Separate moan - very few of the backup tools that use s3 style buckets give a proper description of the IAM security model you should use - I like to understand what object prefixes are used, and what the absolute minimum I require is, as I like to start from minimum privilege - PutObject, GetObject on an index or config file, and nothing else if I can avoid it!
Hopefully we see more focus on backup security after lastpass.
I backup to a Hetzner storage box and a Raspberry Pi at home.
[0] https://github.com/borgbackup/borg/issues/1872
What tool could I use that has a windows client, android and ios?
And borgbase.com has been a good place to host my backups. Really nice onboarding flow.
I believe this is roughly what borgbase does to implement their own backup protection features.
Would I be better off using ZFS native encryption and replicating the encrypted stream without giving a key to the rsync.net side? I worry that one false move would mean I would have to do a full backup again. I think I kind of feel more comfortable having borg as a second technology in the system, too, so that my restore can’t be broken by a ZoL bug. Doing test restores with Borg is also nice because I don’t need a ZFS system to do the restore — restoring to macOS works just fine for example.
Funnily enough, I’ve never had to prune / ruler function any Borg archives. I just don’t have enough churn in my data to need to do it. Everything change is there in full.
It works so well. Thank you everyone involved for donating your time and ideas to providing these tools and services.
I think you mean you are using borgs "repo key" method where the key to the repo is stored, remotely, inside the repo, correct ?
Just to clarify for others:
That repo-stored key has a passphrase so this method still keeps all of your files encrypted in a way that nobody at rsync.net can access.
FWIW, I use this method personally - I like the idea of storing the key with the repo and unlocking it with a passphrase.
As I understood it, if my dataset is encrypted then rsync.net only sees the name of the snapshot and the encrypted data.
I’m not sure this would be a great solution for my most common restore use case though: fetching a single file back with borg while I’m at a third location that is remote to both rsync.net and my ZFS host. The fact that I can do that with just Borg and rsync.net is incredibly useful (yes, I travel with my keys!)
Prior to encrypted datasets in ZFS we did, in fact, have a stopgap measure where we would create your zpool on top of GELI devices such that it was, in fact, encrypted - but we held the keys. This was for HIPAA compliance and such things.
Currently, we run the latest stable/release ZoL codebase which includes bona fide encrypted ZFS datasets where you hold the key and we see nothing.
That being said, we do have a 4TB minimum[1] for these zfs-send enabled accounts where you get your own zpool ... so a smaller account with the (excellent) borg utility is a better bet for most individual users.
[1] https://www.rsync.net/signup/order.html?code=4e9783
It was pretty fast already and recently got multithread support. It has been the only thing usable for backing up a few TB in a raspberry for performance reasons.
Keep in mind it's relatively new and the author does not yet recommend to use in production as the only backup solution.
Maybe it could provide a virtual read only filesystem, mounting the backup at a specific date, using the original atime, mtime and ctime
Also, if you rename a file or folder, that hardlink-based dedup will not work, because it is based on files having the same path. Also, if you change just 1 bit in a huge file, it will add the whole new file because the dedup only works on whole files.
borg does variable size chunk based deduplication. if a big file changed just a bit, your backup repo will also just grow a bit when backing up that file.
also you get compression, encryption, authentication and the ability to check backups if they are ok still.
also, borg won't create a gazillion of files/hardlinks in the destination filesystem, but way fewer files (each about 500MiB big by default).
borg also archives ACLs, xattrs, flags.
No Windows :(
Luckily there are capable alternatives to Borg, including Restic and Kopia.