Seems great, but I don't find the "Quickstart" section helpful. A demo video that shows 2-3 unknown commands without context is not a guide, same with links to the full documentation. Both make sense on the landing page, but both don't tell me how to get started quickly.
This quickstart assumes some context that not every person starting to use Restic has. It could be improved by offering some more context for each line:
# RESTIC QUICKSTART FOR LINUX
# Choose were you want to save your backups
MYREPO=/mnt/mybackupdrive/somefolder
# Initialize a new restic repo at your chosen backup location
restic init -r $MYREPO
# Backup your files to the newly created repo
restic backup /my/files -r $MYREPO
# To restore a backup, first mount the repo
restic mount /mnt/backedup -r $MYREPO
# Browse the latest backup at /mnt/backedup/snapshots/latest
ls -la /mnt/backedup/snapshots/latest
# Copy the files you want to restore out of the repo
cp -r /mnt/backedup/snapshots/latest/directory-to-restore ~/restored
> > unknown commands without context is not a guide
Quick start != guide, to me. A guide will guide you through thing at a slow pace, a quick start is the quickest way to get something running/started. And the commands seem fairly self-explanatory, but I could be suffering from the so-called curse of knowledge.
I don't remember really having that issue, though I don't remember the very first use. I think the only time where I remember being a bit confused, was with the 'forget' command, and I would guess that's just because of its nature rather than due to lacking docs. Testing what I thought was correct with --dry-run solved that problem.
Mostly I just use restic --help or restic subcommand --help anyhow, rather than the docs. Perhaps I'm just not doing fancy enough things with it and that's why I didn't run into edge cases yet where it lacks documentation?
It is never encouraging when I see that a project is at least 6 years old and has unapproachable documentation like you describe, and language about how they'll stop changing the backup format "when 1.0.0 is released."
Like...maybe it's time to stop working on new features, and focus on a release.
Duplicati has a similar problem. They're endlessly tinkering with new features instead of squashing bugs and meeting user expectations. I think it still chokes and permanently corrupts backup archives if you interrupt it during the initial backup.
Like: guys. People expect backup software to be able to handle interruptions and disconnects. It's actually one of the things I liked the most about Crashplan, and it could handle being interrupted without issue...nearly a decade ago.
Really looking forward to the day I can ditch borg backup and switch over but really can't until https://github.com/restic/restic/issues/21 is addressed. I have to pay for cloud storage and the lack of compression would easily double my costs based on my testing.
Holy crap that's just `pkg install borgbackup`. I had no idea (my phone is already rooted anyhow, so this will also be able to get data folders). This changes everything. There is also `pkg install restic` btw. Based on the problems with append-only in borg and the lack of those in restic's implementation (I did a short audit on that part of the `restic/rest-server` code, looks solid but don't take my word for it), I might go with the latter but this is a great tip regardless.
I never wrote a blog post about it, but it is triggered when I plug in my charger and the phone is on Wifi. There are hooks in termux to do so. Thanks for the suggestion to write a blog post about it ;)
Quid pro quo: I've been using Titanium Backup[1] to make backups of all my apps, however it does not working properly with Android 11 and seems to perhaps be abandoned. So I'm now also using OAndBackupX[2] as well, which seems to be doing the job.
I then use FolderSync[3] to SFTP synchronise those two backup folders across to my server regularly when the phone is on the home wifi. (I also two-way sync my photos folder which is really quite handy.)
I use to also occasionally do a full sync of my phone contents to my server using FTP[4] although since upgrading, Android 11 has clobbered access to the Android/data folder making that problematic.
Using Termux + Borg (or Restic) so push full full backups looks attractive. Never seen Termus before. Thanks.
Also a long time user. I'm only speculating on abandonment because: it hasn't had an update since Nov 2019; I believe the fix for Android 11 would be a fairly simple permissions change, and; from the comments, no one has had a response from the author on the issue.
It is a shame, it has been a mainstay for me, restoring apps and data across at least three phones now. I'm hoping OAndBackupX works out, but have not really battle-tested it yet.
I trigger this using cron every night, but systemd timers will work too.
The first neat thing about this setup is that the client never even sees the private key that it uses to authenticate to the borg server - the key stays on the server & authentication is tunnelled between client & server via ssh-agent. You don't even need to be able to make a tcp connection from the client to the server - so long as the borg server can make an outgoing tcp connection to the client then everything just works. The client connects back to the server via a socat connection through a unix socket created by the outgoing ssh connection that tunnels any tcp connection made through it back to the sshd on the server. (You could probably tunnel the repo passphrase through as well, if you really wanted to.)
The second neat thing is the use of authorized_keys commands which are tied to an ssh keypair means that you're giving the minimal possible access - each ssh connection can only trigger that specific command & no other. You can issue ssh keys on a per-host basis & revoke them individually if necessary.
You have to use socat as a proxy program for the return ssh connection as ssh doesn't know how to connect to a unix socket & this setup requires
config
StreamLocalBindUnlink yes
in the .ssh/config on the client (possibly both client + server?), as otherwise the unix socket doesn't get cleaned up after the connection ends & the whole thing only works once before you have to remove the socket by hand. I'm not sure why this isn't the default for ssh to be honest.
If you strip out the unix socket stuff (which I need for oddball network config reasons...) it’s just standard ssh authorised keys configs & ssh-agent working exactly as designed. It’s quite elegant really!
It’s the unix socket dance that introduces the gruesome hackery (imo at least!).
You might like Snebu then, it has always had compression and deduplication, now does public key encryption. No direct cloud support, although you can keep your repository sync'd to a bucket if you want. (Disclosure -- I'm the author)
Restic looked interesting - I would never have guessed it didn't support such a fundamental feature as compression?! So thanks for mentioning this and saving me from wasting my time.
Note: I'm not trying to be horrible or disrespectful to the Restic devs; it's just that backup without compression is a complete show-stopper, especially if you want to use cloud storage (where storage space, storage operation, and bandwidth all eat money).
That's like comparing rsync to google drive. One is an open source tool where you can use whatever back-end you want, the other is a service. (Which is fine, just different kinds of things.)
However, in this case it's the open source tool that has a much easier user interface (I am actually proficient with tar, but still my tarsnap experience is like comparing 'restic backup /my/files --repo /mnt/backupdisk' with https://xkcd.com/1168/)
Indeed. Restic is just something you apt install and nobody provides you any service (you have to organise your storage space yourself); tarsnap is not simply free to use for yourself with your own storage. (Not saying it has to be free, but that's what makes it the definition of a service you have to purchase.)
Tarsnap is not expensive at all for its target audience: folks with highly compressible data. For any data that's not very compressible, it's super costly.
Example, if you are into photography it's not uncommon to generate hundreds of GBs of files _per year_. Only in 2020, I generated over 200GB of photographs. Putting that on Tarsnap would cost me about $60/month. In 5 years time, I could be paying upto $4000/yr. Tiered services like B2 would cost an order of magnitude less.
Tarsnap works out as ~6$/GB/year for me. That’s for a mostly managed backup service. The only thing missing is snapshot pruning which is slow and a bit of pain due to the way tarsnap’s cache works. Restore is on par with restoring from tape — reliable but slow, but who can really complain about how fast disaster recovery is?
Raw managed storage with rsync.net is 0.18$/GB/year.
Do it all yourself, with the associated peril and time sink that entails, and the disks will cost you 0.04$/GB/replica.
Tarsnap has its place and I’m still a happy customer, but it’s one small part of a wider strategy that includes bulk storage elsewhere — rsync.net with borgbackup and plain rsync, on premises ZFS dumpsters, and offsite drives used like they are tapes.
Is the Microsoft OneDrive is $70 for 1TB a good option for the truly paranoid, or Microsoft will share the content in case they get a court order or some with a tank in their front door?
Why on earth someone would pay 10x-20x alternatives for encryption that these days is available in high quality free open source software such as Restic, Borg or Duplicacy?
IIRC restic operates at the file level on the source side and then stores data in blobs, and unchanged blobs are deduplicated (it operates as a content addressed store, kind of like git), which means iterative backups are possible.
Yes, it deduplicates but does not compress. Since most large files (mp3, jpg, mp4, docx, etc.) are already compressed, this is not a big issue but it's not ideal either.
If I remember correctly, another tool either does both or does compression instead of deduplication, this might have been borg or bup. In case that's something you care about.
Sorry about the formatting. But the compression is not completely irrelevant. Dedup of blocks between files and backups is of course the absolutely most crucial part though.
Please correct me if 'one backup' and 'all backups' is an incorrect interpretation of 'all archives'. I wasn't entirely sure what you mean by that but I think I get the point.
So in conclusion, adding compression saves about 17% (a $10 monthly bill would be $8.28 instead, if you pay per GB) for you.
Even if I'd get the better of the two values, I don't have enough systems that a backup being 2/3rds of the original data size reduces the number of backup disks I need to buy, but it's not insignificant either.
To be clear, "every snapshot [is] always a full backup" does not mean that you have to upload your whole terabytes-large drive every time. I guess what you mean is that each backup stands on its own, referring to the chunks it needs. I don't think that's what people mean when they say that each snapshot is a new full backup but rather that this is the definition of it being incremental.
Somewhat offtopic but this reminds me: what's the best way to backup a Raspberry Pi? Mine has a 128GB SD card and that makes backing up the whole thing kind of silly. There has to be a better way.
I've lost a few months of data twice because of my ignorance
I do this too. And you can combine rsync with gocryptfs (in reverse mode) to get strong encryption of your backups as well. This is especially important if you are storing the backups on a remote/untrusted device.
I use rsync to backup my desktop (which is mostly used for development) pretty much just rsync /home to a big enough external LUKS drive.
Has saved my bacon a few times, I don't care about incremental snapshots or delta's (though I've used both rsnapshot and rdiff successfully in the past), I'm covering the "if the SSD blows up, how long to chuck a new one in and be back up" case not the "I might need that file from 6mths ago".
I also have syncthing setup via /home/<user>/Shared/{Personal,Work}/ on every machine and important stuff I just chuck in Shared/ and forget about/it's available wherever I need it at any point.
I've had really ornate bulletproof snapshot based backups but honestly for my particular use case they where more hassle than they were worth, rsync does what I want every time and has never let me down.
Not unless they've improved it substantially in the past decade.
A startup I worked for [0] shipped a backup/"DR" appliance using rdiff-backup behind the scenes that I had the pleasure of inheriting ownership of.
What rdiff-backup was good for was creating the false impression that you had working backups you could restore from. But once your available disk space for backups filled up, which is kind of the whole goal of a backup system; accumulate as many revisions going back as far as you have space for, the thing paints itself into a corner you can't recover from without creating potentially huge amounts of free space first.
Here's why:
1. The backup tree is modified in-place in the course of performing a backup. If the backup is prevented from finishing for any reason (admin/user cancelled, ENOSPC, power loss, backup source became unavailable, etc), the backup tree is left partially in the new revision and partially in the previous revision. Any subsequent operation, restore or backup, must first restore the backup tree to its previous version, using the same primitive restore algorithm rdiff-backup uses for general restores.
2. Unless you're restoring from the latest revision requiring no reassembly from differentials, the restore algorithm requires enough free space to store up to two additional copies of any given file having changes it's reassembling. This doesn't even include the final destination file, if restoring into the backup filesystem (as it does when recovering from an interrupted backup, mentioned in #1), you need space for the third copy too.
Maybe they've fixed these problems since my time dealing with this, it's been years.
I ended up writing a compatible replacement for my employer at the time which used hard link farms to facilitate transactional backups requiring no recovery process when interrupted. This also enabled remote replication to always have a consistent tree to copy offsite while backups were in-progress, something rdiff-backup's in-place modification interfered with. As-is you'd end up just propagating a partially updated backup offsite if it happened to overlap with an ongoing backup.
My replacement also didn't require any temporary space for reassembling arbitrary versions of files from the differentials. So it could always perform a restore, even with no free space available. I even built a FUSE interface and versioned backup fs virtualization shim for QEMU+qcow2 atop those algorithms. But it was all proprietary and some of the stuff got patented unfortunately.
I wouldn't consider rdiff-backup usable if it didn't at least have the ability to restore without free space yet. At least then it might still be able to do its rollback process when ~full, assuming it's still doing the in-place modification of the backup data.
Edit:
In case it's not clear from the above; it's particularly nefarious the way rdiff-backup would fail, since it was typically unattended automated backups that would fill the disks, leaving the backup tree in a rollback-required state to either run another backup OR restore. The customers usually discovered this situation when they urgently needed to restore something, and rdiff-backup couldn't perform any restore without first doing the rollback, which it couldn't do because there was no space available. Not that it could even perform a differential restore without free space, but the rollback-required state almost guaranteed a differential restore was required just to do the rollback.
Back when I was implementing the replacement it was such an urgent crisis that I was logging into customer appliances to manually restore files from sets of differentials without needing temporary space, using unfinished test programs before I had even started on the integration glue to streamline that process.
Basically every back-up system does incremental backups, because indeed uploading or copying everything every time is kind of silly as you say. It's easy to go here "restic does that" but basically every other tool does that as well, so what "the best way" is depends on what other things you need, not on incrementalness.
Personally I would use restic in your situation because I'm familiar with it already and it does what I need (I particularly like the encryption aspect), but that's not to say that borg, bup, rsync, or other tools couldn't also fit your needs.
Use Borg, and copy the whole image. Borg will compress and deduplicate it, so you only backup the changes each time. You'll get point in time historical backups, so if you accidentally delete stuff, you can find it again. Useful because data loss isn't always just hardware failure!
And it's just a single go-binary, i just trow it on a win/bsd/linux machine create a key, start the backup. I love the simplicity of it, however for more complex plans i use git-annex.
I used to use borg and I migrated away from it to Restic when I somehow corrupted my backup archive. I dunno what I did, but I started getting "non-utf-8 filename" python errors every time I tried to access it. It might have had something to do with the archive being on a removable disk.
Anyway! I'm happier with restic now. It's never crashed for me, and it has native cloud backends. But it's ultimately just another backup application.
Is there any statistically-significant data on which backup applications are the most reliable? I'm not married to restic, but I'll judge it by first-hand experience in the absence of anything else.
Not really, no. It usually goes like in this thread: Someone had a problem with software X and switched to software Y. Someone else had the opposite experience. It's worth pointing out that Borg and other hash-deduplicating backup tools regularly find faulty hardware where other backup tools wouldn't notice the data getting corrupted (e.g. many people advocate for "plain" backup tools like rsnapshot or just having an rsync cronjob, but all of these are unable to check the integrity of backups). Sometimes, users point to the backup tool (sometimes they're right and it's a bug, but usually it's a bad stick of RAM or a hard drive loosing a few bits here and there).
Borg is much older and has seen production use for decades and had all the bugs worked out. Iirc rustic is still sub-1.0. Not ideal for backup software.
borg supports compression, restic doesn't (and there is no way to add it without breaking backwards compatibility because of how the file format was designed). That's all I need to know for my use cases.
Speaking of which, borg in Debian is maintained by the Debian Borg Collective, and the nickname of one of the maintainers is Locutus of Borg:
https://tracker.debian.org/teams/borg/
Borg sequentially scans your filesystem for changes and only then starts backing up changed data. (tbf: a lot of backup tools seem to do this)
Restic scans your filesystem for changes, and then also starts backing up the changes it finds in parallel while it is still scanning for more changes.
When you have millions of files, this makes a huge difference.
This is wrong. The difference between the two is that Restic uses multi-threading and Borg currently doesn't. Both just scan the filesystem and add files to the backup set as they go.
Hmm, maybe it has changed or I'm remembering wrong. It's been years since I tried borg, but I remember it taking something like 10 hours to scan for changes, and then another 4 hours or so to actually backup the data.
With restic, it still took around 10 hours to scan for changes, but it was also already done backing up all the data by the time the scan finished.
Bup (Python + C) - https://github.com/bup/bup - de duplicated and compressed. Storage format is a fit repository, an interesting choice that lets you restore using just git tools, “cat” and some effort.
Do you know which of these support a full Windows 10/11 system backup and restore?
I’m trying to avoid the need to reinstall and configure my system (for example, the registry, custom installed and tweaked programs) in case of complete data loss or a migration to new hardware.
My understanding is that full system backups on Windows requires the tool to create VSS snapshots and back up from the snapshot. Any tool that just copies files on the disk won't work.
I use Veeam Agent for this purpose (free, but not open source). It can do full system backups and supports both restoring to the same hardware and new hardware. Restores are done via a bootable WinPE-based image that the tool creates.
One cool thing about it I haven't seen in other backup software is that incremental backups work via a driver that tracks which disk blocks are changed as the system is running. It avoids the need to rescan the disk to detect what has been changed (though it will still do that if the filesystem is modified outside of Windows, eg. if dual booting).
The biggest downside is Veeam's website. It's pretty "enterprisey" and they want you to register to be able to download. I install via the Chocolatey package manager to avoid this. Chocolatey's package source has a direct link to the official installer [0].
There are no ads, nagging, nor upselling in the software itself. I have not seen it making any network connections outside of connecting to my backup target host and the auto-updates server.
I've been looking for open source alternative with a similar feature set, but haven't had too much luck. There's Bacula, but that seems to very much be designed for an enterprise use case.
Windows system backup requires support for correctly handling pretty much every NTFS feature, even (especially) the most obscure ones. While a generic file backup tool works fine for Linux and BSD system backups, it's hopeless for Windows. You need a tool that's specifically designed to do that.
My full-system backups are done with btrfs snapshots synced to an external disk (actually two disks to have two backup locations). It's nice because you can keep the snapshot on your system and don't need the external disk as long as you have enough space, and both filesystems are in almost exactly the same state which makes it easy to mount it for copying a single file or to even boot your system from the external disk.
Well it is written in Go instead of Python.
If I remember correctly, borg is single threaded and quite slow. Kopia is really fast.
Other than performance, the biggest benefit is that multiple clients can write to one repository at once. There is a designated maintenance user, though.
When it comes to backup software, do yourself a favor and do extensive research on their stability and reliability.
From what I learnt, borg has least problem of all the open source backup software with the most wanted features (encryption, dedup, compression and rotation) and others all have their quirks, be it huge memory usage, data loss to worst being corruption of the backup repository.
Things may have improved since I've looked but you want user feedbacks saying so.
It's too late to know that backup software has been failing when you need it as the original data is already unavailable, so choosing by the look of landing page is a bad way when it comes to backup.
kopia is still new and last time I used about a year ago, it still had basic problems, so I would never use it in place of borg.
restic, duplicati, duplicity and duplicacy all have some sort of problems especially when the repo gets large but of course there are cases where things are working fine.
borg also has change logs with any major reliability problems mentioned up front which gives you more confidence than serious bugs buried in GitHub issues like other tools.
The only downside of borg is it can only target ssh host natively, but there are services like rsync.net (with special borg pricing), borgbase or you could locally run borg and rclone the entirety to anywhere you want.
There are a bunch of options to borg check depending on what you want. A full check with --verify-data is indeed rather slow, because it checks everything (essentially equivalent to seeing if all archives can be extracted plus a bunch of extra checks). If you only want to detect e.g. bit-rot, --repository-only will be sufficient in most cases and will be I/O limited.
There are certain priority to what you treat as issues.
Slowness is the least of the problem against data loss or repo corruption and perhaps you may be able to somewhat circumvent it by splitting the repo to different locations or disks.
You mention duplicity but do not give a link to an issue. Do you know if there is something wrong with it? Would really appreciate you pointing me towards that if so.
Sorry I didn't link to an issue but a concern with duplicity is more about its implementation which needs incremental backups to depend on the base full backup
(that you would make the first time) which means if you want to prune that base backup later on, you need to take a new full backup for later incremental backups to depend on instead of taking continuous incremental backups indefinitely.
I'm really glad that you find it interesting, it was my solo project for the long time. Feel free to contact me on discord stan#9673 if you need any help or just for the general shell-scripting related chat :)
I've always liked the look of Restic. I should really start using it to backup my Linux servers.
However for desktop use, I've always really struggled with the idea of not having a UI for my backup client. I'm not afraid of the command line, but the idea of browsing backup archives without a GUI feels awkward to me.
I wonder if there is room for some sort of add-on GUI for Restic for those that are more visual (unless such a thing already exists?)
I’ve thought a lot about this. One alternative solution could be this: Instead of backing the laptop up directly, sync it to a NAS. (Using rsync or a similar tool.) Then run Restic on the NAS and back up the data from the NAS to S3 or similar.
I think many more people would use Restic if this was called out prominently on the home page.
I've known about Restic for years and would likely be using it by now had I realised that!
EDIT: Ah looks like it may not work on Windows. Part of the appeal of Restic, for me, would be being able to cover all of my Windows, Mac and Linux machines with the same system.
Restic doesn’t have a GUI for backup/restore configuration and actions (yet), but I found that Borg has a GUI called Vorta [1] for macOS and Linux. This was mentioned on a HN comment several days ago by @crossroadsguy here. [2]
Indeed, I feel the same way. I used to be a big user of CrashPlan, largely because it had a straightforward UI: highlight a bunch of directories; choose a backup destination; choose a backup frequency.
I do like the idea of being able to tune and configure things from the CLI. It was frustrating configuring CrashPlan on a remote computer.
With that said, I feel like both should be possible. Even just a basic wrapper GUI would be a start.
Edit: and some basic searching has lead me to lots of options! Time to do some more research.
I am a restic user. Use it to backup to b2 and Scaleway. It is not without hiccups, but once setup, and as long as I don’t backup anything from those protected Mac folders, it has worked smooth so far.
However I also acutely feel the lack of a standalone GUI so that I can get rid of the scrips and custom setup or at
least that can be an option. (There’s a commercial third party UI I think which is a subscription)
Another tool I’m looking at closely is https://kopia.io. It comes with a UI by default (Electron I guess). Though its UI and logo has quite some work left.
I trialed restic, kopia and borg and ended up with kopia, backing up to backblaze + opportunistic external hdd. Costs me cents a month for tb+ between all our devices. Agreed on the kopia UI (and logo, was only thinking that the other day!), it's pretty basic .. but to be honest it's all you need. I use it on my family's machines as well, including wife and parents (same bblaze, dif HDDs), and it means they can pretty much manage it without me. I liked restic and probably would have ended up with it if there was a decent oss/stock UI. Borg was significantly slower on my machine for backup and restore.
So every time my PC is on @ 20:00, a shell window will pop-up, asking me for password and runs the backup :). Since they are incremental, it takes maybe 10-15 minutes top.
Never mentioned it's super simple. It's DIY that I stitched together in 1-2 hrs of incremental upgrades and it's been running like this for months. To me it's simple enough but YMMV
Yeah I bet there are good GUI tools out there, but I always want to go for the stuff I can script myself, so I can hook desktop notifications into it and such.
It makes it hard for me to recommend backup tools to the non-technical people in my life, because they're looking for GUI solutions with the corners rounded off, and I want something crunchy and scriptable.
It doesn't have to be one or the other. The GUI would be for one-off or intermittent usage, and the CLI program would still come in as the primary tool for scripting or recurring use.
Well, to be fair, your does a bit more than the simple solution above. If you backup system configs and stuff with restic it wouldn't be much shorter than what you do here.
Love seeing a code example. It's one thing to hear "restic is fantastic, super easy to set up", it is another to see an example of HOW simple it is. Thank you for sharing.
You might be also interested in how I use restic to backup PostgreSQL and other data onto Backblaze for security and to save cloud costs as Cloud providers charge exuberant fees for backups[1].
Note that `restic check` only verifies that the repository metadata is correct, and doesn't detect, say, bit swaps in actual packfiles, which would render your backup unrestorable. You might be interested in the `--read-data` or `--read-data-subset` flags to help double check your backups!
If so, I think an important difference is that mine won't hide the exit code, and you can then answer a sibling commenter (on forgetting without checking) with 'I omitted set -e at the top'.
I have a Makefile to invoke restic on just about every Linux machine I have (even Raspberry Pis), usually to do incremental backups to Azure blob storage. It is a great, no-fuss, “forever” tool I’ve relied upon for years, and has made it trivial to clone or restore some pretty weird setups.
Just mind you exclude node_modules and the like from it :)
Do you also run things like 'restic check' on it from time to time? I once managed to corrupt a restic repository after ctrl+c'ing the program and pulling a disk before it was apparently done syncing the write cache (I wanted to leave and catch a bus but noticed that this was still running), not sure if there are other conditions where that might happen. Better to know ahead of time to make a new backup before the primary copy breaks.
I've used restic for over four years. It's been rock solid on all platforms (mac/windows/linux). I've set it up on everything from beefy linux servers to 10 year old Windows machines (scheduled job running for 3+ years, pruning on a separate machine, never had any problems). The support for B2 and S3 compatible backends as well as rclone makes it a breeze to set up. The community and maintainers are also very friendly and helpful. Highly recommended!
I'm still waiting for backup tool that uses asymmetric encryption (data encrypted using public key, decrypted using private key) and having write-only server mode (so bad actor can't remove backups)
Why would that not be possible? Especially since you say borg can do it, and borg is said to be incremental and deduplicated?
Note that write-only or append-only is a bit of a misnomer, since reading the files is fine (you need the decryption key anyhow before they're of any use). It's about not being able to overwrite or remove backup data without some verification that you're not ransomware or similar.
Hmm. If anybody can encrypt backups indistinguishably, and you want write-only so that bad guys can't remove stuff, surely you can incur unlimited costs as bad guys fill it with gibberish and you can't stop them?
You would run out of remote storage space quickly and then find out you have been compromised. But your backups that you made up to the point of being compromised are still in tact, which seems to be the best you can hope for.
> incur unlimited costs as bad guys fill it with gibberish
That's always an option, no matter if it's set to append-only or not. If you don't want to pay infinite costs for storage, you will need to limit it using software on the server.
The remote backup store can easily choose to only store things you actually encrypted, but this isn't possible for a simple public key setup. If you wanted to get fancy, you could use a sign + encrypt setup with separate keys so the store can tell if this is a real backup from you, and not allow things to get stored unless they've got such a signature, yet it still can't actually decrypt the backups it has been given.
As a proof of concept, take a look at a Certificate Transparency log server. Most CT logs are configured to only accept certificates meeting certain criteria. They'll log any such certificates (their SLAs only apply to contractual users, but you don't need an SLA you're probably just writing one certificate to the log to see it works) but you cannot fill them with garbage because you can't make any certificates they'd accept, only the legitimate CAs can do that†
† The CAs have their own reasons not to let you produce heaps of garbage, even Let's Encrypt has finite resources and so it imposes rate limits.
But if your box gets compromised, then whatever it was using to prove to the server that it's legit is also going to be available to the attacker. I guess you're thinking of a scenario where backups are not automated and the user either unerringly knows whether their box is compromised (and doesn't type in said proof when that happens) or uses some 2FA hardware device.
Given the chance, there certainly will be people that use 2FA when making an append-only regular backup, but even among command line restic users I expect this will be the exception rather than the rule.
Indeed, this is one of the major advantages of tarsnap, though a friend also mentioned borg can do this apparently. I should really look into borg (again).
The append-only mode can be implemented using https://github.com/restic/rest-server or services like rsync.net that offer read-only zfs snapshots. Doesn’t solve the asymmetric crypto of course.
(I make Arq, a backup app that supports S3's object-lock API for immutable backups). I can't see how to do incremental backups without using the private key to read the previous backup record. Can you explain how that would work?
"Once version 1.0.0 is released, we guarantee backward compatibility of all repositories within one major version; as long as we do not increment the major version, data can be read and restored. We strive to be fully backward compatible to all prior versions.
During initial development (versions prior to 1.0.0), maintainers and developers will do their utmost to keep backwards compatibility and stability, although there might be breaking changes without increasing the major version.
"
For what it's worth, it has a good track record, but yes you're totally right that restic is not exactly commercial-grade software with proper guarantees.
In one of the early talks at a local hackerspace, the author did also demo decrypting the data manually if something got somehow broken. The tool is just a layer on top of relatively straightforward cryptography. From what I remember (I looked at this in 2018 so forgive any errors), you'd have to write a script that iterates over the index where it says which blocks are in which files (since it's deduplicated) and decrypt it with some standard AES library. Perhaps as a security consultant this seems easier to me than it does to others, though, but it's not as if you're without hope if the tool did break, or as if you couldn't just download a previous version from the GitHub releases page.
One of the reasons I prefer restic to borg is it is trivial to maintain a standalone copy of the executable. Wherever I put backups, I keep a copy of the restic executable used to generate the dump.
For extra paranoid, could clone the restic source tree (with vendored dependencies). Go language backwards compatibility is such that I should always be able to read my data.
Restic is a great backup program. I use it (via rclone) to back up my laptop and it truly saved my bacon when I attached my main hard disk to a VM I was playing with and accidentally formatted it!
Restic can use rclone to access cloud providers it can't access and that's something we worked together on.
I use rclone directly for backing up media and files which don't change much, but restic nails that incremental backup at the cost of no longer mapping one file to one object on the storage which is where rclone shines.
I used to religiously make backups, and I still rsync my homedir onto a server occasionally.
That said, I keep ~/Documents and ~/dev these days in Syncthing directories, and one of my syncthing nodes is an Ubuntu LTS server with zfs, with the zfs-auto-snapshot package installed.
I still run my old backup system periodically (once or twice a month) but I now think Syncthing is at a point of reliability where realtime cross-machine sync is now my primary safety net wrt "the machine in front of me has turned to entropy", versus some point-in-time backup.
> No. Syncthing is not a great backup application because all changes to your files (modifications, deletions, etc.) will be propagated to all your devices. You can enable versioning, but we encourage you to use other tools to keep your data safe from your (or our) mistakes.
Even if you format your drive and Syncthing copies over the empty files you will still have a history of snapshots on the other host running ZFS snapshots.
What if you accidentally format the drive on the host with the ZFS snapshots? Won't that result in synchronizing the deletion of those files to all your other devices?
Yes, for his setup (if I understand right). I'm suggesting having two hosts with ZFS snapshots running (independent of each other) with Syncthing between. So if you delete the files on one host and they get Syncthing'd across there will still be a ZFS snapshot history on the other host. I'd also have a cold storage backup on hand that is also ZFS. Having your backup as a regular filesystem is a very nice feature once it comes to recovery.
syncthing has send-only and receive-only folders. You can set a receive-only folder on your backup device and send-only on your phone or whatever. With incremental backups it's not an issue. Syncthing is just the tool to get your data to the place it needs to be. Just like rsync or cp
Anyone taking backups seriously already knows the 3-2-1 rule anyway.
Syncthing frequently stops syncing on my phone and requires me to delete everything and resync. I'm using receive-only folders. I think I'm an exception but it's been a pain in the ass.
I'm coming to the same conclusion. I was running Freenas and my file server died and then my files were locked up on that host. Luckily I was using Syncthing to the central file server but I just added new links between individual devices and then it didn't matter that my file server died (except for files that were not on any non-server device). File servers are annoying for disconnected usage. I was using duplicacy for backups but the storage format is annoying (not just regular files) and nothing is more reliable than regular file systems. So now with my new setup I'm using a Rpi4b with NixOS and ZFS data drive over a USB3 dual drive docking station with UASP. I can boot from a USB3 thumb drive (which I do) or an SSD in the docking station and still have another SATA port for the data drive. (The downside of this is RAID doesn't work over USB.)
I'll have two of the same RPi servers in different locations with all software running (like a hot swap) with Syncthing keeping them in sync while ZFS snapshots keeps a history. I can plug in a cold storage drive in the second dock slot once in a while too. I'll have a spare RPi4b on the shelf in case it dies. If my server dies I can take the off site hot backup home and reconfig the network and then it is my primary server. With remote duplicacy backup I'm days away from getting going again. So ZFS snapshots + Syncthing and cold storage is where I'm going (for home use). Also I want to stick with Linux because I set up Freenas 5 years ago and now I forget how to admin it so I'd rather just keep with ZFS and Linux (the zfs send from freenas to zfs recv on Linux works perfectly).
Well, for one Tarsnap is incredibly expensive. I get 6TB of cloud storage for roughly $65/year with unlimited transfer. Tarsnap would be in excess of $1500 per month.
295 comments
[ 2.5 ms ] story [ 284 ms ] threadEdit: interestingly the restic repo's Readme is better in that regard. It's the demo video but in text form with explanations: https://github.com/restic/restic#quick-start
There are a ton of other commands and options, but to literally just get started, this is all you need for making and restoring data from a backup.
This quickstart assumes some context that not every person starting to use Restic has. It could be improved by offering some more context for each line:
Quick start != guide, to me. A guide will guide you through thing at a slow pace, a quick start is the quickest way to get something running/started. And the commands seem fairly self-explanatory, but I could be suffering from the so-called curse of knowledge.
Fwiw, I've used restic for a few years, and (still, if perhaps less) agree the docs are not great.
Mostly I just use restic --help or restic subcommand --help anyhow, rather than the docs. Perhaps I'm just not doing fancy enough things with it and that's why I didn't run into edge cases yet where it lacks documentation?
Like...maybe it's time to stop working on new features, and focus on a release.
Duplicati has a similar problem. They're endlessly tinkering with new features instead of squashing bugs and meeting user expectations. I think it still chokes and permanently corrupts backup archives if you interrupt it during the initial backup.
Like: guys. People expect backup software to be able to handle interruptions and disconnects. It's actually one of the things I liked the most about Crashplan, and it could handle being interrupted without issue...nearly a decade ago.
I then use FolderSync[3] to SFTP synchronise those two backup folders across to my server regularly when the phone is on the home wifi. (I also two-way sync my photos folder which is really quite handy.)
I use to also occasionally do a full sync of my phone contents to my server using FTP[4] although since upgrading, Android 11 has clobbered access to the Android/data folder making that problematic.
Using Termux + Borg (or Restic) so push full full backups looks attractive. Never seen Termus before. Thanks.
[1] https://play.google.com/store/apps/details?id=com.keramidas....
[2] https://github.com/machiav3lli/oandbackupx
[3] https://play.google.com/store/apps/details?id=dk.tacit.andro...
[4] https://play.google.com/store/apps/details?id=com.theolivetr...
It is a shame, it has been a mainstay for me, restoring apps and data across at least three phones now. I'm hoping OAndBackupX works out, but have not really battle-tested it yet.
It is possible to do pull backups with borg, with some gruesome ssh hackery.
On the backup client side, you need to have a /root/.authorized_keys line like this (edit borg options to suit):
Then, on your borg server, you need an authorized_keys file like this: Finally, run a small shell script like this from the borg server to trigger a pull backup: I trigger this using cron every night, but systemd timers will work too.The first neat thing about this setup is that the client never even sees the private key that it uses to authenticate to the borg server - the key stays on the server & authentication is tunnelled between client & server via ssh-agent. You don't even need to be able to make a tcp connection from the client to the server - so long as the borg server can make an outgoing tcp connection to the client then everything just works. The client connects back to the server via a socat connection through a unix socket created by the outgoing ssh connection that tunnels any tcp connection made through it back to the sshd on the server. (You could probably tunnel the repo passphrase through as well, if you really wanted to.)
The second neat thing is the use of authorized_keys commands which are tied to an ssh keypair means that you're giving the minimal possible access - each ssh connection can only trigger that specific command & no other. You can issue ssh keys on a per-host basis & revoke them individually if necessary.
You have to use socat as a proxy program for the return ssh connection as ssh doesn't know how to connect to a unix socket & this setup requires
in the .ssh/config on the client (possibly both client + server?), as otherwise the unix socket doesn't get cleaned up after the connection ends & the whole thing only works once before you have to remove the socket by hand. I'm not sure why this isn't the default for ssh to be honest.This method is outlined in the ssh-agent section of https://borgbackup.readthedocs.io/en/stable/deployment/pull-... but the docs don't really call it out as a method of getting pull backups working properly. It's a bit convoluted, but it does work!
(If your client can make a direct tcp connection to the server you can skip the whole song + dance with the unix sockets of course.)
It’s the unix socket dance that introduces the gruesome hackery (imo at least!).
Note: I'm not trying to be horrible or disrespectful to the Restic devs; it's just that backup without compression is a complete show-stopper, especially if you want to use cloud storage (where storage space, storage operation, and bandwidth all eat money).
I don't know if they apply to this specific case, but I assumed that the restic developers were being extra paranoid.
Previously on HN:
https://news.ycombinator.com/item?id=21410833
However, in this case it's the open source tool that has a much easier user interface (I am actually proficient with tar, but still my tarsnap experience is like comparing 'restic backup /my/files --repo /mnt/backupdisk' with https://xkcd.com/1168/)
What is your definition of "service" that makes tarsnap - a company that asks you to pay them over time to provide an, uh, service - not one?
For years we had a running discussion here, where patio11 writes a nice article called something like:
fake patio11> "10 reasons why tarsnap must raise the price and stop using funny units"
And a week later cperciva writes another article called something like:
fake cperviva> "Nah. Amazon reduced the storage price 10%, so I'm reducing the price of tarsnap in 1 picodollar/byte"
I would be sad if either of those ceased.
Example, if you are into photography it's not uncommon to generate hundreds of GBs of files _per year_. Only in 2020, I generated over 200GB of photographs. Putting that on Tarsnap would cost me about $60/month. In 5 years time, I could be paying upto $4000/yr. Tiered services like B2 would cost an order of magnitude less.
Raw managed storage with rsync.net is 0.18$/GB/year.
Do it all yourself, with the associated peril and time sink that entails, and the disks will cost you 0.04$/GB/replica.
Tarsnap has its place and I’m still a happy customer, but it’s one small part of a wider strategy that includes bulk storage elsewhere — rsync.net with borgbackup and plain rsync, on premises ZFS dumpsters, and offsite drives used like they are tapes.
i have 700G so it would cost me ~ $230/year (and you have to buy a min of 400G)
microsoft onedrive is $70 for 1TB and you don't pay for bandwidth. can use rclone.
you get some other goodies too (office) which i don't use but i'd imagine it's a nice to have for some people.
I pay around 0.40$ a month to have a backup of around 70 GB.
(I use Borg and the feature is an absolute must for me)
EDIT: see "blob" https://restic.readthedocs.io/en/latest/100_references.html#... and "efficient" https://github.com/restic/restic#design-principles
If I remember correctly, another tool either does both or does compression instead of deduplication, this might have been borg or bup. In case that's something you care about.
All archives: 2.93 TB 2.38 TB 133.66 GB
Sorry about the formatting. But the compression is not completely irrelevant. Dedup of blocks between files and backups is of course the absolutely most crucial part though.
So in conclusion, adding compression saves about 17% (a $10 monthly bill would be $8.28 instead, if you pay per GB) for you.
Even if I'd get the better of the two values, I don't have enough systems that a backup being 2/3rds of the original data size reduces the number of backup disks I need to buy, but it's not insignificant either.
Deduplication means ~20x smaller backups so this is absolutely key for me.
I've lost a few months of data twice because of my ignorance
https://wiki.archlinux.org/title/Gocryptfs#Usage
Has saved my bacon a few times, I don't care about incremental snapshots or delta's (though I've used both rsnapshot and rdiff successfully in the past), I'm covering the "if the SSD blows up, how long to chuck a new one in and be back up" case not the "I might need that file from 6mths ago".
I also have syncthing setup via /home/<user>/Shared/{Personal,Work}/ on every machine and important stuff I just chuck in Shared/ and forget about/it's available wherever I need it at any point.
I've had really ornate bulletproof snapshot based backups but honestly for my particular use case they where more hassle than they were worth, rsync does what I want every time and has never let me down.
https://rdiff-backup.net/docs/features.html
A startup I worked for [0] shipped a backup/"DR" appliance using rdiff-backup behind the scenes that I had the pleasure of inheriting ownership of.
What rdiff-backup was good for was creating the false impression that you had working backups you could restore from. But once your available disk space for backups filled up, which is kind of the whole goal of a backup system; accumulate as many revisions going back as far as you have space for, the thing paints itself into a corner you can't recover from without creating potentially huge amounts of free space first.
Here's why:
1. The backup tree is modified in-place in the course of performing a backup. If the backup is prevented from finishing for any reason (admin/user cancelled, ENOSPC, power loss, backup source became unavailable, etc), the backup tree is left partially in the new revision and partially in the previous revision. Any subsequent operation, restore or backup, must first restore the backup tree to its previous version, using the same primitive restore algorithm rdiff-backup uses for general restores.
2. Unless you're restoring from the latest revision requiring no reassembly from differentials, the restore algorithm requires enough free space to store up to two additional copies of any given file having changes it's reassembling. This doesn't even include the final destination file, if restoring into the backup filesystem (as it does when recovering from an interrupted backup, mentioned in #1), you need space for the third copy too.
Maybe they've fixed these problems since my time dealing with this, it's been years.
I ended up writing a compatible replacement for my employer at the time which used hard link farms to facilitate transactional backups requiring no recovery process when interrupted. This also enabled remote replication to always have a consistent tree to copy offsite while backups were in-progress, something rdiff-backup's in-place modification interfered with. As-is you'd end up just propagating a partially updated backup offsite if it happened to overlap with an ongoing backup.
My replacement also didn't require any temporary space for reassembling arbitrary versions of files from the differentials. So it could always perform a restore, even with no free space available. I even built a FUSE interface and versioned backup fs virtualization shim for QEMU+qcow2 atop those algorithms. But it was all proprietary and some of the stuff got patented unfortunately.
I wouldn't consider rdiff-backup usable if it didn't at least have the ability to restore without free space yet. At least then it might still be able to do its rollback process when ~full, assuming it's still doing the in-place modification of the backup data.
Edit:
In case it's not clear from the above; it's particularly nefarious the way rdiff-backup would fail, since it was typically unattended automated backups that would fill the disks, leaving the backup tree in a rollback-required state to either run another backup OR restore. The customers usually discovered this situation when they urgently needed to restore something, and rdiff-backup couldn't perform any restore without first doing the rollback, which it couldn't do because there was no space available. Not that it could even perform a differential restore without free space, but the rollback-required state almost guaranteed a differential restore was required just to do the rollback.
Back when I was implementing the replacement it was such an urgent crisis that I was logging into customer appliances to manually restore files from sets of differentials without needing temporary space, using unfinished test programs before I had even started on the integration glue to streamline that process.
[0] https://ww...
Personally I would use restic in your situation because I'm familiar with it already and it does what I need (I particularly like the encryption aspect), but that's not to say that borg, bup, rsync, or other tools couldn't also fit your needs.
Anyway! I'm happier with restic now. It's never crashed for me, and it has native cloud backends. But it's ultimately just another backup application.
I use it over NFS.
To which the conclusion is: test your backups!
Still, some might corrupt more easily than others. People just confuse first-hand problems with statistical significance.
Restic scans your filesystem for changes, and then also starts backing up the changes it finds in parallel while it is still scanning for more changes.
When you have millions of files, this makes a huge difference.
With restic, it still took around 10 hours to scan for changes, but it was also already done backing up all the data by the time the scan finished.
Related: "Restic Cryptography" by FiloSottile (2017), 36 comments https://news.ycombinator.com/item?id=15131310
attic (python) - https://github.com/jborg/attic
borg (c) - https://github.com/borgbackup/borg
bupstash (rust) - https://github.com/andrewchambers/bupstash
duplicacy (go) - https://github.com/gilbertchen/duplicacy
duplicati (c#) - https://github.com/duplicati/duplicati
duplicity (python) - https://github.com/henrysher/duplicity
kopia (go) - https://github.com/kopia/kopia
nfreezer (python) - https://github.com/josephernest/nfreezer
rdedup (rust) - https://github.com/dpc/rdedup
restic (go) - https://github.com/restic/restic
rclone (go) - https://github.com/rclone/rclone
rsnapshot (perl) - https://github.com/rsnapshot/rsnapshot
snebu (c) - https://github.com/derekp7/snebu
tarsnap (c) - https://github.com/Tarsnap/tarsnap
I think there are many more out there (https://github.com/restic/others) - I personally use
while technology wise (speed, only restore needs password) i would prefer which is an impressive piece of software but unfortunately without file iterator... :-)zpaq (C++) - http://mattmahoney.net/dc/zpaq.html
to the list.
But zpaq is pure magic...a slow one but with massive compression.
borg is python and c
bupstash is rust and c
Maybe update your list a bit ;)
I’m trying to avoid the need to reinstall and configure my system (for example, the registry, custom installed and tweaked programs) in case of complete data loss or a migration to new hardware.
I use Veeam Agent for this purpose (free, but not open source). It can do full system backups and supports both restoring to the same hardware and new hardware. Restores are done via a bootable WinPE-based image that the tool creates.
One cool thing about it I haven't seen in other backup software is that incremental backups work via a driver that tracks which disk blocks are changed as the system is running. It avoids the need to rescan the disk to detect what has been changed (though it will still do that if the filesystem is modified outside of Windows, eg. if dual booting).
The biggest downside is Veeam's website. It's pretty "enterprisey" and they want you to register to be able to download. I install via the Chocolatey package manager to avoid this. Chocolatey's package source has a direct link to the official installer [0].
There are no ads, nagging, nor upselling in the software itself. I have not seen it making any network connections outside of connecting to my backup target host and the auto-updates server.
I've been looking for open source alternative with a similar feature set, but haven't had too much luck. There's Bacula, but that seems to very much be designed for an enterprise use case.
[0] https://github.com/sbaerlocher/chocolatey.veeam-agent/blob/m...
I do miss Obnam, nevertheless. The interface was the best.
https://kopia.io/
Quite happy with it.
Other than performance, the biggest benefit is that multiple clients can write to one repository at once. There is a designated maintenance user, though.
From what I learnt, borg has least problem of all the open source backup software with the most wanted features (encryption, dedup, compression and rotation) and others all have their quirks, be it huge memory usage, data loss to worst being corruption of the backup repository.
Things may have improved since I've looked but you want user feedbacks saying so.
It's too late to know that backup software has been failing when you need it as the original data is already unavailable, so choosing by the look of landing page is a bad way when it comes to backup.
kopia is still new and last time I used about a year ago, it still had basic problems, so I would never use it in place of borg.
restic, duplicati, duplicity and duplicacy all have some sort of problems especially when the repo gets large but of course there are cases where things are working fine.
https://forum.rclone.org/t/rclone-as-destination-for-borgbac... (Restic issues)
https://www.reddit.com/r/Backup/comments/opu1ep/comment/h67n... (Kopia issues)
https://forum.duplicacy.com/t/memory-usage/623/24 (Duplicacy doing a big rewrite of the engine to fix memory issues.)
https://forum.duplicati.com/t/is-duplicati-2-ready-for-produ... (duplicati stability issues)
https://www.reddit.com/r/unRAID/comments/eg0zpe/duplicati_se... (Another duplicati issues)
borg also has change logs with any major reliability problems mentioned up front which gives you more confidence than serious bugs buried in GitHub issues like other tools.
https://borgbackup.readthedocs.io/en/stable/changes.html
The only downside of borg is it can only target ssh host natively, but there are services like rsync.net (with special borg pricing), borgbase or you could locally run borg and rclone the entirety to anywhere you want.
Every software has its issues. That's why it's important to test restores regularly.
Slowness is the least of the problem against data loss or repo corruption and perhaps you may be able to somewhat circumvent it by splitting the repo to different locations or disks.
https://duplicity.gitlab.io/duplicity-web/vers8/duplicity.1....
> When restoring, duplicity applies patches in order, so deleting, for instance, a full backup set may make related incremental backup sets unusable.
Other software mentioned have each incremental backups independent of each others, so you can prune any of it and the last one is still retrievable.
I usually work in multiple Linux virtual machines and I have a Bash script to setup regular backup of all my many $HOMEs.
I did not yet have the script for verification (did it manually just to be sure), but the rest is here if someone is interested https://github.com/senotrusov/sopkafile/blob/main/lib/ubuntu...
However for desktop use, I've always really struggled with the idea of not having a UI for my backup client. I'm not afraid of the command line, but the idea of browsing backup archives without a GUI feels awkward to me.
I wonder if there is room for some sort of add-on GUI for Restic for those that are more visual (unless such a thing already exists?)
I think many more people would use Restic if this was called out prominently on the home page.
I've known about Restic for years and would likely be using it by now had I realised that!
EDIT: Ah looks like it may not work on Windows. Part of the appeal of Restic, for me, would be being able to cover all of my Windows, Mac and Linux machines with the same system.
[1]: https://vorta.borgbase.com
[2]: https://news.ycombinator.com/item?id=29089947
I do like the idea of being able to tune and configure things from the CLI. It was frustrating configuring CrashPlan on a remote computer.
With that said, I feel like both should be possible. Even just a basic wrapper GUI would be a start.
Edit: and some basic searching has lead me to lots of options! Time to do some more research.
However I also acutely feel the lack of a standalone GUI so that I can get rid of the scrips and custom setup or at least that can be an option. (There’s a commercial third party UI I think which is a subscription)
https://vorta.borgbase.com (a third party qt GUI for borg backup) has been really awesome.
Another tool I’m looking at closely is https://kopia.io. It comes with a UI by default (Electron I guess). Though its UI and logo has quite some work left.
restic -r <repo_name> --verbose --exclude="$HOME/snap" --exclude="$HOME/Android" --exclude="$HOME/.android" --exclude="$HOME/ApkProjects" backup ~/
To me if it requires complicated shell commands it is not really "done right".
The snippet GP mentioned includes only shell functions, invocations of restic, and echo commands. How could it be simpler?
Simple for someone who is comfortable with the terminal. Scary for those who are oyherwise familiar with GUIs only.
It makes it hard for me to recommend backup tools to the non-technical people in my life, because they're looking for GUI solutions with the corners rounded off, and I want something crunchy and scriptable.
GUIs for simple stuff and 99% of the population, CLIs for hackermans who need the advanced stuff.
A lot of backup services (tarsnap comes to mind) prioritize scriptability, as I imagine many people run backups from cron or a systemd timer.
restic -r reponame /path-to-files-to-be-backuped
Everything else in the mentioned script is optional stuff for scheduling, removing old backups, etc.
# tar cvf /dev/st0 .
[1] https://abishekmuthian.com/backup-postgresql-to-cloud/
Relevant manual page in case anyone's interested: https://restic.readthedocs.io/en/latest/045_working_with_rep...
If so, I think an important difference is that mine won't hide the exit code, and you can then answer a sibling commenter (on forgetting without checking) with 'I omitted set -e at the top'.
It looks like in this example it's used simply to trim whitespace. E.g.
Just mind you exclude node_modules and the like from it :)
Also Borg has an append only mode.
Note that write-only or append-only is a bit of a misnomer, since reading the files is fine (you need the decryption key anyhow before they're of any use). It's about not being able to overwrite or remove backup data without some verification that you're not ransomware or similar.
You could do deduplication on the encrypted blocks I suppose (is that secure?)
That's always an option, no matter if it's set to append-only or not. If you don't want to pay infinite costs for storage, you will need to limit it using software on the server.
As a proof of concept, take a look at a Certificate Transparency log server. Most CT logs are configured to only accept certificates meeting certain criteria. They'll log any such certificates (their SLAs only apply to contractual users, but you don't need an SLA you're probably just writing one certificate to the log to see it works) but you cannot fill them with garbage because you can't make any certificates they'd accept, only the legitimate CAs can do that†
† The CAs have their own reasons not to let you produce heaps of garbage, even Let's Encrypt has finite resources and so it imposes rate limits.
Given the chance, there certainly will be people that use 2FA when making an append-only regular backup, but even among command line restic users I expect this will be the exception rather than the rule.
[1] https://www.tarsnap.com/tips.html#write-only-keys
That said, Tarsnap isn't free software, and doesn't come with the ability for me to self-host backups. Thus, I am somewhat at the mercy of Amazon.
Still, the advantages of Tarsnap currently outweigh the disadvantages in my opinion.
For the record, this is not a shill. I don't even know who Jenny is.
During initial development (versions prior to 1.0.0), maintainers and developers will do their utmost to keep backwards compatibility and stability, although there might be breaking changes without increasing the major version. "
they are on 0.12 now
In one of the early talks at a local hackerspace, the author did also demo decrypting the data manually if something got somehow broken. The tool is just a layer on top of relatively straightforward cryptography. From what I remember (I looked at this in 2018 so forgive any errors), you'd have to write a script that iterates over the index where it says which blocks are in which files (since it's deduplicated) and decrypt it with some standard AES library. Perhaps as a security consultant this seems easier to me than it does to others, though, but it's not as if you're without hope if the tool did break, or as if you couldn't just download a previous version from the GitHub releases page.
For extra paranoid, could clone the restic source tree (with vendored dependencies). Go language backwards compatibility is such that I should always be able to read my data.
Restic can use rclone to access cloud providers it can't access and that's something we worked together on.
I use rclone directly for backing up media and files which don't change much, but restic nails that incremental backup at the cost of no longer mapping one file to one object on the storage which is where rclone shines.
(rclone author)
Is there any interaction between these two layers of encryption, or are they completely independent?
Also, I have a crypt set up for other stuff and rather deal with one remote.
Otherwise, cascades encryption is generally not recommended.
I've been using it for a few years and was not aware of it so would be great to know if any backups were affected.
That said, I keep ~/Documents and ~/dev these days in Syncthing directories, and one of my syncthing nodes is an Ubuntu LTS server with zfs, with the zfs-auto-snapshot package installed.
I still run my old backup system periodically (once or twice a month) but I now think Syncthing is at a point of reliability where realtime cross-machine sync is now my primary safety net wrt "the machine in front of me has turned to entropy", versus some point-in-time backup.
> Is Syncthing my ideal backup application?
> No. Syncthing is not a great backup application because all changes to your files (modifications, deletions, etc.) will be propagated to all your devices. You can enable versioning, but we encourage you to use other tools to keep your data safe from your (or our) mistakes.
I'm really not super familiar with Syncthing, but it sounds like it's niche is availability as opposed to durability.
Anyone taking backups seriously already knows the 3-2-1 rule anyway.
This is the curse of sharp tools. Don't run unix if you don't want rm to rm.
I still also have backups, and zfs snapshots on a different independent machine.
I'll have two of the same RPi servers in different locations with all software running (like a hot swap) with Syncthing keeping them in sync while ZFS snapshots keeps a history. I can plug in a cold storage drive in the second dock slot once in a while too. I'll have a spare RPi4b on the shelf in case it dies. If my server dies I can take the off site hot backup home and reconfig the network and then it is my primary server. With remote duplicacy backup I'm days away from getting going again. So ZFS snapshots + Syncthing and cold storage is where I'm going (for home use). Also I want to stick with Linux because I set up Freenas 5 years ago and now I forget how to admin it so I'd rather just keep with ZFS and Linux (the zfs send from freenas to zfs recv on Linux works perfectly).
https://emborg.readthedocs.io/en/latest/
https://www.tarsnap.com/