I agree for this volume of data incremental is the best choice.
My projects so far ranged from 1-100GB so a full daily backup is quite quick done and its more easy to recover compared to applying incremental backups individually.
> compared to applying incremental backups individually.
At least one style of incremental backups can be restored in a single copy - I use rsync to create hardlinks to the previous day if the file is unchanged (based on modified time). This results in around 400M of additional disk usage per day due to changes, for around 90G of data in my home directory.
This is a good point; he could have physically removed one pair of drives, at least, at which point he would have had an actual backup sitting on the shelf.
Not to be a grammar Nazi, but I believe the word you're looking for is definitely. I wouldn't have normally picked it out, but I did see you use it like that in the article as well.
It was a good read! Thanks for taking your time to write it up.
That sounds like a very complicated way to do a full OS backup.
Dead easy, 10 min, can't fail alternative for home projects: use a VM instead, switch off the VM, copy the disk file, switch the VM back on. Use NVMe SSDs and depending on the size, you can probably do an automated daily backup with minimal downtime.
Anyone have a solution that can snapshot a running VM that has a PCI or PCIe card passed through to the guest OS? I use snapshots on ESXi, but they don't work for guests that have a card passed-through.
Well, it is not like I didn't do any stupid things in the past, but booting into to a different OS (live system) with such a setup is obviously risky.
I am a bit conservative when it comes to backups and I do not trust myself very much. So I am doing backups on the block device level to avoid not copying certain file types, file attributes or whatever feature the filesystem in question posses.
In order to avoid, risky OS reboots, I am using lvm snapshots like this:
# sync the filesystem to disk
sync
# create a snapshot with a 10GB buffer
lvcreate -L10G -s -n "$snapshot" "/dev/vg/$lv"
# write the backup to a backup disk
dd if="/dev/vg/$snapshot" of="/mnt/backups/lvm-vg-${lv}_backup-$(date +%Y%m%d).dd" bs=64k
# remove the snapshot
lvremove -f "/dev/vg/$snapshot"
That method is certainly not perfect, as you are still taking the backup with a mounted FS in place and the snapshot size must be adapted to the use-case, but in the end, this is a method to create backups which are complete (independent of FS), easily accessible (e.g. mount via loop device) and do not interrupt the service.
There's now way he could have possible known simply booting the Ubuntu live disk would do this. He could have done that without any consideration to doing a backup. However regular tried and true backups are the way to go, preferably automated.
My worst horror story wasn't really mine. I did desktop support in my first job. The library had their own system and their own support arrangements but I had a good relationship with them and would advise from time to time. One day they said the PC running their archive management system wouldn't boot. Could I take a quick look before they called the company on their support contract? Sure enough the PC wouldn't boot, the hard drive had died. I asked if they had recent backups? Oh yes the backups were fine, in fact it had got much better recently. Before the backups used to take an hour, but for for the last few weeks it only took 2 minutes. They only had 2 weeks of tapes on rotation.
Needless to say, the backups hadn't been running at all and failed silently, probably due to early symptoms of the hard drive failure. They lost everything.
I have seen one case like this, too. Every week, the customer put a tape into the drive, the machine did something with the tape, but apparently not what it was supposed to do. So when the application running on that machine managed to scramble its (FoxPro) database, they were quite surprised to find their tapes empty. I was really happy I did not have to work on that case. ;-)
There's a collision here between Ubuntu trying to do the right thing automatically (that's part of Ubuntu's design philosophy AIUI) and the author's need to have nothing done at all.
I think the Ubuntu live image is therefore unsuitable for this kind of work. What was needed here is a more specialist "rescue" system image that has a "forensic" mode. For example, Finnix has this: https://www.finnix.org/Forensics
Getting backup right is extremely important, that is why you need proper tools. For the last 10 years I have been using ZFS snapshots and replication. I keep 180 days with snapshots on the remote backup and 30 days for the local.
I use it to take backup of virtual machines, different databases including "running", sql servers. And of course important files, documents and photos.
I've also configured a daily mail with a log about how the backup/snapshot/replication went. Even if it went fine. So if I don't get mail, I know something is up and can fix it asap.
I have never slept so well after doing this with ZFS, even backup for Windows SQL Server with iSCSI volumes works great!
Do you use ZFS on the desktop too, or something different? I've used btrfs raid on a server before, and it worked apart one incident which took it two weeks (for 4TB of data) to recover from, so I'm probably going to move to ZFS on the next upgrade. On the desktop there seem to be many more tools for btrfs though, like timeshift [0] - effectively Time Machine for Linux.
I use NFS, which works really great and is faster than what most people believe. With the exception of no local page cache which impact latency for some applications. iSCSI is another great alternative, and with iSCSI you will also benefit from your local page cache. Though only one machine can use one iSCSI volume at the time. NFS can be shared between different machines.
In my setup there is no special network tuning either, it is completely basic MTU is 1500, NO VLAN's, both iSCSI and regular network traffic works great without impacting for each other. Performance is excellent with the cheap Mellanox ConnextX-2 ethernet cards you find on Ebay for 20 USD. There are some really excellent, cheap, silent and with 10G SFP+ ports network switches available too for just 100USD
Is it safe to backup a database by just straight copying its files? Wouldn't it be better to use a specific backup tool, e.g. pg_dump for Postgres or mongodump for Mongo?
(Hopefully you've tested recovery procedures, because as the saying goes, "your backups are only as good as your last restore")
If you use a consistent snapshot of the volume it should be ok. The database would perform crash recovery on startup but data should still be consistent.
With Postgres, it's perfectly safe and supported if you just do it according to the documentation using pg_start/stop_backup and also archive your WALs properly. Most databases have something similar.
Depends on the database / disk setup. For Mongo you should probably do an fsync/lock prior to taking the snapshot then release it, this will ensure the data on disk is in a consistent state. I don't think this is technically required with a single disk setup (at least with LVM, I've not used ZFS with mongo) but it's probably a good idea anyway and only causes a few ms hiccup.
> Is it safe to backup a database by just straight copying its files?
ZFS snapshots are atomic. Recovering from a snapshot is the same as recovering from a loss of power. All serious databases don't acknowledge commits until either the transaction log or the container is synced to disk and can handle an unexpected loss of power/reboot/system crash without significant data loss.
However, it is absolutely better to put the database into some kind of backup/write-suspend mode when taking the snapshot.
ZFS has no room for configuration errors. A common critical mistake that required making a new pool was to add a non-redundant vdev to a redundant pool, as vdevs could not be removed from pools, permanently rendering this pool non-redundant due to its weakest link.
IIRC, the ability to remove a vdev has just recently been added as a feature for ZFS-on-Linux on master, but I do not believe it is released yet.
What exactly do you do to verify that whatever you backed up was indeed backed up bit-for-bit and that the backup medium didn't just report success even though it didn't get some data stored correctly? Do you verify every backup using some tool? If yes, how long does that take? The reason I ask is because so many sites talk about the benefits of backing up regularly, using backup rotation and retention policies, etc., but they usually don't talk about verification of the backups themselves. Even many popular backup tools don't seem to handle this part (sometimes due to complexity introduced by buffer caches).
ZFS has checksums, so you get a message if some bits are off. Backups has to be verified regulary, there is no exception even with ZFS.
For example once a week I fire up a PostgreSQL instance on my remote backup server by cloning one of the snapshots and then I use pg_dump so I also have a "traditional" backup of the database. I do not spend time on verifying this dump though, as the ZFS snapshot already worked fine :-) But I sometimes use the dump to test new hardware or a new PostgreSQL version.
If you haven’t restored your data you aren’t doing backups.
At work once I set up data backups, the first thing I do is try and replace the backed up server with a restored server. And you should regularly do that if the data is at all important.
If the article writer didn't have 2 copies of his data, he didn't have a backup.
I keep a running linux server that all my desktops, phones, and other devices back up to, it's a sort of home NAS.
I have a 2nd internal drive that gets mounted and all the important photes and less personal stuff gets rsynced to it. My main drive is XFS, but the backup drive is btrfs. I use btrfs to maintain snapshots and unmount the drive when it's not in use.
I also take that same set of important stuff and rsync it to a cloud server. I have a TB "storage server" from Time4vps (affiliate link: https://billing.time4vps.eu/?affid=1881). I then use borg to backup my more sensitive personal data to my cloud server and 2nd drive.
I had several copies of my digital archive going back to 1996, including all of the art and music I made as a kid, as well as the first programs I wrote. A few weeks ago I reformatted one of my backup drives so I could use it with my roku TV to watch The Crimes of Grindlewald. Shortly after formatting the disk I realized that I had been mistaken and it was actually the only copy. The worst part is, the movie wasn't even that good.
You might be able to recover some of your info with testdisk, a linux utility. Download the system rescue cd (www.system-rescue-cd.org) if you need a rescue disk.
It is very important to elaborate that a snapshot is not a backup of anything.
If you snapshot a backup to provide a history of backups, then the resulting snapshot is also a backup (albeit not a new one, merely part of the original). However, if what you snapshot is not a backup, then neither is the snapshot.
As a good rule of thumb: If you lose data when the machine suddenly bursts into flames in the most literal sense, then that machine cannot even in the slightest be considered a backup. Not even for home use. While the thought of spontaneous combustion seems drastic, total death situations are very real and not rare at all. Lightning, power surge, power supply failure, drive controller failure, simultaneous disk failure, filesystem bugs, and the list goes.
A good backup for things that truly matter is also resistant to the entire building being lost in the flames. For ZFS, the obvious choice would be to `zfs send` to replicate to a machine in a remote location location, but beware of the incremental send hole birth issues.
If you do not use ZFS, or do not have multiple ZFS machines for replication, look at restic (https://restic.net/) and rclone (https://rclone.org/). Restic can through rclone do nice, incremental (snapshot based), deduplicated and encrypted backup to anything from google drive to your toaster. For small payloads (<5TB), consumer cloud solutions such as OneDrive ends up being quite affordable.
Similar story but way shorter. Was working on the beginning of Typeform years ago and we had some issues in production, wanted to take a look at the logs. The logs were in a tar archive, so just needed to untar it. Mixed up the arguments in the tar command and instead overwrote the logs with a empty archive. Doh. In the end, lesson learned: be 100% sure arguments are correct if you're stupid enough to run terminal commands in production environments. Obviously, duplicate data (pull down logs locally) if you're planning to read something that can possibly write instead of just reading is another lesson.
Should open source software fill up with quirks for the mistakes of other software?
This looks like a bug report to other RAID software that should clear up any obsolete flags and enforce a sane condition of the RAID.
# dmsetup remove_all
# dmraid --activate y --format isw
Is that y a "yes" flag !? eg answer Yes on any question.
In my experience, working with anything beside simple mirrors in a RAID config will eventually result in loss of data, and that most data loss is because of human failure rather then disk failure.
I think the moral of the story is; always use software RAID.
Hardware RAID (even fake ones like Intel) are time bombs, either you have fun configurations like this or the controller dies and no replacement exists.
Unless you'd absolutely need a HW RAID, I'd go for a Software RAID that mounts and works natively in Linux. I can put the harddrives into a fully foreign computer and it would work.
I’d also argue that for most individuals who don’t really need to worry about an hour or two of downtime, it’s a lot more useful to have good (multiple) current backups than potentially complex RAID configurations.
RAID is the zeroth-tier backup. At least it will prevent most of the horror stories where people accidentally destroy their backups when trying to restore them.
No. RAID allows you to combine make many highly fallible disks into one big less fallible disk.
Making independent devices into a unified device is a concept that is the polar opposite to a backup.
Your example is also why people say that two backups are needed. Once your live system fails, if you only had one backup, you are now left with none. What was a backup is now your only copy, leaving you very vulnerable.
I had this happen eleven years ago, when an integrated hardware RAID controller in a Dell server decided to literally blow up. Oddly enough, despite all the black burn marks on the mainboard, rest of the server seemed to work just okay.
Another identical (hardware-wise) cold spare was already in the rack right below the unlucky one, but it wouldn't accept the transplant license dongle because of service tag mismatch. First I had to install Windows 95 just to run the service tag changer tool and flash the firmware on the spare controller to match the original before it recognized the old disks as an existing array.
In around one week we were back up and running (this was a non-workplace environment so there were no concerned execs bugging us for status updates every fifteen minutes). Had we been using mdraid, we could just have inserted the disks in just about any commodity PC hardware. In hindsight, just restoring the week-old backups would have been easier.
Lesson learned: even if you are using hardware RAID and have an ostensibly identical hardware spare available, it won't necessarily save you.
Depends on the use-case really. I use raid (if any) for performance reasons, so I prefer hardware raid with a nice controller and some lots of spindles where flash is not an option. I would never implement this with a software raid solution for reasons.
Likewise. If it’s not offline, verified and off-site, it’s not a backup.
Raid is useful for availability as well as performance though.
Also while we’re discussing raid, the price/gb of a mirrored set of larger disks vs a raid 5/6/10 of smaller disks (for the same total capacity) is always worth checking - usually comes out in favour of mirrored pair in my experience which also has better performance (especially when resilvering) and resilience and lower energy costs.
seconded - storage is cheap, and with ZFS, mirrors can be grown, wheras arrays can't. i recently grew a mirror from 3 TB to 8 TB. strongly recommend using a three-way mirror though. don't be cheap.
I use Code42 CrashPlan[0]. This has clients for MacOS, Linux and Windows and it's been flawless since I started using it years ago for personal and business use.
Going by the names and the descriptions I've read, the overloaded term 'backup' only applies to Carbon Copy Cloner and Clonezilla in the sense that they make a copy of your storage, so don't protect against corruption (such as ransomware, or you accidentally corrupting files, or deleting files...) Please let me know if I'm wrong on this. Of course you can do real backups with a clone-only tool if you rotate the backup media yourself, but I prefer this aspect to be handled automatically.
For macOS, Time Machine can do automatic backups to a mounted NAS volume over WiFi. So my Macbook backs up to a ZFS 3-way mirror. Important files are also stored in DropBox. That's how I get relatively hassle free 3-2-1 backups.
For Windows, a similar strategy is possible, but I consider my Windows box disposable (only gaming), so don't have much recent experience.
I stopped using TM. It had (maybe still does) the really nasty habit of corrupting the virtual backup volume when my laptop's connection to the server was interrupted. You know, like closing the lid and taking it in to work. The TM volumes were on a Netatalk server; _it_ had no problem maintaining a consistent wrapper volume (the files inside it, including the TM virtual containers, were always perfect), but TM apparently made/makes non-journaled, non-atomic internal writes.
Just a risk I see here, you have everything IN your home.. laptop, backup.
Unless your NAS is somewhere (not your home's internal wifi network but a WLAN). Consider a Carbonite (or similar) that has infinite backup space to make sure you have the 'offsite' covered.
Acronis: I use (monthly) Acronis to drop a .tib file to an external hard disk Z: (license came free with an external drive)
Carbonite: I use carbonite to backup all my C: drive stuff (documents, desktop, favorites, portables) and my Z: drive that has the trove of photos, videos, archives, audiobooks, tools, setups, etc. PLUS the Acronis .tib file that contains the full drive clone.
Carbonite backups up 24/7 so I have even the latest changes.
The only painful thing would be to find a new/similar laptop to restore my data.
If it would be a different brand laptop, then I would have to go through the pain to setup most stuff. Chances are that restoring to a new laptop may work right off the bat, even if I will have to spend X hours finding appropriate drivers (better than setting up everything from scratch).
I've used Acronis for about 10 years now. Got one set going to the cloud, one to the NAS, and once in a while a third to an external HDD I keep elsewhere.
It has saved me a few times. In particular it saved my ass when my brand new OCZ SSD died on me on day 2. Due to being full-disk backups, I was back up and running in 30 minutes on a spare HDD with no essential data loss.
- Your data is at most risk from yourself. Other risks such as hackers, burlgars, fire or hardware failures aren't nearly as dangerous to your data as you are.
- Backup is HARD. Have someone else do it.
- Mirroring is fine for one first copy. Whether that copy is raid or just another NAS or something, but there should always be a proper BACKUP too.
- To be a proper backup it should be 1) Write only/snapshotting, so even if you remove all your local files, the deltions aren't mirrored. 2) Have long retention. When you mess up a document you want to be able to notice 2 years later and just fetch that version 3) Be off site so theft and fire doesn't risk the primary copy and the backups.
- Get backups sorted out before you consider using it as a production system.
If OP had iterated on having a backup system BEFORE adding important data, they wouldn't have consequences to trying to implement it. At the very least, do an online, filesystem level backup of the things that are important before trying to block level disk copy things. Run a database dump, rsync it to an external location, put important things in version control and push to a remote origin. And rsync or tarball your home directory and any other important directories(whole system if possible) and push it out somewhere, then and ONLY then should you feel semi-comfortable to start messing around with RAID settings/LVM/fdisk/ etc.
The last time I lost all most of my data was when I tried upgrading Ubuntu to the next major version using the upgrade manager in Ubuntu.
I thought that since the UI was so friendly looking, it must be the safest route. WRONG. It was the second time that I trusted Ubuntu's UI for doing something critical. Never again.
Now I always back up all the files I need onto an external drive and then I install the OS from scratch. Only trust the command line.
No mention of offsite backup. Article could have stopped there.
More subtly, he is the epitome of what I call "cowboy programmer/sysadmin", who doesn't really know what he's doing, just always improvising the next step based on minimal research. Which is fine and productive, until you are dealing with critical data/systems.
I've witnessed so many times during my career this kind of people royally fuck things up with this "I'm a terminal God, of course I'm doing it live on production" attitude.
Everybody I know who used hardware RAID for their personal setup ended up losing their data EVERY SINGLE TIME. Hardware RAID has zero tolerance for human error, it's like the Suicide Linux distribution.
Personal attacks are not ok here, regardless of how bad you feel someone is at backups.
Please review https://news.ycombinator.com/newsguidelines.html and follow the rules when posting here. We've already had to ask you many times. When accounts continue to violate the guidelines, we end up banning them, and I don't want to do that.
To me the takeaway is that backup system should be designed with the storage system, when the storage system is still empty of valuable data. If you change the backup system, you should treat that as a change of the whole storage system, so first you would run the former backup, archive that securely, set up the new system, check it and only then release the former backup as not critical anymore.
Raid is not backup... Your data isn't backed up unless there are at least 3 full copies in 2 physical locations. I have to admit I'm slightly paranoid about it... 20-something years ago I was working in tech support for iomega (mostly os/2 and jazz calls). Nothing taught me the importance of true backups more than that job.
I do leverage a few things though... for the most part, I keep stuff on my nas. Projects are in git remotes as well as several local copies. Serial numbers, access keys etc are encrypted and stored in dropbox and copied to google drive.
These days since most of my work is in source control, I'm less worried about that aspect. That said, however, RAID is not backup.
I understand... by 3 full copies, that can include the active in-use copy. I just don't consider something backed up short of having the data in a separate device and location.
It has to be on a separate device to be a backup indeed, just like a spare wheel isn't a spare if you're driving on it.
The location thing is trickier, and while you can't have a any backup on the same device, you can have a bad backup on the same location. A bad backup is better than no backup by a long shot.
I'm fine with a backup on-site... I just feel that data isn't safely backed up unless it's off-site too. Not that everything needs to go that far. But it's usually useful to think in those terms imho.
I back up all my personal data from each associated desktop via SpiderOak. Personal server data I rely on a combination of rsnapshot, rsync, and SpiderOak.
I also rsync to an external drive bay that I swap out weekly.
At the end of the day, every single machine in my network of things, including my desktop can light itself on fire - and I don't care. If my office or house is bombed, I don't care. Now - if my city were bombed.. I'd argue that I probably have other fish to fry.
I don't have backups myself, but I have 5 copies of my data at 3 different locations + snapshots everywhere: desktop, laptop (linux, BTRFS), NAS at home (FreeBSD, ZFS); off-site dedicated server at OVH (FreeBSD, ZFS); at school (FreeBSD, ZFS/netapp, managed by school IT).
Syncthing [0] + whatever handles local snapshots [1,2] work wonders [3]
It's commendable that the OP was looking to backup but has the reliability of RAID against disk failure contributed to this being a "black swan" event?
Do we reduce failure rates to "acceptable" at the expense of neglecting recovery?
125 comments
[ 2.9 ms ] story [ 505 ms ] threadi'm basically asking what the impetus was for "just going for it" which is probably going to be a reason that other people share...
Personally I prefer full backups - they feel more complete.
But also, there are different styles of incremental. Some are dodgier than others.
My projects so far ranged from 1-100GB so a full daily backup is quite quick done and its more easy to recover compared to applying incremental backups individually.
Well, always choose the right tool for the task.
At least one style of incremental backups can be restored in a single copy - I use rsync to create hardlinks to the previous day if the file is unchanged (based on modified time). This results in around 400M of additional disk usage per day due to changes, for around 90G of data in my home directory.
The bright side is, you'll never make this mistake again. The downside is, for me at least, I end up with triple+ backups of everything..
It was a good read! Thanks for taking your time to write it up.
Dead easy, 10 min, can't fail alternative for home projects: use a VM instead, switch off the VM, copy the disk file, switch the VM back on. Use NVMe SSDs and depending on the size, you can probably do an automated daily backup with minimal downtime.
I am a bit conservative when it comes to backups and I do not trust myself very much. So I am doing backups on the block device level to avoid not copying certain file types, file attributes or whatever feature the filesystem in question posses.
In order to avoid, risky OS reboots, I am using lvm snapshots like this:
That method is certainly not perfect, as you are still taking the backup with a mounted FS in place and the snapshot size must be adapted to the use-case, but in the end, this is a method to create backups which are complete (independent of FS), easily accessible (e.g. mount via loop device) and do not interrupt the service.My worst horror story wasn't really mine. I did desktop support in my first job. The library had their own system and their own support arrangements but I had a good relationship with them and would advise from time to time. One day they said the PC running their archive management system wouldn't boot. Could I take a quick look before they called the company on their support contract? Sure enough the PC wouldn't boot, the hard drive had died. I asked if they had recent backups? Oh yes the backups were fine, in fact it had got much better recently. Before the backups used to take an hour, but for for the last few weeks it only took 2 minutes. They only had 2 weeks of tapes on rotation.
Needless to say, the backups hadn't been running at all and failed silently, probably due to early symptoms of the hard drive failure. They lost everything.
I think the Ubuntu live image is therefore unsuitable for this kind of work. What was needed here is a more specialist "rescue" system image that has a "forensic" mode. For example, Finnix has this: https://www.finnix.org/Forensics
I've also configured a daily mail with a log about how the backup/snapshot/replication went. Even if it went fine. So if I don't get mail, I know something is up and can fix it asap.
I have never slept so well after doing this with ZFS, even backup for Windows SQL Server with iSCSI volumes works great!
More people should do backup with ZFS!
[0] https://github.com/teejee2008/timeshift
"The parity RAID code has multiple serious data-loss bugs in it. It should not be used for anything other than testing purposes" [1]
[1] https://btrfs.wiki.kernel.org/index.php/RAID56
In my setup there is no special network tuning either, it is completely basic MTU is 1500, NO VLAN's, both iSCSI and regular network traffic works great without impacting for each other. Performance is excellent with the cheap Mellanox ConnextX-2 ethernet cards you find on Ebay for 20 USD. There are some really excellent, cheap, silent and with 10G SFP+ ports network switches available too for just 100USD
(Hopefully you've tested recovery procedures, because as the saying goes, "your backups are only as good as your last restore")
ZFS snapshots are atomic. Recovering from a snapshot is the same as recovering from a loss of power. All serious databases don't acknowledge commits until either the transaction log or the container is synced to disk and can handle an unexpected loss of power/reboot/system crash without significant data loss.
However, it is absolutely better to put the database into some kind of backup/write-suspend mode when taking the snapshot.
Shut down the DB, or use a CoW filesystem which can do atomic snapshots (zfs, btrfs, bcachefs, apfs, ...)
Even an email saying 'things went well' doesn't tell you that you can restore your files.
IIRC, the ability to remove a vdev has just recently been added as a feature for ZFS-on-Linux on master, but I do not believe it is released yet.
For example once a week I fire up a PostgreSQL instance on my remote backup server by cloning one of the snapshots and then I use pg_dump so I also have a "traditional" backup of the database. I do not spend time on verifying this dump though, as the ZFS snapshot already worked fine :-) But I sometimes use the dump to test new hardware or a new PostgreSQL version.
https://github.com/elliottslaughter/integrity-checker
At work once I set up data backups, the first thing I do is try and replace the backed up server with a restored server. And you should regularly do that if the data is at all important.
I also take that same set of important stuff and rsync it to a cloud server. I have a TB "storage server" from Time4vps (affiliate link: https://billing.time4vps.eu/?affid=1881). I then use borg to backup my more sensitive personal data to my cloud server and 2nd drive.
If you snapshot a backup to provide a history of backups, then the resulting snapshot is also a backup (albeit not a new one, merely part of the original). However, if what you snapshot is not a backup, then neither is the snapshot.
As a good rule of thumb: If you lose data when the machine suddenly bursts into flames in the most literal sense, then that machine cannot even in the slightest be considered a backup. Not even for home use. While the thought of spontaneous combustion seems drastic, total death situations are very real and not rare at all. Lightning, power surge, power supply failure, drive controller failure, simultaneous disk failure, filesystem bugs, and the list goes.
A good backup for things that truly matter is also resistant to the entire building being lost in the flames. For ZFS, the obvious choice would be to `zfs send` to replicate to a machine in a remote location location, but beware of the incremental send hole birth issues.
If you do not use ZFS, or do not have multiple ZFS machines for replication, look at restic (https://restic.net/) and rclone (https://rclone.org/). Restic can through rclone do nice, incremental (snapshot based), deduplicated and encrypted backup to anything from google drive to your toaster. For small payloads (<5TB), consumer cloud solutions such as OneDrive ends up being quite affordable.
> I keep 180 days with snapshots on the remote backup and 30 days for the local.
That required some great caution.
https://news.ycombinator.com/item?id=18541493
Should open source software fill up with quirks for the mistakes of other software? This looks like a bug report to other RAID software that should clear up any obsolete flags and enforce a sane condition of the RAID.
In my experience, working with anything beside simple mirrors in a RAID config will eventually result in loss of data, and that most data loss is because of human failure rather then disk failure.
Hardware RAID (even fake ones like Intel) are time bombs, either you have fun configurations like this or the controller dies and no replacement exists.
Unless you'd absolutely need a HW RAID, I'd go for a Software RAID that mounts and works natively in Linux. I can put the harddrives into a fully foreign computer and it would work.
I’d also argue that for most individuals who don’t really need to worry about an hour or two of downtime, it’s a lot more useful to have good (multiple) current backups than potentially complex RAID configurations.
Making independent devices into a unified device is a concept that is the polar opposite to a backup.
Your example is also why people say that two backups are needed. Once your live system fails, if you only had one backup, you are now left with none. What was a backup is now your only copy, leaving you very vulnerable.
Another identical (hardware-wise) cold spare was already in the rack right below the unlucky one, but it wouldn't accept the transplant license dongle because of service tag mismatch. First I had to install Windows 95 just to run the service tag changer tool and flash the firmware on the spare controller to match the original before it recognized the old disks as an existing array.
In around one week we were back up and running (this was a non-workplace environment so there were no concerned execs bugging us for status updates every fifteen minutes). Had we been using mdraid, we could just have inserted the disks in just about any commodity PC hardware. In hindsight, just restoring the week-old backups would have been easier.
Lesson learned: even if you are using hardware RAID and have an ostensibly identical hardware spare available, it won't necessarily save you.
Raid is useful for availability as well as performance though.
Also while we’re discussing raid, the price/gb of a mirrored set of larger disks vs a raid 5/6/10 of smaller disks (for the same total capacity) is always worth checking - usually comes out in favour of mirrored pair in my experience which also has better performance (especially when resilvering) and resilience and lower energy costs.
Also for data integrity, with something like ZFS.
For home desktops, that is. Server end is different.
For complete system backup on Windows use Acronis [0], on MacOS use Carbon Copy Cloner [1] and on Linux Clonezilla [2].
Also never forget the 3-2-1 rule of backups [3], 3 copies, 2 local copies on different mediums and 1 copy offsite (cloud, remote) :-)
[0] https://www.acronis.com/en-us/personal/computer-backup/
[1] https://bombich.com/
[2] https://clonezilla.org/
[3] https://www.backblaze.com/blog/the-3-2-1-backup-strategy/
Going by the names and the descriptions I've read, the overloaded term 'backup' only applies to Carbon Copy Cloner and Clonezilla in the sense that they make a copy of your storage, so don't protect against corruption (such as ransomware, or you accidentally corrupting files, or deleting files...) Please let me know if I'm wrong on this. Of course you can do real backups with a clone-only tool if you rotate the backup media yourself, but I prefer this aspect to be handled automatically.
[0] https://www.crashplan.com
For Windows, a similar strategy is possible, but I consider my Windows box disposable (only gaming), so don't have much recent experience.
Unless your NAS is somewhere (not your home's internal wifi network but a WLAN). Consider a Carbonite (or similar) that has infinite backup space to make sure you have the 'offsite' covered.
Honestly though, low effort is very important for me. After all, any solution is simply reducing the odds, not completely eliminating them.
Carbonite backups up 24/7 so I have even the latest changes. The only painful thing would be to find a new/similar laptop to restore my data.
If it would be a different brand laptop, then I would have to go through the pain to setup most stuff. Chances are that restoring to a new laptop may work right off the bat, even if I will have to spend X hours finding appropriate drivers (better than setting up everything from scratch).
It has saved me a few times. In particular it saved my ass when my brand new OCZ SSD died on me on day 2. Due to being full-disk backups, I was back up and running in 30 minutes on a spare HDD with no essential data loss.
- Your data is at most risk from yourself. Other risks such as hackers, burlgars, fire or hardware failures aren't nearly as dangerous to your data as you are.
- Backup is HARD. Have someone else do it.
- Mirroring is fine for one first copy. Whether that copy is raid or just another NAS or something, but there should always be a proper BACKUP too.
- To be a proper backup it should be 1) Write only/snapshotting, so even if you remove all your local files, the deltions aren't mirrored. 2) Have long retention. When you mess up a document you want to be able to notice 2 years later and just fetch that version 3) Be off site so theft and fire doesn't risk the primary copy and the backups.
If OP had iterated on having a backup system BEFORE adding important data, they wouldn't have consequences to trying to implement it. At the very least, do an online, filesystem level backup of the things that are important before trying to block level disk copy things. Run a database dump, rsync it to an external location, put important things in version control and push to a remote origin. And rsync or tarball your home directory and any other important directories(whole system if possible) and push it out somewhere, then and ONLY then should you feel semi-comfortable to start messing around with RAID settings/LVM/fdisk/ etc.
Separately, having at least a partial online backup would have minimized the issues caused by the offline backup.
Raid for backup is not so good, specially if you have only 2 drive...
Using software (all provided is too a not good example.)
Just copy to another hare drive (network, other machine) and mirroir rsync that damn folder, the cost is low and easy to setup and recover.
More subtly, he is the epitome of what I call "cowboy programmer/sysadmin", who doesn't really know what he's doing, just always improvising the next step based on minimal research. Which is fine and productive, until you are dealing with critical data/systems.
I've witnessed so many times during my career this kind of people royally fuck things up with this "I'm a terminal God, of course I'm doing it live on production" attitude.
Everybody I know who used hardware RAID for their personal setup ended up losing their data EVERY SINGLE TIME. Hardware RAID has zero tolerance for human error, it's like the Suicide Linux distribution.
Please review https://news.ycombinator.com/newsguidelines.html and follow the rules when posting here. We've already had to ask you many times. When accounts continue to violate the guidelines, we end up banning them, and I don't want to do that.
I do leverage a few things though... for the most part, I keep stuff on my nas. Projects are in git remotes as well as several local copies. Serial numbers, access keys etc are encrypted and stored in dropbox and copied to google drive.
These days since most of my work is in source control, I'm less worried about that aspect. That said, however, RAID is not backup.
A backup is a copy of data on hardware separate from the live system.
A good backup is a backup in in a remote location.
A backup strategy is to maintain a good backup.
A good backup strategy is to have routine restore tests, to make sure that the backups serve their purpose.
A great backup strategy requires at least 2 good backups.
The location thing is trickier, and while you can't have a any backup on the same device, you can have a bad backup on the same location. A bad backup is better than no backup by a long shot.
I also rsync to an external drive bay that I swap out weekly.
At the end of the day, every single machine in my network of things, including my desktop can light itself on fire - and I don't care. If my office or house is bombed, I don't care. Now - if my city were bombed.. I'd argue that I probably have other fish to fry.
I don't have backups myself, but I have 5 copies of my data at 3 different locations + snapshots everywhere: desktop, laptop (linux, BTRFS), NAS at home (FreeBSD, ZFS); off-site dedicated server at OVH (FreeBSD, ZFS); at school (FreeBSD, ZFS/netapp, managed by school IT).
Syncthing [0] + whatever handles local snapshots [1,2] work wonders [3]
[0] https://syncthing.net
[1] https://github.com/zfsnap/zfsnap
[2] https://gitlab.com/moviuro/butter
[3] https://try.popho.be/securing-home.html
Technically this is a booby trap, not a time bomb. ;-)
Do we reduce failure rates to "acceptable" at the expense of neglecting recovery?