ZFS is great, but won't help if there's a power surge that fries all the disks at once. To prevent that you'll want to store snapshots remotely somewhere.
Don't forget to replace your surge protector on an annual basis (or more frequently if you have really dirty power). Also, you can have a nasty fault in your PSU that fries everything also (If anyone knows of a surge protector that sits between the power supply and the motherboard, I'd really like to get one).
> (If anyone knows of a surge protector that sits between the power supply and the motherboard, I'd really like to get one)
I think the best you can do is just get a top-quality power supply. SeaSonic will sell you a ridiculously overengineered box with a 12-year warranty for $160; it's guaranteed to have a longer useful lifetime than any other component in your computer, probably even including the case itself.
Just use a surge protector literally right next to your PSU.
Cheap whole house surge protectors are pretty much useless. It will work for a few times, but the MOVs will degrade and fail pretty quickly. Every time a big inductive load switches on or off its going to cause a voltage spike that is going to trip the MOVs. or the MOV's Trip voltage is so high (to avoid quick degration) that they really don't provide any protect. Really a good whole house protector needs to use SADs (Silicone Avalance Diodes) or a passive LC filter. Transtector, Thor Systems (SADs) or Pricewheeler (LC Filter). These protectors can take many more surge hits than the ones you listed above. Why whole house Surge protectors don't work.https://zerosurge.com/wp-content/uploads/2016/10/USTech.pdfE... with a whole house protector it still would be a good idea to use point of use surge protectors. Since the surge has to reach the Whole House protector before it can be clamped. Surges can reach your electrical devices faster than a whole house protector can clamp. It only takes a few nanoseconds to destroy a microchip. For instance say you have a vacuum cleaner plugged into the same circuit your computer is connected to. the vacuum cleaner jams, causing a surge that will hit your computer (since it closer) than the whole house proetector is (down in your basement).
True story: I've seen Raid5 not recovering after a simple power loss. Software was working but couldn't build it up. Fortunately, we didn't have to dig further than wiping it.
Exactly, my solution right now is to create encrypted backups of my raid-z2 FreeNAS instance (protected by a UPS) and then have a nightly cronjob copy the backup to a 4TB external hard drive using duplicacy (https://duplicacy.com). This drive gets rotated with another drive stored locally within the house, but in a fireproof and fairly waterproof location, then every 3 weeks I rotate the oldest drive out to a safety deposit box and bring the current drive in the safety deposit box back into the in-house rotation. Call me paranoid, but for me it's worth the peace of mind and ease of local restoration. Once you get in the routine it's really not that much work at all. :-)
Sure. Encrypted ZFS with FreeBSD, have a power loss, goodbye data.
Should have read the manual though, it does tell you to make a backup of certain data ranges in case of encrypted ZFS for this specific case, so it's partly my fault.
That said, I'm using ZFS ever since, but on top of LUKs with linux.
Encrypted ZFS with FreeBSD is not actually native ZFS encryption, it uses GELI to handle the encryption part (which may be why it's bad at handling power losses).
I've just rebuilt my home server/nas going from 2x3tb disks in a mirror no encryption to 4 x 4tb disks in a striped mirror with native ZFS encryption on ZOL.
Most of my data is read only media content, 90% read use which is also extremely low io wise and write is only when adding new media. I was thinking power loss would only be an issue loosing data that is currently being written so would not corrupt the entire zpool?
If native ZFS encryption can lead to loosing a zpool on powerloss I might look in to buying a used cheap Dell r210 ii, stick two 8tb drives in as a mirror then stick it in a collocation data center and use native zfs send/recv incremental snapshots for offsite. Looks to be cheaper then rsync.net $30 per TB/month for ZFS when you got 2-3TB+, also can do the initial sync locally. Still looking at $700+ a year after initial hardware costs.
It's like magic. Decide how much redundancy you want. Then just add drives to it. It balances files automatically across drives. You can have remote drives in the mix.
What it is not good at, is many small files. But for my use, media files and backups (tar archives) it's a breeze. And the files are stored as normal file on the drives it distributes too, so there is nothing complicated to dig into should disaster strike. (Not that it has happened to me.)
No affiliation, just finally in a Zen state of mind when it comes to my home NAS.
Next step - make sure all of that is backed up off site too, but that is another thing altogether...
It's a really tough sell for most people when you tell them they need to have at least two copies of their data if they care about it. The vast majority of people will rather "take their chances." Usually that means another $100-500 bucks for most people.
>I don’t know what happened for sure, but I think it may have been a power surge that fried the boards on both the Synology and the USB, as they were plugged in to the same socket.
He didn't have a surge protector? Sweet Jesus. I don't plug my backed-up PC into anything not surge protected.
I have never used a surge protector, and I don't think I've ever seen one in a home or office.
Much of Europe uses buried cables, so there's less risk from lightning strikes, and I think the higher line voltage (230V) reduces the effect of something like a dodgy elevator motor in the same billing.
I have visited the US a few times, where I remember the lights would dim momentarily if the garbage compactor or the dryer were used.
Surge protection isn't a big thing in the UK these days. You can get arresters but it's really rare now to hear of computer damage as a result of mains spikes.
I re-read the article twice and didn’t see him mention he didn’t have a surge suppressor. Did I miss it or are you assuming that he didn’t have a surge suppressor?
As others have said, electronics can get fried by power surges while plugged into even commercial grade surge suppressors. Home grade surge suppressors provide even less protection.
He didn't mention having one either. It's weird not to mention anything at all about surge protection in a story about a power surge destroying important equipment. I'd assume if he had one he'd mention the state it was in after his equipment was destroyed.
Power surges, brownouts etc. don't really exist in large parts of Europe. I also never unplugged my computer(s) in thunderstorms (some of which are on a USP, most of which were simply plugged into mains, no surge protector, no USP) and lost exactly zero computers to that.
One of the few things that actually routinely comes with real surge arresters (i.e. gas-discharge tubes, not the MOVs plastered all over the place) is anything connected to a telephone line, e.g. A/VDSL modems. On some old PBXs for analog lines these were even contained in field-replaceable modules.
Can confirm, I think it's over ten years since my last power glitch. I don't think I've ever even seen a surge protector; UPS I've seen a couple of times at people who run servers at home.
Surge suppression isn't all it's cracked up to be. I work in an MSP and the number of electrically fried components behind Enterprise grade UPSs is pretty astounding.
I'm not saying it's worthless, but it's not silver bullet. You must plan for failure.
Yep, I've had a PSU get friend by a power spike even though it was plugged in through a surge protector. Like you say, I'm sure it helps, but it's not perfect.
Most UPSs don't provide much more protection than a power strip. They have a cheap "surge" protector, if you look it's often just running the tower through a ferrite ring.
Generally the input it directly connected to the output, through the ferrite ring. Only in cases of a power outage does a relay trigger and switch to the batteries. But that doesn't prevent a surge from going through the UPS.
You can hear the relay trigger when you disconnect it from the wall power, the latency is high enough to allow plenty of damage.
There do exist UPSs without this problem, but they are more expensive, heavier, generate more heat, and consume more power. Look for double conversion UPSs that in the normal state go from AC -> DC -> AC.
Kinda weird that while computers are 100% DC, they are fed with 100% AC. I was pleasantly surprised that pretty much all ham radio stuff is DC fed. MUCH easier to do UPSs, solar, wind, or have multiple pieces of equipment share the same power supply. Imagine racks with a big power supply on top (and getting rid of all the AC -> DC conversion heat). I've actually seen these, but they were unfortunately cost prohibitive.
Kudos to Synology for having a process to recover on a Linux box if the Synology box craps the bed. I was reading this story thinking "this is a good lesson on why you don't buy an expensive SAN, unless you can afford to buy two".
But, the other lesson is: Backups. Sounds like backups were shut off 6 months prior.
The other lesson is: Monitoring. Backups were going to the USB drive, but it that stopped working at some point. Unless you have some tested monitoring of your backups, you are likely to lose data.
The nice thing about Synology storage stuff is that it's a nice GUI (seriously, they're about the only company I can think of that's doing appliance management right) on top of standard and battle-tested open source tools.
This was one of the reasons I was okay with paying their prices. Even if the device completely craps the bed, I'll be able to hook the drives containing absolutely normal LVM/btrfs volumes up to another machine and get my data out.
It’s the reason I’m desperate to get my pro-photographer best mate off her old Drobo and on to a new Synology. Should be happening in the next few weeks, and not a minute too soon.
The Drobo is literally a black box to me. It either mounts and you can see disks, or it doesn’t. And if it doesn’t ... well. Yep. Right-o. I’ll try rebooting it, I guess.
We use a box with 2x2TB mirror zpools. It runs Nas4free off an 128GB SSD. Initially we wanted to use Freenas but ir needs 16GB of RAM, while Nas4free works fine with 8. It also does SMART monitoring sending emails when something is up.
It’s nice for running Docker in too and give a beginner a fairly gentle introduction. The boxes are also really nice in that you have a range of configuration options that lets you get what you need.
Don’t put big Seagate drives in it if it’s near where you sleep though as they make a bit of noise as they grind.
The best solution I found as an alternative to an off-site copy is to set up a USB drive and something like a Raspberry Pi in my car. When I pull into the garage, the house server senses it and syncs new data to it.
This can be supplemented further by having it auto sync to a computer in your office whenever you pull into work. Add in some monitoring so you can get a phone alert if any of the synced copies are more than a few days out of date, and you are almost golden.
It is hooked up to a USB battery pack, which in turn is plugged into the accessory port on the car. The hard drive powers down after the data transfer is finished.
All data is encrypted before sending to it. If it gets stolen, well it is only one of the copies of the data (however it lives hidden under the dash, and is small enough that it looks like a piece of car equipment).
I've never specifically used a Li-Ion pack in my car PC setup, storing it in the heat would worry me a bit, but surprisingly, operational temps in a car aren't bad: If he's using it after he drives it, the car has likely been cooled or heated for driver comfort, and it'll take time after he parks for the temperature in the car to return to "outdoor" levels.
My bigger concern is just storing it. It sounds like he’s using just a regular powerbank type battery which is li-ion. Letting it sit in a parking lot for a day in the summer then turning it on probably isn’t a very good thermal situation. I wonder if some sort of sealed lead acid might be better. Far more inconvenient for power management though as some sort of circuit would have to be created.
It doesn't need to, assuming you're using something like rclone [1], where the encryption is done on the "client," which in this case would be the GP's home server. The client does the encryption, treating the disk attached to the Raspberry Pi as an untrusted third-party disk. As long as you keep the decryption key safe (e.g., on a USB stick stored off-site), you should be good to go in the event that you need to pull data and decrypt it.
RaspberryPis can emulate Mario Kart on an N64, run Minecraft on Linux and even Mathematica at decent speeds; I'm willing to bet big bucks they can handle symmetric key encryption (which does not require much power).
If a crappy NAS can run disk encryption, so can a raspberry pi. They're deceivingly potent.
It got significantly better in the last generation. As a result the disk I/O and network are significantly faster. While the last 2 versions had GigE, the throughput almost doubled in the last generation.
Yes they can, and Golden Eye. I have just played Mario Kart with the kid before bedtime. It stutters a little if you have too many karts on screen, but that’s not a problem when you’re winning.
You can do encryption with any amount of CPU power, it’s just slower if you have less. Backups are not typically a case where speed is extremely important.
I've run Truecrypt on mine, and it works with small drives - 100gb or so. But it gets super hot, even with a heatsink. I wouldn't rely on it.
This was so I could use it to back up my Synology box to a set of external USB drives and retire the older Mac Mini I had been using. I'm still using the Mini...
In the case of the author, +1 for doing backups, but -1 for not establishing a chain of off-site copies (a safety deposit box at your bank, a relatives house that's some distance away, etc)
I'm not sure about disk encryption, but I know that the Raspberry Pi didn't have the juice to cut it when Syncthing was running, calculating hashes for my files. I eventually gave up, hoping that some day Go would support the use of GPU hardware to hash combined with accelerated drivers for the Pi's graphics card.
The 3rd and 4th are mutually exclusive.
If you want the power consumption of a smartphone, you also have the computational speed of a smartphone, but that should be enough for many applications.
I have experimented with a very large number of devices between NUC and Pi.
While the best processors for such devices would be ARM processors (with Cortex-A76 or Cortex-A75), nobody offers adequate devices.
You can find only devices with obsolete slow ARM processors, e.g. RK3399, which are much faster than Pi but much slower than x86, or you can find development systems or modules with modern smartphone ARM processors, but those are much more expensive than a faster Intel NUC.
Therefore, for the moment the only sensible choices are devices with either Intel Gemini Lake processors or with Intel Y-Series processors.
The devices with Intel Y-Series processors are much faster, but they are also more expensive. While Zotac has presented at CES one such device, its price and time of availability are unknown.
For now, the only device available with Y-Series is "LattePanda Alpha". Considering that its price includes 8 GB RAM, 64 GB flash and the Windows licence (even if I wiped it and I installed Linux) it is cheaper than an i3 NUC where at the same price you have no memory.
It has everything a NUC has, except that it has one less USB port (it has 1 C + 3 A, while some NUCs have 1 C + 4 A). I have installed in its 2 M.2 connectors a NVMe SSD and a second Ethernet interface. The speed was excellent, because the Kaby Lake CPU was configured for 7 W TDP with 15 W TDP for the first 28 seconds. The included fan does not start unless you run a heavy workload and even then it is silent. The size is smaller than a NUC but larger than a pico-ITX board.
If you want something cheaper, or if you want, like me, both full-size DisplayPort & HDMI connectors (not DisplayPort on USB C), then you must choose a Gemini Lake device, i.e. either Zotac ZBOX PI335 Gemini Lake (with N4100, not to be confused with the obsolete ZBOX PI335 with N3350) or an ODROID H2.
The ODROID H2 is faster and possibly cheaper (depending on the memory you buy or you already have; ZBOX has soldered memory), but it is larger. ODROID H2 is NUC-sized while PI335 Gemini Lake is pico-ITX sized. ODROID H2 has 2 Ethernet, but PI335 Gemini Lake includes WiFi & BT and it comes in a closed case which will protect it in dusty environments (both have passive cooling). PI335-GK uses a 5 V power supply, so it should be easy to be powered from a power bank. The CPU TDP is configured for steady state 6 W and 15 W for the first 28 seconds. The idle CPU power is less than 2 W. I have not measured the total idle power of the computer. With the CPU halted, the power consumption should be much less, but I have not measured it.
Gemini Lake has a speed just a little less than a Snapdragon 845, but it is much faster than older ARM processors and than older Atom processors.
There's the Intel Up board. 4 Atom cores, gigabit ethernet, 2xUSB3, 2xUSB2, HDMI and the same form factor as a Pi. You can get expansion boards that break out more USB ports (not on a hub, independent bandwidth), native UART and I2C. I've been playing with them for onboard computing on a drone, so far it is working really well. Price is about $100 for the low spec ones.
It's an 8 core ARM SBC, with 2GB Ram, Gigabit ethernet on one USB3 bus, and a SATA connector on another USB3 bus. It can fully saturate spinning rush, though it might have problems with SSD drives. In any case, it will fully saturate your GigE link with even a slow harddrive.
It runs a standard Linux kernel, with the "driver" for the hardware added, which is why it requires a special build. Latest kernel version is 4.18. There is also an OMV build for it.
Mine uses around 4-8W, depending on how much the harddrive is being used.
I've got a couple of them running as backup targets (Arq, Borg, etc) with a 4TB WD Red and Btrfs, and they've been rock solid.
If you need hardware AES, you should probably be looking elsewhere.
That’s on-site way too often for my tastes. If your house burns down while your car is in the garage, there’s a good chance you’ll lose the backup too. Adding a computer in your office fixes that, but why not just send your backups to it over the internet instead of using this ad-box vehicular transport protocol?
Initial seeding of your offsite backup over the Internet would be a pain, but most people's valuable data change rate isn't high enough to significantly tax modern broadband connections. I sync a few gigs of data a week and I never even notice it happen.
Even if it does, isn't Syncthing traffic encrypted. Also, 'they' would need to spoof your other syncthing node for the first one to start syncing with it, and if they have that much info, you've already lost.
I use syncthing to sync (mostly important text stuff) between my two machines and my android phone. Once every day one of the machines uploads the changes via duplicati.
When I had my new 'Man Shed' (It's a large workshop really[1]) built at the end of my garden, I ensured I could extend my network to it. I have an old NAS with a couple of large drives in it, and I have scheduled backups that write to it from the main file server in the house.
It's not perfect, but it's way better than not having any offsite backups at all. I have a deep distrust of Cloud storage services so for me I think this setup is the best solution I'm going to get.
---
[1] I'm in the UK, so our typical 'sheds' are absolutely tiny - this one is not.
It no so much that, but more of a control aspect. Once my backups are on someone-elses-computer I've lost control of them, and knowing my luck the day I need them will be the day the cloud provider has an outage, or even worse, goes offline.
There is no way the big cloud providers, AWS, GCP, Azure go offline without a notice period (even medium size one like DO).
Use a solution that you can easily switch from one provider to another one, switch when there is any warning that the one you use might be in trouble. In the same line, make sure people in your family (or technically savvy friends) understand your system, are able to switch and know to pay the could provider bills if you're not around - unless it is ok for your entire storage to disappear when you die of course.
Ha! That sounds like such a neat idea that I'm now skeptical that it is indeed just an idea! Do you have a write up somewhere? That sounds really cool.
And here's your recurring reminder that all cloud storage services fingerprint your uploaded files looking for TOS violations. If you haven't encrypted your backups first you are at the will of the service you are using.
I use Arq to backup, and it’s a lovely little program that isn’t a resource hog, and handles the encryption. I use it to back everything up, a copy to a removable drive, a copy that gets updated every day online, and another once a month online on a seperate server. It seems to work well, and it really is painless,after the initial seeding.
Yip, I also use arq and love it. Everything backed up into my nas and Google drive. The only irritating thing is that there's no Linux client and I suspect my next laptop will be Linux based not a MacBook.
I had a similar experience with a relative's Seagate NAS.
Except the ext filesystem was unreadable because it used a different page size. Required some shenanigans in userland by thankfully I was able to recover the data. Seemed like a software fault on the box.
The chassis had to be destroyed to remove the drive and it was interesting to see the warranty explicitly mention the customer was allowed to do this to recover their data.
First of all, given the price of storage, for backups, I don’t think anything other than mirroring makes sense. Just get 2 big hard drives for your NAS and set them up for mirroring. In the event of a filmier, you can read directly.
Next, you don’t have a full backup without offsite storage. Even if it wasn’t a power surge, there could be flooding or fire at a single location.
Indeed. I'd rather have a bunch of RAID 1s around than a RAID 5 I have to worry about rebuilding. I sync my files to two different RAID 1 setups in two different cities. No hardware failure has ever been worrying.
A surge from a lightning strike near my home travelled over the cable line to kill a network switch and the WAN port on a firewall. (Strangely, the cable modem was spared.)
Everything was on a decent UPS... but I’d completely forgotten the cable line.
You can completely air gap your network from the outside line by converting to fiber at some point (probably between your cable modem and your router, in a DOCSIS setup). Isn't foolproof, because lightning can induce current on your wires directly, but it'll help these kinds of scenarios.
Many years ago I watched someone stuff the DVDRW (remember them?) which contained all his stuff into a work PC’s pioneer slot loader drive to get some music off it. We stood there and the drive went bzzt, clang, bzzt, clang then sped up way faster than it was supposed to go. This was followed by a large bang and bits of DVDRW coming flying out and then a crunching noise.
From this I learned about single points of failure.
Edit: also in the decade and a half since I learned that you should never trust a magic box, magic piece of software or magic container file system for backups. A plain file system you can just copy your shit back from is the closest thing to a guarantee. Also it’s cheaper to curate your data carefully than end up with 4TiB of crap you’re too scared to deal with on your hands.
But define 'plain file system' - is that NTFS? ext4? ZFS? FAT32? HFS? UFS? ISO9660? TAR? cpio? Some of these are more complicated than others, and all of them have catastrophic failure modes...
These kinds of scenarios are why I built Relica [1]. It backs up to local disks, network drives, remote computers (on LAN or anywhere with a public IP address), your own cloud storage, or our own special formula we call the Relica Cloud: one upload, five independent cloud providers -- replicated in real-time.
And restores can use the open-source tool restic [2], so you don't have to be locked into Relica for accessing your data.
We're working on the ability to do byte-for-byte copies of a repository to other destinations to make data even harder to lose in these kinds of disaster scenarios, as well as a new UI to make it more pleasant and powerful.
Anyway, our goal is help make robust backup strategies like what this guy needs really, really painless, because I'm as paranoid as he should have been about losing data.
Can this also be used as a replacement for sync services such as Dropbox or SpiderOak ONE? In other words, can I sync accessible folders between computers in near-realtime, or is it only archival/backup storage?
No, Relica is not a sync service, because we do not want to sync your deletes -- we recommend using backup and sync together, because they serve two different functions.
Relica is archival software, but you can backup+archive your synced folders of course.
Coincidental timing for this article! I just wrote in my own blog yesterday about the level of importance I had placed upon the data I had stored on my PC [0]. (TLDR - Data I thought was unimportant ended up actually being the opposite right after I lost my external drive).
I'm surprised it's not mentioned here but a lot of the Synology nas's have had power supply issues in addition to the prior Intel Atom issues. I swapped out four units twice each (1815+'s) and swore off their hardware permanently.
To be fair, at least with the Intel Atom C2000-series problem, I find it difficult to blame them. We had a network edge device from another vendor die due to the C2000 defect. I figure the best the vendor can do in that case is offer to replace/fix it, which the vendor in question did. Did Synology offer to fix their C2000 devices?
After watching Steve from Gamers Nexus talk about their second Synology NAS failure in a single quarter, I instantly figured that OP was using a Synology product. Their stuff must be bad if I'm making that association.
You need one copy somewhere else! What if there's a break-in or the roof leaks or whatever? My low-tech solution is two external drives, one of which is at my Mom's place and gets swapped every month or two.
Here is what I do to hedge the risk of electrical failure (raid+usb on the same circuit), fire (both copies in same structure), malicious compromise of my machine and theft:
- My workstation has linux software raid-1 of 2x 6TiB drives (this provides robustness and uptime in case of single drive failures and ease of recovery).
- Another machine in my garage doing incremental daily backup pulls over the network. It is setup as multiple discrete hard drives thus partitioning single drive failures (low cost, the garage is a separate building, the host machine is an arm board that actually turns off HD PSU when not backing up, so hard drives are fairly isolated from power spikes).
- I make a monthly incremental backup (three sets) onto an external 6TB usb hard drive encrypted with luks. This drive spends 99% of it's life powered down in a cabinet at my office at work. It is protected against theft, fire, electrical spikes, etc... by my employer.
- The is not a *ucking cloud anywhere in this picture. I can get access to my backups within ~1hr in worst case (round trip drive to office to pickup my drive).
You Kids need to learn how to take care of your shit - now get off my lawn!!
I use Arq Backup on my workstations, along with Resilio Sync.
The NAS holds all our media/documents/music/whatever, and where possible, this gets auto uploaded from workstations to the NAS, mostly through Resilio Sync, but also ChronoSync.
A local Raspberry Pi (different building) acts as a node in the Resilio Sync setup, adding more redundancy.
The NAS backs up to a local USB drive nightly, as well as a remote (4km distance) Odroid HC2 with a WD Red drive. This device also runs Resilio Sync as a redundant node.
All machines run Btrfs where possible, with smart monitoring, daily short smart tests, weekly long smart tests, monthly scrubs, and log monitoring emailed to my inbox every morning.
Finally i make yearly archive discs (100GB M-Disc) with the data from the past 12 months. I burn these in 2 copies, one is stored locally, the other is stored remotely.
Along with these drives, i also maintain a couple of 4TB USB3 drives, which i freshen (nondestructive badblocks) yearly, and update. Again, one is stored locally, the other remotely with the M-discs.
Even with the above setup, there is a theoretical possibility of losing data, but as most data lives on both the NAS and the client machines, as well as a remote target, i would need to lose all 3 at the same time.
The only irreplaceable data would be our family photos, and those are also stored on optical and magnetic media (spending 360/365 days powered off), adding at least a couple more layers of redundancy to the equation.
I had all of my backups on a 2TB external drive, which worked great until the MFT got corrupted somehow. Suddenly all my eggs in one backup basket felt a bit silly. Fortunately I was able to recover all of it. I'm in the process of partitioning out a new backup system to avoid that in the future.
Backblaze is great. But their desktop backup solution for $5/month is not available beyond Windows and MacOS. Their B2 storage is via duplicity, but that quickly exceeds $5/month with typical storage cases.
I have to agree with the guy above. House fires often leave a significant bit of the house structure alone. I also feel that a decent number of scenarios where my house would burn down would be when I'm not there.
Their USB drive was plugged into their NAS. The probability (as they found out in TFA) is very high that a power surge will knock out most of their devices. A house fire wouldn't hesitate in destroying both. I think "close to 100%" is being too generous.
If you don't have 3 copies of your data (with at least 1 offsite) then your data doesn't really exist.
The chances of both failing get even higher when you factor in failure causes like "I was running old firmware and someone's ransomware exploited that."
- SSD + HDD in laptop, this latter for storage, because it actually tells me, if it's unhappy and about to die, unlike the m.2 ssd. The cost of this is to have an older laptop, in my case, a thinkpad x250.
- synced to home server, which, at this point, is a thinkpad x201 on an ultrabase with 2 disks - it has built-in ups, called a battery
- all of this synced to off-site rented server in Germany
- irreplacable photos are on blu-ray on yearly archives
This covers lost laptop, burglary, house fire/flood, etc. To avoid problems with lost ssh keys, I have a few users on that rented server which can log in with a password, in case of emergency.
He doesnt confirm that the drives are faulty, and it isnt clear that they are, as the data is accessible while maybe some partition or boot data was scrambled but repairable
The NAS should also have a warranty of some kind or the controller could be repaired for cheap
He was never in any data peril, so just fix that and then add an offsite backup to the mix
261 comments
[ 4.1 ms ] story [ 198 ms ] threadAnd at least Raid-Z. Raid-5 and Raid-6 are now at probability of failure levels that your rebuild is likely to throw an error.
I think the best you can do is just get a top-quality power supply. SeaSonic will sell you a ridiculously overengineered box with a 12-year warranty for $160; it's guaranteed to have a longer useful lifetime than any other component in your computer, probably even including the case itself.
Cheap whole house surge protectors are pretty much useless. It will work for a few times, but the MOVs will degrade and fail pretty quickly. Every time a big inductive load switches on or off its going to cause a voltage spike that is going to trip the MOVs. or the MOV's Trip voltage is so high (to avoid quick degration) that they really don't provide any protect. Really a good whole house protector needs to use SADs (Silicone Avalance Diodes) or a passive LC filter. Transtector, Thor Systems (SADs) or Pricewheeler (LC Filter). These protectors can take many more surge hits than the ones you listed above. Why whole house Surge protectors don't work.https://zerosurge.com/wp-content/uploads/2016/10/USTech.pdfE... with a whole house protector it still would be a good idea to use point of use surge protectors. Since the surge has to reach the Whole House protector before it can be clamped. Surges can reach your electrical devices faster than a whole house protector can clamp. It only takes a few nanoseconds to destroy a microchip. For instance say you have a vacuum cleaner plugged into the same circuit your computer is connected to. the vacuum cleaner jams, causing a surge that will hit your computer (since it closer) than the whole house proetector is (down in your basement).
Source: comment section on https://www.youtube.com/watch?v=6PqO0aQaGDY
Should have read the manual though, it does tell you to make a backup of certain data ranges in case of encrypted ZFS for this specific case, so it's partly my fault.
That said, I'm using ZFS ever since, but on top of LUKs with linux.
I've just rebuilt my home server/nas going from 2x3tb disks in a mirror no encryption to 4 x 4tb disks in a striped mirror with native ZFS encryption on ZOL.
Most of my data is read only media content, 90% read use which is also extremely low io wise and write is only when adding new media. I was thinking power loss would only be an issue loosing data that is currently being written so would not corrupt the entire zpool?
If native ZFS encryption can lead to loosing a zpool on powerloss I might look in to buying a used cheap Dell r210 ii, stick two 8tb drives in as a mirror then stick it in a collocation data center and use native zfs send/recv incremental snapshots for offsite. Looks to be cheaper then rsync.net $30 per TB/month for ZFS when you got 2-3TB+, also can do the initial sync locally. Still looking at $700+ a year after initial hardware costs.
What finally clicked for me, is https://www.greyhole.net
It's like magic. Decide how much redundancy you want. Then just add drives to it. It balances files automatically across drives. You can have remote drives in the mix.
What it is not good at, is many small files. But for my use, media files and backups (tar archives) it's a breeze. And the files are stored as normal file on the drives it distributes too, so there is nothing complicated to dig into should disaster strike. (Not that it has happened to me.)
No affiliation, just finally in a Zen state of mind when it comes to my home NAS.
Next step - make sure all of that is backed up off site too, but that is another thing altogether...
I'll bet this guy did not have a surge protector in front of his Synology PSU.
He didn't have a surge protector? Sweet Jesus. I don't plug my backed-up PC into anything not surge protected.
Much of Europe uses buried cables, so there's less risk from lightning strikes, and I think the higher line voltage (230V) reduces the effect of something like a dodgy elevator motor in the same billing.
I have visited the US a few times, where I remember the lights would dim momentarily if the garbage compactor or the dryer were used.
As others have said, electronics can get fried by power surges while plugged into even commercial grade surge suppressors. Home grade surge suppressors provide even less protection.
One of the few things that actually routinely comes with real surge arresters (i.e. gas-discharge tubes, not the MOVs plastered all over the place) is anything connected to a telephone line, e.g. A/VDSL modems. On some old PBXs for analog lines these were even contained in field-replaceable modules.
I'm not saying it's worthless, but it's not silver bullet. You must plan for failure.
Generally the input it directly connected to the output, through the ferrite ring. Only in cases of a power outage does a relay trigger and switch to the batteries. But that doesn't prevent a surge from going through the UPS.
You can hear the relay trigger when you disconnect it from the wall power, the latency is high enough to allow plenty of damage.
There do exist UPSs without this problem, but they are more expensive, heavier, generate more heat, and consume more power. Look for double conversion UPSs that in the normal state go from AC -> DC -> AC.
Kinda weird that while computers are 100% DC, they are fed with 100% AC. I was pleasantly surprised that pretty much all ham radio stuff is DC fed. MUCH easier to do UPSs, solar, wind, or have multiple pieces of equipment share the same power supply. Imagine racks with a big power supply on top (and getting rid of all the AC -> DC conversion heat). I've actually seen these, but they were unfortunately cost prohibitive.
But, the other lesson is: Backups. Sounds like backups were shut off 6 months prior.
The other lesson is: Monitoring. Backups were going to the USB drive, but it that stopped working at some point. Unless you have some tested monitoring of your backups, you are likely to lose data.
Glad this story had a happy ending.
This was one of the reasons I was okay with paying their prices. Even if the device completely craps the bed, I'll be able to hook the drives containing absolutely normal LVM/btrfs volumes up to another machine and get my data out.
The Drobo is literally a black box to me. It either mounts and you can see disks, or it doesn’t. And if it doesn’t ... well. Yep. Right-o. I’ll try rebooting it, I guess.
https://btrfs.wiki.kernel.org/index.php/Gotchas
We use a box with 2x2TB mirror zpools. It runs Nas4free off an 128GB SSD. Initially we wanted to use Freenas but ir needs 16GB of RAM, while Nas4free works fine with 8. It also does SMART monitoring sending emails when something is up.
[1] https://www.synology.com/en-global/knowledgebase/DSM/tutoria...
Exactly. Onsite backup is not a backup, especially if it is directly connected to the primary data store.
This can be supplemented further by having it auto sync to a computer in your office whenever you pull into work. Add in some monitoring so you can get a phone alert if any of the synced copies are more than a few days out of date, and you are almost golden.
What if the drive gets stolen?
All data is encrypted before sending to it. If it gets stolen, well it is only one of the copies of the data (however it lives hidden under the dash, and is small enough that it looks like a piece of car equipment).
[1] https://rclone.org/
If a crappy NAS can run disk encryption, so can a raspberry pi. They're deceivingly potent.
Yes they can, and Golden Eye. I have just played Mario Kart with the kid before bedtime. It stutters a little if you have too many karts on screen, but that’s not a problem when you’re winning.
This was so I could use it to back up my Synology box to a set of external USB drives and retire the older Mac Mini I had been using. I'm still using the Mini...
In the case of the author, +1 for doing backups, but -1 for not establishing a chain of off-site copies (a safety deposit box at your bank, a relatives house that's some distance away, etc)
Lot fewer choices are available.
NUC is expensive. I wonder when will 8core ARM based NUC will arrive in the market.
No idea is NUC can be powered by Xiomi Mi 2i powerbank but a Raspberry Pi can be
Priorities:
1. Gigabyte ethernet
2. Small form factor NUC or smaller.
3. 8+ cores
4. Very low power consumption
I have experimented with a very large number of devices between NUC and Pi.
While the best processors for such devices would be ARM processors (with Cortex-A76 or Cortex-A75), nobody offers adequate devices.
You can find only devices with obsolete slow ARM processors, e.g. RK3399, which are much faster than Pi but much slower than x86, or you can find development systems or modules with modern smartphone ARM processors, but those are much more expensive than a faster Intel NUC.
Therefore, for the moment the only sensible choices are devices with either Intel Gemini Lake processors or with Intel Y-Series processors.
The devices with Intel Y-Series processors are much faster, but they are also more expensive. While Zotac has presented at CES one such device, its price and time of availability are unknown.
For now, the only device available with Y-Series is "LattePanda Alpha". Considering that its price includes 8 GB RAM, 64 GB flash and the Windows licence (even if I wiped it and I installed Linux) it is cheaper than an i3 NUC where at the same price you have no memory.
It has everything a NUC has, except that it has one less USB port (it has 1 C + 3 A, while some NUCs have 1 C + 4 A). I have installed in its 2 M.2 connectors a NVMe SSD and a second Ethernet interface. The speed was excellent, because the Kaby Lake CPU was configured for 7 W TDP with 15 W TDP for the first 28 seconds. The included fan does not start unless you run a heavy workload and even then it is silent. The size is smaller than a NUC but larger than a pico-ITX board.
If you want something cheaper, or if you want, like me, both full-size DisplayPort & HDMI connectors (not DisplayPort on USB C), then you must choose a Gemini Lake device, i.e. either Zotac ZBOX PI335 Gemini Lake (with N4100, not to be confused with the obsolete ZBOX PI335 with N3350) or an ODROID H2.
The ODROID H2 is faster and possibly cheaper (depending on the memory you buy or you already have; ZBOX has soldered memory), but it is larger. ODROID H2 is NUC-sized while PI335 Gemini Lake is pico-ITX sized. ODROID H2 has 2 Ethernet, but PI335 Gemini Lake includes WiFi & BT and it comes in a closed case which will protect it in dusty environments (both have passive cooling). PI335-GK uses a 5 V power supply, so it should be easy to be powered from a power bank. The CPU TDP is configured for steady state 6 W and 15 W for the first 28 seconds. The idle CPU power is less than 2 W. I have not measured the total idle power of the computer. With the CPU halted, the power consumption should be much less, but I have not measured it.
Gemini Lake has a speed just a little less than a Snapdragon 845, but it is much faster than older ARM processors and than older Atom processors.
It's an 8 core ARM SBC, with 2GB Ram, Gigabit ethernet on one USB3 bus, and a SATA connector on another USB3 bus. It can fully saturate spinning rush, though it might have problems with SSD drives. In any case, it will fully saturate your GigE link with even a slow harddrive.
It runs a standard Linux kernel, with the "driver" for the hardware added, which is why it requires a special build. Latest kernel version is 4.18. There is also an OMV build for it.
Mine uses around 4-8W, depending on how much the harddrive is being used. I've got a couple of them running as backup targets (Arq, Borg, etc) with a 4TB WD Red and Btrfs, and they've been rock solid.
If you need hardware AES, you should probably be looking elsewhere.
It's susceptible to macspoofing but I hope my Pi won't join a spoofed network.
It's not perfect, but it's way better than not having any offsite backups at all. I have a deep distrust of Cloud storage services so for me I think this setup is the best solution I'm going to get.
---
[1] I'm in the UK, so our typical 'sheds' are absolutely tiny - this one is not.
The data I really care about is encrypted right on my laptop,and I backup that encrypted data.
Use a solution that you can easily switch from one provider to another one, switch when there is any warning that the one you use might be in trouble. In the same line, make sure people in your family (or technically savvy friends) understand your system, are able to switch and know to pay the could provider bills if you're not around - unless it is ok for your entire storage to disappear when you die of course.
Throwing it on multiple clouds with version history isn't an issue.
(I'd recommend an O365 sub + duplicati...in theory you can push like 5TB to MS cloud).
Plus ironically all the content that would score me a TOS violation doesn't get backed up since it's easily replaceable...
Except the ext filesystem was unreadable because it used a different page size. Required some shenanigans in userland by thankfully I was able to recover the data. Seemed like a software fault on the box.
The chassis had to be destroyed to remove the drive and it was interesting to see the warranty explicitly mention the customer was allowed to do this to recover their data.
First of all, given the price of storage, for backups, I don’t think anything other than mirroring makes sense. Just get 2 big hard drives for your NAS and set them up for mirroring. In the event of a filmier, you can read directly.
Next, you don’t have a full backup without offsite storage. Even if it wasn’t a power surge, there could be flooding or fire at a single location.
Always remember the basic 3 2 1 rule!
From my families perspective it’s just one drive, but it’s replicated everywhere.
Everything was on a decent UPS... but I’d completely forgotten the cable line.
You can completely air gap your network from the outside line by converting to fiber at some point (probably between your cable modem and your router, in a DOCSIS setup). Isn't foolproof, because lightning can induce current on your wires directly, but it'll help these kinds of scenarios.
From this I learned about single points of failure.
Edit: also in the decade and a half since I learned that you should never trust a magic box, magic piece of software or magic container file system for backups. A plain file system you can just copy your shit back from is the closest thing to a guarantee. Also it’s cheaper to curate your data carefully than end up with 4TiB of crap you’re too scared to deal with on your hands.
The catastrophic failure modes are present only through hardware and misuse issues on a mature file system.
These days, I'd suggest using a file system with some form of file checksum metadata. If one values the integrity of their data, bit rot is a thing.
And restores can use the open-source tool restic [2], so you don't have to be locked into Relica for accessing your data.
We're working on the ability to do byte-for-byte copies of a repository to other destinations to make data even harder to lose in these kinds of disaster scenarios, as well as a new UI to make it more pleasant and powerful.
Anyway, our goal is help make robust backup strategies like what this guy needs really, really painless, because I'm as paranoid as he should have been about losing data.
[1]: https://relicabackup.com
[2]: https://github.com/restic/restic
Relica is archival software, but you can backup+archive your synced folders of course.
[0] - http://devan.blaze.com.au/blog/2019/1/20/the-folly-of-unimpo...
https://www.youtube.com/watch?v=K7ly8zde3dE
- My workstation has linux software raid-1 of 2x 6TiB drives (this provides robustness and uptime in case of single drive failures and ease of recovery).
- Another machine in my garage doing incremental daily backup pulls over the network. It is setup as multiple discrete hard drives thus partitioning single drive failures (low cost, the garage is a separate building, the host machine is an arm board that actually turns off HD PSU when not backing up, so hard drives are fairly isolated from power spikes).
- I make a monthly incremental backup (three sets) onto an external 6TB usb hard drive encrypted with luks. This drive spends 99% of it's life powered down in a cabinet at my office at work. It is protected against theft, fire, electrical spikes, etc... by my employer.
- The is not a *ucking cloud anywhere in this picture. I can get access to my backups within ~1hr in worst case (round trip drive to office to pickup my drive).
You Kids need to learn how to take care of your shit - now get off my lawn!!
The NAS holds all our media/documents/music/whatever, and where possible, this gets auto uploaded from workstations to the NAS, mostly through Resilio Sync, but also ChronoSync. A local Raspberry Pi (different building) acts as a node in the Resilio Sync setup, adding more redundancy.
The NAS backs up to a local USB drive nightly, as well as a remote (4km distance) Odroid HC2 with a WD Red drive. This device also runs Resilio Sync as a redundant node. All machines run Btrfs where possible, with smart monitoring, daily short smart tests, weekly long smart tests, monthly scrubs, and log monitoring emailed to my inbox every morning.
Finally i make yearly archive discs (100GB M-Disc) with the data from the past 12 months. I burn these in 2 copies, one is stored locally, the other is stored remotely. Along with these drives, i also maintain a couple of 4TB USB3 drives, which i freshen (nondestructive badblocks) yearly, and update. Again, one is stored locally, the other remotely with the M-discs.
Even with the above setup, there is a theoretical possibility of losing data, but as most data lives on both the NAS and the client machines, as well as a remote target, i would need to lose all 3 at the same time. The only irreplaceable data would be our family photos, and those are also stored on optical and magnetic media (spending 360/365 days powered off), adding at least a couple more layers of redundancy to the equation.
Also ... RAID is not a backup.
So I guess the options would be desktop backup on the various computers that feed the current NAS solution.
Probably close to 100% if your house catches on fire.
Wildfires for example often destroy homes, but generally allow most people to evacuate.
If you don't have 3 copies of your data (with at least 1 offsite) then your data doesn't really exist.
- synced to home server, which, at this point, is a thinkpad x201 on an ultrabase with 2 disks - it has built-in ups, called a battery
- all of this synced to off-site rented server in Germany
- irreplacable photos are on blu-ray on yearly archives
This covers lost laptop, burglary, house fire/flood, etc. To avoid problems with lost ssh keys, I have a few users on that rented server which can log in with a password, in case of emergency.
Never worry about this again.
The NAS should also have a warranty of some kind or the controller could be repaired for cheap
He was never in any data peril, so just fix that and then add an offsite backup to the mix