Very interested to see what the root cause ends up being - this is mind bogglingly bad. I know several people who use these devices because of their simplicity, and they’d be devastated.
I run a FreeNAS setup at home on an HP MicroServer. From time to time, I lament the amount of time I’ve spent over the years maintaining it, and after a recent drive failure (and wishing for more performance), I started considering consumer options like Synology/QNAP.
Articles like this always give me pause, and remind me why I invest the time I do to maintain my own thing. I’d rather fuck something up myself than fall victim to an issue like this, and the time spent setting it up likely pales in comparison to dealing with data recovery.
I really hope these users can get their data back.
Qnap had a lot of security issues in the past, like unpatched RCE and backdoors. I wouldn't recommend, (and never ever use qnap cloud or upnp) as it seems they don't really do a good job on software QA or penetration testing or security audits.
Hardware is fine but it's much more expensive than just building it yourself.
QNAp takes the ‘just ship it’ approach. They have a lot of functionality, but it’s clear if you deal with their software much that it was a rush job. Lots of random breakages, bugs, stability issues, etc.
You’re wise to not trust it with anything important.
Some of their hardware is tough to do yourself. e.g. I've got a TL-D800S hooked up to my home server. I know I could build something comparable, but once hardware involves actual "building" rather than just "assembly", it's past the level of effort I want to spend.
I've never tried FreeNAS, but am surprised it took so much maintenance. I've been using FreeBSD with an HP Microserver and ZFS for the same purpose for about 10 years. I just install the updates and everything just works. Curious what kind of maintenance is taking up your time?
For me, the issues were mostly related to the period of time I started using FreeNAS, and getting a bit too creative with plugins and jails.
On the first part, there was a period of time when the project was going through some upheaval (quite a few years ago now) and I had some issues with upgrades.
This goes hand in hand with the 2nd thing - I was trying all sorts of plugins and using it as a BT and newsgroup client. These did not upgrade well.
I suspect that a fresh vanilla build (TrueNAS now) would probable be zero maintenance. I need to upgrade my drives (a bunch of 1TB drives from the era I built it), and I’ll probably use this opportunity to start fresh.
Those projects that start easy, and then suffer from:
hmm, let's try that,
then let's try these add ons, etc.
And everything works, for years, and then you need to return to them, but can't remember anything!
And then, it turns out, that you don't have the time you used to have.
And even though you loved tinkering late into the night, 10 years later, hight has become precious.
For sleeping. Or trying to sleep!
I converted a QNAP device to Ubuntu after the updates stopped coming. It’s decent and fits perfectly where it lives in my house.
Of all things, it has a weird boot-up bug where it fails to boot to Ubuntu during about a half-hour period (around :15 and :45) each hour and lands on a kernel panic within a second after grub. I have no idea where to start. Firmware is latest.
I’m about 3/4 certain that it’s between those times, and never at the top of the hour.
Getting the exact panic would unfortunately mean to have to get it down from the shelf where it lives, and plugging it into a monitor to replicate. It doesn’t make it out of low-res text mode when it does.
Most of the time, I’m usually in a rush to get it back up as my family wants lights and Plex to work normally.
If you get a chance to test, I'd recommend changing the time on the device and seeing if that makes a difference, should clue you in on if it's a software or environment issue.
My guess is someone figured out how to bounce commands off WD command server (customer reps helping people reset their devices) and they are spamming customer ids.
Either that or these devices have unique ports on the WAN with a 0-day but that would take a lot more effort to exploit.
That CVE only shows how to factory-reset a single MyBook to which you can connect and send HTTP requests. There's still something missing to explain how it could happen to seemingly all MyBook's at once, including those behind a NAT.
There are comments in this thread from people who read about this in the news, checked their own device, and found it was wiped as well. This seems like it would be very unlikely to happen if this were just happening in isolated cases.
>Very interested to see what the root cause ends up being
The article says:
>Multiple users reported that the data loss coincided with a factory reset that was performed on their devices. One person posted a log that showed unexplained behavior occurring on Wednesday:
I'm leaning more towards 0day than "software bug", mainly because I can't really think of a reason why a bug would cause the factory reset to get triggered, but I can think of plenty of ways an internet connected NAS can be exploited.
For certain notions of "read/write", the risk can be lower. You might make backups to storage, but not have permission to delete files. Or when you delete files, they are kept for a certain safety window (e.g. if you delete a file on June 24, the file is instead moved and can be restored any time before July 24, at which point it is purged).
>I am guessing this is normal? My data backups are offline. The idea of making my backups read/write on the internet seems insane.
It's a NAS. Backup isn't the only thing you can do with it. A personal cloud (aka dropbox) is also a valid use case, and requires internet access to work.
There are countless systems out there which are far older and also more secure. Age is of zero relevance except to those who propagate forced obolescence, mindless consumption, and generation of massive amounts of e-waste. This "update mentality" is a cancer itself. Especially when this is supposed to be a simple device.
That said, relying on the same credentials for a long time is more problematic. My bet is on credentials being bruteforced, especially if many people have left them at their default or used easily guessable ones.
Also - age is of zero relevance to internet connected linux devices?
I rely on many "8-10 year old device that hadn’t received a security update in 6+ years". My coffee machine. My fridge. My hifi. All my motorcycles. None of them run linux and are directly connected to the internet.
But think about this from the perspective of the average buyer/user. Hard drives are traditionally "local". Local = safe. Now, a new generation of drives are introduced with this awesome capability to get access to data remotely. "Cool", they think. They don't understand how the Internet works, and so they don't really see the issue, or if they have a basic understanding of networks they might understandably believe that "it's behind the router, so it's safe".
The remote access feature was disabled by default. For many people, the only reason it was accessing the internet was so it could get security updates.
I can't imagine a scenario where people assume an 8 y.o. device is automatically trash, only because it's old.
I'm typing this on a 10 y.o. laptop. It still works. Sure, the manufacturer and browser vendor released updates, but it wouldn't be too out of the ordinary if it didn't.
That being said, I think its predecessor would still work (for posting a comment to HN at least)
QNAP's recent situation was hard to believe. Having only just fixed a significant SQL injection vulnerability[0], a spate of ransomware hit. It was originally announced to be this issue that people rushed out to patch[1].
It ultimately turned out to be a backdoor account[2]. Backdoors really upset me. Every reasonable "you can't blame people for mistakes" goes out the window when it's not a mistake.
Following the ransomware, they released an auto-installing malware remover. A Python script that detected one particular piece of code associated with this recent attacker. That malware remover was a python script full of vulnerable exec calls introducing multiple new RCEs.
Wow. Back door accounts are just totally inexcusable these days - manufacturers should be held to account if their back doors end up getting exploited by threat actors. (Granted, you could argue it’s hard to distinguish between a really terrible bug and a back door, sometimes, but “intent” should cover this difference…)
Would you happen to have a write up/article about this series of events that isn’t a sanitized series of security bulletins? Sounds like it’d make for good reading.
Reading further along in the forum, the `walter` thing sounds like it's only present in test code and comments.
The actual backdoor looks like the `jisoosocoolhbsmgnt` session ID [1] that was removed in the update [2]. It looks like a hardcoded session ID used for tests [3]. Leaving something like that hardcoded and active in the production code is inexcusable.
I wouldn't call it a backup without further clarifications. Like, if it can be killed by the same power surge that takes your main server out, I wouldn't call it a backup.
There are a lot of ways to introduce redundancy in the system that would make backups needed much less frequently (if ever).
But what a backup does is it allows you to have your data in case your system and accounts are compromised, whether through a misuse, failure, or an attack vector.
ANY backup solution can go down with a big enough disaster.
An on-site backup is A backup, but it shouldn't be the ONLY backup. Typically you want a hierarchy of backups, one of which is on-site and on-line but a different machine, another of which is off-site but on-line, and possibly one or more offline backups. But on-line backups are fine, IF (and only if) they're copy-on-write/append-only.
Of course if the data can be deleted or overwritten too soon (even tape drive backup systems have some period over which they start re-using tapes) it's worthless. But that's an entirely separate concern from whether it's always-on or internet-connected or on the same national power grid segment as the original.
I was with you through the rest of the thread until here, but this is too extreme. By your definition there does not exist such a thing as an in-site backup, does there?
Take it to the extreme and everything is connected somehow at some point, thereby making a True Backup by your definition physically impossible, even off-site. The point where the backup is made could have a zero-day propagate to the tape machine, killing it and the tape by spinning up the motors. A blueray drive could have the firmware pwned.
If not even incremental snapshots synced to a remote count as backups, you are probably the only one to harbor this interpretation of the word and it arguably loses its usefulness as a word.
Well, as long as you can recover from an onsite backup after logging into your system and completely destroying everything you can, it counts.
An incremental snapshot to a remote is as good as a backup as WD users just found out.
>The point where the backup is made could have a zero-day propagate to the tape machine
Again, the point here is the failure of the backup is decoupled from the failure of your system.
Again, everything is on a spectrum, and we can argue about definitions, however, a copy on a hard drive that's sitting on your bookshelf is decoupled from whatever happens to your computer unless both get destroyed (or stolen) - and that's how good that backup solution is.
The data on a RAID array will go poof if you accidentally rm -rf that partition, so the chances of it failing when the system fails are very high. Hence it's an awful backup solution.
What you have to consider is not whether the chances of failure are high or low, but where the chances of failure of both the system and the backup at the same time are non-negligible.
Well, as long as you can recover from an onsite backup after logging into your system and completely destroying everything you can, it counts.
An incremental snapshot to a remote is as good as a backup as WD users just found out.
>The point where the backup is made could have a zero-day propagate to the tape machine
Again, the point here is the failure of the backup is decoupled from the failure of your system.
Again, everything is on a spectrum, and we can argue about definitions, however, a copy on a hard drive that's sitting on your bookshelf is decoupled from whatever happens to your computer unless both get destroyed (or stolen) - and that's how good that backup solution is.
The data on a RAID array will go poof if you accidentally rm -rf that partition, so the chances of it failing when the system fails are very high. Hence it's an awful backup solution.
What you have to consider is not whether the chances of failure are high or low, but where the chances of failure of both the system and the backup at the same time are non-negligible.p
I don't know if this has gone to court yet - but I'd assume that if Dropbox does a thing to delete a bunch of data that causes you a great loss of value when you're paying them specifically to store a bunch of data - then there would be enough of an implied contract in place that you could extract a bunch of pain from them regardless of what their TOS tries to say.
The law is complicated and the details are extremely important - but intent is a pretty big factor.
Also - I'd strongly advise someone who is deciding between buying an external hard drive or using a third party for backup to use the third party. There are definitely risks that remain, but their SLA is likely going to end up being a lot stronger than your hard drive's reliability.
Then people irresponsibly storing single copies of critical data on Dropbox raise the cost for more responsible users, because upon loss Dropbox has to pay out the high value of the loss to the irresponsible user, but pay next to nothing to the reponsible users, who already have multiple backups or store less critical data.
It should at least be a tiered offering with different liability caps.
That has gone to court. They're using the same limitation of liability (including data loss), as-is clause, and waiver of warranties found in every software license. It's entirely enforceable.
Read the TOS. You're not getting anything from them if they delete your data.
I am very sorry to break it to you, but there is no Santa, and you'll be SOL if Dropbox's SLA hits a SNAFU.
Also, from another comment in this thread:
>I've "lost" (had to restore from personal backup drives) data from Dropbox due to an error made by their support staff during a mass rollback, which was itself needed due to the Dropbox client interpreting a drive disconnection as a mass file deletion.
By that logic _nothing_ is a backup. A disconnected hard-drive could be lost in a fire, theft, physical damage, you could stop paying your rent and your landlord could repossess it...
Telling people that any redundant copy that isn't 100% reliable (i.e., everything) isn't a backup is not a good way to get people to take backups.
While a hard-drive could be lost in a fire, etc., the state of availability of the data on it is decoupled from that of the system that has the data that's being backed up.
And that's the entire point! Take that as a definition: a backup is a redundant copy whose availability is independent of that of the original.
FYI, from one of the comments below:
>I've "lost" (had to restore from personal backup drives) data from Dropbox due to an error made by their support staff during a mass rollback, which was itself needed due to the Dropbox client interpreting a drive disconnection as a mass file deletion.
How decoupled is decoupled enough though? Does it need to be air-gapped?
You could access your dropbox data from computer B even while it is syncing files from computer A. Together with the 30 day file recovery window, this seems fairly decoupled to me. Assuming that the dropbox file history and the dropbox normal sync is seen as two separate systems.
Now you shouldn't have this as your only backup, but combine it with a second service with similar guarantees and even if you wipe your local drive and the wipe get propagated to dropbox, you are still backed by the 30 day history inside both dropbox and the second service, which are decoupled from each other.
I think it is correct that, without unusual precautions, Dropbox should not be counted as a distinct backup copy for satisfying the 3-copies rule. I've "lost" (had to restore from personal backup drives) data from Dropbox due to an error made by their support staff during a mass rollback, which was itself needed due to the Dropbox client interpreting a drive disconnection as a mass file deletion. (This was before they implemented mass rollback through the UI; it's a safer process now.) And this is fine. Dropbox does file synchronization, which means it propagates data loss as well as intentional changes.
You know, Dropbox functions in part on the assumption that users act independently. I wonder if they can tell, and if so what happens, if a vendor bug like the one in the article cause a large fraction of users to take the same action simultaneously.
>Dropbox allows you to roll back changes and undelete files
What you are saying, is that Dropbox allows you to purchase a backup service from them (without any guarantees, at that).
Cloud sync isn't backup. If someone performs backups on the either side of the cloud, that's great, but that's an extra, which is what I'm saying to begin with :)
It's more nuanced than that. Many of those things you mention are backups, just not necessarily that reliable for certain threats. By your standards, you'd probably say tapes, removed from their drive, are a backup. Yet, buildings burn down, taking the tapes with them. If restoring backups from off-line, off-site storage is going to take days or weeks, then for some organisations, you may as well not have a backup. Cloud sync, duplication, connected devices are fine for a backup, but probably not as the only backup. But they still reduce the likelihood of total data loss compared to not having them.
Diversity and risk analysis are what's important here.
I agree with what you say, I'm just nitpicking on differentiating redundancy from backups.
Yes, a building with tapes can burn down. But whether it does so or not is not correlated with the state of the system that the original data lives on.
Redundancy is a great step to prevent data loss. But redundancy won't do what a backup would: keep your data safe even if your system is fucked.
Cloud sync is a double-edged sword, because you make Dropbox a part of your system; their failures are not decoupled from yours anymore. Say, someone hacks into your Dropbox account, encrypts/deletes all data, and then downgrades/cancels your plan so that their backups go poof.
Then you'll find yourself with no data - and no backup.
Compare that to the tape burning down. You just make another tape. The chances of both your system going down, and the tape building having a fire at the same time are astronomically low, because these events are independent.
TL;DR: backups are decoupled, which is why you need them.
I'd say there's an even better distinction: backups keep your data safe even if you manually delete it. With only redundancy if you delete a file it's gone. With a backup, it's still in the last backup. Of course there's usually some maximum retention period for backups, but that can get quite long.
Everything else is secondary and can be covered by both. The chance of your redundant data center burning down at the same time as your primary is also low. If one burns down, you just bring up another. Likewise for smaller-scale users, where the chance of your cloud provider being down when your house burns down is also incredibly low.
So it's really the versioning or snapshot nature that's the key aspect of backups as opposed to simple redundancy.
Agreed that RAID isn't backup, but other online systems, while not completely decoupled, are less-coupled, which serves as a convenient middle ground for some cases.
And then because that is so cumbersome and expensive, it gets done only once and when you have a disaster, you can restore your backup from a few years ago.
There just seems to be no way to square the curcle - if the backup is trully offline, it's clumbersome to make if thr backup is effortless, it's online and vulnerable.
I think this backup maximalism can be taken further. Say if it’s not one of many tapes stored offsite at an Iron Mountain repository, it’s not a backup. Or multiple locations at that, can’t discount redundancy (what if one location gets nuked)
OK, looks like many responses have the same misunderstanding here.
To find out if something is a backup or not, ask a simple question: does the availability of the data on it correlate with what's happening to the system it's backing up?
If the answer is no, then it's not a backup.
Whether your tape storage gets nuked or not does not, in any way, correlate with whether your system gets hacked into, or whether someone who has a grudge against you messes with your system.
Then P(data loss) = P(system is fucked) x P(backup is fucked).
If your backup is not decoupled from the system, then P(data loss) = P(system is fucked) -- which is a far larger.
You don't need to take your tapes to Everest. You just need to decouple them from the source.
TL;DR: if you can destroy a backup from the system it's backing up, it's not a backup :)
I back up to Dropbox and an external hard drive next to my machine. I've used both to restore my data on hard drive death.
I can't imagine the amount of time and/or money it would take to do what you want. It certainly doesn't seem like it is something most computer users would do.
I can set anyone up with a (for example) iCloud + time machine backup with very little work, and it will keep ticking over, backing up. What's your suggestion exactly?
> There are two kinds of users: those who back up their data, and those who haven't experienced a data loss yet.
I have never had a significant data loss, and I back up my data.
The weaknesses of cloud are obvious, and alone it's not enough, but leaving it on the table is dumb unless you have some other way of practically and reliably maintaining a far-offsite copy. Most people don't, and even fewer people have a good reason to bother when they're employing offsite backups as part of a broader strategy like 3-2-1.
Sounds like its affecting people using the My Book live service to access it over the internet
Just wow though... Hopefully its not actually doing something like overwriting the data with random bits and just deleting the pointers to the file in the MFT. If the latter case, the data can be easily recovered
The article mentions that a factory reset was performed on the devices. Factory resets usually leave the data in a readable state depending on how paranoid the developers coding them are - you're not usually going to run shred -n 11
I don't see anything. The site isn't DRMed either; I'll demonstrate by pasting the text of the article:
> It doesn't matter what the files are: If you try to share these formats over a network, Western Digital assumes not just that you're a criminal, but that it is its job to police users. You see, MP3, DivX, AVI, WMV and Quicktime files are copy-protected formats.
> The list of banned filetypes includes more than thirty extensions. Some of them are bizarre: .IT files are banned — these are Amiga-style music modules composed with Impulse Tracker, a particularly well-loved tracking sequencer that hasn't been updated in almost a decade. I composed with IT myself, back in the day, and still have all my shitty compositions, none of which Western Digital would have me share. (Try MOD vs. Speak&Spell masterpiece Eddie Dreams of Women, if you dare: IT, MP3)
> Isn't it cute how the only data it views as worthy of policing are music and movies? These are the only copyrights that matter under corporate monkey law.
> It's the most astonishing example of crippled equipment I've ever seen. A DRM'd hard drive! Whatever next?
Dreaming meat?
> *UPDATE: The manual's appendix and online support site provide setup instructions for SAMBA, allowing access over IP instead of with the DRM-infested and poorly-reviewed client app, elsewhere claimed to be "required."
*
> *MOAR! Samba not enough? Gut the firmware and install made-to-measure Linux: An entire community of folks is here to help you hack your MyBook: mybookworld.wikidot.com.
Internal formatting was stripped in the copy-and-paste process. I could add it back, but that would detract from the purity of the approach. ;D
My money is on "hardcoded admin backdoor password leaked".
Edit: Found the manual[1]
"Changing a User’s Password: When viewing details about a user, the Administrator can change the user’s password (no password is the default setting)."
Alternatively - a backdoor without any authentication requirements at all was discovered.
There have been times when web crawlers have found pages with delete icons on them that are pointing to GET requests and, after dutifully following them to index the data, resulted in a server being wiped.
A quick search through Shodan will reveal thousands of unsecured storage devices just sitting open on the internet. I'm really surprised it took this long for something like this to happen.
I used to have one of those, and it bricked after a power outage. Luckly it only contained media I could download again. I opened it to try recovering my files from its HDD directly, but to my surprise it merges the HDD with a portion of internal flash in a single volume. I didn't try recovering it further because I didn't care of the data, but I was really expecting to see my files in that HDD... After that I ditched its mother board and connected its HDD to a raspberry pi.
> “It is very scary and devastating that someone can do factory restore on my drive without any permission granted from the end user,” one user wrote.
I'll say again: backup drives must have a physical write enable switch on them. To all the people who argue against this - just you wait till it happens to you!
Not having multiple backups is also a very bad idea. One day my 8T drive slipped out of my hand and smacked on the floor. That was the end of it. But I wasn't out my data, just out the cost of another drive.
Cloud backup is a stupid idea. At any moment it could go dark, for any reason, and you have no recourse. Might as well bare your throat and hand some random stranger a razor.
> Cloud backup is a stupid idea. At any moment it could go dark, for any reason, and you have no recourse. Might as well bare your throat and hand some random stranger a razor.
I really disagree - physical storage in your home is the common alternative for a lot of consumer users - and that physical storage will involve maybe unplugging your external hard drive between backups - but otherwise never checking for the consistency and accuracy of the data nor the hardware. If you're working in a data center then it's your job to do these things and it doesn't take very much time... for normal folks cloud backups are likely going to be more reliable.
If my cloud backup backend provider cancels my account, my backup software will quickly complain. There will be a window of exposure, but I can act quickly.
If my backup harddrive is in the same house as my main storage and my house burns down, I'm fucked.
If I'm fancy, I may have multiple backend storage providers, too. Lot easier to do that then to have multiple houses. I wanted to use a safe deposit box in the past, but last time I called banks near me, they didn't even have safe deposit boxes available to rent! I can ask some friends, but then will be limited in frequency of updates, still.
I didn't think syncthing supported encrypted replicas - that's why I passed on it years ago for a dropbox-like use case. I wanted to have a replica in the cloud as a relay, without having to fully trust the machine. A quick google suggests they may have something beta along these lines, so this may change.
I believe Resilio can do it, but it is closed source.
If you're looking for backups and not dropbox-like functionality, then something like restic might be more appropriate. It does actual content addressed backups, so you've got history for those times when you realized you messed something up six backups ago.
"Redundant Array of Inexpensive Cloud Storage Providers is not a backup!!!" <grin>
(Can we make RAICSP a thing? Or do multi-cloud solution vendors already has a snappy marketing term for storing everything on all of S3, Azure Blob, and Google Storage at once?)
Yeah, I’m falling on this too. A client with write-only privs to a buckets on multiple cloud storage providers is literally the most durable backup in human existence right now. Like nothing else will get you worldwide DR and 30+ 9’s of durability.
3 (designed for) 10 9’s systems whose failure modes are wholly independent might be 30 9s.
I strongly suspect (as in “for the right price would stake my life on it”) that there are some common failure modes across the major cloud vendors’ offerings.
Your advice is directionally sound (and is what I do for family docs and photos); I just think it has fewer meaningless 9s than you’re imagining.
Companies like Iron Mountain provide this service to businesses. Even if a bank no longer offers safe deposit boxes (and you were relying on their "safe", which for small banks was an arbitrary thing), there are usually facilities in larger cities that offer the same sort of service.
Just went through this with my family (in Australia) when the local bank branch said they were shutting down the service. It was basically a locked room in their branch, there was no "safe" involved.
I think you've fallen into the trap of being scared by the news. Normal hard drive failures don't make it to HN because they're so boring and common. You even had one yourself. That's how unreliable maintaining your own backup device is.
This very story is just that - people had their own hard drive and it wasn't maintained for security so it got erased. At least this article needs to be added to your mental list of failures of local storage to balance your news-driven bias.
"At least this article needs to be added to your mental list of failures of local storage to balance your news-driven bias." What a toxic comment. You are saying far more about yourself than you are of someone you don't know at all. Projection and a lot of anger/hate inside.
Backblaze's latest drive report shows an annual failure rate for hard drives of 0.85%. I very strongly suspect the total of all failures of the type you're listing there is many orders of magnitude lower than that across all cloud storage users.
If you care about your backups, a single cloud vendor/account vanishing isn't going to be a problem, in exactly the same way that if you care about your backups,m a single drive failure will not be a problem. They are both predictable and mitigable risks.
Yeah, but it certainly feels in the right ballpark from my personal experience. I’ve normally got 20 or so drives in use at any time (perhaps fewer spinning rust drives not that ssds are more common), and I have a drive go bad occasionally - not every year or two, but certainly every 5 years. So I reckon they’re at least order of magnitude right.
That's why companies need to start paid/premium customer support: when you lost access to your account, you should be able to pay 10, 100, 1000 of money to talk to a real person who understand how the system works and where there a mistake was made.
Like a hacker stole your account and you cannot take it back? A special person will to do a background check (e.g. call your employer) to verify that you is actually you.
Robot erroneously blocked your account because of misclassified spam-like activity? Your account is restored, and you are fully refunded.
do you know of any tools for consistency/accuracy? I've been meaning to do some basic md5 checks for all my files but haven't gotten around to it. Every now and then I find a corrupt jpg image that has a rectangular band running through it or something of the like.
Generally I'd recommend using a filesystem designed for that. Something like btrfs or (I think) ZFS, which have checksums built into the filesystem (and if you set them up in a RAID configuration, these checksums can be used to correct data as well)
I believe that's some form of bitrot. AFAIK only ZFS can deal with this (assuming you use ECC RAM).
EDIT: According to Wikipedia, ZFS, Btrfs and ReFS offer strategies to deal with various forms of data degradation, although it appears as if ZFS is still king.
For all my important photos I use par files. Using command-line tools like par2create and par2repair is simply part of the routine of storing photos from the camera on the NAS.
I also use it for music files, in the past too many mp3s got broken.
Why not as a backup strategy? I had my desktop harddrive fail and while I had some local backups but the cloud backup restored me to almost the minute of the failure. The only downside to cloud backup is the slow restore process.
> The only downside to cloud backup is the slow restore process.
If you use BackBlaze, they offer multiple options to restore data quicker if you're willing to shell out more money, including AWS or sending you a physical drive with your data on it, albeit the latter is kinda expensive, but if you have a ton of data and need it ASAP, is probably worth the cost.
I recently lost a 16TB drive for the second time in 6 months and my local backups were out of date by a couple months, meaning I needed to restore ~200GB of data. Aside from the web interface file explorer window being cramped (and support telling me to basically go fuck myself after suggesting that), I was able to make ZIP files from recently changed data and use their downloader tool which IMHO is a piece of crap and probably designed by some intern, but at least the download speed was fairly fast, even in Europe, in contrast to upload speeds capped at around 250 KB/s (though thankfully support multiple threads).
Cloud backups are just one layer of a robust backup strategy. A copy on your desk, one in your closet, one at your parents, one in Backblaze, one in Glacier, etc.
I know you're joking but to stay on topic, you still have to even get them to even admit they have your data, since it's supposedly illegal. Thanks to Snowden we know, but even in light of those leaks, they have and will probably continue to lie during Senate/Congressional hearings. Once we get past that hurdle, then we can talk about restoring your illegally captured data ;)
"Cloud backup is a stupid idea. At any moment it could go dark, for any reason, and you have no recourse."
If you are aware of the issue, and get kicked off AWS, or whatever, you move the backup. It is only a problem if your data catxhes fire and you get removed from cloud at the same time
> Cloud backup is a stupid idea. At any moment it could go dark, for any reason, and you have no recourse. Might as well bare your throat and hand some random stranger a razor.
It's not stupid but redundancy is important, don't rely on cloud only.
My own anecdata: I lost a couple TBs with Streamload[0] ~2007 when they switched over to MediaMax. They wouldn't admit it was their fault and I never got any kind of compensation - even though I had just renewed my yearly subscription a couple months prior. Thankfully I had offline backups on CD/DVD but that was definitely no fun. More recently CrashPlan decided to no longer offer unlimited backup for non-enterprise users i.e. I would have to pay a LOT more money to back up multiple TBs. At least they offered me a partial refund for the remaining months in my yearly subscription. I really really hope Backblaze keeps their unlimited plan.
As for physical backups, my 8-drive RAID-5 GPT partition got somehow nuked when I was trying out a new Windows 8 installation (one of the first official builds). I never found out what actually caused it, but I immediately went back to Windows 7 and waited for 10 to be out for a couple years before finally switching over. I'm still paranoid and keep my backup server separate and off the network except to make diff backups.
Bit-rot is also becoming a serious issue. I hope to finally build a home ZFS server (with ECC RAM of course) to deal with this, as I've noticed non-streaming encoded videos sometimes just stop working after 15+ years. I'm not sure if it's actually bit-rot but I can't think of anything else that would cause this to happen.
In any case, keep multiple backups: offline and online!
Hard disk drives go on sale regularly. I replace mine usually once a year or so. They're not expensive relative to the loss of the data. I just bought a 500G SSD drive on sale for $60.
I've read that SSDs are also generally more susceptible to data degradation earlier than spinning rust, although I'm not sure how true that is. Wikipedia[0] says:
- Solid-state media...store data using electrical charges, which can slowly leak away due to imperfect insulation.
- Magnetic media...may experience data decay as bits lose their magnetic orientation. Periodic refreshing by rewriting the data can alleviate this problem. In warm/humid conditions these media, especially those poorly protected against ambient air, are prone to the physical decomposition of the storage medium.
In either case, only ZFS/Btrfs/ReFS seem to implement strategies to deal with data corruption. Kinda sad that Apple recently released a new FS and didn't bother to address this.
WHAT!? How the heck did you get that conclusion from this story? These are literally your people, who backed up to a local device and essentially chose their own local NAS rather then the cloud. Which unfortunately for them came from a company with bad support that dropped it 6 years ago and likely did a shitty job even before that. But not everyone can be a tech person! Even tech people can't be that knowledgeable about more than a fraction of information technology, the field is crazy vast [0].
The failure modes here are exactly why "cloud" makes so much sense for so many. Keeping stuff up to date, or migrating if the company drops it (even noticing that support has stopped). Securing access. Verifying integrity. Amortizing the cost of different media cold storage fallbacks with things like tape robot facilities. And on and on. You wanna opine a bit on the relative probability that Amazon, Apple, Backblaze, Dropbox, Google, Microsoft, rsync.net, or a host of others just "at any moment going dark, for any reason" vs Western fucking Digital My Dropped-in-2015-Book dying? Think carefully.
I mean shit, I actually have a custom Epyc-based TrueNAS Core (not FreeNAS anymore! gotta keep up!) system with tens of terabytes as my main backing store, and replication to a remote site, with more usage of network segmentation, wireguard, ZFS user privileges so compromise of the replication credentials can't delete old stuff, then I'd expect nearly anyone to have any idea about. I still have it going to Backblaze B2 as well, and I pay to have them hang onto it for a year rather then 30 days. And the only reason I can swing it all is that I can amortize a lot of it professionally, get other usage from it, and draw upon a bunch of both knowledge and metaknowledge that my particular life path happens to have granted me.
For 99% of the people in my life "turn on iCloud/Backblaze/whatever" is the right advice. Backups are very risky without heavy automation and regular attention on top.
----
0: One thing working at various levels has given me a deep appreciation for is what broad shoulders I stand upon, many layers deep, and how valuable and deep the knowledge of so many other tradespeople I interact with in life is. The mechanics, electricians, plumbers, general contractors, structural engineers, medical folks, researchers and so on I work with or have worked with all possess skills and knowledge that I never will. That's why we're social animals. I've traded on my skills helping someone improve their network for a hand with skilled carpentry or electrical work or the like. I don't see any shame in that.
They hooked the backup to the internet. Not a good idea. Nothing connected to the internet is secure, especially since nothing comes with physical write-enable switches.
Consider that I worked on the 757 stabilizer trim gearbox. The idea is no matter what fails, the airplane lands safely. I've been trained to think that way.
Consider also that I've worked with computers for 45 years. I've seen about every failure you can think of, including fires and floods, explosions and earthquakes, viruses and phishing.
The most unreliable, by far, part of computing is the internet.
Incredibly shallow and dangerously bad hot take. I think you have a very poor understanding of what "security" is, nor the concept of backups in general which necessarily (like security) have a very strong human factors requirement. A backup which is too much of a PITA and requires nearly any level of manual effort simply isn't going to get used much or maintained well at all by the vast majority of the population. Tools and systems exist to serve humanity, not the other way around, and when a system fails badly for many it's not humanity that must all change, it's a shitty system.
Fact: if these people had been running to a decent cloud service, they'd probably still have their data. Really "don't connect backup to the internet" is like, fractally stupid. The more one digs the more stupid angles turn up recursively similar to higher level stupid. "Don't connect backup to the internet" means what to you? What about the computer itself making the backup? If that is connected it could get infected as well, now what? Everyone is just supposed to give up on all networks entirely because "hurr durr nothing on a network is secure"? Which is wrong anyway since security an economic equation, not some ethereal absolute. I hope you never give this advice to anyone IRL.
you do need some sort of off-site backup for disaster recovery. House can burn down, get burglarized, flooded, and you absolutely must account for that if you really want to mitigate the risk
My backup drive has a physical switch. I used to be meticulous in setting it to read-only except while making a backup. But now I just leave it in read/write mode because I lost my enthusiasm. It's not plugged in when not in use though, so it's a bit safer from this sort of thing.
How would you do automated backups with a physical switch? Are you proposing nobody does automated daily or more frequent backups?
I suggest rolling backups. Tapes are also a possibility, as they can be written in append-only mode.
Just remember that ransomware will encrypt any drive you hook up to it, including your backup drives. A physical write-enable will prevent that, and it should be disabled anytime you are restoring from backup.
I want a write-enable switch even for simple tasks like copying one drive to another, just so I don't mistakenly copy the wrong direction.
> and it should be disabled anytime you are restoring from backup.
What I was alluding to but didn't make clear was that physical switches are easily defeated by human laziness or mistakes. Why have a physical switch on drives you're copying if you might accidentally switch the wrong one or eventually stop bothering? This already happens with warning popups in Windows when you try to run an untrusted program, for instance. People get trained to bypass the security because it's just a tedious obstacle.
You personally might be careful enough to forever set the switch correctly, but people who didn't even know their hard drive was years-un-patched and internet connected yet left it there to get hacked also wouldn't reliably set the switches every time either.
Regarding accidentally copying in the wrong direction, I think a safer way than a physical switch is to show some details of what will get deleted. Maybe previews of images, tiny snippets of text from the files, a graphical view of the space they occupy, etc. Make it more visceral, like throwing away an actual book where you can see how many pages it has and what the picture on the cover is.
> backup drives must have a physical write enable switch on them. To all the people who argue against this - just you wait till it happens to you!
Yep, I once reformatted the wrong drive. I plugged in a firewire drive into a friend's Mac and didn't realize he had an internal drive that apparently also showed up with a firewire icon. Whoops!
> I'll say again: backup drives must have a physical write enable switch on them. To all the people who argue against this - just you wait till it happens to you!
Reminds me of the little sliders on memory cards. Though I think those were software driven. It didn’t actually lock it out from writing.
Are there any portable USB style disks that have a hardware enforced read only switch?
Any single backup is a stupid idea. Combine them, and you've got a great backup solution. Why would I keep important data in one physical location?
My house could burn down tomorrow, but I'd still have my cloud backups. My cloud backups could roast in a fire tomorrow, but I'd still have my local backups. And probably several other cloud backups.
If you put all your eggs in one backup, sooner or later, you're going to get screwed.
He only reliable backup mechanism is starting a successful cult whose main tenet is that your data is valuable communication from God, so that they continually duplicate it. Anything else will go bust long, long before the heat death of the universe.
Amazon just announced AWS Cult-As-A-Service. For the small fee of just $0.05 per Gigabyte per month they will store your data in three separate cults located in geographically distinct regions.
Your faith in the internet doesn't match mine. (Far too many single points of failure.) But suppose you're right. I can buy 10 drives for cheap, but not 10 different cloud accounts.
I also don't care to have the internet snooping on my data. Yes, I know about encryption. Enough to know it cannot be trusted.
>Cloud backup is a stupid idea. At any moment it could go dark, for any reason, and you have no recourse.
I wrote something about it [1] just a few days ago. And I have been banging on about it for years if not a whole decade.
>What we need is something like iOS TimeCapsule. Let the Phone do the Photo Management on Devices, and have an on site ( Home ) DataBackup as well as offsite ( iCloud ) Data Backup. Apple could even make iCloud Backup subscription a requirement for TimeCapsule to function.
We need both. In a simple, safe, easy to use fashion.
I don't know what you mean by "cloud", but an off-site backup is essential. Your home can be burglarized, or burnt down, or flooded. Basically anything you insurance covers for your home you should cover regarding your data.
> At any moment it could go dark, for any reason, and you have no recourse.
There is no difference between your backup going dark or your main data source going dark. The whole point is that the data is duplicated and the odds of both of them going dark at the same time is low. This only means that you should check your backup regularly.
Almost bought one of these, but opted for a raspi and usb3 external drive instead. Works great with Time Machine. https://saschaeggi.medium.com/use-a-raspberry-pi-4-for-time-... was my guide, though I deviated by using ext4 instead of hfs, and didn't need avahi.
But use ext4 instead of HFS+, and use Samba instead of Netatalk. With HFS+, the bundle would get corrupt after a few days. With Netatalk, a sudden power outage would corrupt the whole thing.
Don't listen to 99% of blog posts on this. They all tell you to use HFS+/Netatalk. None of those authors have actually used that backup solution for any extended period of time, because if they did, they'd find that it's unusable. Use this one instead:
Thanks for the recommendation. We work pretty closely with Apple's SMB2 client developers to make sure everything works well together with Samba. They're great fun to work with !
I'm inclined to write off (anyone using a WD "My Book" device) as beyond help but a simple SFTP client like WinSCP or Cyberduck or Filezilla is comprehensible to just about everyone.
You drag and drop to the SFTP destination[1] which covers "History"[2] with immutable ZFS snapshots and "Integrity"[3] with ZFS itself and "Security"[4] by running nothing but OpenSSH.
If you're going to set your parents up with something, you could do worse than that ...
I do both: I backup to my Synology NAS, which then in-turn uses Synology's Hyper-Backup (which is very nice, btw) to my Azure storage account - costs me about $15/mo to store a few terabytes with PITR recovery back to when I started doing this in 2018.
The thing is... I can't help but worry someone's going to compromise my NAS and DBAN the drives and then extract the Azure storage key and use that to delete all my backup blobs...
(Yes, the backup client needs read, write, and delete permissions, unfortunately - and Azure doesn't offer a Blockchain-style "append-only" mode for blob storage, unfortunately - still, better than nothing).
UPDATE: Apparently Azure Blob Storage does support strict append-only blobs that cannot be mutated or deleted, only appended - so I wonder if Hyper-Backup can use that…
Is there anyone out there with $$$$ who will stop at nothing to part you, a rando, from your old data? Probably not. Are there sophisticated attackers who will burn a couple 0-days to build a botnet for the sole purpose of randsoming NASes AND attached cloud accounts AND the origin systems, accounting for tons of possible configurations? Still pretty unlikely — this is NotPetya level stuff with small payoff.
If you find yourself in the crosshairs of a sophisticated, dedicated attacker (perhaps one in possession of a 0-day), you’re pretty much done. Offline write-only backups stored offsite are the only defense.
However, is there a bug lurking in Hyper-backup that might accidentally wipe stuff from Azure storage, and the bug hits a month before your house gets struck by lightning? Maybe…
> However, is there a bug lurking in Hyper-backup that might accidentally wipe stuff from Azure storage, and the bug hits a month before your house gets struck by lightning? Maybe…
Are you forgetting the recent post about how kids these days don't even have any concept of files? You're over-estimating most people in their technical competence and the required level of understanding and patience needed to do anything with SFTP. You say "drag and drop" and I think of the hours of troubleshooting and explaining (and re-explaining) I'd have to do if I were to try to implement this with family.
A "backup plan" which involves manually copying files around is no plan at all. It's basically guaranteed to fail when you forget to copy the files for a while, or when you give up on it because it takes too long.
Discontinued in 2015, so no security patches. That's the problem with all of these purpose built IoT devices, a general lack of security updates even when they're still supported, and of course no chance at all if 6 months or a year after you buy it they decide to drop the product line.
"Western Digital has determined that some My Book Live devices are being compromised by malicious software. In some cases, this compromise has led to a factory reset that appears to erase all data on the device. The My Book Live device received its final firmware update in 2015. We understand that our customers’ data is very important. At this time, we recommend you disconnect your My Book Live from the Internet to protect your data on the device. We are actively investigating and we will provide updates to this thread when they are available."
I'm using a WD My Cloud for non-essential data. I was wondering if my unit would be affected as well. As far as I can tell, the latest firmware update completed January 01, 2020. I'm safe for now but I better build something more robust.
It runs Linux, and they could have at least opened up the firmware for opensource updating. WDs response to Windows 10 blocking network access because the My Clouds use an old version of SMB is to buy a new one (and trash working equipment) or enable a not particularly secure SMB version in Windows.
Of course they issue a CYA about their cloud systems not being compromised, but then bury the fact that other systems were compromised by using the passive voice when they point to "threat actors".
I'm assuming the My Books phone home in some way to facilitate file access over the internet. If so, that WD system got cracked. That seems more likely since-- at least for my ISP and I'm guessing most home ISPs-- don't expose individual device ports to the world unless the user sets NAT for that.
Anyone know if NAT configs are required to use the feature that let's you access files from anywhere? If so, someone simply cracked bad security 6 years out of date and scanned for devices.
Given typical IoT security, it might only have been using the default user & password.
I wasn't sure because apparently there can be issues with multiplayer or online play with some game consoles and people will be directed to mess about in their router settings. There were some stories about problems with the Xbox like that.
The vast majority of users assuredly have an Internet connection they already use with at least one other machine on it, so unless ISPs are handing out another public IP for their NAS, it's almost certainly behind a NAT.
If so, someone simply cracked bad security 6 years out of date and scanned for devices.
Maybe the majority of them aren't so much an "exploit" as they are "easily guessed credentials". The fact that anyone else, anywhere else on the Internet can also use those same credentials to access their data is something that probably few people realise when they set one of these things up.
Judging by the CVE they linked, you need local access. But, since it's a GET parameter, this could be a drive by attack where someone puts all the links to common home networks in a page and the browser just takes care of that (and there are ways to guess the local network). Though, judging by the number of people affected, it might be something else.
I just bought a large external hard drive for exactly this reason in case things fail. I did not use this live feature but I feel much better with an offline copy of my data safe. This lesson has been played time and time again when will people learn? You need to be in physical possession of your data and if it is truly critical you better have a second copy kept in another location in case of theft or fire.
Folks, buy 2 similar drives & backup stuff monthly to both & have your local workspace synced to another machine local or cloud using any tool like rsync, Cyberduck, iCloud whatever.
By definition "directly connected to the internet" means if a device can on its on accord, direct requests to an entity, ask it a question, and act upon it is true. From what I understand these WD boxes go to a management service in the cloud. and were told they should factory reset. Whether something is pull-only (as in this case) or push (say allows HTTP or SSH access from a random on the internet) is irrelavnt if either result in unauthorised activity on a box in your home.
In defense of the parent comment, there is a meaningful difference between a device acting as the terminating IP meaning any open services are directly probe-able and a device sitting behind a firewall.
For this particular attack (assuming c2 server compromise?) that might not matter, but ultimately there is a massive difference in attack surface when comparing “direct” with “NATed”
How good do you reckon a 6 years past EOL consumer linux device's defences against a browser running 3rd (or 1st) party javascript making http requests to http://192.168.0.1..254]/cgi-bin/factoryRestore.sh?
How much would you bet against that being an unauthenticated call or one with leaked hard coded reds?
gives me a 403 Forbidden error, and if there was a known default password to my router - it'd try and do a factory reset on it. (It actually wouldn't, it'd send back a confirmation popup, but...)
Not sure this makes any sense, the 6 years past EOL consumer linux device isn't running a browser.
Or are you assuming the user's browser itself is compromised and is running random javascript hitting the NAS address? That would be unfortunate, but I'm not sure I'd blame it on the "6 years past EOL consumer linux device"
Doesn’t need any browser compromise as such, just a user on the same wifi network running a browser and visiting a site with malicious JavaScript (possible a malicious site, possible a benign site with us delivered by a shitty ad network, possible a poorly secured site with persistent xss flaws).
Classic old cross origin request forgery. It ranks #7 I owasp’s top 10 website security flaws, and they have this to say about it:
“XSS is the second most prevalent issue in the OWASP Top 10, and is found in around two thirds of all applications.“
I remember Opera showing an error when I tried to follow a link from Internet to a private addresses (192.168.0.0/16 and such). Don't browsers enforce that anymore?
Yeah difference in threat model of “evil internet can make tcp connection to me” vs “basically need a c2 compromise” is huge. Sucks for those that lost data either way.
This doesn't appear to have anything directly to do with Linux, just crappy software written by WD running on it. Please be a little more careful with your criticism.
I don’t think the commenter was criticizing Linux. Seemed more a comment on the manufacturers who lazily slap these products together with minimal effort. Taking a free OS, sprinkling a thin layer of their software on top, and then abandoning their responsibility to maintain the full software stack being a common example of that minimal effort.
So, yes, essentially you end up with a bunch of devices running vendor-specific unmaintained Linux distros on the wide open Internet. The “unmaintained” part of that sentence is the problem, not the “Linux” part.
I agree, but it also makes me consider what the role of software engineering (as a discipline) is in this disaster.
Shouldn't we design systems that are hard to break by default? Shouldn't the OS assume that terrible things are going to happen anyway, and provide protection from bad faith actors in case the OS is indeed left unpatched for 10 years while being fully exposed to the internet? Is it even possible to design a system that provides this level of security so that we can get away with near-zero additional security expertise from product designers who build on top of it?
I think that, first of all, it's Western Digital's responsibility that things went south here. But shouldn't we build systems that provide bomb-proof security for the many companies that build on top of it? Is it even possible? And if so, how? In the end we would be doing ourselves (as consumers) a favor.
Current mainline Linux still has support for the powerpc SoC in those drives. It may take a week for someone to prepare alternative firmware for the device, that is modern, uptodate and safer. U-Boot dropped the support in 2017.
So if anyone wants to support their 11 year old drive they can do so.
You're correct that this would be very nice to have.
However this is a problem the industry has been struggling with for decades. It's simply not easy (and maybe not even possible) to achieve what you claimed "should" happen. Nobody knows how to produce bug-free software at scale.
I know. I'm just pondering if it's possible to come up with a design that guarantees a secure system even if you assume that all of your protective layers will have security holes in them that you will not be able to patch. Does or can such an architecture exist?
> Is it even possible to design a system that provides this level of security so that we can get away with near-zero additional security expertise from product designers who build on top of it?
I highly doubt it, at least not with 100% certainty. We build bridges to last and to stand weather. These are enormous constructions with large safety margins, teams designated just for security (against weather, earthquakes etc) with peoples lives on the line. Yet, we need to inspect them on the regular to make sure no assumption broke, no safety system was triggered and nothing unexpected happened.
If we can't make this work at this scale, I have no hopes that software for comparatively cheap consumer devices manages to achieve this.
Read this, then checked my old My Book Live.
Sure enough... everything's gone.
I just used it for continuous laptop backups over my LAN, so unless one of them crashes tonight, I should be good. But, this will certainly give me pause when considering WD products, and this type of product in particular.
This is very bad, WD! Are your other products also vulnerable in this way? Why should anyone ever again trust your company to keep their data safe?
Seagate mitigated their most famous problems. They were a good bargain buy for a while after that as they tried to rebuild their brand. Issue is now they're too proud of themselves, you're better off buying something else. They might still be the cheapest per GB for certain sizes, just look at the backblaze stats.
One option is shucking a high capacity external. Those are more likely to be high quality. Some are rebranded enterprise drives. But the hd model isn't guaranteed, and warranty support might be iffy.
for my raid i try to buy disks from different manufacturers. with the limited selection this is extremely hard. i have to fall back to different models from the same manufacturer and even buying the same model from different shops to get at least disks from a different batch.
As a part of the sale of HGST to WD, Toshiba got assets to produce 3.5” hard drives from HGST. I’d consider them the true heirs of HGST, and their good reliability as shown in Backblaze’s studies bear that out.
Unless WD changed their mind again, HGST HDD is still available as of 2021 because customer has been demanding the exact same HDD with exact same model number from the exact same plant. Basically they want to know it is the same old HGST and not WD. And some ( if not most ) of them are large, enterprise customers. So they brought it back in ~2019 (without an announcement).
The affected model here, known as My Book Live, uses an ethernet cable to connect to a local network. From there, users can remotely access their files and make configuration changes through Western Digital cloud infrastructure. Western Digital discontinued the My Book Live in 2015. The support forum thread was first reported by Bleeping Computer.
The affected product is not a drive but a NAS solution. In terms of being a reasonable consumer my thought process is to evaluate distinctive products differently.
NAS is a cheap computer with attached drives. It’s going to have computer-like failure modes.
Well, the GP said "why should anyone ever again trust [Western Digital] to keep their data safe?" Using a Western Digital hard drive, internet connected or not, very much means trusting a WD product with your data.
But I might be reading too literally. And personally, I do intend to continue purchasing Western Digital hard drives, just nothing internet connected, because I agree they're different products with different risks.
Non-technical users don't care about this distinction. The name Western Digital is on both, and that name's brand value is destroyed forever if this indeed turns out to be irreversible. Though they could probably pivot to selling their drives in other's NAS devices without much difficulty if they want to remain in this market.
Non-technical users don't really buy hard drives individually, they would at most ask their computer-savvy friend or go to a shop who will replace it for them.
I highly doubt the average non-technical user even knows who produced the hard drive they own.
> Non-technical users don't really buy hard drives individually
Yes, they do. My dad bought two WD My Book drives by himself as backup drives. He luckily was not aware that those things have an extra software for backups, nor how to install it, but those things are just sold like that targeted to the average user in your average computer store.
Internal drives, you're right, likely wouldn't be bought by a non-techie alone.
But THIS affected type of drive is specifically marketed to a very non-technical user, and I can certainly confirm my non-technical friends and family have bought products in this segment independently, whereas my techie friends steer clear.
It's a NAS that's made to look like a drive. My techie friends would never buy it - they'd get Synology/QNAP, or do their own over-complicated time-consuming solution (slight editorial opinion there ;), or use cloud backup, or some combo. But my dad, mother in law, and other relatives have products like this, and have purchased them on their own. In fact, I think when asked, most techies would in fact go against the notion of buying cheap but internet-exposed storage device for a non-technical friend :O
Your slight editorial opinion reminded me of this HN hot-take from the DropBox announcement:
> For a Linux user, you can already build such a system yourself quite trivially by getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem. From Windows or Mac, this FTP account could be accessed through built-in software.
Somebody actually poisoned Tylenol. People realize that all the food they eat could have been poisoned in the grocery store. There's no easy mapping to "every external hard drive manufacturer could stop updating their products and your data could spontaneously delete itself."
There have been incidents of (other manufacturers') hard drives shipping with pre-installed malware though [e.g. 0]. I think that's a good analogue for Tylenol.
True, a small computer running an Internet connected general purpose operating system should have a prominently labeled expiration date of some short time after the last patch. Use after that date is not recommended and potentially dangerous. WD sold something that looked like a toaster to the consumer but was actually a lump of cheese.
A NAS isn't actually distinct from a plain drive in that particular way. A plain SATA hard drive is a cheap computer with some attached hardware, just like a NAS. Literally, there's probably at least one ARM CPU core in there controlling everything and communicating over the link to your main computer.
The difference is the protocol that goes over the link. In a NAS, it's SMB/NFS access to files over ethernet/wifi, and for a SATA drive it is the SATA command set accessing blocks. The difference is small.
Public-facing Synology devices got hit in the past too, but they just used it to mine Litecoin. I think someone calculated it and figured out that it cost something like $400k in electricity to mine $100k of coins on the boxes.
And of course Qnap in the recent month. It keeps repeating, and keeps happening, and some ( me ) keep ranting about it and no one / company is doing anything much with it.
Majority of people buy NAS only use it to access their Data within their internal network. But somehow they all include Internet / Cloud features as upsell.
They used to make hard drives years ago, and I still have three old 1.5 TB units kicking around that I can't seem to throw out, because they won't die.
This is the third such major scandal from WD in as many years. First, selling SMR drives as CMR. Second, selling 7200RPM drives as 5400RPM. Now this. They've lost a lot of credibility, and I don't see any of the other manufacturers having issues rivaling WD.
I suppose you mean the other way around? Otherwise, what's the problem with selling a higher RPM device as a lower one? Isn't higher RPM better? (Genuine question, I don't know much about HDD).
WD Red is marketed for NAS/storage, where it's not uncommon to have 4/8/12/16 drives in the same chassis where noise/heat becomes a very different problem. It turns out that the drives generated the same heat and consumed the same power as 7200RPM drives, while the data sheets were outright lies and BS. WD decided on double down on the BS for several months, and only recently released new SKUs where the specs matched the marketing.
It’s correct, and higher is better, 7K2 is faster and often longer lasting somehow. Generally better built. Downsides are noise, power draw and thermal.
They sourced drives from former HGST under WD but 5K4 we’re not available so marketing rebadged 7K2 as “5400rpm-class” and lucky buyers were hit with better performances.
Sort of. The WD Red drives outperformed actual 5400 drives but underperformed other 7200 drives by a large margin (180MB/s vs. 230MB/s), while not necessarily offering heat or noise benefits that matched. My current 8x8TB WD Red Unraid build will likely be the last WD drives I buy for a long time.
The faster devices generally vibrate more and give off more heat, which might make them unsuitable for mass storage (i.e. in a NAS box or other multi-drive arrangement). The difference might technically even invalidate warranties.
They also draw more power, which might mean they simply don't work properly in some low-power devices, or mean that any UPS someone might have one connected to will not last as long as they expect during a power interruption.
Verify that your previous backup drive is readable before writing to the next backup drive.
But hey, if you just got a ransomware note, and think "I'm good, I have my backups!", wouldn't you want to flip the read-only switch before plugging in those backup drives? I would. In fact, I'd flip that switch always before trying to read from a backup drive.
How many My Book customers would even understand the meaning of your [correct] advice? When companies fuck people over with a defective product, we should resist the urge to tell the victims to be more tech savvy and not use those sort of products. Particularly when those products are intended for the general public.
It's always the same old thing.
But the fundamental problem will never vanish: computers are complex, and no matter how hard you try with neat packaging and software, this complexity cannot be hidden. Sooner or later the illusion bursts at its seams and the user discovers another failure mode that they weren't even aware of.
WD really messed up there - but they and others will mess up again, so if the user's goal is not losing any data they'll still need to do more than buy the next shiny thing and click "accept" on the EULA. Because in the end pushing around the blame won't get you the files back.
Problem is that whoever designed the system should have done a better job. Computers are still (and probably will always remain) a niche skill so the blame lies completely on the shoulders of the WD engineers/designers who left this option open on the device
In this particular case, there were My Book drives and My Book Live. When the Live part was configured, you would be creating an entry-point into your network for WD to run code on your drive. I know this, because I purchased one, and read the small getting started guide that came with it.
Needless to say, I never ran any of the Live code. Several of these sorts of things have come up in the industry that always made me recall how happy I was to not have those drives with their backdoors on my network.
The CVE says it needs the IP address. How did the entry point work? Unless it was something like NAT port-forwarding I don't know how the attack could punch through to whatever port the device was using to expose the API.
Seagate is the other big consumer brand and they also have their own set of issues. Anecdotally, I've had multiple Seagate drives die within a year or 2 whereas WD ones have kept going for years longer.
Seagate's anecdotal failure rate was why I kept to WD. Backblaze's numbers convinced me this is not a problem I need to worry about in the present. Seagate had much better price/performance than Toshiba in my local market when I last needed drives, so it's what I got.
I am afraid to buy WD drives since the mid 2000s. I used to store all my media in them, and with WD every 1 to 2 years, my drive would be corrupted to the point that it’d need formatting. Having to rebuild my collections multiple, with always “improved” WD models moved me away from them forever. I don’t even buy server grade WDs.
There are a limited number of hard drive manufacturers. All of them had reliability issues at some point. Like the IBM "deathstar" and the Seagate "failacuda", I don't know about an equivalent from WD. Reliability come and go, and you don't even know when you buy because problems can take years to develop, and it is dependent on usage.
They are all in the same ballpark, with Seagate a bit worse than the others but not terrible like it was a few years ago. WD tends to be consistently in the middle of the pack.
The latest fail from WD (as a hard drive manufacturer) was when they sneaked inadequate SMR technology into "red" consumer-grade NAS drives. But others did more or less the same. Now, all manufacturers are transparent about the technology used and newer "red plus" drives from WD are CMR.
So I'd say, for now, I'd say WD is as good (or as bad) as any other and I would let price decide. For reliability, that's what RAID is for. And in any case, RAID or not, and no matter how reliable your disks are, if your data is precious, you need backups too.
Mine is still prompting me to register my device. It's been a secondary Time Machine backup for 10+(?) years, but really is too slow to ever do a restore from. This is a good reminder to replace it.
"Western Digital has determined that some My Book Live devices are being compromised by malicious software. In some cases, this compromise has led to a factory reset that appears to erase all data on the device. The My Book Live device received its final firmware update in 2015. We understand that our customers’ data is very important. At this time, we recommend you disconnect your My Book Live from the Internet to protect your data on the device. We are actively investigating and we will provide updates to this thread when they are available." - Western Digital
Could this happen if your router is not open to the world? I dont get it to be honest, maybe the device sends queries commands home? that seems unlikely
700 comments
[ 5.8 ms ] story [ 366 ms ] threadI run a FreeNAS setup at home on an HP MicroServer. From time to time, I lament the amount of time I’ve spent over the years maintaining it, and after a recent drive failure (and wishing for more performance), I started considering consumer options like Synology/QNAP.
Articles like this always give me pause, and remind me why I invest the time I do to maintain my own thing. I’d rather fuck something up myself than fall victim to an issue like this, and the time spent setting it up likely pales in comparison to dealing with data recovery.
I really hope these users can get their data back.
Hardware is fine but it's much more expensive than just building it yourself.
You’re wise to not trust it with anything important.
The only fiddly bit to setup for me was LetsEncrypt auto-renewals, but once I had it automated I never needed to touch it again.
On the first part, there was a period of time when the project was going through some upheaval (quite a few years ago now) and I had some issues with upgrades.
This goes hand in hand with the 2nd thing - I was trying all sorts of plugins and using it as a BT and newsgroup client. These did not upgrade well.
I suspect that a fresh vanilla build (TrueNAS now) would probable be zero maintenance. I need to upgrade my drives (a bunch of 1TB drives from the era I built it), and I’ll probably use this opportunity to start fresh.
Of all things, it has a weird boot-up bug where it fails to boot to Ubuntu during about a half-hour period (around :15 and :45) each hour and lands on a kernel panic within a second after grub. I have no idea where to start. Firmware is latest.
Getting the exact panic would unfortunately mean to have to get it down from the shelf where it lives, and plugging it into a monitor to replicate. It doesn’t make it out of low-res text mode when it does.
Most of the time, I’m usually in a rush to get it back up as my family wants lights and Plex to work normally.
I do my best not to reboot to keep the peace!
Explain what makes you think this happened to all of them at once. That's a huge leap
The article says:
>Multiple users reported that the data loss coincided with a factory reset that was performed on their devices. One person posted a log that showed unexplained behavior occurring on Wednesday:
I'm leaning more towards 0day than "software bug", mainly because I can't really think of a reason why a bug would cause the factory reset to get triggered, but I can think of plenty of ways an internet connected NAS can be exploited.
I am guessing this is normal? My data backups are offline. The idea of making my backups read/write on the internet seems insane.
It's a NAS. Backup isn't the only thing you can do with it. A personal cloud (aka dropbox) is also a valid use case, and requires internet access to work.
I can’t imagine a scenario where I’d rely on an 8-10 year old device that hadn’t received a security update in 6+ years…
That said, relying on the same credentials for a long time is more problematic. My bet is on credentials being bruteforced, especially if many people have left them at their default or used easily guessable ones.
I rely on many "8-10 year old device that hadn’t received a security update in 6+ years". My coffee machine. My fridge. My hifi. All my motorcycles. None of them run linux and are directly connected to the internet.
I'm typing this on a 10 y.o. laptop. It still works. Sure, the manufacturer and browser vendor released updates, but it wouldn't be too out of the ordinary if it didn't.
That being said, I think its predecessor would still work (for posting a comment to HN at least)
It ultimately turned out to be a backdoor account[2]. Backdoors really upset me. Every reasonable "you can't blame people for mistakes" goes out the window when it's not a mistake.
Following the ransomware, they released an auto-installing malware remover. A Python script that detected one particular piece of code associated with this recent attacker. That malware remover was a python script full of vulnerable exec calls introducing multiple new RCEs.
[0] https://www.qnap.com/en/security-advisory/qsa-21-11 [1] https://www.qnap.com/de-de/security-advisory/qsa-21-05 [2] https://www.bleepingcomputer.com/news/security/qnap-confirms... [3] https://www.qnap.com/en/security-advisory/qsa-21-16
Would you happen to have a write up/article about this series of events that isn’t a sanitized series of security bulletins? Sounds like it’d make for good reading.
https://forum.qnap.com/viewtopic.php?f=45&t=160849&start=450...
The actual backdoor looks like the `jisoosocoolhbsmgnt` session ID [1] that was removed in the update [2]. It looks like a hardcoded session ID used for tests [3]. Leaving something like that hardcoded and active in the production code is inexcusable.
[1] https://forum.qnap.com/viewtopic.php?f=45&t=160849&start=495...
[2] https://forum.qnap.com/viewtopic.php?f=45&t=160849&start=555...
[3] https://forum.qnap.com/viewtopic.php?f=45&t=160849&start=495...
This is your daily reminder that anything that's connected is not a backup.
Yes, that includes the hard drive in your machine. For that matter, RAID is not a backup solution.
Yes, that includes cloud sync (Dropbox/OneDrive/GDrive/etc) that propagates changes automatically. That's not backup.
And that includes WD My Book, as it (unsurprisingly) turns out.
Take this as a PSA.
If you want to be more enlightened: http://www.taobackup.com/
There are a lot of ways to introduce redundancy in the system that would make backups needed much less frequently (if ever).
But what a backup does is it allows you to have your data in case your system and accounts are compromised, whether through a misuse, failure, or an attack vector.
An on-site backup is A backup, but it shouldn't be the ONLY backup. Typically you want a hierarchy of backups, one of which is on-site and on-line but a different machine, another of which is off-site but on-line, and possibly one or more offline backups. But on-line backups are fine, IF (and only if) they're copy-on-write/append-only.
Of course if the data can be deleted or overwritten too soon (even tape drive backup systems have some period over which they start re-using tapes) it's worthless. But that's an entirely separate concern from whether it's always-on or internet-connected or on the same national power grid segment as the original.
Take it to the extreme and everything is connected somehow at some point, thereby making a True Backup by your definition physically impossible, even off-site. The point where the backup is made could have a zero-day propagate to the tape machine, killing it and the tape by spinning up the motors. A blueray drive could have the firmware pwned.
If not even incremental snapshots synced to a remote count as backups, you are probably the only one to harbor this interpretation of the word and it arguably loses its usefulness as a word.
An incremental snapshot to a remote is as good as a backup as WD users just found out.
>The point where the backup is made could have a zero-day propagate to the tape machine
Again, the point here is the failure of the backup is decoupled from the failure of your system.
Again, everything is on a spectrum, and we can argue about definitions, however, a copy on a hard drive that's sitting on your bookshelf is decoupled from whatever happens to your computer unless both get destroyed (or stolen) - and that's how good that backup solution is.
The data on a RAID array will go poof if you accidentally rm -rf that partition, so the chances of it failing when the system fails are very high. Hence it's an awful backup solution.
What you have to consider is not whether the chances of failure are high or low, but where the chances of failure of both the system and the backup at the same time are non-negligible.
An incremental snapshot to a remote is as good as a backup as WD users just found out.
>The point where the backup is made could have a zero-day propagate to the tape machine
Again, the point here is the failure of the backup is decoupled from the failure of your system.
Again, everything is on a spectrum, and we can argue about definitions, however, a copy on a hard drive that's sitting on your bookshelf is decoupled from whatever happens to your computer unless both get destroyed (or stolen) - and that's how good that backup solution is.
The data on a RAID array will go poof if you accidentally rm -rf that partition, so the chances of it failing when the system fails are very high. Hence it's an awful backup solution.
What you have to consider is not whether the chances of failure are high or low, but where the chances of failure of both the system and the backup at the same time are non-negligible.p
Dropbox allows you to roll back changes and undelete files
https://www.dropbox.com/features/cloud-storage/file-recovery...
That's to say, Dropbox is not a backup.
The law is complicated and the details are extremely important - but intent is a pretty big factor.
Also - I'd strongly advise someone who is deciding between buying an external hard drive or using a third party for backup to use the third party. There are definitely risks that remain, but their SLA is likely going to end up being a lot stronger than your hard drive's reliability.
It should at least be a tiered offering with different liability caps.
Read the TOS. You're not getting anything from them if they delete your data.
Also, from another comment in this thread:
>I've "lost" (had to restore from personal backup drives) data from Dropbox due to an error made by their support staff during a mass rollback, which was itself needed due to the Dropbox client interpreting a drive disconnection as a mass file deletion.
One does not safely rely on one third party for backups. You should do both the local drive and the remote, not one or the other.
Telling people that any redundant copy that isn't 100% reliable (i.e., everything) isn't a backup is not a good way to get people to take backups.
While a hard-drive could be lost in a fire, etc., the state of availability of the data on it is decoupled from that of the system that has the data that's being backed up.
And that's the entire point! Take that as a definition: a backup is a redundant copy whose availability is independent of that of the original.
FYI, from one of the comments below:
>I've "lost" (had to restore from personal backup drives) data from Dropbox due to an error made by their support staff during a mass rollback, which was itself needed due to the Dropbox client interpreting a drive disconnection as a mass file deletion.
You could access your dropbox data from computer B even while it is syncing files from computer A. Together with the 30 day file recovery window, this seems fairly decoupled to me. Assuming that the dropbox file history and the dropbox normal sync is seen as two separate systems.
Now you shouldn't have this as your only backup, but combine it with a second service with similar guarantees and even if you wipe your local drive and the wipe get propagated to dropbox, you are still backed by the 30 day history inside both dropbox and the second service, which are decoupled from each other.
What you are saying, is that Dropbox allows you to purchase a backup service from them (without any guarantees, at that).
Cloud sync isn't backup. If someone performs backups on the either side of the cloud, that's great, but that's an extra, which is what I'm saying to begin with :)
Diversity and risk analysis are what's important here.
Yes, a building with tapes can burn down. But whether it does so or not is not correlated with the state of the system that the original data lives on.
Redundancy is a great step to prevent data loss. But redundancy won't do what a backup would: keep your data safe even if your system is fucked.
Cloud sync is a double-edged sword, because you make Dropbox a part of your system; their failures are not decoupled from yours anymore. Say, someone hacks into your Dropbox account, encrypts/deletes all data, and then downgrades/cancels your plan so that their backups go poof.
Then you'll find yourself with no data - and no backup.
Compare that to the tape burning down. You just make another tape. The chances of both your system going down, and the tape building having a fire at the same time are astronomically low, because these events are independent.
TL;DR: backups are decoupled, which is why you need them.
Everything else is secondary and can be covered by both. The chance of your redundant data center burning down at the same time as your primary is also low. If one burns down, you just bring up another. Likewise for smaller-scale users, where the chance of your cloud provider being down when your house burns down is also incredibly low.
So it's really the versioning or snapshot nature that's the key aspect of backups as opposed to simple redundancy.
To find out if something is a backup or not, ask a simple question: does the availability of the data on it correlate with what's happening to the system it's backing up?
If the answer is no, then it's not a backup.
Whether your tape storage gets nuked or not does not, in any way, correlate with whether your system gets hacked into, or whether someone who has a grudge against you messes with your system.
Then P(data loss) = P(system is fucked) x P(backup is fucked).
If your backup is not decoupled from the system, then P(data loss) = P(system is fucked) -- which is a far larger.
You don't need to take your tapes to Everest. You just need to decouple them from the source.
TL;DR: if you can destroy a backup from the system it's backing up, it's not a backup :)
I back up to Dropbox and an external hard drive next to my machine. I've used both to restore my data on hard drive death.
I can't imagine the amount of time and/or money it would take to do what you want. It certainly doesn't seem like it is something most computer users would do.
I can set anyone up with a (for example) iCloud + time machine backup with very little work, and it will keep ticking over, backing up. What's your suggestion exactly?
I had to explain, at length, to supposedly smart people that there are no time machines.
I've had people state with a straight face that they'll deploy the backup system if they lose some data. As in: afterwards.
I have never had a significant data loss, and I back up my data.
The weaknesses of cloud are obvious, and alone it's not enough, but leaving it on the table is dumb unless you have some other way of practically and reliably maintaining a far-offsite copy. Most people don't, and even fewer people have a good reason to bother when they're employing offsite backups as part of a broader strategy like 3-2-1.
Planning to go down this route except incremental instead of propagated
Just wow though... Hopefully its not actually doing something like overwriting the data with random bits and just deleting the pointers to the file in the MFT. If the latter case, the data can be easily recovered
I don't know if they still do this, but there was a time their NAS would refuse to allow access to MP3 files over a network. To fight piracy.
https://www.theregister.com/2007/12/07/western_digital_drm_c...
(Was www.wired.com/2007/12/western-digital)
Better link? https://www.theregister.com/2007/12/07/western_digital_drm_c...
> It doesn't matter what the files are: If you try to share these formats over a network, Western Digital assumes not just that you're a criminal, but that it is its job to police users. You see, MP3, DivX, AVI, WMV and Quicktime files are copy-protected formats.
> The list of banned filetypes includes more than thirty extensions. Some of them are bizarre: .IT files are banned — these are Amiga-style music modules composed with Impulse Tracker, a particularly well-loved tracking sequencer that hasn't been updated in almost a decade. I composed with IT myself, back in the day, and still have all my shitty compositions, none of which Western Digital would have me share. (Try MOD vs. Speak&Spell masterpiece Eddie Dreams of Women, if you dare: IT, MP3)
> Isn't it cute how the only data it views as worthy of policing are music and movies? These are the only copyrights that matter under corporate monkey law.
> It's the most astonishing example of crippled equipment I've ever seen. A DRM'd hard drive! Whatever next? Dreaming meat?
> *UPDATE: The manual's appendix and online support site provide setup instructions for SAMBA, allowing access over IP instead of with the DRM-infested and poorly-reviewed client app, elsewhere claimed to be "required." *
> *MOAR! Samba not enough? Gut the firmware and install made-to-measure Linux: An entire community of folks is here to help you hack your MyBook: mybookworld.wikidot.com.
Internal formatting was stripped in the copy-and-paste process. I could add it back, but that would detract from the purity of the approach. ;D
But I have no idea if they would've even had to pay that in the first place on these devices.
https://en.wikipedia.org/wiki/Private_copying_levy#Regulatio...
Edit: Found the manual[1]
"Changing a User’s Password: When viewing details about a user, the Administrator can change the user’s password (no password is the default setting)."
Oof.
[1] https://media.flixcar.com/f360cdn/Western_Digital-905161691-...
There have been times when web crawlers have found pages with delete icons on them that are pointing to GET requests and, after dutifully following them to index the data, resulted in a server being wiped.
Mine are on UPnP and shitty firmware web server
I'll say again: backup drives must have a physical write enable switch on them. To all the people who argue against this - just you wait till it happens to you!
Not having multiple backups is also a very bad idea. One day my 8T drive slipped out of my hand and smacked on the floor. That was the end of it. But I wasn't out my data, just out the cost of another drive.
Cloud backup is a stupid idea. At any moment it could go dark, for any reason, and you have no recourse. Might as well bare your throat and hand some random stranger a razor.
I really disagree - physical storage in your home is the common alternative for a lot of consumer users - and that physical storage will involve maybe unplugging your external hard drive between backups - but otherwise never checking for the consistency and accuracy of the data nor the hardware. If you're working in a data center then it's your job to do these things and it doesn't take very much time... for normal folks cloud backups are likely going to be more reliable.
I've seen enough HackerNews articles:
1. My account got deleted and I can't find a human to talk to and all my pictures going back 20 years are gone!
2. A hacker guessed my password and deleted my online presence! It's all gone!
3. My storage provider went bust and all my stuff is gone!
If my backup harddrive is in the same house as my main storage and my house burns down, I'm fucked.
If I'm fancy, I may have multiple backend storage providers, too. Lot easier to do that then to have multiple houses. I wanted to use a safe deposit box in the past, but last time I called banks near me, they didn't even have safe deposit boxes available to rent! I can ask some friends, but then will be limited in frequency of updates, still.
I believe Resilio can do it, but it is closed source.
If you're looking for backups and not dropbox-like functionality, then something like restic might be more appropriate. It does actual content addressed backups, so you've got history for those times when you realized you messed something up six backups ago.
(Can we make RAICSP a thing? Or do multi-cloud solution vendors already has a snappy marketing term for storing everything on all of S3, Azure Blob, and Google Storage at once?)
I strongly suspect (as in “for the right price would stake my life on it”) that there are some common failure modes across the major cloud vendors’ offerings.
Your advice is directionally sound (and is what I do for family docs and photos); I just think it has fewer meaningless 9s than you’re imagining.
Just went through this with my family (in Australia) when the local bank branch said they were shutting down the service. It was basically a locked room in their branch, there was no "safe" involved.
4. My crypto exchange disappeared/went bankrupt/was a scam and now my crypto is all gone.
I swear, some people never learn.
This very story is just that - people had their own hard drive and it wasn't maintained for security so it got erased. At least this article needs to be added to your mental list of failures of local storage to balance your news-driven bias.
Two wrongs don't make a right.
If you care about your backups, a single cloud vendor/account vanishing isn't going to be a problem, in exactly the same way that if you care about your backups,m a single drive failure will not be a problem. They are both predictable and mitigable risks.
0.85% is extremely high. I've rolled enough 1s on d100s to know that I wouldn't take those odds!
Obviously customer support is a finite resource.
That's why companies need to start paid/premium customer support: when you lost access to your account, you should be able to pay 10, 100, 1000 of money to talk to a real person who understand how the system works and where there a mistake was made.
Like a hacker stole your account and you cannot take it back? A special person will to do a background check (e.g. call your employer) to verify that you is actually you.
Robot erroneously blocked your account because of misclassified spam-like activity? Your account is restored, and you are fully refunded.
EDIT: According to Wikipedia, ZFS, Btrfs and ReFS offer strategies to deal with various forms of data degradation, although it appears as if ZFS is still king.
https://en.wikipedia.org/wiki/Data_degradation
I also use it for music files, in the past too many mp3s got broken.
Before I did this, in the past, I restored from backup and found I had damaged jpeg files. With par2, there's at least a recourse.
http://md5deep.sourceforge.net/
Cloud backup is perfectly fine as long as it's one of those copies -- I cloud backup my desktop and it saved me from a lot of loss.
But not as a backup strategy.
If you use BackBlaze, they offer multiple options to restore data quicker if you're willing to shell out more money, including AWS or sending you a physical drive with your data on it, albeit the latter is kinda expensive, but if you have a ton of data and need it ASAP, is probably worth the cost.
I recently lost a 16TB drive for the second time in 6 months and my local backups were out of date by a couple months, meaning I needed to restore ~200GB of data. Aside from the web interface file explorer window being cramped (and support telling me to basically go fuck myself after suggesting that), I was able to make ZIP files from recently changed data and use their downloader tool which IMHO is a piece of crap and probably designed by some intern, but at least the download speed was fairly fast, even in Europe, in contrast to upload speeds capped at around 250 KB/s (though thankfully support multiple threads).
Data theft is another one. Encryption might help, but you have no reasonable way of knowing if the encryption is secure.
Nonsense. Any robust backup strategy has multiple copies, and multiple locations. Cloud is a sensible component to this.
“It’s not a backup until you restore it.”
If you are aware of the issue, and get kicked off AWS, or whatever, you move the backup. It is only a problem if your data catxhes fire and you get removed from cloud at the same time
It's not stupid but redundancy is important, don't rely on cloud only.
My own anecdata: I lost a couple TBs with Streamload[0] ~2007 when they switched over to MediaMax. They wouldn't admit it was their fault and I never got any kind of compensation - even though I had just renewed my yearly subscription a couple months prior. Thankfully I had offline backups on CD/DVD but that was definitely no fun. More recently CrashPlan decided to no longer offer unlimited backup for non-enterprise users i.e. I would have to pay a LOT more money to back up multiple TBs. At least they offered me a partial refund for the remaining months in my yearly subscription. I really really hope Backblaze keeps their unlimited plan.
As for physical backups, my 8-drive RAID-5 GPT partition got somehow nuked when I was trying out a new Windows 8 installation (one of the first official builds). I never found out what actually caused it, but I immediately went back to Windows 7 and waited for 10 to be out for a couple years before finally switching over. I'm still paranoid and keep my backup server separate and off the network except to make diff backups.
Bit-rot is also becoming a serious issue. I hope to finally build a home ZFS server (with ECC RAM of course) to deal with this, as I've noticed non-streaming encoded videos sometimes just stop working after 15+ years. I'm not sure if it's actually bit-rot but I can't think of anything else that would cause this to happen.
In any case, keep multiple backups: offline and online!
[0] https://en.wikipedia.org/wiki/Nirvanix
Hard disk drives go on sale regularly. I replace mine usually once a year or so. They're not expensive relative to the loss of the data. I just bought a 500G SSD drive on sale for $60.
- Solid-state media...store data using electrical charges, which can slowly leak away due to imperfect insulation.
- Magnetic media...may experience data decay as bits lose their magnetic orientation. Periodic refreshing by rewriting the data can alleviate this problem. In warm/humid conditions these media, especially those poorly protected against ambient air, are prone to the physical decomposition of the storage medium.
In either case, only ZFS/Btrfs/ReFS seem to implement strategies to deal with data corruption. Kinda sad that Apple recently released a new FS and didn't bother to address this.
[0] https://en.wikipedia.org/wiki/Data_degradation
WHAT!? How the heck did you get that conclusion from this story? These are literally your people, who backed up to a local device and essentially chose their own local NAS rather then the cloud. Which unfortunately for them came from a company with bad support that dropped it 6 years ago and likely did a shitty job even before that. But not everyone can be a tech person! Even tech people can't be that knowledgeable about more than a fraction of information technology, the field is crazy vast [0].
The failure modes here are exactly why "cloud" makes so much sense for so many. Keeping stuff up to date, or migrating if the company drops it (even noticing that support has stopped). Securing access. Verifying integrity. Amortizing the cost of different media cold storage fallbacks with things like tape robot facilities. And on and on. You wanna opine a bit on the relative probability that Amazon, Apple, Backblaze, Dropbox, Google, Microsoft, rsync.net, or a host of others just "at any moment going dark, for any reason" vs Western fucking Digital My Dropped-in-2015-Book dying? Think carefully.
I mean shit, I actually have a custom Epyc-based TrueNAS Core (not FreeNAS anymore! gotta keep up!) system with tens of terabytes as my main backing store, and replication to a remote site, with more usage of network segmentation, wireguard, ZFS user privileges so compromise of the replication credentials can't delete old stuff, then I'd expect nearly anyone to have any idea about. I still have it going to Backblaze B2 as well, and I pay to have them hang onto it for a year rather then 30 days. And the only reason I can swing it all is that I can amortize a lot of it professionally, get other usage from it, and draw upon a bunch of both knowledge and metaknowledge that my particular life path happens to have granted me.
For 99% of the people in my life "turn on iCloud/Backblaze/whatever" is the right advice. Backups are very risky without heavy automation and regular attention on top.
----
0: One thing working at various levels has given me a deep appreciation for is what broad shoulders I stand upon, many layers deep, and how valuable and deep the knowledge of so many other tradespeople I interact with in life is. The mechanics, electricians, plumbers, general contractors, structural engineers, medical folks, researchers and so on I work with or have worked with all possess skills and knowledge that I never will. That's why we're social animals. I've traded on my skills helping someone improve their network for a hand with skilled carpentry or electrical work or the like. I don't see any shame in that.
Consider that I worked on the 757 stabilizer trim gearbox. The idea is no matter what fails, the airplane lands safely. I've been trained to think that way.
Consider also that I've worked with computers for 45 years. I've seen about every failure you can think of, including fires and floods, explosions and earthquakes, viruses and phishing.
The most unreliable, by far, part of computing is the internet.
Fact: if these people had been running to a decent cloud service, they'd probably still have their data. Really "don't connect backup to the internet" is like, fractally stupid. The more one digs the more stupid angles turn up recursively similar to higher level stupid. "Don't connect backup to the internet" means what to you? What about the computer itself making the backup? If that is connected it could get infected as well, now what? Everyone is just supposed to give up on all networks entirely because "hurr durr nothing on a network is secure"? Which is wrong anyway since security an economic equation, not some ethereal absolute. I hope you never give this advice to anyone IRL.
> if these people had been running to a decent cloud service
They thought they were.
> If that is connected it could get infected as well, now what?
That's what the physical write-enable switch is for. A hardware read-only device cannot be written to.
How would you do automated backups with a physical switch? Are you proposing nobody does automated daily or more frequent backups?
Just remember that ransomware will encrypt any drive you hook up to it, including your backup drives. A physical write-enable will prevent that, and it should be disabled anytime you are restoring from backup.
I want a write-enable switch even for simple tasks like copying one drive to another, just so I don't mistakenly copy the wrong direction.
What I was alluding to but didn't make clear was that physical switches are easily defeated by human laziness or mistakes. Why have a physical switch on drives you're copying if you might accidentally switch the wrong one or eventually stop bothering? This already happens with warning popups in Windows when you try to run an untrusted program, for instance. People get trained to bypass the security because it's just a tedious obstacle.
You personally might be careful enough to forever set the switch correctly, but people who didn't even know their hard drive was years-un-patched and internet connected yet left it there to get hacked also wouldn't reliably set the switches every time either.
Regarding accidentally copying in the wrong direction, I think a safer way than a physical switch is to show some details of what will get deleted. Maybe previews of images, tiny snippets of text from the files, a graphical view of the space they occupy, etc. Make it more visceral, like throwing away an actual book where you can see how many pages it has and what the picture on the cover is.
Yep, I once reformatted the wrong drive. I plugged in a firewire drive into a friend's Mac and didn't realize he had an internal drive that apparently also showed up with a firewire icon. Whoops!
Reminds me of the little sliders on memory cards. Though I think those were software driven. It didn’t actually lock it out from writing.
Are there any portable USB style disks that have a hardware enforced read only switch?
My house could burn down tomorrow, but I'd still have my cloud backups. My cloud backups could roast in a fire tomorrow, but I'd still have my local backups. And probably several other cloud backups.
If you put all your eggs in one backup, sooner or later, you're going to get screwed.
As opposed to what? I think you've got too much faith in your method.
(Data transfer charges not included)
I suspect your drive is more likely to die in its own that AWS to fail.
I also don't care to have the internet snooping on my data. Yes, I know about encryption. Enough to know it cannot be trusted.
Home backup is a stupid idea. Any any moment your place could catch fire, and you no recourse.
I wrote something about it [1] just a few days ago. And I have been banging on about it for years if not a whole decade.
>What we need is something like iOS TimeCapsule. Let the Phone do the Photo Management on Devices, and have an on site ( Home ) DataBackup as well as offsite ( iCloud ) Data Backup. Apple could even make iCloud Backup subscription a requirement for TimeCapsule to function.
We need both. In a simple, safe, easy to use fashion.
[1] https://news.ycombinator.com/item?id=27605365
I don't know what you mean by "cloud", but an off-site backup is essential. Your home can be burglarized, or burnt down, or flooded. Basically anything you insurance covers for your home you should cover regarding your data.
> At any moment it could go dark, for any reason, and you have no recourse.
There is no difference between your backup going dark or your main data source going dark. The whole point is that the data is duplicated and the odds of both of them going dark at the same time is low. This only means that you should check your backup regularly.
But use ext4 instead of HFS+, and use Samba instead of Netatalk. With HFS+, the bundle would get corrupt after a few days. With Netatalk, a sudden power outage would corrupt the whole thing.
Don't listen to 99% of blog posts on this. They all tell you to use HFS+/Netatalk. None of those authors have actually used that backup solution for any extended period of time, because if they did, they'd find that it's unusable. Use this one instead:
https://mudge.name/2019/11/12/using-a-raspberry-pi-for-time-...
Samba. Ext4.
You drag and drop to the SFTP destination[1] which covers "History"[2] with immutable ZFS snapshots and "Integrity"[3] with ZFS itself and "Security"[4] by running nothing but OpenSSH.
If you're going to set your parents up with something, you could do worse than that ...
[1] https://www.rsync.net/products/sftp.html
[2] http://taobackup.com/history.html
[3] http://taobackup.com/integrity.html
[4] http://taobackup.com/security.html
The thing is... I can't help but worry someone's going to compromise my NAS and DBAN the drives and then extract the Azure storage key and use that to delete all my backup blobs...
(Yes, the backup client needs read, write, and delete permissions, unfortunately - and Azure doesn't offer a Blockchain-style "append-only" mode for blob storage, unfortunately - still, better than nothing).
UPDATE: Apparently Azure Blob Storage does support strict append-only blobs that cannot be mutated or deleted, only appended - so I wonder if Hyper-Backup can use that…
Is there anyone out there with $$$$ who will stop at nothing to part you, a rando, from your old data? Probably not. Are there sophisticated attackers who will burn a couple 0-days to build a botnet for the sole purpose of randsoming NASes AND attached cloud accounts AND the origin systems, accounting for tons of possible configurations? Still pretty unlikely — this is NotPetya level stuff with small payoff.
If you find yourself in the crosshairs of a sophisticated, dedicated attacker (perhaps one in possession of a 0-day), you’re pretty much done. Offline write-only backups stored offsite are the only defense.
However, is there a bug lurking in Hyper-backup that might accidentally wipe stuff from Azure storage, and the bug hits a month before your house gets struck by lightning? Maybe…
Brb calling my psychiatrist for more xanax
Are you forgetting the recent post about how kids these days don't even have any concept of files? You're over-estimating most people in their technical competence and the required level of understanding and patience needed to do anything with SFTP. You say "drag and drop" and I think of the hours of troubleshooting and explaining (and re-explaining) I'd have to do if I were to try to implement this with family.
I'm using a WD My Cloud for non-essential data. I was wondering if my unit would be affected as well. As far as I can tell, the latest firmware update completed January 01, 2020. I'm safe for now but I better build something more robust.
I'm assuming the My Books phone home in some way to facilitate file access over the internet. If so, that WD system got cracked. That seems more likely since-- at least for my ISP and I'm guessing most home ISPs-- don't expose individual device ports to the world unless the user sets NAT for that.
Anyone know if NAT configs are required to use the feature that let's you access files from anywhere? If so, someone simply cracked bad security 6 years out of date and scanned for devices.
Given typical IoT security, it might only have been using the default user & password.
I doubt it. The specs list UPnP support. DLNA on some versions too. No consumer device is going to expect the average punter to mess with NAT.
If so, someone simply cracked bad security 6 years out of date and scanned for devices.
Maybe the majority of them aren't so much an "exploit" as they are "easily guessed credentials". The fact that anyone else, anywhere else on the Internet can also use those same credentials to access their data is something that probably few people realise when they set one of these things up.
A unpatched in 6 years linux device directly connected to the internet? What could possibly go wrong?
"There is no IoT, there is only the internet of unpatched linux boxes."
For this particular attack (assuming c2 server compromise?) that might not matter, but ultimately there is a massive difference in attack surface when comparing “direct” with “NATed”
How much would you bet against that being an unauthenticated call or one with leaked hard coded reds?
Assume some javascript that loops through:
http://192.168.0.1/cgi-bin/factoryRestore.sh
http://192.168.0.2/cgi-bin/factoryRestore.sh
http://192.168.0.3/cgi-bin/factoryRestore.sh
up to
http://192.168.0.254/cgi-bin/factoryRestore.sh
Those just give connection errors for me, but:
http://192.168.1.1/cgi/ACT_FACTORY_RESET
gives me a 403 Forbidden error, and if there was a known default password to my router - it'd try and do a factory reset on it. (It actually wouldn't, it'd send back a confirmation popup, but...)
Or are you assuming the user's browser itself is compromised and is running random javascript hitting the NAS address? That would be unfortunate, but I'm not sure I'd blame it on the "6 years past EOL consumer linux device"
Classic old cross origin request forgery. It ranks #7 I owasp’s top 10 website security flaws, and they have this to say about it:
“XSS is the second most prevalent issue in the OWASP Top 10, and is found in around two thirds of all applications.“
So, yes, essentially you end up with a bunch of devices running vendor-specific unmaintained Linux distros on the wide open Internet. The “unmaintained” part of that sentence is the problem, not the “Linux” part.
Shouldn't we design systems that are hard to break by default? Shouldn't the OS assume that terrible things are going to happen anyway, and provide protection from bad faith actors in case the OS is indeed left unpatched for 10 years while being fully exposed to the internet? Is it even possible to design a system that provides this level of security so that we can get away with near-zero additional security expertise from product designers who build on top of it?
I think that, first of all, it's Western Digital's responsibility that things went south here. But shouldn't we build systems that provide bomb-proof security for the many companies that build on top of it? Is it even possible? And if so, how? In the end we would be doing ourselves (as consumers) a favor.
So if anyone wants to support their 11 year old drive they can do so.
However this is a problem the industry has been struggling with for decades. It's simply not easy (and maybe not even possible) to achieve what you claimed "should" happen. Nobody knows how to produce bug-free software at scale.
I highly doubt it, at least not with 100% certainty. We build bridges to last and to stand weather. These are enormous constructions with large safety margins, teams designated just for security (against weather, earthquakes etc) with peoples lives on the line. Yet, we need to inspect them on the regular to make sure no assumption broke, no safety system was triggered and nothing unexpected happened.
If we can't make this work at this scale, I have no hopes that software for comparatively cheap consumer devices manages to achieve this.
[1] https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=OpenSSH
I just used it for continuous laptop backups over my LAN, so unless one of them crashes tonight, I should be good. But, this will certainly give me pause when considering WD products, and this type of product in particular.
This is very bad, WD! Are your other products also vulnerable in this way? Why should anyone ever again trust your company to keep their data safe?
Because if you need a hard drive, your options are pretty much Western Digital, Seagate, and Toshiba, and they've all had their own problems.
Unless WD changed their mind again, HGST HDD is still available as of 2021 because customer has been demanding the exact same HDD with exact same model number from the exact same plant. Basically they want to know it is the same old HGST and not WD. And some ( if not most ) of them are large, enterprise customers. So they brought it back in ~2019 (without an announcement).
Edit: To be clear, it was my mistake, my post made no sense before I fixed it. The parent doesn't deserve the downvotes.
The affected product is not a drive but a NAS solution. In terms of being a reasonable consumer my thought process is to evaluate distinctive products differently.
NAS is a cheap computer with attached drives. It’s going to have computer-like failure modes.
But I might be reading too literally. And personally, I do intend to continue purchasing Western Digital hard drives, just nothing internet connected, because I agree they're different products with different risks.
I highly doubt the average non-technical user even knows who produced the hard drive they own.
Yes, they do. My dad bought two WD My Book drives by himself as backup drives. He luckily was not aware that those things have an extra software for backups, nor how to install it, but those things are just sold like that targeted to the average user in your average computer store.
But THIS affected type of drive is specifically marketed to a very non-technical user, and I can certainly confirm my non-technical friends and family have bought products in this segment independently, whereas my techie friends steer clear.
It's a NAS that's made to look like a drive. My techie friends would never buy it - they'd get Synology/QNAP, or do their own over-complicated time-consuming solution (slight editorial opinion there ;), or use cloud backup, or some combo. But my dad, mother in law, and other relatives have products like this, and have purchased them on their own. In fact, I think when asked, most techies would in fact go against the notion of buying cheap but internet-exposed storage device for a non-technical friend :O
> For a Linux user, you can already build such a system yourself quite trivially by getting an FTP account, mounting it locally with curlftpfs, and then using SVN or CVS on the mounted filesystem. From Windows or Mac, this FTP account could be accessed through built-in software.
Source: https://news.ycombinator.com/item?id=9224
[0] https://www.zdnet.com/article/malware-found-on-new-hard-driv...
The difference is the protocol that goes over the link. In a NAS, it's SMB/NFS access to files over ethernet/wifi, and for a SATA drive it is the SATA command set accessing blocks. The difference is small.
One of these things is talking to the internet on its own accord. The other is not. Small difference, huge implications.
There are quite a few other brands offering NAS devices e.g. Synology, QNAP, Thecus, Pegasus and more.
Aha! It was $600k in dogecoin in 2014
https://www.pcworld.com/article/2364120/hacked-synology-nas-...
That’s worth around $125m today (assuming straight hodling).
https://www.synology.com/en-us/company/news/article/2019July...
Majority of people buy NAS only use it to access their Data within their internal network. But somehow they all include Internet / Cloud features as upsell.
Of course the issues shouldn't be there in the first place, but it's not like they're ignoring the problem.
What? They backdoored every device.
https://www.bleepingcomputer.com/news/security/qnap-removes-...
Using 500GB SSD for many years as an external backup.
This is the third such major scandal from WD in as many years. First, selling SMR drives as CMR. Second, selling 7200RPM drives as 5400RPM. Now this. They've lost a lot of credibility, and I don't see any of the other manufacturers having issues rivaling WD.
I suppose you mean the other way around? Otherwise, what's the problem with selling a higher RPM device as a lower one? Isn't higher RPM better? (Genuine question, I don't know much about HDD).
Less noise and usually higher durability.
WD cut down the number of real SKUs and then slowed down a 7200 RPM drive in firmware calling them "5400rpm class"
WD Red is marketed for NAS/storage, where it's not uncommon to have 4/8/12/16 drives in the same chassis where noise/heat becomes a very different problem. It turns out that the drives generated the same heat and consumed the same power as 7200RPM drives, while the data sheets were outright lies and BS. WD decided on double down on the BS for several months, and only recently released new SKUs where the specs matched the marketing.
They sourced drives from former HGST under WD but 5K4 we’re not available so marketing rebadged 7K2 as “5400rpm-class” and lucky buyers were hit with better performances.
Sort of. The WD Red drives outperformed actual 5400 drives but underperformed other 7200 drives by a large margin (180MB/s vs. 230MB/s), while not necessarily offering heat or noise benefits that matched. My current 8x8TB WD Red Unraid build will likely be the last WD drives I buy for a long time.
They also draw more power, which might mean they simply don't work properly in some low-power devices, or mean that any UPS someone might have one connected to will not last as long as they expect during a power interruption.
(Not that I would put anything like this past them; Japanese companies are notorious for leaving ancient buggy software at EoL)
The drive happens to be Toshiba, but that is just a medium
But hey, if you just got a ransomware note, and think "I'm good, I have my backups!", wouldn't you want to flip the read-only switch before plugging in those backup drives? I would. In fact, I'd flip that switch always before trying to read from a backup drive.
WD really messed up there - but they and others will mess up again, so if the user's goal is not losing any data they'll still need to do more than buy the next shiny thing and click "accept" on the EULA. Because in the end pushing around the blame won't get you the files back.
Needless to say, I never ran any of the Live code. Several of these sorts of things have come up in the industry that always made me recall how happy I was to not have those drives with their backdoors on my network.
I presume you mean shouldn't rather than should.
[1]: https://blocksandfiles.com/2020/04/14/wd-red-nas-drives-shin...
Seagate is the other big consumer brand and they also have their own set of issues. Anecdotally, I've had multiple Seagate drives die within a year or 2 whereas WD ones have kept going for years longer.
Backblaze does a survey every quarter, here is the latest one https://www.backblaze.com/blog/backblaze-hard-drive-stats-q1...
They are all in the same ballpark, with Seagate a bit worse than the others but not terrible like it was a few years ago. WD tends to be consistently in the middle of the pack.
The latest fail from WD (as a hard drive manufacturer) was when they sneaked inadequate SMR technology into "red" consumer-grade NAS drives. But others did more or less the same. Now, all manufacturers are transparent about the technology used and newer "red plus" drives from WD are CMR.
So I'd say, for now, I'd say WD is as good (or as bad) as any other and I would let price decide. For reliability, that's what RAID is for. And in any case, RAID or not, and no matter how reliable your disks are, if your data is precious, you need backups too.
I'd hope it gives you some pause when considering "cloud" products as well, that was the reason I avoided this.