Yes, there's nothing wrong with SMR at all, if it's not your primary storage device (though I doubt anyone reading this is in danger of making that choice in 2020).
It's roughly a 25% boost from the same physical platters. If you're storing large files with a primarily read workload, it's almost certainly the right choice.
Except it means you can't use the drives for RAID, as they'll fail the initial resilvering. Worse yet, their firmware appears to be buggy enough that they're not just slow, they start throwing errors once the PMR buffer zone is full.
Sounds like we'll need to add some sort of buffer congestion avoidance (anti-bufferbloat) logic as part of storage drivers to solve this issue. It can crop up in any use of SMR tech, not just RAID rebuilds. And of course solving this would also mitigate the issue w/ rebuilding drives dropping out of RAID arrays altogether.
RAID Controllers (Software and Hardware) need to be aware of SMR. The easiest fix is that the controller will not allow you to mix SMR and CMR technologies in a single array.
ZFS and friends need to check that too and if there is an SMR drive, give warning and handle it appropriately. You can't resilver an SMR like a CMR; the performance is bad enough that a few controllers will think the drive is dead too.
Nothing wrong with shipping slow SMR drives, the issue is, it was never explicitly disclosed to the customers/end users.
It's not that simple. Even it the letters "SMR" were used in marketing, most customers won't know what the technical term means or what its implications are. In reality, it means some drives using it, such as WD Reds, are not suitable for their advertised purpose. I don't know what the law says about this in any particular part of the USA, but certainly in some other countries that would be a breach of consumer protection rules even if the technology had been disclosed.
I'm non-US, and I already have data on them -- they are part of 8 drive RAID6, the rest are the "old" EFRX drives -- so need to rotate them with new drives
Anyway, thanks, I will contact them and try to find out what the options are.
Where's the Seagate lawsuit? They've been doing this too for years now. AFAIK you can no longer buy 4TB Seagate consumer 3.5" drives now, the current product line is all undisclosed SMR unless you go to higher tiers.
E.g the ST4000DM000 (which I have a pile of in an array) was replaced with the ST4000DM004, which is sold as a direct replacement in that product line but is SMR.
That's fair, though it's still sad that you need to spend more money for a Seagate NAS drive today when 4 years ago a Barracuda cost 20% less than that and worked just fine.
They'll work fine (although with the normal expected slower writes of SMR drives) until a drive fails normally and the raid array has to rebuild (which is kind of the reason you use a raid array), then it could take a week+ and there's a very good chance you'll get more failures during and lose all your data.
> the ST4000DM000 (which I have a pile of in an array) was replaced with the ST4000DM004
So I had no idea about this, and was curious if this is why I've been having so many issues with 3x4TB Seagate drives in FreeNAS. Turns out all three are the SMR ST4000DM004 model, and I will be looking to replace these post-haste. My own fault using consumer drives!
If anyone is curious about said issues, it usually starts within 24 hours of booting the server. Eventually it will start throwing checksum errors on that array, even with as few as two disks in RAID1. Scrubs/resilvers generally have errors (10+ daily), and if your pool survives "too long," it will start kicking disks out of the array, and will not re-initialize them on a reboot.
I'm pretty sure that page is a recent addition after all the bad press this issue has gotten. It does not contain proper model numbers, and the info isn't (or wasn't) in the drives' spec sheets, which is where it should be.
Both the ST4000DM000 and the ST4000DM004 are not marketed for NAS usage, because they are part of the Barracuda line. You should've gotten IronWolf drives.
I shouldn't "have gotten" NAS drives, because I have 15 of the DM000s in an array and only one has failed so far. That means, assuming getting NAS drives would've earned me a 0% failure rate (which they wouldn't have), I would've spent two to four times as much money on the NAS markup as I have on replacing the single failed drive.
Just because Seagate says they want to upsell me for dubious market segmentation reasons doesn't mean I need to pay attention to them. I know what I'm doing, I know desktop drives work just fine for my use case.
Before SMR. Now with SMR, they've found the perfect way of getting rid of everyone who was buying desktop drives for NAS use they were perfectly suitable for, and force them into the NAS upsell. 4 years later and it still costs more to buy a NAS drive today than it cost to buy an equally fit por purpose non SMR desktop drive back then.
Which is strange, honestly. RAID rebuild is a streaming write from one end of the disk to the other; something which should be the ideal case for SMR, if the firmware catches on.
That's only the case in the first write. Rewritten blocks suffer from severe write amplification in SMR drives due to much larger segments needing to be written even for small changes.
Rather than ideal, a rebuild is actually the worst case scenario as the RAID may detect the drive as unresponsive as it scrambles to catch up and thus gets dropped, exacerbating the problem for the other drives in the array. This is HL2 Resonance Cascade bad.
If it is software raid (like ZFS or Btrfs), the writes are random.
Plus, it's not clear SMR can be made to accept heavy write load even when the load was predictable.
- rebuilding on a non-fresh/cleard hard drive can have negative performance implications due to how SMR works
So my guess is that it's a problem with the firmware which can potentially be fixed. After which they could be used in a NAS, just aren't necessary as far as you might think. So I guess it's probably not looking too good for the lawsuites chances to cause much yield/discount.
> That 9 day and almost 14-hour rebuild means that using the WD Red 4TB SMR drive inadvertently in an array would lead to your data being vulnerable for around 9 days longer than the WD Red 4TB CMR drive or Seagate IronWolf.
The other 4TB drives they compared against completed in around 15 hours, meaning the SMR drive could potentially take 14x longer to rebuild a RAID.
WD richly deserves to be raked over the coals for not labeling these drives clearly.
Their initial customer-facing reaction was even worse if I recall: One customer escalated his issue and asked WD to identify which drives used SMR, WD said something along the lines of We don't share that information with customers.
(If anyone has a pointer to that story I'd appreciate it. I can't find it at the moment).
"""
Also: “Well the higher team contacted me back and informed me that the information I requested about whether or not
the WD60EFAX was a SMR or PMR would not be provided to me. They said that information is not disclosed to consumers. LOL. WOW."
"""
I can't see any information about when they started doing this? The Red drives have been available for a long time, have they always been SMR or did WD switch them out at some point?
They switched them, quietly, but only for some products at some sizes. A subtle change in the model numbers is the only clue when you're ordering.
There was a lot of press coverage of this a few weeks ago when the story first broke, including here on HN. It looks like the lawsuit reported here is the first formal legal action in response.
From what I've read, the issue is that the reduced performance has a direct effect on stability: When used as part of a RAID array then these disks can exhibit unexpected long pauses under load, which causes the RAID controller to mark them as faulty and kick them out of the array.
There are quite a few stories of people being unable to use these HDDs for archival, for this reason.
If you use them as JBOD, they should be perfectly stable, just a little slower at writing than at reading. But as said by sibling, not stable in a RAID.
SMR isn't "a little slower" depending on your workload, but massively slower (RAID rebuilds are just one example).
Do not buy SMR if you have any kind of write-heavy workload, or if you rely on consistent performance (e.g. any kind of streaming / latency sensitive usage).
The Red series was originally marketed and positioned specifically for Home/SMB NAS use. I have 4 of these (the old versions from before the SMR switch) running in my Synology NAS for years, and a 5th lying spare in its original sealed packaging just in case.
When WD stealthily swapped to using device managed SMR technology in the series, they hoped that typical users wouldn't notice the change in behavior of the drives, as they assumed a small NAS would have enough downtime for the drive to manage the data being behind the scenes rearranged towards the slow shingled sections.
This assumption is not always true, but specifically it doesn't hold at all when an array has to be rebuild e.g. because of a drive failure in a raid array, as that requires writing data continuously and for a long time.
WD was caught not just using a technology that made their drives unsuited for the advertised purpose, but initially even denying that they had done so.
Or, they may have actually tested raid rebuild on a system that did not have a single write during the rebuild and only with fresh drives - under this very specific use case, if the RAID controller does things right, all writes are sequential from the first disk sector until the last -- which is a use case that SMR doesn't slow down in; essentially, the only such use case.
On a related note, RAID rebuild failing on slower drives is ridiculous these days. Is it so hard to implement something like buffer underrun protection in CD/DVD writers? Or just longer/adjustable wait times?
These issues mostly crop up with hardware RAID controllers that were not designed for anything like that. And what you'd need is actually the opposite of buffer underrun protection; you have to protect against congestion, whereas having the drive idle would not per se be an issue.
The enterprise hardware RAID controllers have battery backed memory that is heavily utilized during a rebuild. Operations that would take days without it are reduced to hours. I'm not sure what a typical NAS does though.
In the case of drive failure, the RAID rebuild will probably take days, not hours. If failure isn't something you're concerned with, or you only ever read and never write, it's not an issue.
Otherwise, you will probably be interested in replacing them sooner rather than later.
I would have expected that for a 2-drive system, a RAID rebuild would only be writing to the replacement drive. What data would be written to the old one?
Under that assumption, my strategy is to isolate these drives in their own 2-drive RAID volume until they fail.
Sometime ago was reading some zfs issues and it was first time I got to know there are SMR/CMR drives. Anyways, they mention exact serial numbers I see in this article. [WD WDx0EFAX drive unable to resilver](https://github.com/openzfs/zfs/issues/10214)
I don't know if iXsystems (the company behind FreeNAS) has a deal with WD, but their product page[0] still reads:
> WD Red drives are specifically designed to handle the rigors of 24×7 FreeNAS workloads, offering excellent data protection and the highest level of performance possible.
Remember when drive makers were constantly looking to one-up each other on reliability? Those were the good 'ole days. Bad press for consistently losing users' data back then could put them out of business. Now they're so big that even uber amounts of bad press and deliberately shipping inferior products can't do that - they'll pay a few million dollars in fines and go right back to doing what they were doing.
Honestly no I don’t, as far as I remember it has always been a competition on price and capacity. Maxtor, IBM DeathStars, neither killed those manufacturers (IBM lives on as HGST, Maxstor as Seagate). Maybe it was different in the enterprise markets, but Red wasn’t an enterprise drive.
It typically took a really, really, really bad debacle to get that level of attention. Back when I was selling hardware, I can think of a few incidents where certain drives were just known for data loss.
Early WD IDE 20GB drives - absurdly high 90d failure rate. Shop stopped carrying WD for over a year after these
Quantum Bigfoots - At best slow, and not -terrible- for reliability but I remember them having a really nasty failure mode.
IBM Deathstars - Everyone knows about these.
Fujitsu drives in the 8-30gb era - I think these actually wound up having a lawsuit over them
Heck, back then, on top of having to worry about all of that, Creative lab sound cards could cause data corruption in some motherboards!
And yet, only IBM suffered severe damage. People kept buying bigfoots. People kept buying Early model WD drives that bit them. People kept buying creative labs cards that had bad bus mastering....
In the late 90s on efnet's #computers we used to say "don't get a maxtor, just write your data to a dead fish instead. It's cheaper, and you have about the same chance of reading your data back."
I am curious as to when these good ol days are for you.
What would you rather have, then? No lawsuit at all and no check? A bigger check for you? Who would pay this? WD? The lawyers? Keep in mind that lawsuits are expensive and the lawyers are essentially working on contingency. Lower payouts for them might dissuade them from suing altogether.
That would need a dramatic change in the legal system. Either making the losing side pay the other side's legal fees, or increasing the amount of punitive damages that a judge can award. Both of these are non-trivial changes to get through congress, and have numerous side effects.
While seemingly unfair for you, the lawyers and the big incentives they get do provide a valuable societal check on company behavior.
For example, I imagine your typical executive at a big company looks at the list of salaries of their employees over 45 and thinks to themselves, hey, why don't we just fire these people and replace them with young people?
Then, they think about the aforementioned lawyers and subsequent class action lawsuits, and then decide not to. Sometimes.
After this upcoming class action, you can bet the next WD exec is going to think twice before screwing over their customers.
Consumer protection agencies and similar can potential force some "free exchange" program on them.
But here is the think: From the iXsystems blog post you can learn that not all SMT drivers so lead to errors on rebuild. Just some of the FW versions do. Other drivers might still lead to big slow downs but it's not clear if all SMT drivers slow down to a unacceptable degree. Weather it not a newly added drive is fresh/was cleaned seems to also affect this. Lastly while the drives are marketed as NAS usable they have a list of supported NAS systems. And don't give any guarantee for performance on other systems.
Given all this my guess is that after fixing faulty firmware version they will at most provide some form of we update the firmware for you service if you can show that you are using a supported NAS system. ...
And then if any consumer protection agency or similar acts in the EU that program might also be available there.
Through that just my sold guess. Thinks can be very different, e.g. not just a firmware bug.
I just overhauled my entire synology array and my original purchase was two reds and two seagate ironwolves. I ended up returning the reds because they were disconcertingly loud and performed poorly, which it turns out was probably because they were SMR drives!
I'm using four of the seagate ironwolf drives now and they have been fine so far. Equivalent price point and capacity, slightly quieter than the WD. Just make sure to check whether you're getting SMR or not - the ironwolf NAS drive line is not SMR according to Seagate's docs that I read but definitely verify for yoruself.
If you want to stick with WD, you can still buy WD40EFRX, which are not SMR, at many retailers. Of WD's current product line, you want to avoid the EFAX, EZAZ, SPZX, and SPSX models.
I don't see how this succeeds. If you read the WD Red product brief, it says "without being tested for compatibility with your NAS system, optimum performance is not guaranteed." Then, the compatibility list I can find shows the Red's have limited options. FreeBSD and Debian weren't compatible, and I gave up after that.
Unless you can show that your hard drive is causing problems on something they claim compatibility with, it doesn't seem like they'd be liable. One of those times where they're clearly in the wrong, but not legally at fault.
For anyone still doubting WD is at fault here, as I was (because I thought these were obviously inferior drives not appropriate for RAID or 24/7 operation), just look at the product page [1] which is still, as of Fri 29 May 13:50:16 UTC 2020, claiming that WD Red drives are "Built for NAS compatibility" and "Designed for RAID environments".
If some Red drives (most probably the SMR ones) require many day rebuild times or even fail to rebuild in common commercial consumer NAS solutions, they are obviously not viable for RAID environments and then WD is guilty of false advertising.
88 comments
[ 3.2 ms ] story [ 157 ms ] threadFor that, they deserve the lawsuit.
It's roughly a 25% boost from the same physical platters. If you're storing large files with a primarily read workload, it's almost certainly the right choice.
ZFS and friends need to check that too and if there is an SMR drive, give warning and handle it appropriately. You can't resilver an SMR like a CMR; the performance is bad enough that a few controllers will think the drive is dead too.
I consider that the biggest issue. Drive manufacturers are trying to charge CMR prices for SMR drives.
It's not that simple. Even it the letters "SMR" were used in marketing, most customers won't know what the technical term means or what its implications are. In reality, it means some drives using it, such as WD Reds, are not suitable for their advertised purpose. I don't know what the law says about this in any particular part of the USA, but certainly in some other countries that would be a breach of consumer protection rules even if the technology had been disclosed.
Anyway, thanks, I will contact them and try to find out what the options are.
Oh boy refrain me from celebration.
E.g the ST4000DM000 (which I have a pile of in an array) was replaced with the ST4000DM004, which is sold as a direct replacement in that product line but is SMR.
https://arstechnica.com/information-technology/2020/04/seaga...
So I had no idea about this, and was curious if this is why I've been having so many issues with 3x4TB Seagate drives in FreeNAS. Turns out all three are the SMR ST4000DM004 model, and I will be looking to replace these post-haste. My own fault using consumer drives!
If anyone is curious about said issues, it usually starts within 24 hours of booting the server. Eventually it will start throwing checksum errors on that array, even with as few as two disks in RAID1. Scrubs/resilvers generally have errors (10+ daily), and if your pool survives "too long," it will start kicking disks out of the array, and will not re-initialize them on a reboot.
https://www.seagate.com/internal-hard-drives/cmr-smr-list/
Just because Seagate says they want to upsell me for dubious market segmentation reasons doesn't mean I need to pay attention to them. I know what I'm doing, I know desktop drives work just fine for my use case.
Before SMR. Now with SMR, they've found the perfect way of getting rid of everyone who was buying desktop drives for NAS use they were perfectly suitable for, and force them into the NAS upsell. 4 years later and it still costs more to buy a NAS drive today than it cost to buy an equally fit por purpose non SMR desktop drive back then.
Rather than ideal, a rebuild is actually the worst case scenario as the RAID may detect the drive as unresponsive as it scrambles to catch up and thus gets dropped, exacerbating the problem for the other drives in the array. This is HL2 Resonance Cascade bad.
- not all SMR drives seem to cause problems
- rebuilding on a non-fresh/cleard hard drive can have negative performance implications due to how SMR works
So my guess is that it's a problem with the firmware which can potentially be fixed. After which they could be used in a NAS, just aren't necessary as far as you might think. So I guess it's probably not looking too good for the lawsuites chances to cause much yield/discount.
> That 9 day and almost 14-hour rebuild means that using the WD Red 4TB SMR drive inadvertently in an array would lead to your data being vulnerable for around 9 days longer than the WD Red 4TB CMR drive or Seagate IronWolf.
The other 4TB drives they compared against completed in around 15 hours, meaning the SMR drive could potentially take 14x longer to rebuild a RAID.
Their initial customer-facing reaction was even worse if I recall: One customer escalated his issue and asked WD to identify which drives used SMR, WD said something along the lines of We don't share that information with customers.
(If anyone has a pointer to that story I'd appreciate it. I can't find it at the moment).
EDIT:
https://blocksandfiles.com/2020/04/14/wd-red-nas-drives-shin... cites a comment on a synology board:
""" Also: “Well the higher team contacted me back and informed me that the information I requested about whether or not the WD60EFAX was a SMR or PMR would not be provided to me. They said that information is not disclosed to consumers. LOL. WOW." """
There was a lot of press coverage of this a few weeks ago when the story first broke, including here on HN. It looks like the lawsuit reported here is the first formal legal action in response.
There are quite a few stories of people being unable to use these HDDs for archival, for this reason.
Do not buy SMR if you have any kind of write-heavy workload, or if you rely on consistent performance (e.g. any kind of streaming / latency sensitive usage).
When WD stealthily swapped to using device managed SMR technology in the series, they hoped that typical users wouldn't notice the change in behavior of the drives, as they assumed a small NAS would have enough downtime for the drive to manage the data being behind the scenes rearranged towards the slow shingled sections.
This assumption is not always true, but specifically it doesn't hold at all when an array has to be rebuild e.g. because of a drive failure in a raid array, as that requires writing data continuously and for a long time.
WD was caught not just using a technology that made their drives unsuited for the advertised purpose, but initially even denying that they had done so.
/s
https://nascompares.com/answer/list-of-wd-cmr-and-smr-hard-d...
Know anything about EFBX? SMR I assume? I literally have one arriving today to replace a disk in my single disk redundancy JBOD
The whole disk buying process was irksome because I up until WD shut them down always depended on HGST.
Does anybody know if that's going to be a problem? Or am I fine as long as the next drive I add is CMR?
Otherwise, you will probably be interested in replacing them sooner rather than later.
Under that assumption, my strategy is to isolate these drives in their own 2-drive RAID volume until they fail.
> WD Red drives are specifically designed to handle the rigors of 24×7 FreeNAS workloads, offering excellent data protection and the highest level of performance possible.
They probably should adapt their wording.
[0] https://www.freenas.org/freenas-mini/
It typically took a really, really, really bad debacle to get that level of attention. Back when I was selling hardware, I can think of a few incidents where certain drives were just known for data loss.
Early WD IDE 20GB drives - absurdly high 90d failure rate. Shop stopped carrying WD for over a year after these
Quantum Bigfoots - At best slow, and not -terrible- for reliability but I remember them having a really nasty failure mode.
IBM Deathstars - Everyone knows about these.
Fujitsu drives in the 8-30gb era - I think these actually wound up having a lawsuit over them
Heck, back then, on top of having to worry about all of that, Creative lab sound cards could cause data corruption in some motherboards!
And yet, only IBM suffered severe damage. People kept buying bigfoots. People kept buying Early model WD drives that bit them. People kept buying creative labs cards that had bad bus mastering....
$$$
I am curious as to when these good ol days are for you.
WD, obviously.
They won't like being made to pay, but that's rather the point of fines.
For example, I imagine your typical executive at a big company looks at the list of salaries of their employees over 45 and thinks to themselves, hey, why don't we just fire these people and replace them with young people?
Then, they think about the aforementioned lawyers and subsequent class action lawsuits, and then decide not to. Sometimes.
After this upcoming class action, you can bet the next WD exec is going to think twice before screwing over their customers.
But here is the think: From the iXsystems blog post you can learn that not all SMT drivers so lead to errors on rebuild. Just some of the FW versions do. Other drivers might still lead to big slow downs but it's not clear if all SMT drivers slow down to a unacceptable degree. Weather it not a newly added drive is fresh/was cleaned seems to also affect this. Lastly while the drives are marketed as NAS usable they have a list of supported NAS systems. And don't give any guarantee for performance on other systems.
Given all this my guess is that after fixing faulty firmware version they will at most provide some form of we update the firmware for you service if you can show that you are using a supported NAS system. ...
And then if any consumer protection agency or similar acts in the EU that program might also be available there.
Through that just my sold guess. Thinks can be very different, e.g. not just a firmware bug.
https://nascompares.com/answer/list-of-wd-cmr-and-smr-hard-d...
I'm using four of the seagate ironwolf drives now and they have been fine so far. Equivalent price point and capacity, slightly quieter than the WD. Just make sure to check whether you're getting SMR or not - the ironwolf NAS drive line is not SMR according to Seagate's docs that I read but definitely verify for yoruself.
https://www.extremetech.com/extreme/184253-ssd-shadiness-kin...
Earlier story: https://news.ycombinator.com/item?id=22875094
Unless you can show that your hard drive is causing problems on something they claim compatibility with, it doesn't seem like they'd be liable. One of those times where they're clearly in the wrong, but not legally at fault.
[1] https://www.westerndigital.com/products/internal-drives/wd-r...
If some Red drives (most probably the SMR ones) require many day rebuild times or even fail to rebuild in common commercial consumer NAS solutions, they are obviously not viable for RAID environments and then WD is guilty of false advertising.
This long time WD customer is no longer a customer.