Disclaimer: I work at Backblaze but more on the backup client side that runs on laptops, not in the datacenter storage side.
> Is there a failure list for write heavy drives?
To be clear about what these drive failure stats are and what they are not: Backblaze runs a data storage service with about 214,000 hard drives in it right now. We don't run any specific tests or induce issues on purpose FOR the drive failure stats, we just report what occurred in our datacenter.
Sometimes readers think we're carefully running a "study", but it's more just what we have experienced as honestly as we can offer it up. If the reads and writes and seeks our drives experience matches your particular application, great! Or maybe it is just interesting to read.
Now, we do save (and publish) all the raw data, and some other awesome people out there have done various analysis on it, which always makes us happy also. You can find the raw data here: https://www.backblaze.com/b2/hard-drive-test-data.html At this point it goes back almost a full decade.
Short summary is there are a few SMART stats that seem to predict failure way more than others, which is probably obvious. But we aren't PhDs in statistics and it isn't our area of focus, so....
I can't comment on their findings, but it's DEFINITELY an interesting thing to study now that we have almost 10 years of these drive stats across a pretty big drive farm.
Is there something you published regarding regular consumer drives? I ask this because I cannot find / afford high capacity enterprise drives in India, and I am forced to make multiple backups on cheap external hard drives.
I'm building https://github.com/stonecharioteer/renfield for this purpose. Before I get around to it, I'm trying out git-annex, but I must say I don't like the git approach to files.
I wonder if they've done any proper Cox proportional hazards lifetime analysis across the different models? The stats for this sort of problem are well developed. Of course, the hazard is not going to be proportional, I suspect, bit I'd love to see that come out of the regression...
The whole reason why the market still demands HGST. There is a whole segment / sector / section of industry that refuse to call it anything other than HGST. And they are willing to pay for it.
These have been popular for some time now and I've been wondering what are the chances of manufacturers sending better batches to Backblaze. I imagine you run some batch and some percentage of disks don't pass tests, you may adjust something or maybe it's the last one before routine machine maintenance, etc. No idea if there really even is such thing as batches which manufacturers know have higher failure rates. But it would be interesting.
I had osmosed that the bathtub curve isn't real, but the new time-trail quadrant graph certainly looks like evidence for at least the infant-mortality section.
The thing people care about is the AFR over time: like actuarial life tables. These updates always make it a bit hard to figure that one out: the tables show "average life" and AFR; it makes no sense to compare drives of different ages.
The "Age vs AFR" plot is getting there but it should be flipped and is hard to read; error bars would also help.
Also percentiles would be useful: people want to minimize the probability of failure.
Based on these reports and pricing, I've currently filled my NAS with only Toshiba drives. 1 4TB, 1 5TB and 2 14TB drives and all of them have been great. These enterprise drives are quite noisy, but my oldest already runs for over 4 years without any reallocated sectors or other issues. Apart from that, Toshiba provides free 5 year warranty on these drives.
And the best thing? These bad boys are only 250 euro's for 14TB or 300 euro's for 18TB of raw storage capacity.
Just to add: Consumer drives are not necessarily sub-par quality-wise just because they're consumer drives. Sometimes what it comes down to is firmware; WD NAS drives used to be almost identical to conumer drives except their firmware would report errors back to the SATA controller directly instead of continually trying to read the block.
Continually reading the block seems doomed but it works a surprisingly large number of times, and when it's successful the block is often put somewhere else on the drive transparently (and the old block is marked as "bad"). This drastically improves the lifetime of the drives since they're slowly wearing down vs "going bad" because of a single block being unrecoverable.
This is not the only difference, but it's one thing to consider.
It's similar to how the linux kernel can be compiled with low latency in mind; which causes lower throughput but makes the desktop experience much nicer (ime). It's just trade-offs. Nothing is just "objectively better" it's a different trade off.
I wonder if they consider doing something like putting data more likely to be accessed on the more reliable drives so that that ones that wear out easier are used less.
21 comments
[ 4.0 ms ] story [ 31.7 ms ] thread> Is there a failure list for write heavy drives?
To be clear about what these drive failure stats are and what they are not: Backblaze runs a data storage service with about 214,000 hard drives in it right now. We don't run any specific tests or induce issues on purpose FOR the drive failure stats, we just report what occurred in our datacenter.
Sometimes readers think we're carefully running a "study", but it's more just what we have experienced as honestly as we can offer it up. If the reads and writes and seeks our drives experience matches your particular application, great! Or maybe it is just interesting to read.
Now, we do save (and publish) all the raw data, and some other awesome people out there have done various analysis on it, which always makes us happy also. You can find the raw data here: https://www.backblaze.com/b2/hard-drive-test-data.html At this point it goes back almost a full decade.
Someone with some time could correlate that to failure rate. My hypothesis is of course it's correlated--but by how much?
We looked into it a little, some notes written up here: https://www.backblaze.com/blog/what-smart-stats-indicate-har...
Short summary is there are a few SMART stats that seem to predict failure way more than others, which is probably obvious. But we aren't PhDs in statistics and it isn't our area of focus, so....
This guy wrote a paper based on the Backblaze SMART data: https://etd.ohiolink.edu/apexprod/rws_etd/send_file/send?acc...
These 5 guys wrote another paper based on the Backblaze SMART data to train up a Bayesian network to predict failures: https://ieeexplore.ieee.org/document/8489097
This is another article of predicting hard drive failures using the Backblaze SMART data: https://karthikna.github.io/Prediction-of-Hard-Drive-Failure...
I can't comment on their findings, but it's DEFINITELY an interesting thing to study now that we have almost 10 years of these drive stats across a pretty big drive farm.
I'm building https://github.com/stonecharioteer/renfield for this purpose. Before I get around to it, I'm trying out git-annex, but I must say I don't like the git approach to files.
I wonder if they've done any proper Cox proportional hazards lifetime analysis across the different models? The stats for this sort of problem are well developed. Of course, the hazard is not going to be proportional, I suspect, bit I'd love to see that come out of the regression...
Backblaze shouldn't complain though ;)
Some interesting answers to this and other questions.
"Due to the quantities we need and the visibility of the posts, drive manufacturers are motivated to give us their best possible product." ....
The "Age vs AFR" plot is getting there but it should be flipped and is hard to read; error bars would also help.
Also percentiles would be useful: people want to minimize the probability of failure.
And the best thing? These bad boys are only 250 euro's for 14TB or 300 euro's for 18TB of raw storage capacity.
Generally I only buy enterprise drives
Continually reading the block seems doomed but it works a surprisingly large number of times, and when it's successful the block is often put somewhere else on the drive transparently (and the old block is marked as "bad"). This drastically improves the lifetime of the drives since they're slowly wearing down vs "going bad" because of a single block being unrecoverable.
This is not the only difference, but it's one thing to consider.
It's similar to how the linux kernel can be compiled with low latency in mind; which causes lower throughput but makes the desktop experience much nicer (ime). It's just trade-offs. Nothing is just "objectively better" it's a different trade off.
According to the link I shared, they aren't lower quality, they're higher quality than enterprise drives.
But that's an interesting note about the drive firmware!