Are these consumer drives being used in an industrial setting?
I'm surprised to see such high AFR from Seagate and WD. These brands have had decades of well-deserved loyalty and trust, and I'd never suspected some of their drives have such high failure rates.
Why are HGST's AFR numbers so much more preferable?
I wonder if that's due to over-built drives (a quality standard independent of other factors) or if it's related to the specific controllers and other hardware in the Backblaze environment (independent of each manufacturer's drive quality).
HGST was originally IBM's hard drive division. They probably have inherited much of the reliability secrets that the other manufacturers never bothered with for their consumer class drives (which Backblaze uses). Either that, or they learned a lot of stuff with the "DeathStar" model[0].
Yes, but nowadays HGST is no more. The company has been bought by WD, the 2.5" and smaller HDDs are now part of Western Digital. The 3.5" HDDs are now Toshiba due regulatory requirements.
The bit about the ST3000DM001 is fascinating - they've a longer article on it at https://www.backblaze.com/blog/3tb-hard-drive-failure - I have had four (out of the four I bought - two internal, two "shucked") of these die on me over the years, on the first occasion simultaneously in a raid-5 array that I'd just downloaded several TB of irreplaceable video to from SD cards. Now at least I know why.
SD cards are pretty easy to recover media from, if you haven't overwritten them.
Also, I'm a bit surprised by their topology. Reading the bit about the 3TB drives, it sounds like three drive failures in their raid5 array would result in lost customer data.
I hope they have a 2nd line of defense, e.g. tapes. One company I used to work for ran into sufficient failures (particularly of the living-dead variety, where a second drive fails during rebuild or after reboot) they made tapes the 3rd line of defense, and maintained two copies of all data at the cluster level, so that the total failure of an entire machine was acceptable.
"A Backblaze Vault is comprised of 20 Storage Pods, with the data evenly spread across all 20 pods. ... Drives in the same drive position in each of the 20 Storage Pods are grouped together into a storage unit we call a “tome”. ... Every file uploaded to a Vault is broken into pieces before being stored. Each of those pieces is called a “shard”. ... Each file is stored as 20 shards: 17 data shards and 3 parity shards. Because those shards are distributed across 20 storage pods in 20 cabinets, the Vault is resilient to the failure of a storage pod, or even a power loss to an entire cabinet."
From https://www.backblaze.com/blog/vault-cloud-storage-architect..., which is well worth taking a few minutes to read. Their approach to data resiliency is pretty neat and full of the kind of common-sense reasoning that warms the very bottoms of my ol' sysadmin's heart ("use simple parts" and "build the tools you need to run your infrastructure", among others).
I thought it was the same article I'd seen before since I've seen this URL appear on HN before at https://news.ycombinator.com/item?id=8923535, but apparently it's not. Only the small "May 21st, 2015" text in the article is any indication.
I think they really need to put a date in the URL and not just overwrite the previous version, so we can see how things change over time.
Edit: in fact they had a nice bargraph of the failure rates by drive model in their previous post, but it's gone now. Fortunately someone seems to have archived a copy of it here: http://thetrendythings.com/read/28144
What is the formula used for confidence interval ? "0.0%-106.9%" interval for low numbers is somewhat disturbing. What approximation cause this problem ?
The 106.9% isn't a problem of mathematics; it just means an average of more than one failure per drive per year, or alternatively an average lifespan of less than one year. It comes from the fact that they've only had those models in service for a very short time, and so they can only constrain the lifetime through the same reasoning used to estimate a MTBF of multiple years with less than a year of testing. With zero failures so far, it's obvious why the bottom of the confidence interval is at 0%.
They only have 50 Toshiba drives, which have only been used for a couple of months.
With 0% failures over the 50 drives, you've got a statistical nightmare. You really have _no_ idea how reliable or unreliable the drives are. So the "confidence interval" is huge.
The other drives have thousands and thousands of samples, and have been used for many years. They are confident in the measurement of the reliability.
Toshiba has taken over the former Hitachi factories (along with HGST). So I'd expect Toshiba drives to be highly reliable. I hope that Backblaze can purchase more Toshiba drives so that they can gather some better statistics.
For now, I'm confident in their 50 hard drives that haven't failed. That's a pretty good measurement, even if its not statistically reliable quite yet.
Please rename the HGST drives. There are only Seagate, Western Digital and Toshiba - HGST is no more.
Hitachi Global Storage Technologies (HGST) is now a wholly owned subsidiary of Western Digital! To address the requirements of regulatory agencies, in May 2012 WD divested to Toshiba assets that enabled Toshiba to manufacture and sell 3.5-inch hard drives for the desktop and consumer electronics markets. So all HGST 3.5" HDDs are now Toshiba, whereas the HGST 2.5" and smaller HDDs from Western Digital.
Yev from Backblaze here -> They are run as separate entities: "It was agreed that WD would operate with WD Technologies and HGST as wholly owned subsidiaries and they would compete in the marketplace with separate brands and product lines.[4][5][6]" So, they compete with each other and use different supply-chains, so they are quite different, even if the money eventually flows in to the same conglomerate.
If you read my explanation, and the wiki article you would see it is wrong.
"Some of the HGST drives listed were manufactured under their previous brand, Hitachi. We’ve been asked to use the HGST name and we have honored that request." As I mentioned even if backblaze honors a request of Western Digital, it doesn't make the claim true. I researched on this topic some months ago, after I read backblaze previous article from late 2014 and bought several Toshiba (former HGST) drives.
"Western Digital will be allowed to acquire Hitachi’s 2.5” and SSD businesses, but not the 3.5” business. Instead Western Digital will be selling that business to Toshiba – factories and all – along with granting licenses for the necessary patents, which would allow Toshiba to effectively continue in the 3.5” market from where Hitachi left off. This would firmly establish Seagate, Western Digital, and Toshiba as the 3 major players in the hard drive business across all product segments."
So former Hitachi and HGST 3.5" drives are now Toshiba drives.
"The 3.5" desktop and server Hitachi drives are all manufactured by Toshiba. WD wanted to buy Hitachi's drive division so there would be a WD/Seagate drive duopoly, European regulators said they had to sell Hitachi's 3.5" to someone else before they would approve the merger. Toshiba stepped up and got it. So yes it is somewhat confusing. You buy one of these drives and it will say "HGST a Western Digital company" on the box, when in reality it's all made by Toshiba, probably in an old Hitachi factory." -- a comment, from backblaze HN post in Sept 2014: https://news.ycombinator.com/item?id=8355860
I got the quote from the same wikipedia article ;-) Plus HGST has always been quite vocal about the companies being separate. We stopped calling them Hitachi drives as a result, since the new entity is called HGST.
This comes across as a pet peeve. If the drives are branded, marketed, and sold as HGST, then they are HGST drives. Even if Yev is wrong, it's possible that WD or Toshiba are maintaining HGST drives to different specifications or QA standards.
Either call them Hitachi or HGST or Toshiba. But as HGST still exists as a brand name of Western Digital and new HGST drives are either 2.5 or don't resemble the old ones mentioned in the statistics, it get's quite confusing - as the statistics is often used as a buyer guide. Adding a remark somewhere on his article would help to avoid the confusion.
26 comments
[ 2.9 ms ] story [ 66.4 ms ] threadI'm surprised to see such high AFR from Seagate and WD. These brands have had decades of well-deserved loyalty and trust, and I'd never suspected some of their drives have such high failure rates.
Why are HGST's AFR numbers so much more preferable?
Yes, Backblaze is built around consumer hardware and a custom RAID(ish) setup.
As far as why other drives have such high AFR, historically Seagate and WD have had much better QA on their server class drives.
https://en.wikipedia.org/wiki/Hitachi_Deskstar#IBM_Deskstar_...
http://en.wikipedia.org/wiki/HGST
https://en.wikipedia.org/wiki/File:Diagram_of_Hard_Disk_Driv...
Those things were indestructible. But now my cellphone has more storage than these golden drives.
https://www.backblaze.com/blog/enterprise-drive-reliability/
The TL;DR is that the consumer drives had a better ROI since they failed at similar rates and were cheaper.
Also, I'm a bit surprised by their topology. Reading the bit about the 3TB drives, it sounds like three drive failures in their raid5 array would result in lost customer data.
I hope they have a 2nd line of defense, e.g. tapes. One company I used to work for ran into sufficient failures (particularly of the living-dead variety, where a second drive fails during rebuild or after reboot) they made tapes the 3rd line of defense, and maintained two copies of all data at the cluster level, so that the total failure of an entire machine was acceptable.
From https://www.backblaze.com/blog/vault-cloud-storage-architect..., which is well worth taking a few minutes to read. Their approach to data resiliency is pretty neat and full of the kind of common-sense reasoning that warms the very bottoms of my ol' sysadmin's heart ("use simple parts" and "build the tools you need to run your infrastructure", among others).
I think they really need to put a date in the URL and not just overwrite the previous version, so we can see how things change over time.
Edit: in fact they had a nice bargraph of the failure rates by drive model in their previous post, but it's gone now. Fortunately someone seems to have archived a copy of it here: http://thetrendythings.com/read/28144
https://www.backblaze.com/blog/best-hard-drive-q4-2014/
https://www.backblaze.com/blog/hard-drive-reliability-update...
https://www.backblaze.com/blog/what-hard-drive-should-i-buy/
https://www.backblaze.com/blog/how-long-do-disk-drives-last/
You can also sift through the data yourself: https://www.backblaze.com/hard-drive-test-data.html
With 0% failures over the 50 drives, you've got a statistical nightmare. You really have _no_ idea how reliable or unreliable the drives are. So the "confidence interval" is huge.
The other drives have thousands and thousands of samples, and have been used for many years. They are confident in the measurement of the reliability.
Toshiba has taken over the former Hitachi factories (along with HGST). So I'd expect Toshiba drives to be highly reliable. I hope that Backblaze can purchase more Toshiba drives so that they can gather some better statistics.
For now, I'm confident in their 50 hard drives that haven't failed. That's a pretty good measurement, even if its not statistically reliable quite yet.
Hitachi Global Storage Technologies (HGST) is now a wholly owned subsidiary of Western Digital! To address the requirements of regulatory agencies, in May 2012 WD divested to Toshiba assets that enabled Toshiba to manufacture and sell 3.5-inch hard drives for the desktop and consumer electronics markets. So all HGST 3.5" HDDs are now Toshiba, whereas the HGST 2.5" and smaller HDDs from Western Digital.
See: http://en.wikipedia.org/wiki/HGST
If you read my explanation, and the wiki article you would see it is wrong.
"Some of the HGST drives listed were manufactured under their previous brand, Hitachi. We’ve been asked to use the HGST name and we have honored that request." As I mentioned even if backblaze honors a request of Western Digital, it doesn't make the claim true. I researched on this topic some months ago, after I read backblaze previous article from late 2014 and bought several Toshiba (former HGST) drives.
http://www.anandtech.com/show/5635/western-digital-to-sell-h... mentions it too:
"Western Digital will be allowed to acquire Hitachi’s 2.5” and SSD businesses, but not the 3.5” business. Instead Western Digital will be selling that business to Toshiba – factories and all – along with granting licenses for the necessary patents, which would allow Toshiba to effectively continue in the 3.5” market from where Hitachi left off. This would firmly establish Seagate, Western Digital, and Toshiba as the 3 major players in the hard drive business across all product segments."
Confirmation is the press release by WD: http://www.wdc.com/en/company/pressroom/releases/?release=f8...
So former Hitachi and HGST 3.5" drives are now Toshiba drives.
"The 3.5" desktop and server Hitachi drives are all manufactured by Toshiba. WD wanted to buy Hitachi's drive division so there would be a WD/Seagate drive duopoly, European regulators said they had to sell Hitachi's 3.5" to someone else before they would approve the merger. Toshiba stepped up and got it. So yes it is somewhat confusing. You buy one of these drives and it will say "HGST a Western Digital company" on the box, when in reality it's all made by Toshiba, probably in an old Hitachi factory." -- a comment, from backblaze HN post in Sept 2014: https://news.ycombinator.com/item?id=8355860
And I don't think Yev's wrong, I'm pretty sure I remember reading/hearing the same, and http://www.zdnet.com/article/western-digitals-hgst-cuts-500-... also suggests that WD is maintaining HGST as a separate entity.
Calling HGST drives Western Digital or Toshiba would be more confusing, not less.