Upvoted, but I'd like to suggest a title change, since this is just the Q3 report. The next report will be covering all of 2017.
>Our next drive stats post will be in January, when we’ll review the data for Q4 and all of 2017, and we’ll update our lifetime stats for all of the drives we have ever used. In addition, we’ll get our first real look at the 12 TB drives.
Do you have any info on what temperature your drives are running on normally?
Love to see some performance stat to see if read /write IOP, MB/second, etc change over time on per model base.
Other than complete failure, do you collect check the any other early warning failure data such as ECC change over the life time of the drive? --- Would be a fun big data exercise for an intern to work on.
Yev, would you ever consider opening the software and hardware design of the vault?
It seems that you could lead this and give the industry a standard way, with vendors producing the fitting hardware, and allowing others to maintain their own vaults. Maybe this is a business advantage you don't want to lose right now. Then, I'm sorry to have asked :). If not then I think you might benefit by lower prices if the hardware is readily available and not a Backblaze custom production run.
There are many places who cannot for technical or won't for political reasons offload their storage to S3 and friends and need it all locally most of the time which would use a vault or two.
As for getting the servers, there's an entire company that popped up from one of our early storage pod producers, Protocase, you can buy servers (very reasonably priced btw) based on our designs here -> http://www.45drives.com/.
I have a somewhat off-topic question: are the new 10/12 TB HDD filled with helium (He)? Last time I've checked on WesternDigital and Seagate sites I couldn't find any mention of it.
Three feedback points for B2. In case anyone from BB is reading:
- (mentioned this before) invoices need improvement. No accountant will accept this as of now. Please take a look at e.g. Digitalocean for how to do this right. I mentioned this at least 5 times to support...but nobody seems to care.
- upload speed is really bad from Europe (might be purely due to distance). Tried uploading one TB (@1.25MB/s), did around 25GB per day. Switched to a Hetzner Storage box, doing consistently 75GB per day right now.
- deleting your bucket (with data in it) requires to install a command-line tool which is really inconvenient. Took me 30 minutes+ and still threw some weird java.exception afterwards when clicking on „delete bucket“. Why not put a simple „delete“ button in the web ui?
Deleting a bucket in an object store is surprisingly non-trivial. See Amazon’s UI - it works fine for small object counts but is very buggy for larger buckets.
> - upload speed is really bad from Europe (might be purely due to distance). Tried uploading one TB (@1.25MB/s), did around 25GB per day
That's really useful to know. I am looking to switch from Amazon Cloud Drive for my cloud backups (now their pricing scheme has changed) and one criteria was much faster uploads. Sounds like B2 won't be it then, shame.
You should see the invoices from LastPass for "Enterprise".. it's hilarious. By default they dont send anything, the invoice link from their enterprise dashboard has been broken for months (support know) and they give you some other weird login to workaround and it loads a poor html page with tables, and a button to switch what kind of tax you want the Invoice to show (VAT, GST, etc)
It's so bad.
Side note: One thing that terrifies me about B2 is the ability to make a bucket public (when it's private and intended to be). Would be nice to have some kind of lock on it to say "never private" or at least make it harder to change that opening the same settings page as other stuff.
I love these reports! I'm always in the market for new storage solutions and find these statistics very useful in determining which brands and drive sizes provide the best reliability.
One statistic I'd really like to see included is the disk usage of each drive as a percentage of their total capacity. Would there be any correlation between that and the failure rate?
Ahh yeah I was looking through the SMART stats but wasn't sure which item denoted this, is it perhaps: "Minimum Spares Remaining - The Minimum Spares Remaining attribute indicates the number of remaining spare blocks as a percentage of the total number of spare blocks available."?
I believe smart would not be able to account for the use size correctly. Since that system lives in the drive itself, it can't really tell the difference between "this block has been initialised and is unused" and "this block is storing some data".
Backblaze, from what I gather, mostly just leave dead drives in place for a while.
They have to maintain more racks to do that (how much does it cost them to have their drive arrays running st 90%?) but that drops the amortized cost of replacing the drives.
But for me it might be cheaper to have one big drive that fails twice as often, because the odds of a drive being dead on any given Tuesday and the costs of addressing that might be low enough to offset the price differential.
I see Seagate is continuing their track record of Being Shit, as usual. Yet again they're the only brand where a couple models are exceeding the normal failure rate by a factor of 50-100x.
They've been shitty for 10 years now. I had a bunch of bad experiences with their 7200.11 drives too. I swore off Seagate forever after the 3 TBs started shitting the bed and I literally haven't had another drive die prematurely since then. It's not worth the risk, I'll pay an extra 10% for Toshiba or WD thanks.
Am I misreading the data (Cumulative 2013->2017) or something? The Seagate 4TB are horrific, yes, but they look to have completely solved their issues above 4TB.
e.g.
6TB - BB have 4 times as many Seagate drives (and nearly 4x logged time) as WDC, but more WDC drives have failed. (4.6%WDC versus 1.2%)
8TB - Annualised failure is lower than HGST (1.1% & 1.2% versus 1.7%). Nearly 25,000 Seagate drives and 150 odd failures.
10TB - 0 failures over 13,720 drive days so far
12TB - 0 failures over 500 drive days so far.
Maybe I'm misreading something, but it looks from their data that the major Seagate issues have been put behind them now?
> Am I misreading the data (Cumulative 2013->2017) or something?
Yes. Per the description, that table is only drives that remain in service as of 2017. There have been "troublesome" Seagate models that failed at such high rates that Backblaze finally just pre-emptively pulled them, and those drives are not in the "cumulative" table they show in the OP.
> Maybe I'm misreading something, but it looks from their data that the major Seagate issues have been put behind them now?
That's what people said after the 7200.11 series. And after the 1.5 TBs shit the bed. And after the 3 TBs shit the bed. They're good for a while, then Seagate releases yet another model that shits the bed.
You can actually see that in the link above.
> While this particular 3TB model had a painfully high rate of failure, subsequent Seagate models such as their 4TB drive, model: ST4000DM000, are performing well with an annualized 2014 failure rate of just 2.6% as of December 31, 2014. These drives come with 3-year warranties and show no signs of hitting the wall.
2 years later they've released two new 4 TB models that fail at 50-100x the norm again.
To add to the other comment, it's not just price but also volume. It's easy enough for me as an individual to buy 10 Toshiba HDDs... but Backblaze would like to buy them in quantities of 10k or more (if at all possible - obviously during the Thailand Drive Crisis they had to deal in small quantities).
It's a lot easier to get Seagates in large volumes than some of the other brands - Toshiba specifically is problematic, iirc HGST maybe also.
Also, I didn't say Seagates were always shit. But some of their models are, and they are the only brand with these issues, and it's been a repeated problem for them over the last 10 years.
Those 10gb Seagates have to be pretty new still. I'm curious about them over time. It would be nice if that's one like their old pre 2006 hdds that hardly ever failed.
Anecdata, but my work's just passed the 1 year anniversary of running 12 of those and they're all still in perfect health. I'm very impressed by Seagate's enterprise drives.
48 comments
[ 2.8 ms ] story [ 34.3 ms ] thread>Our next drive stats post will be in January, when we’ll review the data for Q4 and all of 2017, and we’ll update our lifetime stats for all of the drives we have ever used. In addition, we’ll get our first real look at the 12 TB drives.
Love to see some performance stat to see if read /write IOP, MB/second, etc change over time on per model base.
Other than complete failure, do you collect check the any other early warning failure data such as ECC change over the life time of the drive? --- Would be a fun big data exercise for an intern to work on.
Also what's the usage patterns for your HDDs? read/write/spin down/sleep time in term of life time of the HDD.
I assume mostly write, some read. But also very low percentage is active?
Yes, we collect temperature data. If you download the dataset, you'll find a daily sample of the SMART data (including temperature) for every drive.
It seems that you could lead this and give the industry a standard way, with vendors producing the fitting hardware, and allowing others to maintain their own vaults. Maybe this is a business advantage you don't want to lose right now. Then, I'm sorry to have asked :). If not then I think you might benefit by lower prices if the hardware is readily available and not a Backblaze custom production run.
There are many places who cannot for technical or won't for political reasons offload their storage to S3 and friends and need it all locally most of the time which would use a vault or two.
As for getting the servers, there's an entire company that popped up from one of our early storage pod producers, Protocase, you can buy servers (very reasonably priced btw) based on our designs here -> http://www.45drives.com/.
For now I'll stick with the plain offer. ^__^;
- (mentioned this before) invoices need improvement. No accountant will accept this as of now. Please take a look at e.g. Digitalocean for how to do this right. I mentioned this at least 5 times to support...but nobody seems to care.
- upload speed is really bad from Europe (might be purely due to distance). Tried uploading one TB (@1.25MB/s), did around 25GB per day. Switched to a Hetzner Storage box, doing consistently 75GB per day right now.
- deleting your bucket (with data in it) requires to install a command-line tool which is really inconvenient. Took me 30 minutes+ and still threw some weird java.exception afterwards when clicking on „delete bucket“. Why not put a simple „delete“ button in the web ui?
Btw, what is missing in the invoices?
For us to be able to move from S3 we would also need to be able to create multiple API keys where we can set what buckets they have access to.
That's really useful to know. I am looking to switch from Amazon Cloud Drive for my cloud backups (now their pricing scheme has changed) and one criteria was much faster uploads. Sounds like B2 won't be it then, shame.
It's so bad.
Side note: One thing that terrifies me about B2 is the ability to make a bucket public (when it's private and intended to be). Would be nice to have some kind of lock on it to say "never private" or at least make it harder to change that opening the same settings page as other stuff.
Per-bucket keys would be nice too.
One statistic I'd really like to see included is the disk usage of each drive as a percentage of their total capacity. Would there be any correlation between that and the failure rate?
(Source: https://en.wikipedia.org/wiki/S.M.A.R.T.)
edit: No that can't be it. Maybe this data isn't available, hmm...
When 12TB HDD fails, it's equal to four 3TB HDD's failing at once.
They have to maintain more racks to do that (how much does it cost them to have their drive arrays running st 90%?) but that drops the amortized cost of replacing the drives.
But for me it might be cheaper to have one big drive that fails twice as often, because the odds of a drive being dead on any given Tuesday and the costs of addressing that might be low enough to offset the price differential.
I even had RMA'd ones die within a week.
e.g.
6TB - BB have 4 times as many Seagate drives (and nearly 4x logged time) as WDC, but more WDC drives have failed. (4.6%WDC versus 1.2%)
8TB - Annualised failure is lower than HGST (1.1% & 1.2% versus 1.7%). Nearly 25,000 Seagate drives and 150 odd failures.
10TB - 0 failures over 13,720 drive days so far
12TB - 0 failures over 500 drive days so far.
Maybe I'm misreading something, but it looks from their data that the major Seagate issues have been put behind them now?
Yes. Per the description, that table is only drives that remain in service as of 2017. There have been "troublesome" Seagate models that failed at such high rates that Backblaze finally just pre-emptively pulled them, and those drives are not in the "cumulative" table they show in the OP.
https://www.backblaze.com/blog/3tb-hard-drive-failure/
> Maybe I'm misreading something, but it looks from their data that the major Seagate issues have been put behind them now?
That's what people said after the 7200.11 series. And after the 1.5 TBs shit the bed. And after the 3 TBs shit the bed. They're good for a while, then Seagate releases yet another model that shits the bed.
You can actually see that in the link above.
> While this particular 3TB model had a painfully high rate of failure, subsequent Seagate models such as their 4TB drive, model: ST4000DM000, are performing well with an annualized 2014 failure rate of just 2.6% as of December 31, 2014. These drives come with 3-year warranties and show no signs of hitting the wall.
2 years later they've released two new 4 TB models that fail at 50-100x the norm again.
While they may have a greater failure rate percentage, the price often offsets that for Backblaze's needs. Your needs may vary.
It's a lot easier to get Seagates in large volumes than some of the other brands - Toshiba specifically is problematic, iirc HGST maybe also.
Also, I didn't say Seagates were always shit. But some of their models are, and they are the only brand with these issues, and it's been a repeated problem for them over the last 10 years.