The problem with the statistics provided is that they don't account for how long the drives have been running. Eventually all hard drives will fail but what matters is how long it will take until they do. From the charts I can't understand which is the most reliable over time. Or am I just misinterpreting the data?
It also doesn't seem to take into consideration the use-case for the drives. Are they in write-heavy or read-heavy roles? Is it balanced between? Are there regular, punctuated periods of heavy load? Is every drive in the same style of SAN/RAID across the enterprise? (doubtful on that last one)
Maybe it doesn't matter due to the sheer number of drives being reported on here?
My understanding is that that Backblaze handles data integrity at a higher level and just treats the disks as JBOD, so I would expect the "SAN/RAID" situation across all their disks to be roughly the same. Their storage pods have always had little more than SATA controllers and port multipliers and in later iterations, direct-wired drives w/ no port multipliers.
Loosely speaking files are encoded in to to chunks and scattered across multiple disks (actual stored data is more than the size of the original object, but less than full duplication, triplication or other redundant ways to store it).
That blog suggests Backblaze use a Reed-Solomon ratio of 17:20 ratio, i.e. every object is encoded in to 20 chunks, for which you only need to retrieve 17 different chunks to be able to decode and get the full object.
If they're putting only one chunk per drive, that gives any object stored resilience from up to 3 disks failing, at a storage cost of only 1.17x original object size (20/17 ~= 1.176). They'll likely have repair processes to proactively handle any loss of shards from a disk failure, too.
Reed-Solomon is giving them something akin to triplication, while using significantly less disk space than duplicating.
Nothing comes for free, though. The additional cost here is the overhead of encoding and decoding the object, and on top of that the co-ordination etc required from requesting chunks from multiple locations. That adds some unavoidable latency, but that article suggest Backblaze find the JVM to be extremely fast (as fast as a pure C implementation of Reed-Solomon).
Yev from Backblaze here -> the "drive days" should give you an idea of how long those were running. In the raw data you can actually go through and look for the failures and then see the drive's "time to failure"!
Kinda interesting.. if you multiply Drive Size by Drive Count and sum the result, you get 625 PB. Wonder what the expected time until they have a full Exabyte of raw storage is
To put this in some scale, back in 2013 Randall Munroe estimated[[1] that Google had 10 exabytes on disk. That has to be significantly greater these days.
Interesting thoughts on SSDs. I was under the impression that it wasn't so much the speed improvements they provide as the _power_savings_ that made SSDs useful in a Data Center environment. It was a single piece of anecdotal evidence, but the payoff due to power savings was roughly 8 months.
It depends on the use case. IO-heavy workloads (like databases) benefit from the better performance (relative to traditional hard disks), though ramdisks would give even more performance gains (so long as your data - or the most actively used subset thereof - fits in RAM).
Another factor is the form factor (pun intended; I'll be here all week). SSDs can get a lot smaller than traditional hard drives, which opens up the floodgates for tiny servers (blades, single-board machines, etc.) as well as for increasingly-dense solid-state SANs.
But yes, power savings are yet another huge benefit - and one which is still such even as a side effect of going with SSDs for a different reason (like physical space or needing high-speed disk access).
Does Backblaze support Linux yet? I have store almost all of my data on a NAS, and it would be great to have that automatically backed up online, but the last time I looked into it they only supported Mac/Windows. I assume the reason for this was to keep out users with large NAS capacity and allow them to keep advertising unlimited storage. I would be ok with having Linux computers capped, I'm not storing more than about 2TB of data.
They don't have their own client offering, but super-easy to roll your own with their B2 cloud service. There's support for backblaze targets in duplicity, hashbackup, restic and rclone -- my org uses it as a special remote for git-annex, and also use it to back up our synology units (see: https://www.backblaze.com/b2/integrations.html).
2TB would cost you about $10/month for the storage ($0.005/GB), plus $0.01/GB for downloads.
As much as I love B2, their SDK support is pretty terrible. There are a handful of half-baked and abandoned libraries. We've been using an internal python library which we're starting to open source and plan on maintaining. We've also forked wal-e to support this library and B2 endpoint (currently have basic upload functionality working). It would not take much effort from Backblaze to throw some resources behind these efforts, or hell, to host their own S3 proxy (I'll never understand the design choices behind their API).
> Backblaze's APIs are not S3 compatible. .... I'll never understand the design choices behind their API
It is all clear if you understand our APIs are cheaper to implement than Amazon S3's APIs. I freely admit the APIs are LITTLE BIT more complicated than S3 (which is not a good thing). Let me start from the beginning and explain why:
We originally never planned to open the APIs up for general use. We created the Backblaze Personal Backup client and we owned both the client and the server, so we were originally willing to do a tiny bit more programming to get rid of some expensive equipment and choke points in the datacenter.
In the S3's APIs, you simply "upload" and you are done. This means there is a very high availability choke point on upload. With Backblaze client (and later "B2" APIs), there are no choke points. The Backblaze client contacts (HTTPS) a central dispatch server and asks the dispatch server for a pod (computer) that contains some spare disk space. Then something very very important occurs that does not occur with Amazon S3 -> the client entirely stops talking with the central dispatch server. And then the client contacts the pod DIRECTLY to perform the uploads.
Now the "contract" with the storage pods is that if that one storage pod crashes, or fills up, or decides not to talk with the client anymore, the client is absolutely responsible to go back to the "dispatch server" and request a new pod somewhere else in the Backblaze datacenter to upload data into.
By requiring these one or two extra steps, Backblaze does not have to buy any expensive load balancing hardware, and we only need 10 Gbit network cards, there are no central choke points requiring multi-Terabit networking speeds like Amazon. Instead we have several thousand individual 10 Gbit network cards which cost us almost nothing because they come built into the Intel motherboards nowadays. The switches cost a few bucks per port.
I assume that Amazon S3 accepts the data, then has to "move it" to the final destination with a network copy. In Backblaze, the data literally lands in the final destination coming straight from the client. Fewer copies means lower cost and higher performance.
Backblaze purchases no load balancers, except for some "tricks" to make the "dispatch servers" highly redundant. But the dispatch servers have only TINY trickles of info coming and going, and the clients only have to contact the dispatch server ONCE every few days (then upload endlessly to the final destination pods), so even the dispatch servers can be "only" 10 Gbit.
> Backblaze is incompatible with Amazon S3
TL;DR - by requiring clients to do a small amount of extra programming and making programmers understand the "contract" (which is the client must retry some things and handle a few more errors), Backblaze cuts out a lot of cost out of our datacenter while INCREASING scalability over S3. We pass the money savings along to our customers. We throw in the increase scalability over S3 for free. :-)
This is kind of depressing. Per the stats here if I'm reading it correctly, it appears WD Red disks have pretty high failure rates relative to competitors and that is what I've been buying to put in my NAS.
Why is there such a general WD > Seagate slant if the stats don't seem to back it up?
31 comments
[ 2.8 ms ] story [ 82.7 ms ] threadMaybe it doesn't matter due to the sheer number of drives being reported on here?
Loosely speaking files are encoded in to to chunks and scattered across multiple disks (actual stored data is more than the size of the original object, but less than full duplication, triplication or other redundant ways to store it).
That blog suggests Backblaze use a Reed-Solomon ratio of 17:20 ratio, i.e. every object is encoded in to 20 chunks, for which you only need to retrieve 17 different chunks to be able to decode and get the full object.
If they're putting only one chunk per drive, that gives any object stored resilience from up to 3 disks failing, at a storage cost of only 1.17x original object size (20/17 ~= 1.176). They'll likely have repair processes to proactively handle any loss of shards from a disk failure, too.
Reed-Solomon is giving them something akin to triplication, while using significantly less disk space than duplicating.
Nothing comes for free, though. The additional cost here is the overhead of encoding and decoding the object, and on top of that the co-ordination etc required from requesting chunks from multiple locations. That adds some unavoidable latency, but that article suggest Backblaze find the JVM to be extremely fast (as fast as a pure C implementation of Reed-Solomon).
[1] https://what-if.xkcd.com/63/
Another factor is the form factor (pun intended; I'll be here all week). SSDs can get a lot smaller than traditional hard drives, which opens up the floodgates for tiny servers (blades, single-board machines, etc.) as well as for increasingly-dense solid-state SANs.
But yes, power savings are yet another huge benefit - and one which is still such even as a side effect of going with SSDs for a different reason (like physical space or needing high-speed disk access).
2TB would cost you about $10/month for the storage ($0.005/GB), plus $0.01/GB for downloads.
https://www.backblaze.com/b2/docs/quick_command_line.html
> Backblaze's APIs are not S3 compatible. .... I'll never understand the design choices behind their API
It is all clear if you understand our APIs are cheaper to implement than Amazon S3's APIs. I freely admit the APIs are LITTLE BIT more complicated than S3 (which is not a good thing). Let me start from the beginning and explain why:
We originally never planned to open the APIs up for general use. We created the Backblaze Personal Backup client and we owned both the client and the server, so we were originally willing to do a tiny bit more programming to get rid of some expensive equipment and choke points in the datacenter.
In the S3's APIs, you simply "upload" and you are done. This means there is a very high availability choke point on upload. With Backblaze client (and later "B2" APIs), there are no choke points. The Backblaze client contacts (HTTPS) a central dispatch server and asks the dispatch server for a pod (computer) that contains some spare disk space. Then something very very important occurs that does not occur with Amazon S3 -> the client entirely stops talking with the central dispatch server. And then the client contacts the pod DIRECTLY to perform the uploads.
Now the "contract" with the storage pods is that if that one storage pod crashes, or fills up, or decides not to talk with the client anymore, the client is absolutely responsible to go back to the "dispatch server" and request a new pod somewhere else in the Backblaze datacenter to upload data into.
By requiring these one or two extra steps, Backblaze does not have to buy any expensive load balancing hardware, and we only need 10 Gbit network cards, there are no central choke points requiring multi-Terabit networking speeds like Amazon. Instead we have several thousand individual 10 Gbit network cards which cost us almost nothing because they come built into the Intel motherboards nowadays. The switches cost a few bucks per port.
I assume that Amazon S3 accepts the data, then has to "move it" to the final destination with a network copy. In Backblaze, the data literally lands in the final destination coming straight from the client. Fewer copies means lower cost and higher performance.
Backblaze purchases no load balancers, except for some "tricks" to make the "dispatch servers" highly redundant. But the dispatch servers have only TINY trickles of info coming and going, and the clients only have to contact the dispatch server ONCE every few days (then upload endlessly to the final destination pods), so even the dispatch servers can be "only" 10 Gbit.
> Backblaze is incompatible with Amazon S3
TL;DR - by requiring clients to do a small amount of extra programming and making programmers understand the "contract" (which is the client must retry some things and handle a few more errors), Backblaze cuts out a lot of cost out of our datacenter while INCREASING scalability over S3. We pass the money savings along to our customers. We throw in the increase scalability over S3 for free. :-)
Why is there such a general WD > Seagate slant if the stats don't seem to back it up?
it'd be nice if folks would repeat the more in-depth analyses more often, as they can produce better answers to the questions folks really want.