Ask HN: Cheap Bulk Storage?
2 years ago, BackBlaze released specs for their storage pod, which is basically huge tract of drives wired together.
It's not fast, but it's cheap, and it's great for bulk-archiving documents.
Now it's 2011- Is there a better way of solving this problem? S3 is crazy expensive for massive (100+TB) archiving, and I'd rather go one-step above building my own system using custom frames and components.
Are there any web-services which offer bulk storage at somewhat reasonable rates? Are there off-the-shelf cases which offer 50+ HDs for a non-insane price?
42 comments
[ 249 ms ] story [ 1148 ms ] threadHP has a 70-bay JBOD that works out to under 30 cents/GB: http://h10010.www1.hp.com/wwpc/us/en/sm/WF25a/12169-304616-3...
Where did you get the pricing from, to generate your cost estimates (ie. the 10/30c/gig)?
http://www.google.com/search?q=SC847E16-RJBOD1&ie=utf-8&...
2K - Case + Backplanes 1K - CPU, MB, RAM 3.3K - 45 1TB drives at 75 ea --
6.6K - 45TB RAID0 If I want it to be even somewhat sane, I'd add every 4th drive as spare/parity, which drops to 33G
$6600 / 35TB gives $.19/GB
That's higher than the .12 from the 2 year old BackBlaze box, but not much.
If instead we go with the $179 3TB WD drives, we can get slightly better than the pod pricing.
($179 * 45) / (45 * 3TB * .75) in dollars per gigabyte = $.08/GB
It doesn't feel like 2 years worth of progress, though- The numbers are pretty much still where they were 2 years ago ;)
Thanks Again.
Some quick estimates: You can get a 36 bay server with an attached 45 bay JBOD for about 8.5k [2]. If you use 2TB consumer drives, they'll run about $140 a piece[3]. This comes out to $0.12 / GB, which is a damn good deal, IMO.
[0] http://www.siliconmechanics.com/
[1] http://www.aberdeeninc.com/
[2] http://www.siliconmechanics.com/i28693/4u-storage-server.php and http://www.siliconmechanics.com/i19897/4u-45-drive-jbod-sas-...
[3] WD Caviar Black drives ( http://www.newegg.com/Product/Product.aspx? Item=N82E16822136824 ). You can get WD Greens for $80 a piece, but I don't know how their variable RPM holds up in a RAID.
I think the WD's are the way to go. Enterprise-grade drives aren't really much more reliable, and you need to plan capacity assuming they'll fail anyway. Better to get consumer drives that are at least mid-range (avoid Hitachi), and run with it.
http://www.supermicro.com/products/chassis/4U/417/SC417E16-R...
Using that, and 72 1TB 2.5" drives, (http://www.newegg.com/Product/Product.aspx?Item=N82E16822136...)
(($109 * 72) / (72 * 1TB * .75) in dollars per gigabyte) = $.15/Gb
Alternatively we could consider what level of reliability you want and how that will be tested.
You may want to take a look at http://www.wuala.com, although I'm not sure it will fit at your scale.
Besides other features, they offer two not-so-common things:
1) you can trade your own space for remote storage
2) they have local encryption (at your end)
So people (myself included) buy physical HDDs for local backup -- simply because every online alternative jacks up the price by maintaining 3+ copies and keeping everything permanently online.
I'm pretty sure you can find an X which won't cost too much, and will get most of the complaints off your back
Backblaze (no affiliation) is $5/month or $50/yr. That's pretty hard to beat.
Also, running a static web-server is fairly trivial to setup these days, what would a service not running a static web-server do/have?
Every online alternative has a vested interest in both A. saving money, and B. not losing your data. As much as you trust a business to perform B adequately, you implicitly trust it to perform A at least as well.
http://www.diomedestorage.com/
Edit: Actually, it looks like they closed on Mar 19, according to an email they sent to existing users. They say they might reopen "with a new approach" near the end of the year.
How much data are you willing to lose per day out of your 100TB?
If there were a system where it lost even 20TB/day, I'd be interested, depending on what the tradeoffs were! I'd expect that to be pretty seriously cheap to offset it.
That's why I'm asking what options people know about. If you know one that is randomly lossy but cheap, I'd be REALLY curious as to what their backend is!
Amazon S3 costs around $1/gb/year. Let's say it has 0% chance of data loss Suppose I can store something without any replication. Cost drops to $0.50/gb/year, 2% chance of losing all my data. If the value of my data is less than $25/gb, it makes financial sense to go for the unreliable option.
Much of my data is worth less than $25/gb. photos, video, old mail. Storing it on RAID is a waste of money.
besides, the chance of me deleting everything by human error is often much higher than the chance of hardware failure.
and some decent portion of those failures come with advance warning, so that you can save at least some of the data.
It's also entirely open source, so you could run it on your own hardware if you wanted. It uses parity instead of replication for storing data (but with arbitrary effectively replication levels.) So with R=3, you need a minimum of about 10 nodes to be efficient.
The beta uses AMQP internally, and the upcoming release version is Python/zeromq/gevent based.
https://spideroak.com/diy/
The pricing is not entirely out of line compared with what I can get in-house, and it's a fantastic solution for storing horribly bulk,low-reliability data.
$10 / 100GB comes out to .01/GB/Month. That means that you pay the total cost of an in-house solution, every month you use their service..
That's 12x the price of doing it yourself, but you don't need to buy replacement drives, pay for electricity, or send someone to go replace drives.
It's not that much cheaper than Amazon in the base pricing, but it looks like you can relax the number of copies for them to keep from 3(default) to 1, and cut the price down to 1/3?
Is this your experience? Thanks for the link.
For the storage cost calculations: electricity, cooling, and bandwidth end up costing more than the drives, even once you include all additional hardware to keep them running.
Note of caution: I would recommend that shops build bulk storage hardware themselves only if someone on their team has an intensive storage background. There are a number of gotchas that are distinctly non obvious; they are capital intensive lessons to learn, and endanger data integrity. You saw what Backblaze went through to design their storage pods: all of the self hosted offsite backup companies do likewise. It's a core competency.
Typically it's much better to outsource or buy the more expensive business storage products with hardware RAID, enterprise class drives, cache batteries, and all that.
SpiderOak's DIY is designed for high reliability, but not high performance. In price comparisons to S3, there's no option for reduced-reliability storage, but also no charge for bandwidth in our out (up to a reasonable point.)
Having said all that, if you are interested in building 100+ TB of storage hardware in house, feel free to send me a mail; I may be able to save you some difficulty. If you'd rather us host it, we do discounts for startups. :)
Oops.
Thanks for the awesome service. I've used SO for backups on and off, and I appreciate the tech work you do. I particularly like that you do client-side encryption. I just wish your UI was a bit better ;)
I must have been mixing up your service with diomede; IIRC, their techcrunch article was about letting you choose the levels of redundancy you want to keep.
I'll grant you that the bandwidth is certainly a price-factor, but your page talks a LOT about how you're higher latency than S3, and not everyone needs that speed for that price, etc, etc, and then your price comes out to be about 80% of theirs. Honestly, after reading your product positioning, I was envisioning it coming out to like 10%.
You're absolutely right that building a huge in-house storage network is a huge ordeal.. If you're worried about making it fast and scalable. What I want to do is to treat it as the equivalent of a huge tape drive- Just throw stuff onto it, and hope for the best.
I've been toying around with the idea of dealing with it entirely in application logic. Rather than RAIDing the disks, mount each one separately, and keep a hashtable of my files in memory with their full path, including redundant copies. Then, if a file fails, I have application logic try the backup copy.
Anyway, you know a LOT more about this than I do, so I'll shut up before I embarrass myself further.
My point is just that there's a lot of people in the market who have huge datasets currently, for various reasons. Lots of science departments, analytics groups, etc. I'd love to be able to keep an nearline copy of work.
I don't mind if it takes a good 15-20 minutes to be able to access the first byte, I just want to know that I can if I need to, faster than waiting for Iron Mountain to ship me something.
Side note- I considered trying to justify it under BackBlaze's "$5, unlimited space" offer, but I didn't think they'd go for it. ;)