Ask HN: How would you store 10PB of data for your startup today?
Min requirements of AWS S3 One Zone IA (https://aws.amazon.com/s3/storage-classes/?nc=sn&loc=3)
How would you store >10PB if you'd be in my shoes? Thought experiment can be with and without data transfer cost our of current S3 buckets. Please mention also what your experience is based on. Ideally you store large amounts of data yourself and speak of first hand experience.
Thank you for your support!! I will post a thread once we got to a decision on what we ended up doing.
Update: Should have mentioned earlier, data needs to be accessible at all time. It’s user generated data that is downloaded in the background to a mobile phone, so super low latency is not important, but less than 1000ms required.
The data is all images and videos, and no queries need to be performed on the data.
373 comments
[ 5.1 ms ] story [ 281 ms ] threadAs an aside, you can often get nice credits for moving off of AWS to Azure or GCP. I recommend the later.
10PB seems like a lot to store in S3 buckets. I assume much of that data is not accessed frequently or would be used in a big data scenario. Maybe some other services like Glacier or RedShift (I think).
Wasabi and Glacier would be my 2nd choices.
AFAIK cloudflare ToS prohibits you from using it as a file hosting proxy. You might not run into issues if you're transferring a few gigabytes a month, but if you're transferring multiple terabytes it's just asking for trouble.
edit:
https://www.cloudflare.com/terms/ section 2.8 Limitation on Serving Non-HTML Content
I'm not... super confident in that answer, because despite that being a use case they promote on the site the terms seem a bit murkier, and the page on that use-case doesn't say much about which plan(s) they expect you to use (I'd have expected an "enterprise" plan for serving hundreds of TB of transfer of game-assets per month, but they said no, any normal plan's fine, which... I was up front with them about what our usage would look like, and they held that line, but that seems too good to be true).
I haven't tested these claims yet.
[1] https://www.cloudflare.com/gaming/
- https://www.backblaze.com/blog/backblaze-and-cloudflare-part...
- https://www.cloudflare.com/partners/technology-partners/back...
- https://www.cloudflare.com/bandwidth-alliance/backblaze/
Are these files WORM?
I'd recommend reaching out to some data eng in the various Bigs, they certainly have more clear numbers. Happy to make an intro if you need, feel free to dm me.
And this doesn't even cover how you'd fit 40,000 sticks of RAM together.
edit: perhaps their RCS option would be cheaper if you know exactly how much data you need to store in advance.
600x WESTERN DIGITAL Ultrastar DC HC550 18TB (10800PB in total) - $500 each
~$350k in hardware, up to 20kW energy consumption, should fit in two rack towers. You can host it for about $1.5k somewhere. All assuming no redundancy :)
So toss in at least one SRE type person. Say $200k/year.
Since you only have one, they are gonna be on call 24/7, so assume you’ll burn them out after a year and a half and need to hire a new one....
Since redundancy is a thing, double that $350k. And 10pb is what they have now so double it again for 20pb. Add in $10k per rack for switches, routers, wires, etc.
So probably you are looking at a million dollars of capital plus labor to actually execute on this. And don’t forget the lead time might be a month to get the hardware and a week or two to install it. Plus all the configuration management that needs to be built up. Not to mention monitoring. So maybe a quarter of work just to have it functional.
I haven’t even factored in opportunity costs. What could this business be doing that adds more value than building out a little data center?
I dunno. Maybe it does make sense to manage your own hardware. But it helps to calculate the entire cost of ownership, not just the cost of the servers.
This person's entire job is managing a few racks of hard drives? How often do you think they're actually going to get called in?
> Since redundancy is a thing, double that $350k.
True, but you can do redundancy for cheaper with parity or tape.
> And 10pb is what they have now so double it again for 20pb.
> So probably you are looking at a million dollars of capital plus labor to actually execute on this.
You can go a couple PB at a time if the upfront cost is daunting.
> Add in $10k per rack for switches, routers, wires, etc.
Yep, though that's not very much in comparison.
> Plus all the configuration management that needs to be built up. Not to mention monitoring. So maybe a quarter of work just to have it functional.
This is the one I'd really worry about.
> I haven’t even factored in opportunity costs. What could this business be doing that adds more value than building out a little data center?
You always have to keep opportunity costs in mind, but something like this can pay for itself in under a year if there's significant bandwidth cost too, and that's an amazing ROI.
Not often. But the server gods are a cruel mistress and it will definitely shit the bed when you are on your honeymoon, or maybe the day after your first kid is born.
At this volumes you probably do want a carbon copy at another site to mitigate disasters like datacenter fires.
How about every day?
Quick guess how often disks need to be replaced when there are thousands of them. ;)
Also for one or two thousand disks I would expect less than one failure per week.
- network switch Juniper EX4600 (10Gbps ports) + 3rd party optics ~$11k
- cheap 1Gbps switch for management access <$1k
- some router for VPN for management network - $500
- 1Gbps (not guaranteed) internet access with few IPs ~$350 / month
- 100Mbps low traffic internet access for the management/OOB network.
Time to get the hardware - 2 months. Time to rent and install hardware in rack - about 1 month. I don't count configuring the software.
This setup is full of single points of failure so I would consider it one "region" and use something like CEPH + some spare servers in each "region". That way you don't need to react immediately to hardware failures. Just send a box of hardware from time to time to the DC and use ~$20-40h/h remote hands service to replace the failed drives or whole servers. You could also buy on-site service from the hardware vendor for 1-3 years adding some cost.
I think the most important thing would be have a cleaver person who design a fault tolerant system, automatic failover, good monitoring and alerting so that any on-call and maintenance job is easy and based on procedures. That way you could outsource it. Only then it might have some sense.
Also, the hardware can be depreciated, which reduces its net (of taxes) cost dramatically over time.
Five years (probably the useful life of the equipment in general) of $210,000 per month is $12.6M. That's a lot of savings.
Regarding accounting, the AWS monthly charges are also net of taxes so it makes no difference.
Is the data cold storage, that is rarely accessed? Is it OK to risk losing a percentage of it? Can you identify that percentage? If it's actively utilized, is it all used, or just a subset? Which subset? How much data is added every day? How much is deleted? What are the I/O patterns?
Etc.
I have direct experience moving big cloud datasets to on-site storage (in my case, RAID arrays), but it was a situation where the data had a long-tail usage pattern, and it didn't really matter if some was lost. YMMV.
If it's the former, then investing in-house might make sense (a la Dropbox's reverse course).
This allows you to read the data into AWS instances at no cost and process it as needed since there is 0 cost for ingress into AWS. I have some experience with this (hosting using Equinix)
We had about 25 Dell R730xd servers. When the cluster would start to fill up, we would just replace drives with larger drives. Upgrading drives with SwiftStack is a piece of cake. When I left we were upgrading to 10TB drives as that was the best pricing. We didn't buy the drives from Dell as they were crazy expensive. We just bought drives from Amazon/New Egg, and kept some spares onsite. We got a better warranty that way too. Dell only had a 1 year warranty, but the drives we were buying had a 5 year warranty.
Idk what your team’s expertise is, but I’d advise avoiding the cloud as long as possible. If you can build out an on-premise infrastructure, it will be a huge competitive advantage for your company because it will allow you to offer features that your competitors can’t.
Examples of this:
- Cloudflare built up their own network and infrastructure and it’s always been their biggest asset. They set the standard for free tier of CDN pricing, and nobody who builds a CDN on top of an existing cloud provider will ever beat it.
- Zoom. By hosting their own servers and network, Zoom is similarly able to offer a free tier where they are not subject to variable costs from free customers losing them money on bandwidth charges.
- WhatsApp. They scaled to hundreds of millions of users with less than a dozen engineers, a few dozen (?) servers, and some Erlang code.
IMO defaulting to the cloud is one of the worst mistakes a young company can make. If your app is not business critical, you can probably afford up to a day of downtime or even some data loss. And that is unlikely to happen anyway, as long as you’ve got a capable team looking after it who chooses standard and robust software.
A startup is a company that might still need to pivot to find its final business model, potentially shedding its entire existing infrastructure base in the process. Start-ups are why IaaS providers don't default to instance reservations — because, as a startup, you might suddenly realize that you won't be needing that $10k/hr of compute, but rather $10k/hr of something else.
That was before the cloud existed. They had to poach experts from hosting companies to build and maintain their gear. They built a 24/7 NOC, did server repair, became network experts, storage experts, database experts. Besides being incredibly complex and burdensome, it was financially risky. If they missed their projections they could over-invest by 1-2 million bucks, or even worse, not have the capacity needed to meet demand.
If somebody told us back then that we could pay a premium to be able to scale at any time as much as we needed, when we needed it? We would have flipped out. We had heard about Amazon building some kind of "grid computing" thing, but it seemed like a pipe dream for universities, like parallel computing. Turns out it was a different kind of grid.
CloudFlare went well beyond leasing servers and built their own POPs with network etc prior to IPO. Much of what they built wouldn't have made economic sense with AWS tax.
Of course not. But the free tier was a vital component of Cloudflare's growth, first-mover advantage and wide adoption.
And cheap.
If you put people in charge who are looking for ways of expanding their empire and budget through spending money on EMC/VMWare/Oracle/etc/etc then you can quickly wind up spending a lot more money.
Simplistic network designs, simplistic server designs, simplistic storage designs with mostly open source software used everywhere can be highly competitive with Cloud services.
Mostly all that Amazon did to create AWS/EC2 was to fire anyone who said words like SAN or EMC and do everything very cheaply using open source software, and evolved away from Enterprise vendors and towards commodity hardware.
If you make "frugality" a core competency in your datacenter design like Amazon did, then you can easily beat the cloud.
You also need to have [dev]ops people who are inclined to say "yes" to the business and who know how to debug things and can operate independently of needing to phone up EMC.
Is EBS not, itself, a SAN?
To me, a "Storage Area Network" is 1. a cluster of disk-servers, serving the role of exposing logical block-storage over a protocol like iSCSI (whether directly to client machines, or managed and dynamically allocated by hypervisor software like vSphere), where 2. machines are connected to that storage cluster over a dedicated network interface, to keep LAN/WAN packets from contending for throughput with SAN packets.
By that definition, EBS is definitely a SAN. (And technically, so is my two-drive NAS, if I configure it as an iSCSI target and then run a second switch that connects to its second network port and my workstation's second network port.)
Does "SAN" imply some specific internal architecture for the storage cluster or something?
And, if so, then what do you call the type of thing that EBS is?
It implies purchasing dedicated hardware. SANs are CAPEX heavy solutions.
> And, if so, then what do you call the type of thing that EBS is?
If you insist, you could call EBS a SAN-as-a-Service, I suppose.
For a SAN, not only do you have to become a "storage expert", but their individual limitations will leave you with thousands of hours of wasted time and effort, constrain your architecture, and hold back your application's development.
For EBS, you don't need to know anything about storage. You just say "Give me some space and attach it to any VM I want" and you have it. "Expand that space" and you have it. "Give me a snapshot" and you have it. "Give me a bunch of performance guarantees" and you have it. "Make it all encrypted": Done.
You don't need to maintain it, repair it, upgrade it. No maintenance windows to apply a firmware patch. No waiting for someone to buy, deliver, and install a new storage array to get more space. No hoping your hardware has the right interconnects. No upgrading switch backbones to deal with performance issues. And I'm not even a storage person! I'm so happy that I don't deal with SANs anymore.
Buy storage servers from 45drives they basically build same hardware as Backblaze uses. Add copper 10G nics to the servers.
https://www.45drives.com/
Get necessary switches 10G with 40G uplink ports. Whatever your favorite. Use 10GBaseT to the servers.
Install hardware in a quality data center. Like one of theirs -
https://www.digitalrealty.com/
And get 10G virtual cross connects to AWS.
Back of the envelope calculation you need 30TB raw, so about 60 servers. They aren’t really that power hungry so 10 per cabinet. 6 cabinets. at least 6+2 switches.
Software wise you have lots of options with this infra. High upfront cost but low MRC vs all other options. Assuming you have skilled sys admins who know what they are doing.
[0] "You pay for all bandwidth into and out of Amazon S3, except for the following: Data transferred in from the internet..." - https://aws.amazon.com/s3/pricing/
How are you storing this data? Is it tons of small objects, or a smaller number of massive objects?
If you can aggregate the small objects into larger ones, can you compress them? Is this 10PB compressed or not? If this is video or photo data, compression won't buy you nearly as much. If you have to access small bits of data, and this data isn't something like Parquet or JSON, S3 won't be a good fit.
Will you access this data for analytics purposes? If so, S3 has querying functionality like Athena and S3 Select. If it's instead for serving small files, S3 may not be a good fit.
Really, at PB scale these questions are all critically important and any one of them completely changes the article. There is no easy "store PB of data" architecture, you're going to need to optimize heavily for your specific use case.
We don’t touch the data at all.
> The data is all images and videos, and no queries need to be performed on the data.
OK, so this definitely helps a bit.
At 10PB my assumption is that storage costs are the major thing to optimize for. Compression is an obvious must, but as it's image and video you're going to have some trouble there.
Aggregation where you can is probably a good idea - like if a user has a photo album, it might make sense to store all of those photos together, compressed, and then store an index of photo ID to album. Deduplication is another thing to consider architecting for - if the user has the same photo, across N albums, you should ensure it's only stored the one time. Depending on what you expect to be more or less common this will change your approach a lot.
Of course, you want to avoid mutating objects in S3 too - so an external index to track all of this will be important. You don't want to have to pull from S3 just to determine that your data was never there. You can also store object metadata and query that first.
AFAIK S3 is the cheapest way to store a huge amount of data other than running your own custom hardware. I don't think you're at that scale yet.
Latency is probably an easy one. Just don't use Glacier, basically, or use it sparingly for data that is extremely rare to access ie: if you back up disabled user accounts in case they come back or something like that.
I think this'll be less of a "do we use S3 or XYZ" and more of a "how do we organize our data so that we can compress as much of it together, deduplicate as much of it as possible, and access the least bytes necessary".
If it's all archival storage then it's pretty straight forward. If you're on GCP you take it all and dump it into archival single region DRA (Durable Reduced Availability) storage for the lowest costs.
Otherwise, identify your segments and figure out a strategy for "load balancing" between standard, nearline, coldline, and archive storage classes. If you can figure out a chronological pattern, you can write a small script that uses the gsutils built-in rsync feature to mirror over data from a higher grade storage class to a lower one at the right time.
The strategy will probably be similar in any of the other big 3 providers as well, but fair warning, some providers archival grade storage does not have immediate availability last I checked.
See: https://cloud.google.com/storage/docs/storage-classes
https://cloud.google.com/storage/docs/gsutil/commands/rsync
This is like an ISP asking how they can get hooked up to the internet.
Maybe I’m wrong though. Perhaps the real secret sauce is the end user experience and the kind of storage you use on the backend doesn’t matter at all.
However I bet that the “cloud storage space” is pretty crowded and lots of people shop on price more than anything. If your business model is all about price, then finding economical storage is critical to your company and needs to be part of your core competency.
If price isn’t that important, perhaps it doesn’t matter... the “winners” would win no matter how expensive their storage solution is.
But honestly.... I feel like part of your core competency needs to be managing the storage system.
It's a bit different nowadays that a lot of scaling tech is commoditized, but still means things like negotiating new contracts, finding & fixing the odd pieces that weren't stressed before, etc.
(congrats on hitting the new usage levels + good luck! we're at a much smaller scale, but trying to figure out some similar questions for stuff like web-scale publishing of data journalism without dying on egress $, so it's an interesting thread...)
Apple didn't start manufacturing with mega Foxconn contracts. They had to figure that out along the way as their scale demanded.
However I share your sentiment: doing things the same way but cheaper is usually not the solution. Doing things differently (in-sourcing) might be the path forward.
Apple created something people wanted and sold at a price that would still make money if it was assembled by hand. They didn't form a company around a commodity like data storage.
Data storage is a commodity. Everyone already has some, online storage companies already exist. If you don't know how to store a lot of data and your company's whole purpose is to store a lot of data, it sounds like something that should have been worked out before making the company.
We are a photo/video storage service.
Because there may be an inflection point that offering monetary compensation for data loss, rather than actually trying to store the data, would make more financial sense. I.e., "All data > than 2 years gets silently expunged, and anyone trying to retrieve it at that point gets $10 per gig in compensation for 'our mistake'".
Please don't actually consider that though.
(And if you don't already, I would also consider making it so items that are in the trash for some period of time, say 30 days, get deleted automatically as well, possibly with a reminder email a few days before)
(And lastly, depending on user profiles and usage, incentives around reducing resolution/quality of photos and video, and automating that in the app as part of the sync process, might provide some opportunities to reduce costs of storage > the lost revenue of cheaper plans.)
Sticking points I see are, 1. If you get it wrong you'll need some form of UX that keeps the users from getting to angry about it. 2. The cost of moving the data between hot/cold storage might make this prohibitive until a much larger scale. 3. User behaviors might not be predictable enough.
"We use ML to ensure we only store the highest quality data, freeing you from the chains of having too much worthless data and nothing to do with it."
Changing that can be very very difficult for not much gain. Plus AWS skills are very easy to recruit for vs Google cloud.
Try being intentional and smart in front of your data pipeline and purge data that is not useful. Too many times people store data "just in case" and that case never happens years later.
Machine data takes a lot of space, depending on how long you need/want to hold onto it - any given even might only be a few 100 bytes, but with 1000s or 10s of 1000s of devices sending even syslog turns into a metric buttload of data :)