Is there an open source service designed with HDDs in mind that achieves similar performance? I know none of the big ones work that well with HDDs: MinIO, Swift, Ceph+RadosGW, SeaweedFS; they all suggest flash-only deployments.
Recently I've been looking into Garage and liking the idea of it, but it seems to have a very different design (no EC).
At a past job we had an object store that used SwiftStack. We just used SSDs for the metadata storage but all the objects were stored on regular HDDs. It worked well enough.
I would say that Ceph+RadosGW works well with HDDs, as long as 1) you use SDDs for the index pool, and 2) you are realistic about the number of IOPs you can get out of your pool of HDDs.
And remember that there's a multiplication of iops for any individual client iop, whether you're using triplicate storare or erasure coding. S3 also has iop multiplication, which they solve with tons of HDDs.
For big object storage that's mostly streaming 4MB chunks, this is no big deal. If you have tons of small random reads and writes across many keys or a single big key, that's when you need to make sure your backing store can keep up.
Apache Ozone has multiple 100+ petabyte clusters in production. The capacity is on HDDs and metadata is on SSDs. Updated docs (staging for new docs): https://kerneltime.github.io/ozone-site/
We've been running a production ceph cluster for 11 years now, with only one full scheduled downtime for a major upgrade in all those years, across three different hardware generations. I wouldn't call it easy, but I also wouldn't call it hard. I used to run it with SSDs for radosgw indexes as well as a fast pool for some VMs, and harddrives for bulk object storage. Since i was only running 5 nodes with 10 drives each, I was tired of occasional iop issues under heavy recovery so on the last upgrade I just migrated to 100% nvme drives. To mitigate the price I just bought used enterprise micron drives off ebay whenever I saw a good deal popup. Haven't had any performance issues since then no matter what we've tossed at it. I'd recommend it, though I don't have experience with the other options. On paper I think it's still the best option. Stay away from CephFS though, performance is truly atrocious and you'll footgun yourself for any use in production.
In their design document at https://garagehq.deuxfleurs.fr/documentation/design/goals/ they state: "erasure coding or any other coding technique both increase the difficulty of placing data and synchronizing; we limit ourselves to duplication"
S3’s KeyMap Index uses SSDs. I also wouldn’t be surprised if at this point SSDs are somewhere along the read path for caching hot objects or in the new one zone product.
The storage itself is probably (mostly) on HDDs, but I'd imagine metadata, indices, etc are stored on much faster flash storage. At least, that's the common advice for small-ish Ceph cluster MDS servers. Obviously S3 is a few orders of magnitude bigger than that...
Repeating a comment I made above - for standard tier, requests are expensive enough that it's cost-effective to let space on the disks go unused if someone wants an IOPS/TB ratio that's higher than what disk drives can provide. But not much more expensive than that.
The latest generation of drives store about 30TB - I don't know how much AWS pays for them, but a wild-ass guess would be $300-$500. That's a lot cheaper than 30TB of SSD.
Also important - you can put those disks in high-density systems (e.g. 100 drives in 4U) that only add maybe 25% to the total cost, at least if you're AWS, a bit more for the rest of us. The per-slot cost of boxes that hold lots of SSDs seems to be a lot higher.
Std has the same performance as every other storage class. There are 2 async classes which you can't read from without retrieving first, but that's not a 'performance' difference as such - GETs aren't slow, they fail.
I worked on lifecycle ~5 years ago and just the Standard -> Glacier transition path involved no fewer than 7 microservices.
Just determining which of the 400 trillion keys are eligible for a lifecycle action (comparing each object's metadata against the lifecycle policy on the bucket) is a massive big data job.
Always was a fun oncall when some bucket added a lifecycle rule that queued 1PB+ of data for transition or deletion on the same day. At the time our queuing had become good enough to handle these queues gracefully but our alarming hadn't figured out how to differentiate between the backlog for a single customer with a huge job and the whole system failing to process quickly enough. IIRC this was being fixed as I left.
At this kind of scale, queues, caches and long running workers ought to be avoided at all costs due to their highly opaque nature which drastically increases the unpredictability in the system's behaviour whilst decreasing the reliability and observability.
I generally don't think about storage I/O speed at that scale (I mean really who does?). I once used a RAID0 to store data to HDDs faster, but that was a long time ago.
I would have naively guessed an interesting caching system, and to some degree tiers of storage for hot vs cold objects.
It was obvious after I read the article that parallelism was a great choice, but I definitely hadn't considered the detailed scheme of S3, or the error correction it used. Parallelism is the one word summary, but the details made the article worth reading. I bet minio also has a similar scaling story: parallelism.
RAID doesn’t exactly make writes faster, it can actually be slower. Depends on if you are using RAID for mirroring or sharding. When you mirror, writes are slower since you have to write to all disks.
I think the article's title question is a bit misleading because it focuses on peak throughput for S3 as a whole. The interesting question is "How can the throughput for a GET exceed the throughput of an HDD?"
If you just replicated, you could still get big throughput for S3 as a whole by doing many reads that target different HDDs. But you'd still be limited to max HDD throughput * number of GETs. S3 is not so limited, and that's interesting and non-obvious!
If we assume enterprise HDDs in the double digit TB range then one can estimate that the total S3 storage volume of AWS is in the triple digit Exabyte range. That's propably the biggest storage system on planet earth.
A few factual inaccuracies in here that don't affect the general thrust. For example, the claim that S3 uses a 5:9 sharding scheme. In fact they use many different sharding schemes, and iirc 5:9 isn't one of them.
The main reason being that a ratio of 1.8 physical bytes to 1 logical byte is awful for HDD costs. You can get that down significantly, and you get wider parallelism and better availability guarantees to boot (consider: if a whole AZ goes down, how many shards can you lose before an object is unavailable for GET?).
Naively it seems difficult to decrease the ratio of 1.8x while simultaneously increasing availability. The less duplication, the greater risk of data loss if an AZ goes down? (I thought AWS promises you have a complete independent copy in all 3 AZs though?)
To me though the idea that to read like a single 16MB chunk you need to actually read like 4MB of data from 5 different hard drives and that this is faster is baffling.
> full-platter seek time: ~8ms; half-platter seek time (avg): ~4ms
Average distance between two points (first is current location, second is target location) when both are uniformly distributed in [ 0 .. +1 ] interval is not 0.5, it’s 1/3. If the full platter seek time is 8ms, average seek time should be 2.666ms.
The platter is a circle so using the uniform distribution [0, 1] is incorrect, you should use the unit circular distribution of [0, 2pi] and also since the platter also spins in a single direction the distance is only computed going one way around (if target is right before current, it's one full spin).
But you can simplify this problem down and ask: with no loss of generality, if your starting point is always 0 degrees, how many degrees clockwise is a random point on average, if the target is uniformly distributed?
Since 0-180 has the same arc length as 180-360 then the average distance is 180 degrees. So average half-platter seek is half of the full-platter seek.
Note that you can kind of infer that S3 is still using hard drives for their basic service by looking at pricing and calculating the IOPS rate that doubles the cost per GB per month.
S3 GET and PUT requests are sufficiently expensive that AWS can afford to let disk space sit idle to satisfy high-performance tenants, but not a lot more expensive than that.
It is interesting that even after falling prices of HDDs, S3 costs have remained the same for at least 8 years. There's just not enough competition to push them to reduce costs. But imagine money it brings in in AWS because of this.
Reducing costs is the wrong incentive. If you look at a modern vendor such as Splunk or CrowdStrike, they have huge estates in AWS. There are huge swaths of repeating data, both within and across tenants. Rather than pointing this out, it is simpler and more effective to charge the customer for this data/usage, and use simple techniques so that it isn't duplicative. Reducing costs would only incentive and increase this asinine usage.
46 comments
[ 5.8 ms ] story [ 75.4 ms ] threadhttps://www.allthingsdistributed.com/2023/07/building-and-op...
Building and operating a pretty big storage system called S3 - https://news.ycombinator.com/item?id=36894932 - July 2023 (160 comments)
Recently I've been looking into Garage and liking the idea of it, but it seems to have a very different design (no EC).
And remember that there's a multiplication of iops for any individual client iop, whether you're using triplicate storare or erasure coding. S3 also has iop multiplication, which they solve with tons of HDDs.
For big object storage that's mostly streaming 4MB chunks, this is no big deal. If you have tons of small random reads and writes across many keys or a single big key, that's when you need to make sure your backing store can keep up.
What you mean by no EC?
I honestly figured that it must be powered by SSD for the standard tier and the slower tiers were the ones using HDD or slower systems.
S3’s KeyMap Index uses SSDs. I also wouldn’t be surprised if at this point SSDs are somewhere along the read path for caching hot objects or in the new one zone product.
The latest generation of drives store about 30TB - I don't know how much AWS pays for them, but a wild-ass guess would be $300-$500. That's a lot cheaper than 30TB of SSD.
Also important - you can put those disks in high-density systems (e.g. 100 drives in 4U) that only add maybe 25% to the total cost, at least if you're AWS, a bit more for the rest of us. The per-slot cost of boxes that hold lots of SSDs seems to be a lot higher.
I assume would have lots of queues, caches and long running workers.
I worked on lifecycle ~5 years ago and just the Standard -> Glacier transition path involved no fewer than 7 microservices.
Just determining which of the 400 trillion keys are eligible for a lifecycle action (comparing each object's metadata against the lifecycle policy on the bucket) is a massive big data job.
Always was a fun oncall when some bucket added a lifecycle rule that queued 1PB+ of data for transition or deletion on the same day. At the time our queuing had become good enough to handle these queues gracefully but our alarming hadn't figured out how to differentiate between the backlog for a single customer with a huge job and the whole system failing to process quickly enough. IIRC this was being fixed as I left.
I was just looking for this video so I can send it to my coworkers as one of the best introductory videos into the basics of cloud computing concepts.
(well, I think they may have submitted one or two others, but this is the only one that got published)
I would have naively guessed an interesting caching system, and to some degree tiers of storage for hot vs cold objects.
It was obvious after I read the article that parallelism was a great choice, but I definitely hadn't considered the detailed scheme of S3, or the error correction it used. Parallelism is the one word summary, but the details made the article worth reading. I bet minio also has a similar scaling story: parallelism.
If you just replicated, you could still get big throughput for S3 as a whole by doing many reads that target different HDDs. But you'd still be limited to max HDD throughput * number of GETs. S3 is not so limited, and that's interesting and non-obvious!
If we assume enterprise HDDs in the double digit TB range then one can estimate that the total S3 storage volume of AWS is in the triple digit Exabyte range. That's propably the biggest storage system on planet earth.
The main reason being that a ratio of 1.8 physical bytes to 1 logical byte is awful for HDD costs. You can get that down significantly, and you get wider parallelism and better availability guarantees to boot (consider: if a whole AZ goes down, how many shards can you lose before an object is unavailable for GET?).
The way it’s worded makes me understand that’s what scheme they’re using. Curious to hear what you know
To me though the idea that to read like a single 16MB chunk you need to actually read like 4MB of data from 5 different hard drives and that this is faster is baffling.
Average distance between two points (first is current location, second is target location) when both are uniformly distributed in [ 0 .. +1 ] interval is not 0.5, it’s 1/3. If the full platter seek time is 8ms, average seek time should be 2.666ms.
But you can simplify this problem down and ask: with no loss of generality, if your starting point is always 0 degrees, how many degrees clockwise is a random point on average, if the target is uniformly distributed?
Since 0-180 has the same arc length as 180-360 then the average distance is 180 degrees. So average half-platter seek is half of the full-platter seek.
S3 GET and PUT requests are sufficiently expensive that AWS can afford to let disk space sit idle to satisfy high-performance tenants, but not a lot more expensive than that.
Because as a shelf-hosted, I had to block a ton of AWS ipv4s because of those.
https://www.allthingsdistributed.com/2023/07/building-and-op...