> That’s almost 13,000 objects for each person in the world, or 50 objects for every one of the roughly two trillion galaxies (according to this 2021 estimate) in the Universe.
That’s mind boggling to think about. I wonder how much paper would’ve been needed to store 100T objects on paper. It’s like the new Library of Alexandria.
What’s way more mind boggling to me is that there are at least 260 galaxies per person in the world. I am not shocked at all that S3 is home to any number of cat pictures many of which are probably duplicates.
It should be read as "100 tera-objects", meaning 100 trillion objects. Writing "K" seems to be so widespread these days (Youtube likes/dislikes are written as 1.5K, for example), I'm surprised there is any confusion here.
There's different storage tiers, with different redundancy. So if anyone has an anecdote, would be curious to know which storage tier was used (e.g. single availability zone)
If it works as designed I guess 1000 have; but how would you know. Did your perfect code running on an perfect CPU and RAM connected though a perfect network upload the file with 11 9's certainty?
This is a real problem, but it is partially accounted for:
> To ensure that data is not corrupted traversing the network, use the Content-MD5 header. When you use this header, Amazon S3 checks the object against the provided MD5 value and, if they do not match, returns an error. Additionally, you can calculate the MD5 while putting an object to Amazon S3 and compare the returned ETag to the calculated MD5 value.
It’s not as obvious with multipart uploads where each chunk is (iirc) individually checksumd and stored, but only a single md5 is returned in the “etag”
I imagine that in most cases it's a giant archive - data that no one really cares about (I mean they do in principal). Then there's probably like 0.1% that's probably operationally important data - which is actually huge.
This is just a guess based on my own usage of it over the last 15 years at different companies.
... this is on you to prove it is their secret sauce. Throwing out some random feature and saying it is their secret sauce and putting the burden of proof on somebody else makes no sense.
But it would be because 1) they probably don't have many duped objects to begin with 2) the system to de-dupe items would be complex, error prone, increase latency, and just not worth it
> 1) they probably don't have many duped objects to begin with
That's crap reasoning. You have no way of knowing that.
> 2) the system to de-dupe items would be complex, error prone, increase latency, and just not worth
Would it really be that complex? I kinda doubt that.
Read latencies wouldn't be affected - except being improved.
Write latencies - the deduplication can happen after the write has been confirmed, so no extra latency added.
Whether or not it's worth it - I think it's very hard to to estimate the deduplication factor. My guess is 1.5x. At that point it would save 33% storage. That would instanly make a whole lot of extra complexity worth it.
At the same time, the S3 api is so ubiquitous that there implementations for it in multiple clouds, it frequently comes with hardware storage vendors as well, and there are projects like minio that provide a solution to host your own s3 compatible data architecture.
All things considered, I’m much more concerned about the centralization of eg browser technologies than this.
Eh… the thing about cloud storage is that it can be efficient in ways that smaller players cannot. A system like Amazon S3 can pack its hard drives to the brim with data and pay a fairly small overhead for replicas & unused drives.
If you try to decentralize the data, or keep your data yourself, you end up with something that uses more resources, has worse durability, or is worse along some other axis.
I am not so worried about centralization yet. There is still healthy competition between different cloud storage vendors. My impression is that cheap storage is used to make other cloud services more attractive—which means that Amazon’s incentives here are somewhat aligned with mine (as a customer). Personally I am much more worried about the centralization of the shopping experience.
I think it’s fine as long as people know what they’re getting themselves into. I’m sure I would’ve found whatever was happening on Parler abhorrent, but in the same way Merkel and Macron were concerned by the tech deplatforming that followed the capitol riots, so am I.
I don’t think filecoin will ever demolish a fantastic business like S3 based on stealing away customers. The interesting value proposition from the protocol is probably more about censorship resistance and adjacent issues.
We will start having trouble with random UUID collisions when we have made 2^64 of them, which is a huge number. It's about as many iron atoms as are in an iron filing.
It is also about 200,000 times 100 trillion, so AWS is just 18 doubling times away from the UUID system breaking down.
Is this a real problem? It should be possible for Amazon to update their code to generate IDs a different way, and for them to update their client libraries well ahead of a time when this becomes a problem. Yes, code that relies on the ID being a particular format will break. But users have to deal with deprecated client library features all the time. It's reasonable to give a few year grace period before old APIs are shut down.
UUIDs are usually not simply random numbers. So it's possible to tell when a particular standard of UUID is used. Amazon's code can branch based on this detection, or even something simpler like "UUID-type", a new field that would be initialized by default to the old UUID type.
If you're curious about some "S3 internals", this is a talk I gave in 2009 [0], based on a ton of interesting stuff that S3's then-GM Alyssa Henry [1] shared with me.
>Most of our customers, if they have on-premises systems—if they're lucky—can store two objects in the same data center, which gives them four 9s. If they're really good, they may have two data centers and actually know how to replicate over two data centers, and that gives them five 9s. But eleven 9's, in terms of durability, is just unparalleled
I have a tough time taking him seriously reading this - I'm not sure if he's being intentionally ignorant or intentionally misleading.
I don't know of a single enterprise customer that doesn't have their data replicated to two datacenters, and then backed up to some other medium (tape or disk-based backup appliance) for anything business critical.
Their goal is 5-9's of UPTIME - the expectation is 100% data durability. You would get fired if you architected a solution keeping two copies of data in one datacenter.
Object storage absolutely has a place and I'm happy amazon was able to push a quasi-standard for the industry. Object storage prior to S3 was a hodge-podge of proprietary plays (EMC Centera, Bytecast, etc) and sort-of standards that nobody really used (CDMI). I just wish they did a better job of fairly representing it vs. turning every opportunity into a sales pitch spreading FUD about the alternatives.
Given that part of what I do is help enterprise customers with cloud adoption, almost all of whom are fortune 500 (and on his radar), with an AWS presence, I strongly doubt it.
> I don't know of a single enterprise customer that doesn't have their data replicated to two datacenters, and then backed up to some other medium (tape or disk-based backup appliance) for anything business critical.
Longtime consultant here with BCP/DR insight into 20+ large F500 companies... I think you would be seriously surprised at how common it is for even enterprise customers to not bother with multi-DC or even offsite backups. And even among the ones that do, many of the ones that are non-cloud-based are so immature at it that I would not put money on their backups being restorable if needed. And this is even more true for your “we’re a startup, we don’t have time to worry about backing up our data!” companies, which are in abundance.
For a small peak, go look at how many people were freaking out about losing their entire business due to the loss of a single OVH data center.
Of course, as a consultant I do naturally skew towards customers that need help with this stuff, so my perspective is probably biased towards the companies that are worse off in this regard. But they’re definitely out there.
> I don't know of a single enterprise customer that doesn't have their data replicated to two datacenters, and then backed up to some other medium (tape or disk-based backup appliance) for anything business critical.
You snipped out the part of the quote that answers your question.
"Most of our customers, if they have on-premises systems—if they're lucky—can store two objects in the same data center, which gives them four 9s. If they're really good, they may have two data centers and actually know how to replicate over two data centers, and that gives them five 9s. But eleven 9's, in terms of durability, is just unparalleled. And it trumps everything."
> I don't know of a single enterprise customer that doesn't have their data replicated to two datacenters...
Except, in AWS' case, each AZs (Availability Zones) is made up of upto 8 DCs (Data Centers), and each full-region has at least 3 AZs and 2 Transit Centers. Amazon S3 replicates data to 3 different AZs (which, I am guessing, is in addition to replicating it across DCs in a single AZ for 'eleven 9s').
With S3 cross-region replication durability may shoot up to 'sixteen 9s'? 100% durability, if it exists, is something the major Cloud providers are yet to offer?
>You snipped out the part of the quote that answers your question.
Replicating across two datacenters isn't "really good" though, that's considered table stakes.
>Except, in AWS' case, each AZs (Availability Zones) is made up of upto 8 DCs (Data Centers), and each full-region has at least 3 AZs and 2 Transit Centers. Amazon S3 replicates data to 3 different AZs (which, I am guessing, is in addition to replicating it across DCs in a single AZ for 'eleven 9s').
In AWS' case, the 8 DCs in an AZ are directly adjacent which isn't really useful for anything beyond metro (active/active) availability. I haven't seen a DR plan that doesn't have a hard requirement for the secondary copy of data to be outside of the metro loop/blast radius generally in a different state.
One of the best things that made AWS what it is today. Sure it can be a little pricey compared to competitors but it was revolutionary when it came out and still is a great product. Also a lot of its competitors still lack certain key features that makes it hard to move away from s3.
When people talk about avoiding vendor lock in with AWS, they tend to talk in terms of using k8s or another compute abstraction layer.
But I think the real lock-in lies in services like S3, making so many workflows simple and intuitive (while controlling your data, which is where the real lock-in lies)
Minio and other S3-compatibles make moving off of AWS that much simpler, though obviously it might be difficult if you're using Amazon-only S3 features extensively.
Yeah, even if you don't use any vendor-specific features, there's data gravity—It's hard to move your data from one location to another. AWS and others increase this cost even further by making egress so expensive.
Precisely - vendor lock-in generally happens at the data level. For example, AWS finally turned off their last Oracle DB server only a year or two ago.
How does something like that get handled? At some point isn't contacting this service really just contacting a handful of IP addresses? Are those just "regular" computers? Specialized hardware?
Also with the amount of centralization here, presumably they are the target of bad actors. How do they defend against things like DDOS's to provide their claimed uptimes?
I’m wildly guessing here, but I assume that there are many tiers of load balancing to turn your initial request into the message that reaches the thing that gives back your file data. They also likely do have specialized hardware in various parts of the AWS infrastructure.
A service that can handle tens of millions of requests per second can probably outscale most DDoS attacks without further engineering.
> At some point isn't contacting this service really just contacting a handful of IP addresses?
"Handful". S3 has 283 subnets allocated to it, biggest one being a /15, so there are some few hunderd thousand IP addresses reserved for S3. While impossible to say how many of them are actively in use, I imagine large fraction of them.
S3 is a great service but it's a double edge sword.
It's easy to start with and "cheap" until you scale and then you come to realize you pay a ton for writes and you have a latency of 100ms+ to retrieve data. Even S3 select which on face value seems great is absurdly slow.
Then they offer you Fsx or elastic cache as a proxy.
Very little, I'm sure. S3 and Ceph were both introduced to the world in 2006, which means they had been in development at the same time - or perhaps S3 even a bit before, since it was in a far more mature state by then. Also, while S3 and Ceph's RADOS both supposedly handle "objects" they're very different kinds of objects with very different APIs. And of course the metadata layers are completely different to serve different needs. It's practically certain that they both drew on the same common ancestors, but it's more likely that lessons learned from S3 informed subsequent development on Ceph than the other way around.
78 comments
[ 141 ms ] story [ 4942 ms ] threadThat’s mind boggling to think about. I wonder how much paper would’ve been needed to store 100T objects on paper. It’s like the new Library of Alexandria.
Presumably the Library of Alexandria didn’t have access control lists
Would be very unfortunate if the S3 story ends like the Library of Alexandria.
I think it's clear as-is.
No, it doesn't. Not more than it reads terameters or teragrams. SI prefixes were not invented specifically for computer memory.
Further, '100 Terabyte Objects' wouldn't really make sense as a sentence anyways.
Has anyone lost data on S3 or know anyone who has?
> To ensure that data is not corrupted traversing the network, use the Content-MD5 header. When you use this header, Amazon S3 checks the object against the provided MD5 value and, if they do not match, returns an error. Additionally, you can calculate the MD5 while putting an object to Amazon S3 and compare the returned ETag to the calculated MD5 value.
It’s not as obvious with multipart uploads where each chunk is (iirc) individually checksumd and stored, but only a single md5 is returned in the “etag”
https://www.quora.com/Has-Amazon-S3-ever-lost-data-permanent...
This is just a guess based on my own usage of it over the last 15 years at different companies.
But it would be because 1) they probably don't have many duped objects to begin with 2) the system to de-dupe items would be complex, error prone, increase latency, and just not worth it
That's crap reasoning. You have no way of knowing that.
> 2) the system to de-dupe items would be complex, error prone, increase latency, and just not worth
Would it really be that complex? I kinda doubt that.
Read latencies wouldn't be affected - except being improved.
Write latencies - the deduplication can happen after the write has been confirmed, so no extra latency added.
Whether or not it's worth it - I think it's very hard to to estimate the deduplication factor. My guess is 1.5x. At that point it would save 33% storage. That would instanly make a whole lot of extra complexity worth it.
I dunno, I kinda think you might be out of your depth on this one, based on that response.
Also: none of your few previous posts are tech-related. They're all about personal finance.
All things considered, I’m much more concerned about the centralization of eg browser technologies than this.
If you try to decentralize the data, or keep your data yourself, you end up with something that uses more resources, has worse durability, or is worse along some other axis.
I am not so worried about centralization yet. There is still healthy competition between different cloud storage vendors. My impression is that cheap storage is used to make other cloud services more attractive—which means that Amazon’s incentives here are somewhat aligned with mine (as a customer). Personally I am much more worried about the centralization of the shopping experience.
I don’t think filecoin will ever demolish a fantastic business like S3 based on stealing away customers. The interesting value proposition from the protocol is probably more about censorship resistance and adjacent issues.
https://www.allthingsdistributed.com/2010/05/amazon_s3_reduc...
We will start having trouble with random UUID collisions when we have made 2^64 of them, which is a huge number. It's about as many iron atoms as are in an iron filing.
It is also about 200,000 times 100 trillion, so AWS is just 18 doubling times away from the UUID system breaking down.
UUIDs are usually not simply random numbers. So it's possible to tell when a particular standard of UUID is used. Amazon's code can branch based on this detection, or even something simpler like "UUID-type", a new field that would be initialized by default to the old UUID type.
I simply don't see this as a problem for anyone.
Also as another commenter mentioned, if they are tied to a namespace prefix, that delays collisions.
[0]: https://vimeo.com/7330740
[1]: http://itc.conversationsnetwork.org/shows/detail5273.html
https://cacm.acm.org/magazines/2021/3/250706-a-second-conver...
https://news.ycombinator.com/item?id=26365873
[1] which you commented on, but others might not have seen.
I have a tough time taking him seriously reading this - I'm not sure if he's being intentionally ignorant or intentionally misleading.
I don't know of a single enterprise customer that doesn't have their data replicated to two datacenters, and then backed up to some other medium (tape or disk-based backup appliance) for anything business critical.
Their goal is 5-9's of UPTIME - the expectation is 100% data durability. You would get fired if you architected a solution keeping two copies of data in one datacenter.
Object storage absolutely has a place and I'm happy amazon was able to push a quasi-standard for the industry. Object storage prior to S3 was a hodge-podge of proprietary plays (EMC Centera, Bytecast, etc) and sort-of standards that nobody really used (CDMI). I just wish they did a better job of fairly representing it vs. turning every opportunity into a sales pitch spreading FUD about the alternatives.
He said:
> Most of our customers, if they have on-premises systems—if they're lucky—can store two objects in the same data center
You said:
> I don't know of a single enterprise customer that doesn't have their data replicated to two datacenters
His customers don't equal your enterprise customers. You are both right most likely.
Longtime consultant here with BCP/DR insight into 20+ large F500 companies... I think you would be seriously surprised at how common it is for even enterprise customers to not bother with multi-DC or even offsite backups. And even among the ones that do, many of the ones that are non-cloud-based are so immature at it that I would not put money on their backups being restorable if needed. And this is even more true for your “we’re a startup, we don’t have time to worry about backing up our data!” companies, which are in abundance.
For a small peak, go look at how many people were freaking out about losing their entire business due to the loss of a single OVH data center.
Of course, as a consultant I do naturally skew towards customers that need help with this stuff, so my perspective is probably biased towards the companies that are worse off in this regard. But they’re definitely out there.
You snipped out the part of the quote that answers your question.
"Most of our customers, if they have on-premises systems—if they're lucky—can store two objects in the same data center, which gives them four 9s. If they're really good, they may have two data centers and actually know how to replicate over two data centers, and that gives them five 9s. But eleven 9's, in terms of durability, is just unparalleled. And it trumps everything."
> I don't know of a single enterprise customer that doesn't have their data replicated to two datacenters...
Except, in AWS' case, each AZs (Availability Zones) is made up of upto 8 DCs (Data Centers), and each full-region has at least 3 AZs and 2 Transit Centers. Amazon S3 replicates data to 3 different AZs (which, I am guessing, is in addition to replicating it across DCs in a single AZ for 'eleven 9s').
With S3 cross-region replication durability may shoot up to 'sixteen 9s'? 100% durability, if it exists, is something the major Cloud providers are yet to offer?
ref: https://maisonbisson.com/post/object-storage-prior-art-and-l...
Replicating across two datacenters isn't "really good" though, that's considered table stakes.
>Except, in AWS' case, each AZs (Availability Zones) is made up of upto 8 DCs (Data Centers), and each full-region has at least 3 AZs and 2 Transit Centers. Amazon S3 replicates data to 3 different AZs (which, I am guessing, is in addition to replicating it across DCs in a single AZ for 'eleven 9s').
In AWS' case, the 8 DCs in an AZ are directly adjacent which isn't really useful for anything beyond metro (active/active) availability. I haven't seen a DR plan that doesn't have a hard requirement for the secondary copy of data to be outside of the metro loop/blast radius generally in a different state.
But I think the real lock-in lies in services like S3, making so many workflows simple and intuitive (while controlling your data, which is where the real lock-in lies)
How does something like that get handled? At some point isn't contacting this service really just contacting a handful of IP addresses? Are those just "regular" computers? Specialized hardware?
Also with the amount of centralization here, presumably they are the target of bad actors. How do they defend against things like DDOS's to provide their claimed uptimes?
A service that can handle tens of millions of requests per second can probably outscale most DDoS attacks without further engineering.
"Handful". S3 has 283 subnets allocated to it, biggest one being a /15, so there are some few hunderd thousand IP addresses reserved for S3. While impossible to say how many of them are actively in use, I imagine large fraction of them.
The AWS IP ranges are documented in https://ip-ranges.amazonaws.com/ip-ranges.json
And this is why GCP will never come close to AWS.