I'm so impressed at how S3 can roll out changes like this on a regularly frequent basis that impact an unimaginably large number of servers and stored data.
The only thing impressive about this is how long it’s taken them to make this the default. This has been available since 2011 and other cloud providers have had encryption at rest by default for years.
The read-after-write consistency was definitely an impressive change. This however, not so much.
Is Amazon-managed encryption (SSE-S3) mostly a checklist/regulatory thing? Like, is it mainly protecting against somebody walking out of a DC with a storage device?
Or some other kind of hack/data theft. That's what it looks like to me.
I don't see it as just a checklist thing: if Amazon ever gets hacked, you wouldn't want your data compromised as a result. Of course, if they manage to steal the keys too, it would still be.
Yes. This is really a non-announcement. S3 has had configurable encryption since at least 2011. And Google cloud storage has had default encryption at rest since 2013. For a decent explanation of encryption at rest check out https://cloud.google.com/docs/security/encryption/default-en...
I would have assumed that's always the case for all storage devices? Or am I naive? I would expected any cloud provider to do that, even if it's just to make disposing old drives easier.
What they won't tell you is that this will use the default KMS key, which cannot be shared across accounts. So your S3 bucket objects cannot be shared across AWS accounts by default, unless you setup a custom KMS key and "copy" everything in place (a standard AWS organisation is made of multiple accounts).
Does anyone have rough numbers on s3 scale? Most storage classes have redundant copies as well so 3x things maybe from that. YouTube / S3 - a few services seem like they must have crazy storage scale
If you're building one of these object storages for yourself (say, using Ceph or OpenStack Swift), a plausible configuration is 12 drives with 8 TB each, or 96 TB per node [1]. Let's round that to 100 TB for simplicity. You obviously want to spread your replicas across multiple datacenters, so let's just say you'll put 10 of these servers (i.e. around half a rack full) each into three different datacenters. That's a very modest deployment in terms of floor space, and yet already has a capacity in the low petabyte range (3 PB raw capacity, 1 PB net capacity).
On the AWS scale, object storage servers will probably be filling entire datacenters (or at least an appreciable portion of each datacenter), so it's very easy to see them being well into the exabyte (= million terabyte) scale, though probably not zettabyte (= billion terabyte) yet.
[1] Source: I help maintain an object storage for a private cloud. Disclaimer: These are not the exact numbers for our nodes, but in the same ballpark.
What does encryption at rest protect against with a Cloud provider?
I assume the read credentials must ipso facto decrypt the object. So encryption at rest protects you against an inside job at AWS (smash and grab drives) or government confiscation. Am I thinking about this correctly?
Yeah - physical drive theft I think. I assume that's well locked down (and a crap shoot whose data someone'd get if they stole some drives) but nice to be protected I guess.
I am thinking that it lowers their cost for recycling drives. If everything is encrypted at rest, you don’t need to go to crazy lengths to wipe the drives.
This is it. I help maintain the object storage for a private cloud. Because we have LUKS, preparing servers for decommissioning is done in a matter of minutes (once the servers have been removed from the cluster configuration, of course). We only have to stop all server processes, unmount the disks, overwrite the LUKS headers with /dev/random. Then server management kicks the nodes out of the Kubernetes cluster, then data center management takes care of the hardware.
When I started working for AWS as an SDE, I was hoping it’d be possible to visit a datacenter. I was surprised to find out that I’d require L11 (!) approval to so so! The only L11 in my reporting chain is Adam Selipsky.
I’m told the AWS data centers has red zones, which no harddrive can be taken out of, without being mechanically and violently destroyed first.
Encryption keys are kept physically and logically separate the majority of the time. Generally on HSMs that you can’t really read the private key from. It’s a huge improvement in security if one of their layers of defenses fails.
I read in the past, yet it's still unclear to me. It is a symmetric encryption, right? And as I understand keys are stored at AWS as well. Am I missing something?
I’d guess that the top tier of S3 physical storage is all SSD, and hardware-driven encryption on such devices is virtually free, as well as desirable for lifecycle reasons.
To allow this to double as ”user level” encryption you need to coordinate & manage the keys used vs. just picking something random when the drive is formatted. This is how Apple’s and others’ full-disk encryption has worked for years.
35 comments
[ 2.8 ms ] story [ 81.4 ms ] threadS3 is frankly amazing technology; probably the most robust and well rounded piece of cloud tech that exists.
This is an incredible piece of engineering. Almost as impressive as when they made S3 read-after-write consistent a couple of years ago: https://aws.amazon.com/blogs/aws/amazon-s3-update-strong-rea...
The read-after-write consistency was definitely an impressive change. This however, not so much.
- [x] Microsoft Azure (https://learn.microsoft.com/en-us/azure/storage/common/stora...)
- [x] Google Cloud Storage (https://cloud.google.com/storage/docs/encryption)
- [x] Backblaze B2 (https://help.backblaze.com/hc/en-us/articles/217973398-Encry...)
- [x] Wasabi (https://wasabi-support.zendesk.com/hc/en-us/articles/1150016...)
- [ ] DigitalOcean (https://www.digitalocean.com/community/tutorials/how-to-crea...)
- [ ] DreamHost (https://help.dreamhost.com/hc/en-us/articles/360001089163-En...)
- [ ] Linode (https://www.linode.com/docs/guides/server-side-encryption/)
I don't see it as just a checklist thing: if Amazon ever gets hacked, you wouldn't want your data compromised as a result. Of course, if they manage to steal the keys too, it would still be.
> Amazon S3 now automatically applies S3 managed server-side encryption (SSE-S3) as a base level of encryption to all new objects added to S3
On the AWS scale, object storage servers will probably be filling entire datacenters (or at least an appreciable portion of each datacenter), so it's very easy to see them being well into the exabyte (= million terabyte) scale, though probably not zettabyte (= billion terabyte) yet.
[1] Source: I help maintain an object storage for a private cloud. Disclaimer: These are not the exact numbers for our nodes, but in the same ballpark.
I assume the read credentials must ipso facto decrypt the object. So encryption at rest protects you against an inside job at AWS (smash and grab drives) or government confiscation. Am I thinking about this correctly?
I’m told the AWS data centers has red zones, which no harddrive can be taken out of, without being mechanically and violently destroyed first.
To allow this to double as ”user level” encryption you need to coordinate & manage the keys used vs. just picking something random when the drive is formatted. This is how Apple’s and others’ full-disk encryption has worked for years.