It’s always nice reading about a company taking security seriously, and Cloudflare has some decent write ups for their hardware/software security, but I sure wish they would take other forms of security more seriously.
I visited Cloudflare’s Austin office. The door to the office is an old unlocked door with a glass pane and an old deadbolt. There is no reception desk or even anyone watching the door. I was able to walk in completely unnoticed and walk around for a couple minutes trying to get someone’s attention to figure out where I needed to go (not the best job interview experience, but that’s a different topic), while desks full of unattended and unlocked computers were fully available to me.
For a company that bills itself as an internet security company, it wasn’t very inspiring security.
edit to add: this was over a year ago so it’s possible things have improved since then. My understanding is that the Austin office is relatively new so maybe at the time they were still working out the kinks (still not great security but more understandable at least)
One of the folks I talked to while there was one of the senior security team members, and I mentioned it to him during the interview but felt like it was brushed off (honestly that’s not that uncommon, I’ve worked in security for years and while software people are always really critical on software security, they really don’t care about physical security). I’ve been back once and nothing had changed at that time, but that was over a year ago so hopefully in the past year things have improved.
I went ahead and copied your comment to our internal security team. We do take this stuff seriously and I was surprised to hear about unlocked machines and easy access.
That’s good to hear! I edited my original comment to say that this was over a year ago and was still when the Austin office was relatively new so perhaps it was just the effects of adjusting to a new space, and hopefully things are different now. Still, glad that it’s taken seriously.
While it may actually mean something here... every CTO says they care. The more candid ones will follow up with comments about prioritization and profitability and cost control. But what matters are the actual actions they take. Now again, I gives CF better than usual chances, just... I wouldn't take it too much to heart.
I have been a customer and follower of CF from almost day one. I also know a couple employees. While they haven't been perfect over the years, I tend to trust him on his word.
We had a guy from another company walk into our office behind someone by mistake, find our it support people behind more internal doors, and asked them for help with with his laptop. Still just as easy, we're a large fintech company.
I work at Cloudflare Austin. I can't speak to this specific experience, but we have badges and a security person like every other office. It is true we don't have a reception desk in the lobby as it's a shared building.
When did you join the Austin office? I'm guessing things changed over time. When I visited there definitely was not a security person (not for the building nor for the Cloudflare suite in the building) and there were no badges. I was able to walk into the Cloudflare office through the unlocked door and walk around inside the office for at least a couple minutes before anyone paid any attention to me (and even that was only because I was trying to get someone's attention to tell me where to go).
Regarding the reception desk, it shouldn't matter if it's a shared building. Unless it's a small company (CF is not), even in shared buildings it's common to have at least one person sitting at a desk in the office suite to act as a gatekeeper and assist visitors, etc.
I really don't see the point of encrypting memory this way, with what boils down to a global key.
Anything breaking security of your OS kernel or some trusted process will still be able to steal all your data, since it's executing on the CPU with access to the keys.
Any attacker with physical access can still grab a hold of some PCIe or LPC port somewhere and try to convince your IOMMU to let it DMA out all memory. Or just manipulate the BIOS to install a permanent rootkit.
Why is this not using per-page or per-process keys, in some kind of secure storage? That'd actually add another barrier even if the kernel is already compromised. And the CPU does actually support this... but only for VMs with SEV, it seems. Would be nice to extend this.
This prevents an attacker from ripping out your RAM and freezing it so they can pull secrets from it, which isn’t really an “exploit” per se. I’m sure they have other mitigations for those kinds of things. Or at least I hope they have something in mind, since they didn’t seem to be hiring for security currently…
> I really don't see the point of encrypting memory this way, with what boils down to a global key.
SME was initially developed for game consoles[1], and was designed to protect security keys used for DRM against hardware probing. The feature makes sense in that environment, or any environment where the computer may realistically fall into the hands of an adversary (e.g., a laptop) while still running.
Plain SME on a server doesn't really make sense for DRAM unless your threat model needs to protect against extremely sophisticated attackers (although if there's no real performance hit, you may as well just enable it). However, it would be useful on a system with NVDIMMs.
> although if there's no real performance hit, you may as well just enable it
Even if it is no-cost performance wise, it still costs energy, which translates to heat and CO2.
> protect against extremely sophisticated attackers
I agree, and I'm very doubtful whether any attacker with this level of sophistication and access (i.e. physical to the DIMMs) won't have quite a few other venues of attack that are still viable.
If each resident VM is a(n attested) hardware enclave with its own memory encryption key (stored in a software-inaccessible register), a privilege escalation attack from one VM would not be able to yield the contents of the other VMs. The best an adversary could do is deny service to the other VMs, but that's always allowed by the threat model.
To my knowledge, VPSes are generally container'd these days, not VM'd, so SME/SEV doesn't actually work. But yes, this seems to be single-targeted at a specific VM threat.
(Add: another reason to extend this to per-process or per-page encryption, since that would work on containers too.)
That's the right idea. How do we allow per-process or per-page encryption of memory with hardware attested keys? Imagine if you can plug in your own key store from an HSM. This starts to really create some interesting defenses against very common security enterprise threat modeling.
> To my knowledge, VPSes are generally container'd these days
Not true. AWS is still largely on Xen, and most others are using KVM. Serverless (Lamdba) is a different story, but EC2 instances are largely Xen VM's, not containers.
I have been following this space very closely as we're planning some killer roadmap use cases at my place of employment. This has been in the works for sometime after the interest in Intel's SGX + AMD's SEV environment that delivered subpar results.
Running "serverless" functions on https://github.com/bytecodealliance/lucet which is running in these enclaves provides a relatively simple way to offer "confidential computing" to offer an alternative against "semi-honest" (or sometimes known as "honest-but-curious") adversarial threat models.
32 comments
[ 3.3 ms ] story [ 86.6 ms ] threadI visited Cloudflare’s Austin office. The door to the office is an old unlocked door with a glass pane and an old deadbolt. There is no reception desk or even anyone watching the door. I was able to walk in completely unnoticed and walk around for a couple minutes trying to get someone’s attention to figure out where I needed to go (not the best job interview experience, but that’s a different topic), while desks full of unattended and unlocked computers were fully available to me.
For a company that bills itself as an internet security company, it wasn’t very inspiring security.
edit to add: this was over a year ago so it’s possible things have improved since then. My understanding is that the Austin office is relatively new so maybe at the time they were still working out the kinks (still not great security but more understandable at least)
Regarding the reception desk, it shouldn't matter if it's a shared building. Unless it's a small company (CF is not), even in shared buildings it's common to have at least one person sitting at a desk in the office suite to act as a gatekeeper and assist visitors, etc.
Anything breaking security of your OS kernel or some trusted process will still be able to steal all your data, since it's executing on the CPU with access to the keys.
Any attacker with physical access can still grab a hold of some PCIe or LPC port somewhere and try to convince your IOMMU to let it DMA out all memory. Or just manipulate the BIOS to install a permanent rootkit.
Why is this not using per-page or per-process keys, in some kind of secure storage? That'd actually add another barrier even if the kernel is already compromised. And the CPU does actually support this... but only for VMs with SEV, it seems. Would be nice to extend this.
I don't know how often forensics teams bother freezing the memory; it's not always necessary.
SME was initially developed for game consoles[1], and was designed to protect security keys used for DRM against hardware probing. The feature makes sense in that environment, or any environment where the computer may realistically fall into the hands of an adversary (e.g., a laptop) while still running.
Plain SME on a server doesn't really make sense for DRAM unless your threat model needs to protect against extremely sophisticated attackers (although if there's no real performance hit, you may as well just enable it). However, it would be useful on a system with NVDIMMs.
[1] https://www.crn.com/news/components-peripherals/amd-s-xbox-p...
Even if it is no-cost performance wise, it still costs energy, which translates to heat and CO2.
> protect against extremely sophisticated attackers
I agree, and I'm very doubtful whether any attacker with this level of sophistication and access (i.e. physical to the DIMMs) won't have quite a few other venues of attack that are still viable.
This is extremely useful for cloud providers that provide Virtual Private Servers.
(Add: another reason to extend this to per-process or per-page encryption, since that would work on containers too.)
Not true. AWS is still largely on Xen, and most others are using KVM. Serverless (Lamdba) is a different story, but EC2 instances are largely Xen VM's, not containers.
Yesterday, Intel announced that Multi-Key Total Memory Encryption is coming to CPUs (https://arstechnica.com/gadgets/2020/02/intel-promises-full-...) and it's very exciting to see this space heat up.
If you're interested in embedded systems and data security is interesting to you, we are hiring! (send me an email - mahmoud @ linux dot com).
Again, if this interests you, we would love to talk to you! mahmoud -@- linux dot com!