Ask HN: Why is there not more concern about the physical security of Cloudflare?
Using Hetzner and Azure, we trust that our unencrypted in-memory data and business logic are housed in professional data centers with strong physical security measures. However, Cloudflare has built its Workers and serverless offerings on top of its Cache/CDN and anti-DDoS infrastructure, which operates out of questionable ISP and IXP colocation facilities in various jurisdictions with dubious standards.
As an EU-based company, whenever we ask Cloudflare about the physical security of their edge locations, they consistently refer to encryption in transit and at rest—measures that do nothing to address threats like RAM interception or other physical security vulnerabilities in these questionable facilities. Moreover, when we raise these concerns, they attempt to upsell us on their Enterprise EU/FedRAMP offerings. Cloudflare has also deliberately restricted our ability to block non-Enterprise Workers, KV, and R2 from specific regions, leaving us with limited control over where our data is processed.
57 comments
[ 2.4 ms ] story [ 142 ms ] threadIf I were to guess CF is locating their PoPs at cheap peering points and the reason they are evading the question is because other customers in the facility have physical access to their equipment, which is both an expensive problem to solve and something that is not even remotely allowed at a real cloud provider.
Even your cheapest of colo's offer locked cage areas. For someone on Cloudflare's scale, the cost is trivial.
I've been inside some really "low rent" colo's and even they would provide an escort to unlock your cabinet.
Obviously standards/expectations will vary from DC to DC. I'd wager the situation might be different in some of the smaller countries CF operates in around the world though.
I've also had DC employees, without authorisation: reboot my machines, give themselves access rights and then tamper with my systems
admittedly the latter was a long time ago, before any of this stuff was considered critical infrastructure
these days they'd probably end up in prison
Notably, while Cloudflare has CDN edge locations in countries like China and Russia they don't appear to run workers there.
EDIT: I was wrong - I misinterpreted the map. A solid border circle around a location indicates "Worker-only Datacenter" (see the map legend) and there are indeed locations with those solid borders in Russia (including Moscow and Yekaterinburg) and China (Haidong, Lanzhou and more).
I doubt we could get them on the record for this, but I suspect this may be very deliberate. Maybe CDN edge locations can be run completely securely with forwarded encrypted traffic, while workers are at a higher risk of physical attack.
Workers run everywhere. On every location shown on that map. Granted, you need a special license and Enterprise feature to use the China ones.
However, these countries are now increasingly using Cloudflare Workers without realizing that this creates total uncertainty about where their data and business logic might end up—including potentially adversarial countries like Russia and China.
Data in transit and data at rest can be encrypted. But Clouflare Workers is neither. It's data being processed in-memory where it will be unencrypted.
Unless it's below L7/HTTP, the possibility of doing it securely is very questionable.
Up until very recently it was very trivial to conduct "domain fronting" of sorts but with colocations in hostile locations. So Chinese or Russian servers decrypting your TLS traffic no questions asked, and that was with their premium (DLS) offerings.
I suspect that if you're in a hostile country where CF announces their prefixes locally, it's still doable. Unfortunately that's a bit more difficult to test than it was before.
Cloudflare's Data Localization Suite gives you some control over where things run: https://www.cloudflare.com/data-localization/
Durable Objects specifically support jurisdiction restrictions, to keep your data strictly inside EU (for GDPR) or FedRAMP-compliant locations. https://developers.cloudflare.com/durable-objects/reference/...
The current set of locations supporting Durable Objects is mostly a function of where we have the resources available to operate the distributed database which we use as the storage back-end for first-generation DOs. We recently announced a new storage backend for Durable Objects, which is based on SQLite and a lot of in-house tech instead of an off-the-shelf distributed database. This new backend gives us a lot more flexibility in terms of locating data. There's a lot of work to do, but I would expect that we'll have more than just EU and FedRAMP as jurisdictions eventually. https://blog.cloudflare.com/sqlite-in-durable-objects/
Not specific to Workers: China is special. Your site will not be served from China at all (and your Workers will never run there) unless you've explicitly signed up for China network access. (I think the Chinese government requires a special licence for it? But I'm not an expert on this.) https://www.cloudflare.com/application-services/products/chi...
I don't personally know the current situation in Russia, or how physical security is managed in general (it's not what I work on, personally). I've heard some talk of trying to get the team to write a blog post about it, which I too would be interested to read!
For cloudflare to refer to encryption in transit is quite something, given that cloudflare is the largest MITM service in the planet. SO much traffic goes through them and they have it all in cleartext due to terminating TLS.
Remember "Encryption removed and added here".
I recovered data of mine from ram many times
I'll mention that unencrypted in-memory data is accessible in pure-software ways too (which usually require root) - debugging tools like GDB, VM snapshots, memory forensics tools, /proc/mem, etc. So unencrypted in-memory data is a problem too if root access (including both by infra-provider insiders and external attackers) is part of your threat model.
> which operates out of questionable ISP and IXP colocation facilities in various jurisdictions with dubious standards.
Why do you say that? Do you have signals that their colo facilities are less secure than they should be, and/or that Cloudflare hasn't gotten those facilities to beef up their security as part of their contract? Again, not saying this to defend Cloudflare. I just hadn't heard this before.
> Moreover, when we raise these concerns, they attempt to upsell us on their Enterprise EU/FedRAMP offerings.
That's going to be the case with almost all providers in the space. If you're asking for special treatment, you're going to have to pay for it. I don't mean that to insult you. At a past job, for various reasons we had strict compliance obligations that our data could not be accessed outside of the US. Some of our vendors used offshore tech support who'd have access to our data, and a couple times we faced a decision: pay that vendor $$$ to special-case our support setup to meet our requirements, or choose another vendor.
> Cloudflare has also deliberately restricted our ability to block non-Enterprise Workers, KV, and R2 from specific regions, leaving us with limited control over where our data is processed.
Same. Those fine-grained controls are often going to be an enterprise feature.
Again, I'm not saying this to defend Cloudflare in particular. They have their own paid spokespeople. I'm not one. Nothing you've said sounds particularly egregious though. If your data is sensitive enough that you're legitimately worried about someone sneaking in undetected and intercepting RAM, prepare to pay the enterprise tax with all your cloud vendors.
> infrastructure, which operates out of questionable ISP and IXP colocation facilities in various jurisdictions with dubious standards
What are these questionable facilities? When CloudFlare has installed those servers and software, and they have also made software that manages those servers, what is the problem?
CloudFlare has written some articles about some of those very many security protections they have. But that is a very lot of technical detail to explain, so if that is so important, their Enterprise/FedRAMP offerings fund CloudFlare to make possible to explain that amount of detail. But question is, do you really need that amount of detail? How much you have expertise to build same amount of security protections? Isn't it better to use CloudFlare Workers security features to concentrate on building your app? Alternative is to get your own bare metal servers, and manage them yourself.
With CloudFlare Workers, they have security features to keep code and data of each customer separate from each other.
Here is Satya's May Post, https://blogs.microsoft.com/blog/2024/05/03/prioritizing-sec...
If physical theft is a concern, how do they prevent someone from hijacking the key distribution process?
There are some emerging technologies like SEV-TIO and PCI-IDE which attempt to extend the trust chain down into PCIe devices, allowing trusted devices to form a relationship with trusted code in a secure execution environment: https://www.amd.com/content/dam/amd/en/documents/developer/s...
Anyway, the best protection here is to terminate protocol or application-level encryption (TLS or whatever sits on top of your protocol) inside of the secure environment, so that decryption only occurs through encrypted memory. It's slower, but this way an attacker sitting as a RAM snoop could only see encrypted network traffic hit the DMA buffers, which wouldn't help them much.
Basically depending on the scenario, DMA region will either be unencrypted or transparently encrypted/decrypted.
https://blog.cloudflare.com/securing-memory-at-epyc-scale/
> Gen 12 Servers are currently deployed and live in multiple Cloudflare data centers worldwide
Judging from this, the fleet isn't uniform. How uniform is the fleet? Is it possible to be running on Gen 9 in some locations?
so you want the cheap plan with the features from the premium plan?
https://blog.cloudflare.com/anchoring-trust-a-hardware-secur...
1) Nothing
2) Visitor logs
3) Locks and alarms on your racks, and/or (if you have enough) the rooms they are in. Remote monitoring is pretty common.
4) First party human security professionals
I don't have any special knowledge of Cloudflare's set-up, but 2 and 3 are by far the most common. Lacking #1 means your hardware just gets stolen. #4 is too expensive. #2 and #3 are where most people are, so probably something around there?
what does this mean?