(Cloudflare team here) cloudflared proxies the traffic through the Cloudflare network to the service behind Access so that Cloudflare can ensure the request is authenticated first and then issue a token to the client through cloudflared.
sshd already has a reasonable authentication protocol (several in fact, including public keys of various kinds) that is already tunneled through an encrypted channel. What additional value does cloudflared provide?
It's a bit like saying HTTPS isn't good enough, so let's tunnel HTTPS inside HTTPS -- unless I misunderstand its purpose.
It's because the IdPs most organizations use don't have the type of SSH flow you're talking about. For Cloudflare to authenticate you, you first have to go through your Okta, Google Apps, etc login flow which is browser-centric.
There are actually quite a few aspects of your blog that I think we will emulate in the near future, with a twist, to solve more similar problems. This problem set was super different though.
Adding a public facing SSH interface to our production hosts was a bit of a non-starter and we would have had to hack together auth on top of that (not just for us, but for our customers too). That's a lot of additional surface area and operational burden we didn't want.
Imagine you have an application like GitHub Enterprise or Phabricator, something that has a web interface and SSH for git access. You might trust SSH, but do you trust the web stack to not have vulnerabilities?
I don't know about you, but I don't trust running third party apps, I won't publicly expose them. Instead I put them behind authenticating proxies, like Cloudflare Access[1], Google Identity-Aware Proxy[2], Bitly oauth2_proxy[3], and Buzzfeed SSO[4]. With an authenticating proxy an attacker can't even hit the web application until authenticating, so they can't exploit a vulnerability in the web application. They will have to pop the auth proxy first, which have less surface area, and then pop the web application.
Applications that combine HTTP and SSH complicate things because the typical HTTP proxy won't proxy your SSH (git) connections, and the web authentication flow doesn't work with SSH.
Cloudflare Access provides the auth proxy, cloudflared does the web authentication flow for a command line app and passes SSH traffic through Cloudflare Access.
Can you explain why this setup is better than LDAP/Ansible/Chef (to provision user accounts) + ssh keys/CA-generated certificates for ssh authentication? This setup is well known in the art and isn't rife with reported vulnerabilities. The value of using a web flow to authenticate isn't clear to me because you have to provision user accounts on Unix systems anyway to have reasonable accountability.
Every time I see a post by Cloudflare, I get more and more nervous.
Why? Because each time they introduce something, they end up at the controls.
What if, if, their internal network is breached; or an unhappy employee is at the wheel?
Not only would it be possible for them to use everything at their disposal, but they can start to sell user data; internal ports exposed.
Considering bugs are a fact of life for developers; how long will it be until there is a big hole? Heartbleed was one; what's the next?
the only reason it feels like this on HN is because cloudflare is much more focused on "retail" level marketing than the competitors. fastly is roughly the same size, akamai and edgecast are bigger than both. all of them are gnats compared to the big three cloud providers.
One of the fundamental truths or the move to the cloud has been large organizations which can focus on security tend to do it better than each of us doing our best alone. It just takes too many people to keep a modern system secure, that's unfortunate, but it seems true. All of the security failures we hear about tend to be in bespoke systems, not GCP or AWS.
For one, the idea of "delegate security to a central authority because they know what they are doing" is one that has failed so catastrophically in history it's surprising anyone still considers it a convincing argument. If there is one thing humanity should have learned, it's that centralizion of power is itself a major security problem. Electing dictators to solve security problems does not work.
Then, "we have to give all power to cloudflare or we are on our own" is obviously a false dichotomy. No, people can work together on fixing security problems without the need to delegate power to a central authority.
Also, it isn't hard to keep "a modern system" secure. It's just that noone cares, in part because they think they can just buy security as a product that they somehow plug into their system to make it secure. But that is just a completely mistaken approach. You can buy "IT security" as much as you can buy "car safety". If the engineering of your car is shoddy, no add-on will make if safe, and it's exactly the same for IT systems--and if the engineering is solid, there is no need for "safety add-ons".
BTW, one particular kind of "security" that large organizations are good at is controlling PR. It's just that that is not in your interest as their customer. It's just one of many conflicts of interest.
I've thought about this as well, but in a different sense. Think if cloudfare were actually the NSA. Seeing as how ssl terminates at their edge server before they pass it off to your actual server (whether they re-encrypt it or not), they have access to all of the decrypted traffic in transit before it reaches your server, don't they?
> When you use Cloudflare, we must decrypt the data at our edge in order to cache and filter any bad traffic. Depending on the SSL setting from the options above, we may re-encrypt or send it as plain text. (full vs. flex) Since each certificate needs a dedicated IP address, we add your domain name and wildcard (*.domain.com) domain in the SAN (Subject Alternative Name) to the certificate.[1]
I think it’s highly likely that Cloudflare is at least associated with the NSA/CIA, maybe more. Also Google, Facebook. One of the reasons I don’t own a Home/Alexa/etc. I value privacy.
It would be perfect for them. Instead of having to hack, just get them to come to you under the guise of DDoS protection.
"Hey, everyone's starting to encrypt and we can't get the data as easy as we used to."
"Well, DDoS is a big problem out there. What if we offered a DDoS protection service because very few companies can afford the huge pipes needed to mitigate those attacks."
"Perfect! We can offer it to free for most sites, and offer "free" SSL encryption that terminates on our end."
"Yes! We'd gain their trust by offering such a service and they'd COME TO US begging for help."
I've wondered about LetsEncrypt as well. Since they're they ones generating these "free ssl certs", don't they then have the means to the decrypt the data? Yes, I'm cynical, but I think we need to be.
Same scenario as above. "Hey, we can just offer free certs and they will just give us the keys."
They couldn't decrypt, but as a CA, like any other trusted CA, they could man-in-the-middle the traffic, read it, then pass it along to your server, re-encrypted.
When a Chrome (and soon to be also Safari, Firefox) connects to a web server over TLS it expects to see SCTs, Signed Certificate Timestamps from some qualified logs (in Chrome's case at least one of these logs must be operated by Google and at least one must not). In most people's setups they get these baked inside their certificates, but you can do it in other ways and that'll work too.
The SCTs can only be produced by these logs, and only when the log is shown the tbsCertificate data. The intent is that the log service promises to log all certificates for which it provides a timestamp. A Merkle Tree is used so that we can examine a log and obtain proof that it does indeed contain every certificate we've seen SCTs for.
So, your conspiracy needs to involve the Certificate Authority (Let's Encrypt in this case), the CT logging team at Google, and at least one third party log operator. Or it would be detected when it was tried.
At least in Let's Encrypt's case, I thought the free SSL cert was handled as a normal CSR request; the private key should never leave the server the certificate is issued for, assuming the client performing the CSR isn't compromised in some way. All Let's Encrypt's validator service should get is a challenge signed by the private key, not the key itself.
A CA compromise is still very bad, but mostly because it allows an attacker to issue valid certificates and set themselves up as a MitM more easily. Let's Encrypt is still subject to this kind of attack, but so is every other CA; it's not a new threat. That can be reasonably worked around with certificate pinning and other techniques.
NSA always wanted to be a so-called global observer. CloudFlare is in the perfect position to be the global observer and it sell it as a feature, not a bug.
NSA budget is big enough that it could buy CloudFlare 10 times a year, or have thousands of undercover agents.
Every "target" website that is not using CF can be cheaply DDoSsed until it switches to CF or disappear.
Are you actually concerned that NSA/CIA has any interest in what you do? If you’re not into terrorism or espionage, I guarantee you are not that important...
Is this one of those "if you're not doing anything wrong you have nothing to worry about" arguments? If I'm (we're) not that "important", why do they need the data in the first place? In order to determine if I'm (we're) "important" enough, instead of having some sort of evidence/proof to begin with like the constitution spells out. It's the biggest fishing expedition in the world, and can also be used in the future based on past data. They have no business looking at my data unless they have a legit reason to, as in I've done something and they have enough evidence to take before a judge and get a warrant. Due Process. Not the other way around.
One interesting benefit of using such a large provider is that if a breach or bad behavior were to happen large portions of the internet would suffer, and you'll receive a much smaller portion of the blame. The S3 outage last year was terrible for my employer, but hardly anyone noticed because much more significant sites were similarly suffering.
It's amazing they are able to use local browser technology(via the popup login) to allow for using a public network as though it were private, virtually of course.
Yeah, but are you Enterprise sales teams still a complete bag of useless dicks? Yes. Is your pricing still laughable? Yes. Is 'last decades security' still better than handing off internal network control to a third party? Yes. As usual, you only solved how to make Cloudflare more money, and not anything g technically impressive.
I do not want to use Cloudflare, because I will want to control it by myself, and also because Cloudflare fails to have some stuff that I would use and also it has stuff that I do not want. Also, SSH and HTTPS are not necessarily the only protocols that you might want.
42 comments
[ 3.1 ms ] story [ 116 ms ] threadIt's a bit like saying HTTPS isn't good enough, so let's tunnel HTTPS inside HTTPS -- unless I misunderstand its purpose.
Adding a public facing SSH interface to our production hosts was a bit of a non-starter and we would have had to hack together auth on top of that (not just for us, but for our customers too). That's a lot of additional surface area and operational burden we didn't want.
BTW we should catch up over a beer sometime =]
I don't know about you, but I don't trust running third party apps, I won't publicly expose them. Instead I put them behind authenticating proxies, like Cloudflare Access[1], Google Identity-Aware Proxy[2], Bitly oauth2_proxy[3], and Buzzfeed SSO[4]. With an authenticating proxy an attacker can't even hit the web application until authenticating, so they can't exploit a vulnerability in the web application. They will have to pop the auth proxy first, which have less surface area, and then pop the web application.
Applications that combine HTTP and SSH complicate things because the typical HTTP proxy won't proxy your SSH (git) connections, and the web authentication flow doesn't work with SSH.
Cloudflare Access provides the auth proxy, cloudflared does the web authentication flow for a command line app and passes SSH traffic through Cloudflare Access.
[1] https://www.cloudflare.com/products/cloudflare-access/
[2] https://cloud.google.com/iap/
[3] https://github.com/bitly/oauth2_proxy
[4] https://github.com/buzzfeed/sso
Why? Because each time they introduce something, they end up at the controls.
What if, if, their internal network is breached; or an unhappy employee is at the wheel? Not only would it be possible for them to use everything at their disposal, but they can start to sell user data; internal ports exposed.
Considering bugs are a fact of life for developers; how long will it be until there is a big hole? Heartbleed was one; what's the next?
(not counting the verizon factor w/EC)
Cloudflare already had one (cloudbleed).
For one, the idea of "delegate security to a central authority because they know what they are doing" is one that has failed so catastrophically in history it's surprising anyone still considers it a convincing argument. If there is one thing humanity should have learned, it's that centralizion of power is itself a major security problem. Electing dictators to solve security problems does not work.
Then, "we have to give all power to cloudflare or we are on our own" is obviously a false dichotomy. No, people can work together on fixing security problems without the need to delegate power to a central authority.
Also, it isn't hard to keep "a modern system" secure. It's just that noone cares, in part because they think they can just buy security as a product that they somehow plug into their system to make it secure. But that is just a completely mistaken approach. You can buy "IT security" as much as you can buy "car safety". If the engineering of your car is shoddy, no add-on will make if safe, and it's exactly the same for IT systems--and if the engineering is solid, there is no need for "safety add-ons".
BTW, one particular kind of "security" that large organizations are good at is controlling PR. It's just that that is not in your interest as their customer. It's just one of many conflicts of interest.
> When you use Cloudflare, we must decrypt the data at our edge in order to cache and filter any bad traffic. Depending on the SSL setting from the options above, we may re-encrypt or send it as plain text. (full vs. flex) Since each certificate needs a dedicated IP address, we add your domain name and wildcard (*.domain.com) domain in the SAN (Subject Alternative Name) to the certificate.[1]
[1] https://support.cloudflare.com/hc/en-us/articles/204144518-S...
"Hey, everyone's starting to encrypt and we can't get the data as easy as we used to."
"Well, DDoS is a big problem out there. What if we offered a DDoS protection service because very few companies can afford the huge pipes needed to mitigate those attacks."
"Perfect! We can offer it to free for most sites, and offer "free" SSL encryption that terminates on our end."
"Yes! We'd gain their trust by offering such a service and they'd COME TO US begging for help."
I've wondered about LetsEncrypt as well. Since they're they ones generating these "free ssl certs", don't they then have the means to the decrypt the data? Yes, I'm cynical, but I think we need to be.
Same scenario as above. "Hey, we can just offer free certs and they will just give us the keys."
The SCTs can only be produced by these logs, and only when the log is shown the tbsCertificate data. The intent is that the log service promises to log all certificates for which it provides a timestamp. A Merkle Tree is used so that we can examine a log and obtain proof that it does indeed contain every certificate we've seen SCTs for.
So, your conspiracy needs to involve the Certificate Authority (Let's Encrypt in this case), the CT logging team at Google, and at least one third party log operator. Or it would be detected when it was tried.
No, as you alluded to, my conspiracy only needs that the user uses any of the many browsers besides chrome.
A CA compromise is still very bad, but mostly because it allows an attacker to issue valid certificates and set themselves up as a MitM more easily. Let's Encrypt is still subject to this kind of attack, but so is every other CA; it's not a new threat. That can be reasonably worked around with certificate pinning and other techniques.
NSA budget is big enough that it could buy CloudFlare 10 times a year, or have thousands of undercover agents.
Every "target" website that is not using CF can be cheaply DDoSsed until it switches to CF or disappear.
A level of diversity is necessary for resiliency and security.
This is true for biological systems, social organizations, complex technologies and of course software as well.