I work at a nonprofit and I've had great success pushing LE there. We used to have an org-wide wildcard certificate and we'd buy additional certs as necessary, and at first there was some serious skepticism about "free certs" even though I pointed out the significant players behind it. The IT people are now on board and looking to spread it to more projects.
It's been great for us in many respects. We no longer have certificates shared between servers, which removes the vulnerability to DROWN and some other problems. We also tend to serve multiple domains from a single IP, which requires a cert that covers all domains. LE lets us trivially get a cert to cover multiple domains, which would be especially expensive with project managers who can't make up their minds on what domains they want to use. And, we get the usual benefits of LE - effective forward secrecy in case someone obtains a single key/etc.
There was a little bit of pain at first getting the renewal infrastructure set up properly. My setup still isn't quite perfect, so far the standalone certbot option is the least painful for me. I have a cronjob which shuts down httpd, starts up certbot-standalone, does the renew, and restarts httpd, so we are down for an additional ~5 seconds per day while we renew. Not the end of the world for our usage.
Depending on what your non-profit is in support of, Koalephant (my company) may be able to provide some pro-bono hours to assist/advice on that 'not quite perfect' setup, if that's at all of interest to you.
My email is in my profile if you want to get in touch.
OT: Anyone know the best architecture using LE with a Docker Swarm cluster. I let one server do the challenge and renew the certificates with cron every three months. Those certificates are synced with every Docker Swarm node through Docker secret.
Once the certificates got renewed the secrets are not working anymore because they must be immutable.
50 Million is a massive number for sure but its a drop in the ocean when you see the number of sites out there. As a user i hope you hit even more bigger numbers in the days to come! :)
Is there a form letter available for sites that don't have certificates or are using self-signed/invalid certs? It would have a brief introduction, background on LE, links to getting started docs, etc. Something like:
"Hi, I noticed your website does not support HTTPS. Have you heard about Let's Encrypt? Let’s Encrypt is a free, automated, and open Certificate Authority. [...]"
I have a couple of Let's Encrypt certs for some subdomains, but honestly I've left some others on plain HTTP simply through laziness.
LE imposes a requirement to renew certificates every 90 days, which is a nuisance. Logging-in, finding my process crib-sheet, running the update requests, updating DNS records, running the validation requests, shifting certs to servers, restarting services... all because the LE project wants to evangelise short certificate lifespans, which seems to have been bolted onto the original goal of 'encrypt all the things' because of someone's personal conviction.
I do wish they'd offer a longer optional lifespan. Even 365 days would be a huge improvement. Until then I really can't recommend it to less-technically-inclined colleagues; I just tell them go and buy a one-year cert from Gandi, chuck it into the ssl directory and forget about it until the renewal reminder arrives 11 months later.
It's almost the same amount of work to deploy a one-year cert as it is to set up an automatic LE cert which is going to save you from renewing the one-year cert each year, and forgetting etc.
> which is a nuisance. Logging-in, finding my process crib-sheet, running the update requests, updating DNS records, running the validation requests, shifting certs to servers, restarting services
You're doing it wrong. There are many ways to use the ACME protocol (and Lets Encrypt), I think pretty much anyone will agree this is simply the wrong way to do it.
The ACME protocol is an API (as opposed to the CA's "upload a manually created, signed CSR, then wait for an HTML email and click the link in it and then download this file" process) so that it can be automated, and hands-off.
I love LetsEncrypt - the only problem I've hit is the rate limits (in particular 20 new certificates per registered domain per week, when moving domains). I guess I could have avoided this by combining names into a single certificate, but I wonder why there's not an option to pay/donate to bump the limit temporarily: when I hit the limit I would gladly pay $50 or $100 instead of going through the manual process with a legacy CA. Even though wildcard certs will make this better, the fear of accidentally hitting a limit and being unable to renew certificates remains a concern for me.
This requires sharing the same cert and key for everything. Lots of software doesn't really like this way of doing things, I've been bit by the 20 limit too - and yes, I could have swapped to SANs, it wouldn't have fit with the deployment tooling for all the different things!
Yes, with Kubernetes, the patterns used are a little different than if you were to setup TLS and haproxy by hand (I actually use nginx, but it doesn't particularly matter).
You deploy software, that software declares it needs TLS for its endpoint, and certs are obtained.
While moving from one cluster to another, a new deployment of everything was done - about 20 different services - a few minutes work with Kubernetes. However, the fetching of TLS certs stopped working for the very last service!
Yes, I should have migrated the certs, however, the migration was happing due to a failure - I couldn't access the old certs.
I now make sure to backup those secrets rather than rely on re-issuance.
Still, for a usecase like Kubernetes, 20 certs per domain per week is limiting. I do totally understand that the quotas are in place because cert issuance is expensive, but I'd happily pay a yearly subscription fee to LE to cover the costs and help fund them if the option was there!
This is kind of what I expected too. Surely that component of the architecture can (even if the OP's doesn't) be smart enough to handle various certificate scenarios - there must be someone using it with a wildcard certificate, for instance.
Each independent application is isolated from each other, sharing a TLS cert/key among them all means we're weakening security - again, the Kubernetes patterns here would mean we need to duplicate the secret data into each applications namespace, allowing a compromise of one to compromise the TLS of all.
There are legitimate reasons to not share a single TLS cert for everything in an environment like Kubernetes.
(I'm nearly certain it's not possible to reference secrets cross namespace when declaring an ingress secret, or mounting a secret)
> the Kubernetes patterns here would mean we need to duplicate the secret data into each applications namespace, allowing a compromise of one to compromise the TLS of all
yeah, this would be wrong indeed.
Is there any requirement for an TLS terminating proxy acting as k8s ingress to actually store the TLS secrets in the same namespace where the requesting ingress object lives?
By "when moving domains" I mean that I'm enabling a production domain (project.com), where previously I was working on a development domain (project-dev.com). More recently, I was deploying gitlab and exhausted my quota simply because of mistakes I made along the way. This was simply my mistake, so I don't think it would be appropriate to request an official rate limit increase. If (say) a $100 donation to the EFF allowed me to specify a domain on which the rate-limit-counters would be reset to zero (as a one-off), I'd likely have done that - it's easier than changing the process of how I'm managing certificates.
Once you hit this limit, you are not prevented from renewing existing certs.
You do have to be a little careful, for example adding a new SAN to a cert might fail due to 20-per-domain-per-week limits, attempting to renew that cert without removing the extra SAN will fail.
kiallmacinnes already mentioned that rate limits have no effect on renewals, but more importantly: There's a fairly simple process to bump rate-limits if your use-case requires it[1].
30 comments
[ 2.8 ms ] story [ 74.8 ms ] threadIt's been great for us in many respects. We no longer have certificates shared between servers, which removes the vulnerability to DROWN and some other problems. We also tend to serve multiple domains from a single IP, which requires a cert that covers all domains. LE lets us trivially get a cert to cover multiple domains, which would be especially expensive with project managers who can't make up their minds on what domains they want to use. And, we get the usual benefits of LE - effective forward secrecy in case someone obtains a single key/etc.
There was a little bit of pain at first getting the renewal infrastructure set up properly. My setup still isn't quite perfect, so far the standalone certbot option is the least painful for me. I have a cronjob which shuts down httpd, starts up certbot-standalone, does the renew, and restarts httpd, so we are down for an additional ~5 seconds per day while we renew. Not the end of the world for our usage.
My email is in my profile if you want to get in touch.
Once the certificates got renewed the secrets are not working anymore because they must be immutable.
"Hi, I noticed your website does not support HTTPS. Have you heard about Let's Encrypt? Let’s Encrypt is a free, automated, and open Certificate Authority. [...]"
Ideally available in multiple languages as well.
LE imposes a requirement to renew certificates every 90 days, which is a nuisance. Logging-in, finding my process crib-sheet, running the update requests, updating DNS records, running the validation requests, shifting certs to servers, restarting services... all because the LE project wants to evangelise short certificate lifespans, which seems to have been bolted onto the original goal of 'encrypt all the things' because of someone's personal conviction.
I do wish they'd offer a longer optional lifespan. Even 365 days would be a huge improvement. Until then I really can't recommend it to less-technically-inclined colleagues; I just tell them go and buy a one-year cert from Gandi, chuck it into the ssl directory and forget about it until the renewal reminder arrives 11 months later.
You're doing it wrong. There are many ways to use the ACME protocol (and Lets Encrypt), I think pretty much anyone will agree this is simply the wrong way to do it.
The ACME protocol is an API (as opposed to the CA's "upload a manually created, signed CSR, then wait for an HTML email and click the link in it and then download this file" process) so that it can be automated, and hands-off.
> I could have avoided this by combining names into a single certificate
You can get a cert (via <=20, 100 SAN combined certs) 2000 subdomains per week.
I actually don't really understand your problem case though. What do you mean by:
> when moving domains
That entirely depends on where/how you're terminating TLS.
Do you really have > 20 different pieces of independent software all doing their own TLS?
This is exactly what things like HAProxy are for.
You deploy software, that software declares it needs TLS for its endpoint, and certs are obtained.
While moving from one cluster to another, a new deployment of everything was done - about 20 different services - a few minutes work with Kubernetes. However, the fetching of TLS certs stopped working for the very last service!
Yes, I should have migrated the certs, however, the migration was happing due to a failure - I couldn't access the old certs.
I now make sure to backup those secrets rather than rely on re-issuance.
Still, for a usecase like Kubernetes, 20 certs per domain per week is limiting. I do totally understand that the quotas are in place because cert issuance is expensive, but I'd happily pay a yearly subscription fee to LE to cover the costs and help fund them if the option was there!
Each independent application is isolated from each other, sharing a TLS cert/key among them all means we're weakening security - again, the Kubernetes patterns here would mean we need to duplicate the secret data into each applications namespace, allowing a compromise of one to compromise the TLS of all.
There are legitimate reasons to not share a single TLS cert for everything in an environment like Kubernetes.
(I'm nearly certain it's not possible to reference secrets cross namespace when declaring an ingress secret, or mounting a secret)
yeah, this would be wrong indeed.
Is there any requirement for an TLS terminating proxy acting as k8s ingress to actually store the TLS secrets in the same namespace where the requesting ingress object lives?
There may be ways around this, however, I've never personally looked for them.
You do have to be a little careful, for example adding a new SAN to a cert might fail due to 20-per-domain-per-week limits, attempting to renew that cert without removing the extra SAN will fail.
[1]: https://docs.google.com/forms/d/e/1FAIpQLSetFLqcyPrnnrom2Kw8...