However, should you fail to renew a cert, your script should just keep retrying until the API is back and you should allow enough time for it to fail. I believe most clients renew certs ~10 days in advance.
Obviously if you're relying on getting a new cert to bring up something like a preview environment or to hand out your own subdomains, this will result in a downtime/delay in provisioning, but most people would be fine with a single wildcard and never really experience a problem, as long as their script runs and they keep it reasonably up-to-date.
Based on the reminder emails they've sent me, they recommend renewing when one-third of the current cert's validity is remaining. That is, 1 month for their 3-month certs.
Yeah... my own setup for a hobby project of mine renew 30 days in advance but I don't think the way it is set up will try again tomorrow if it fails today.
With certbot, the check to see if certificates are close to expiry is done offline. So the service is only hit if a renewal is needed. The developers of certbot actually recommend that you schedule the cron twice a day.
EDIT: Actually I'm wrong. It also checks if the certificate was revoked via OCSP. However I can't imagine that it consumes much resources.
i also have a daily cron. One of the renewals somehow repeatedly reissued the cert for 5 consecutive days without me noticing. For some reason the certificate was also nowhere to be found, and letsencrypt woulnd't let me manually renew the certificate because it had exceeded the limit of 5 issuances/month (or something of that sort). Thankfully the limitation is only for the exact same cert, so you can issue a new one for a slightly different set of domains without waiting for the period to pass.
Good chance one of the classic CAs will use this outage against Let's Encrypt though. Some classic CAs have been spreading a lot of FUD about Let's Encrypt.
Probably. What matters is the community response to the outage though. From what I saw there's been a ton of love for the team working through an entire Sunday.
And this is the reason why Debian (and its derivs) ship a cron and systemd timer for you with certbot. You shouldn't have to think about or worry about this stuff. Certbot handles it for you.
For some reason my cron task always fails to renew, and I have to manually renew at the last minute, when I notice the certificate has expired. This is for a personal site, so it doesn't matter.
I tried many things, and can verify the cron task renews when I test it, but months later it always fails.
Is this on Debian? There should be a log file created in /var/log/letsencrypt on each attempt; if you could send me one at my username @debian.org, or ping me on Freenode IRC, I may be able to help you debug. I'm guessing it's because your renew.conf's trying to use the "simple" authentication method which won't work if you've got a webserver running.
Not a solution to your problem, but if you supply an email when creating a certificate, you will receive a notification plenty in advance (three or four weeks, I think)
The Caddy server can be configured to request a certificate while the user makes a request. That means things hosted in that manner might have downtime from this.
Good for you? I guess? Not sure what to do with that comment of yours. Is it an attempt at venting or constructing a situation where Let's Encrypt is responsible for your server?
1. The certbot client is not a product of the Let's Encrypt CA and is not mandatory. You can use another client.
2. With the certbot client, instead of using a deep integration with your http server, you can use `certbot certonly --webroot --webroot-path /var/www/html/.well-known/ -d example.com` (or similar) and provided you told your web server to serve file from the /var/www/html/.well-known/ filesystem directory under the /.well-known/ url path prefix on your domain, cert issuance and renewal will work. This should be easy to do with any web server. certbot will not know which web server you happen to use and will not care. You can replace the web server and renewal will keep working.
You can use dehydrated [1] which is a single shell script that you can trigger via cron to do the whole let's encrypt stuff for you. No dependancies other than curl.
These look like "canned" responses - one of the terror inducing parts of managing an emergency response is, in the middle of everything, choosing the right set of words to use when telling loads of really worried people - and it is dry easy to put it off and that lack of communication makes things bad
this smells like a tickbox on an incident response webform - the responder is just clicking "we have identified", "production database" (or network issue or ...) and the response handler does the rest
Yeah, we have our 'Shit's on fire' one.... generates a message like 'We have some unscheduled maintenance of databases to improve your customer experience'.
Once a post-mortem is done there will be more information.
Status.io is where we post status information, not long-form explanations. The responses aren't canned the way I think you mean it, they're just meant to tell you status information quickly and clearly - is it up, is it down, what services are affected, how long until they're back up...
Once we have a fairly complete picture of what happened to communicate, we communicate that on our community site.
Some of these can (and should) be written ahead of time, at least as templates.
Not only does doing so save someone from having to write them on the fly when they're worrying about other things, they can encourage you to communicate specific factors ("OK, we think we've found the problem, let's publish the message to that effect now"), remind you to think of certain things ("we think we'll be up within the hour" "we don't know yet how long it will take to fix" etc) and for that matter, if you have further failures, help someone look back to see how the process went.
51 comments
[ 4.9 ms ] story [ 116 ms ] threadhttps://medium.com/enigma-shards/lets-encrypt-uptime-and-ope...
However, should you fail to renew a cert, your script should just keep retrying until the API is back and you should allow enough time for it to fail. I believe most clients renew certs ~10 days in advance.
Obviously if you're relying on getting a new cert to bring up something like a preview environment or to hand out your own subdomains, this will result in a downtime/delay in provisioning, but most people would be fine with a single wildcard and never really experience a problem, as long as their script runs and they keep it reasonably up-to-date.
The recommendation and the time certbot uses is 30 days.
It will predictibly fail until D-30,and then there will be 30 attempts to renew (in case something went wrong firth the 1st,2nd etc attempt)
EDIT: Actually I'm wrong. It also checks if the certificate was revoked via OCSP. However I can't imagine that it consumes much resources.
I tried many things, and can verify the cron task renews when I test it, but months later it always fails.
I also set up traefik with auto renewal so it's a setup and forget thing now.
It does your auto renews in the background.
2. With the certbot client, instead of using a deep integration with your http server, you can use `certbot certonly --webroot --webroot-path /var/www/html/.well-known/ -d example.com` (or similar) and provided you told your web server to serve file from the /var/www/html/.well-known/ filesystem directory under the /.well-known/ url path prefix on your domain, cert issuance and renewal will work. This should be easy to do with any web server. certbot will not know which web server you happen to use and will not care. You can replace the web server and renewal will keep working.
Lots of instructions for many common situations here: https://certbot.eff.org/all-instructions
Did you check that you were running it with the right command flags?
If certbot has a bug that breaks things in a situation it's intended for do file a bug or even better a patch to fix the bug.
[1] https://github.com/lukas2511/dehydrated
this smells like a tickbox on an incident response webform - the responder is just clicking "we have identified", "production database" (or network issue or ...) and the response handler does the rest
would love to hear more
https://community.letsencrypt.org/t/2019-11-17-autoincrement...
Once a post-mortem is done there will be more information.
Status.io is where we post status information, not long-form explanations. The responses aren't canned the way I think you mean it, they're just meant to tell you status information quickly and clearly - is it up, is it down, what services are affected, how long until they're back up...
Once we have a fairly complete picture of what happened to communicate, we communicate that on our community site.
> November 17, 2019 19:33 UTC > [Resolved] We have resolved the issue that was affecting production certificate issuance. Service is restored.
Not only does doing so save someone from having to write them on the fly when they're worrying about other things, they can encourage you to communicate specific factors ("OK, we think we've found the problem, let's publish the message to that effect now"), remind you to think of certain things ("we think we'll be up within the hour" "we don't know yet how long it will take to fix" etc) and for that matter, if you have further failures, help someone look back to see how the process went.