Recently my Synology NAS failed to automatically renew its Let's Encrypt certificate for my domain name and the certificate expired on my blog. I caught it the next day when my GoAccess metrics cratered (took some time to figure out since I normally use the QuickConnect domain name myself, whose certificate was fine), but it could've stayed broken for a very long time otherwise without me noticing.
Even though the renewal app runs as a cron job weekly, it occasionally breaks due to OS updates or some other issue so the email from Lets encrypt that warns me at least a week or before the expiration has been fantastic.
I did get an email, but it was triaged under the update category inside Gmail and thus buried under a metric ton of other updates (the account is over 14 years old and it has accumulated a lot of crap over the years).
That's totally on me for missing it. On the other hand I only follow a couple of RSS feeds, so it's a notification channel with a far higher signal-to-noise ratio for me.
I've disabled it just now. I was basically only using it as an alias anyways.
I did take some very basic precautions otherwise (its firewall is configured to drop all non-local packets but for TCP ports 80 and 443), but at some point I'll have to host my blog properly instead of piggy-backing on a dinky, always-on NAS...
Interesting. Choice of rss is nice because there are already a good number of "convert/insert rss into x" tools that can be used to generate other modes of monitoring/alerts.
Super neat tool, but given that I use Caddy, that kinda prevents this issue from happening for me. While a monitoring tool is always a good idea, maybe the best long-term solution would be to encourage certificate auto-renewal tools. OTOH, I have only worked with this on a personal level, so maybe there's problems with auto-renewal that I haven't learned about.
Are there still instances where you would want an Extended Validation (EV) certificate? If so, that’s one case where certificate monitoring could be relevant.
Browsers today no longer provide visual indicators for EV certificates [1] so I don’t know if they’re still in common use.
Cool! I have a strange affinity for RSS and created* a small plugin to subscribe to feeds within Event-Driven Ansible** and run actions on new feed posts. I didn't create it with specific utility in mind, certificate monitoring via RSS fits right in there - much to my surprise.
This is a double negative. Depending on how you interpret the comma, it could mean "guarantees are given for everything." (Pointing this out in case you intend to protect yourself from liability with this statement.)
You monitor for the failures ($currentDate > $cert.NotAfter), great.
What about soft failures, like connection problems? What if the cert is available but actually garbage? What if between 30 and 7 days the cert is changed?
No need to be snarky, clearly monitoring end user connections is a must. But the general idea of using RSS for monitoring is new to me, thanks for sharing!
I used crtsh to discover certificates which were created in my previous company but I found about 20% of the time it returns some type of error (which is recoverable with a simple retry). Not sure if they fixed that, but I wouldn’t be surprised if a lot of companies use them and even profit from it somehow.
29 comments
[ 4.8 ms ] story [ 71.3 ms ] threadRecently my Synology NAS failed to automatically renew its Let's Encrypt certificate for my domain name and the certificate expired on my blog. I caught it the next day when my GoAccess metrics cratered (took some time to figure out since I normally use the QuickConnect domain name myself, whose certificate was fine), but it could've stayed broken for a very long time otherwise without me noticing.
You got yourself a subscriber.
https://letsencrypt.org/docs/expiration-emails/
Even though the renewal app runs as a cron job weekly, it occasionally breaks due to OS updates or some other issue so the email from Lets encrypt that warns me at least a week or before the expiration has been fantastic.
That's totally on me for missing it. On the other hand I only follow a couple of RSS feeds, so it's a notification channel with a far higher signal-to-noise ratio for me.
I did take some very basic precautions otherwise (its firewall is configured to drop all non-local packets but for TCP ports 80 and 443), but at some point I'll have to host my blog properly instead of piggy-backing on a dinky, always-on NAS...
This is what I use for my monitoring solutions
I also have monitoring that alerts me if a cert is nearing expiry.
I’ve been alerted several times and been able to correct bugs or hiccups that would have caused the live cert to expire.
Automation is not a replacement for monitoring: they are complementary.
absolutely. there are any number of reasons Caddy would be unable to renew the cert, just off the top of my head:
- LetsEncrypt has downtime or unavailability
- If you're doing dns-01 challenges for LE, whatever cred Caddy uses for that might expire / become invalidated.
- disk fills up (or gets unexpectedly remounted read-only) and Caddy is unable to write the renewed certs
Browsers today no longer provide visual indicators for EV certificates [1] so I don’t know if they’re still in common use.
[1]: https://en.wikipedia.org/wiki/Extended_Validation_Certificat... "Removal of special UI indicators"
Not really.
> [...] I don’t know if they’re still in common use.
They are. The myth that they are somehow inherently more secure is still widespread.
I have to submit a change request to get this added to our monitoring platform, and this is just so much simpler.
Thank you!
* - https://github.com/cloin/cloin.eda/blob/main/docs/rss.rst
** - https://github.com/ansible/ansible-rulebook
This is a double negative. Depending on how you interpret the comma, it could mean "guarantees are given for everything." (Pointing this out in case you intend to protect yourself from liability with this statement.)
And perhaps also specifying a port, for services not on 443?
https://github.com/louislam/uptime-kuma
What about soft failures, like connection problems? What if the cert is available but actually garbage? What if between 30 and 7 days the cert is changed?
And no, not checking FQDN against SAN is...
And finally, who monitors the monitoring?