This seems unnecessarily hostile. Certbot is a complex beast that generates TLS configurations, has multiple plugins that likely require specific system configuration, and is I think charitably more than "a glorified Python script".
The rationale makes sense for me:
"While the Certbot team tries to keep the Certbot packages offered by various operating systems working in the most basic sense, due to distribution policies and/or the limited resources of distribution maintainers, Certbot OS packages often have problems that other distribution mechanisms do not. The packages are often old resulting in a lack of bug fixes and features and a worse TLS configuration than is generated by newer versions of Certbot. They also may not configure certificate renewal for you or have all of Certbot’s plugins available. For reasons like these, we recommend most users follow the instructions at https://certbot.eff.org/instructions and OS packages are only documented here as an alternative."
Snapd is not a solution to anything. It might be distro-agnostic, but it's tied to Canonical. The server is closed source and the client doesn't support multiple servers.
I'm not saying Flatpak is great, but it's the best we have, in my opinion.
That said, having published one application on Flathub, packaging Flatpak is not a great experience, especially when the two major repositories: Flathub and Fedora both use two completely different build processes.
Is Flatpak better than snapd, if we set aside that one is ‘open’ and one is ‘closed’ and evaluate them by the merits of their functionality in service of the goal?
AppImage? I don't see the need for much sandboxing, since certbot requires to have the keys to the kingdom anyway. systemd can whitelist a single folder for certbot to have write permissions.
> This seems unnecessarily hostile. Certbot is a complex beast that generates TLS configurations, has multiple plugins that likely require specific system configuration, and is I think charitably more than "a glorified Python script".
I didn't meant it as a knock on certbot, which I use and cherish. It's just not something that I would install and entire parallel package management system over, lest of all some ugly thing like snap.
> This seems unnecessarily hostile. Certbot is a complex beast that generates TLS configurations, has multiple plugins that likely require specific system configuration, and is I think charitably more than "a glorified Python script".
Yeah, it is complex. Far too complex for what it does.
What it does is implement a not particularly complex IETF protocol between your server and letsencrypt's servers, and run a few of your scripts when the protocol wants something done, like "install DNS entry".
In fact the protocol is so simple a bash script can do it. Dehydrated https://dehydrated.io/ is just that - a bash script that implements ACME. It also is in Debian. I moved to dehydrated when certbot was slow to implement DNS wildcards.
The nice thing about simplicity is it infects everything else: the document becomes simple, which is nice because I had to read certbot's doco several times just to figure out what scenario would work for me. And distribution packaging becomes simple, so no snap's needed. And installation becomes simple.
snaps are tied to the proprietary Ubuntu store and they are not available on all of the Linux flavors. For instance, I don't see snapd on Alpine Linux:
This has likely been asked and answered before, but why not write something like this in something that can be statically compiled like go? Isn’t the whole thing just a couple of web requests and cert operations?
Debian is still on version 0.x when the snap is at 1.1 ... this is one of those cases where the Debian stable branch is too stable when you need to ship software that interacts with a lot of remote APIs the way the certbot-dns packages do. It’s impossible to use the old Debian versions with the more secure API tokens from CloudFlare, for example.
The certbot-dns packages are certainly the most fragile part of the process -- however, I don't have any bug reports showing that the version in stable is currently broken for any of the dns plugins.
If you're experiencing that, please file a bug against them and I can look into having a stable backport done for them.
I’ll see what I can do, I’m not familiar with Debian package bug tracking. For reference, the incompatibility I faced was for the CloudFlare DNS plugin, which added support for API Tokens that had granular permissions somewhere between version 0.23 distributed in stable and 1.11 distributed via snap. Certbot docs and CloudFlare docs both encourage you to use API tokens instead of account-wide API secrets and the previous install instructions for wildcard DNS support suggested installing the CloudFlare DNS plugin by default on the Certbot site. As CloudFlare is used by a lot of folks for DNS, and was a suggested default install, I’m probably not the only one that encountered problems setting it up, without a backport available for a newer package of Certbot.
Hello there! As the maintainer of certbot for Debian, I promise you that there's no intention of deprecating the certbot packages that exist in the archive. We will still maintain them going forward in line with standard Debian practices, including security backports to stable as necessary.
Upstream fully supports this decision and has promised to continue to assist. Their recommendation for snaps is meant to target new and inexperienced users who may not understand why they are not getting the latest software when they see it was updated "on the website".
Snaps require you to allow each publisher to automatically push arbitrary software updates to your machine, and snap deliberately does not allow you to disable this misfeature. It's not a bug or an oversight -- it's part of the design of snap, and it's not going to change.
Handing out the keys to the kingdom like this is supremely annoying on a desktop (e.g. Windows 10), but it's completely unacceptable on a server.
This is a terrible move, and they should absolutely walk it back.
But I would also point out that on stable Debian it’s near impossible to deploy Certbot because the versions available are too old, especially the plugins. I’ve been routinely installing the snap instead because it’s a quick and easy way to deploy the service at the latest version.
The only other alternative to suggest is to request the Certbot project deploys custom deb repos or similar. But can we expect that of every project? Perhaps we need a new standard to replace snap, if so the only folks I can see large enough to promote it would be GitHub...
Since that’s unlikely to happen, maybe just noting the auto-update “feature” more prominently in their is all they should need to do?
I’m not sure, but it’s possible. The version in stable was 0.31 or so, while the version in snap and unstable/sid is 1.11. The specific incompatibility I faced was for the CloudFlare DNS plugin, which added support for API Tokens that had granular permissions somewhere between the 0.23 version in stable and the current version of the plugin at 1.11. Current Certbot docs and CloudFlare docs both encourage you to use API tokens instead of account-wide API secrets and the previous install instructions for wildcard DNS support suggested installing the CloudFlare DNS plugin by default on the Certbot site.
And if all that wasn't bad enough - snapd also purges package change history after something like 72 hours so you can't even tell what updates were applied and when to your system.
I agree with the rationale that software should be kept up to date. However, if this was their only concern, they would make it easy for updates to be applied automatically, and then would make this the default.
But the rationale for completely removing your ability to opt-out of an update is different. Snapd is not actually providing a service to users, at least not in earnest. They're providing a service to software publishers.
They're making it possible for publishers to ship software that isn't stable. This is a useful service for publishers, because stabilizing, releasing, and maintaining software is hard work, and requires resources that might otherwise be used implementing new features or writing new software. The life of a software publisher would indeed be easier if everyone was on the latest version as quickly as possible, and if multiple versions with conflicting feature sets were never allowed to exist at the same time.
In order to convince software publishers that snapd is capable of providing this service, snapd needs to convince users that running unstable software from these publishers is a reasonable approach. But for many server operators, it isn't reasonable.
I'm not here to talk about possible workarounds, or to argue about whether a delay is an adequate workaround, or about whether the delay interval is long enough. That would miss the point.
I'm here to assert that snapd's philosophy is wrong, that it's design is broken, and that this makes snapd unfit for purpose.
Here are three scenarios:
1. A security vulnerability is found in some stable piece of software. A minimal fix that mitigates only this vulnerability (without changing other behavior) is written and backported to the stable release.
In this case, it makes sense to apply the targeted patch as quickly as possible. It's time sensitive, and the risk of unintended fallout is minimal. You can do this automatically with debian-security and unattended-upgrades. Even in industries with complex software compliance processes, there's usually a fast-track exception for targeted fixes to critical security vulnerabilities.
2. A feature was improved, added, or removed. It might improve your server's performance. It might improve performance for a different system that needs to talk to your server. It might reduce the load on the software publisher's server. It might change or remove behavior that you rely on, causing your system to break in obvious or not-so-obvious ways. It might introduce a new feature that you've been desperately waiting for.
Everybody wants these updates to happen.
But it takes time and resources to figure out what's changed, to figure out whether the change breaks your system, and then to implement and deploy a fix for your system that allows you to upgrade.
3. Adobe or Autodesk want to silently "upgrade" your perpetually licensed software to their new monthly SaaS version.
lol, fuck that, and fuck them.
----
The problem with this approach (for software publishers) is that writing and backporting targeted fixes is hard, boring, thankless work. Similarly, it can be difficult and expensive for a publisher to have to deal with multiple versions of their software existing at the same time. They might interact in conflicting ways, they might require lots of ugly, bug-prone version-checking in client software so that different feature sets can be handled correctly. The publisher may desperately want to turn off a legacy feature on their servers, but can't until the last stable releases that rely on it are properly deprecated.
But this is how stable software works. If you don't want to deal with these things, and want to insist that everyone needs to be on the latest version as quickly as possible, then your software isn't stable, and has no business running on a server.
> Snaps require you to allow each publisher to automatically push arbitrary software updates to your machine, and snap deliberately does not allow you to disable this misfeature.
Your emphasis is completely unwarranted here, since it's easy and trivial to install a snap without it being tied to the store such that it will never update:
Now certbot is installed as a snap, but software updates for the snap are disabled.
Edit: I wouldn't recommend this, of course, because telling users to never update their Internet-connected software is a really bad idea, both for their security and for everyone else. More conventionally you'd want finer grained control over update schedules for production use (https://snapcraft.io/docs/keeping-snaps-up-to-date) rather than block updates entirely. Claiming that you cannot do it, with emphasis, though, is outright false and is an invalid claim only used by the haters. I bet you're now going to come back and tell me how the control isn't good enough somehow, but that doesn't matter. The point is that you can control updates, and you're being outright misleading by claiming otherwise.
Just to clarify something I think a lot of replies don't understand: The Ubuntu Certbot packages have been horribly broken for well over a year, probably at least two.
The PPA for Certbot includes updates of a lot of OS packages, with with incompatibilities with other Ubuntu provided packages. I've spent hours tracking down and fixing the problems that have come from installing certbot via apt.
Usually, I'm much more a fan of native packages than snaps or similar, but in this case it makes a lot of sense.
I formerly maintained this PPA before turning it over to some of the Ubuntu people. It's my understanding that certbot is going to be migrated out of universe soon and into the main Ubuntu archive. That will cause it to gain full support from Canonical. Either way, we will continue to support certbot in Debian (upstream of Ubuntu, if downstream from /actual/ upstream) indefinitely. The Ubuntu maintainers and I work closely on packaging.
Thanks for your work on that. I imagine it will be easier moving forward as most of the issues seemed to be related to pulling in newer versions of Python packages, which broke supplied versions of other packages (boto-related tools ISTR were breaking).
> The Ubuntu Certbot packages have been horribly broken for well over a year, probably at least two.
I think you're mistaken. The Ubuntu Certbot packages are maintained and work fine. What you're referring to is the PPA, which is no longer required (since Ubuntu's distribution-shipped certbot packages have worked fine for a long time now).
Or, if not, can you point to specific issues with Ubuntu's own certbot packages please?
If y'all want to try a high-magic solution based on a single-host docker-compose setup with traefik 2+ as ingress, here is the paragraph for configuring Let's encrypt certs: https://github.com/rocdata/rocserver/blob/main/docker-compos... It works great.
PS: In case I can save anyone days of debugging, if you want to run a wordpress container using this setup, you'll need to do some manual config to convince wp it's running in HTTPS mode, see https://wordpress.org/support/article/administration-over-ss...
Is there anything out of the ordinary here? Most upstream projects don't officially support distro packages and distros don't support non-distro packages. It's been that way for as long as I can remember.
As the maintainer of certbot for Debian, I promise you that there's no intention of deprecating the certbot packages that exist in the archive. The EFF fully supports this decision and has promised to continue to assist. Their recommendation for snaps is meant to target new and inexperienced users who may not understand why they are not getting the latest software when they see it was updated "on the website".
If any of you have concerns specifically about certbot on Debian, please feel free to contact me directly at my username @debian.org.
56 comments
[ 4.5 ms ] story [ 144 ms ] threadhttps://news.ycombinator.com/item?id=25141524
This is a ridiculous requirement for a glorified Python script, over-engineering at its worst.
The rationale makes sense for me:
"While the Certbot team tries to keep the Certbot packages offered by various operating systems working in the most basic sense, due to distribution policies and/or the limited resources of distribution maintainers, Certbot OS packages often have problems that other distribution mechanisms do not. The packages are often old resulting in a lack of bug fixes and features and a worse TLS configuration than is generated by newer versions of Certbot. They also may not configure certificate renewal for you or have all of Certbot’s plugins available. For reasons like these, we recommend most users follow the instructions at https://certbot.eff.org/instructions and OS packages are only documented here as an alternative."
I'm not saying Flatpak is great, but it's the best we have, in my opinion.
That said, having published one application on Flathub, packaging Flatpak is not a great experience, especially when the two major repositories: Flathub and Fedora both use two completely different build processes.
I didn't meant it as a knock on certbot, which I use and cherish. It's just not something that I would install and entire parallel package management system over, lest of all some ugly thing like snap.
Yeah, it is complex. Far too complex for what it does.
What it does is implement a not particularly complex IETF protocol between your server and letsencrypt's servers, and run a few of your scripts when the protocol wants something done, like "install DNS entry".
In fact the protocol is so simple a bash script can do it. Dehydrated https://dehydrated.io/ is just that - a bash script that implements ACME. It also is in Debian. I moved to dehydrated when certbot was slow to implement DNS wildcards.
The nice thing about simplicity is it infects everything else: the document becomes simple, which is nice because I had to read certbot's doco several times just to figure out what scenario would work for me. And distribution packaging becomes simple, so no snap's needed. And installation becomes simple.
The snap is very different and relies on systemd which isn’t commonly found in Alpine.
How about flatpak? Or anything out open?
After hearing some of the issues with it (here and elsewhere), I'm kinda grateful to them for making the effort.
Though this concerns me -- I use certbot on some debian servers, I hope that at least stays up to date.
If you're experiencing that, please file a bug against them and I can look into having a stable backport done for them.
Upstream fully supports this decision and has promised to continue to assist. Their recommendation for snaps is meant to target new and inexperienced users who may not understand why they are not getting the latest software when they see it was updated "on the website".
Thank you for the good work!
That's good news & will have a go at the apt version then.
http://acme.sh
Handing out the keys to the kingdom like this is supremely annoying on a desktop (e.g. Windows 10), but it's completely unacceptable on a server.
This is a terrible move, and they should absolutely walk it back.
But I would also point out that on stable Debian it’s near impossible to deploy Certbot because the versions available are too old, especially the plugins. I’ve been routinely installing the snap instead because it’s a quick and easy way to deploy the service at the latest version.
The only other alternative to suggest is to request the Certbot project deploys custom deb repos or similar. But can we expect that of every project? Perhaps we need a new standard to replace snap, if so the only folks I can see large enough to promote it would be GitHub...
Since that’s unlikely to happen, maybe just noting the auto-update “feature” more prominently in their is all they should need to do?
I suppose the rational is that updates should happen. You can take your time to hold updates for up to sixty days but they should happen anyway.
The certbot snap package is published by the EFF/LetsEncrypt themselves.
You could also trick snapd that you are on a metered Internet connection, therefore avoid updates.
But the rationale for completely removing your ability to opt-out of an update is different. Snapd is not actually providing a service to users, at least not in earnest. They're providing a service to software publishers.
They're making it possible for publishers to ship software that isn't stable. This is a useful service for publishers, because stabilizing, releasing, and maintaining software is hard work, and requires resources that might otherwise be used implementing new features or writing new software. The life of a software publisher would indeed be easier if everyone was on the latest version as quickly as possible, and if multiple versions with conflicting feature sets were never allowed to exist at the same time.
In order to convince software publishers that snapd is capable of providing this service, snapd needs to convince users that running unstable software from these publishers is a reasonable approach. But for many server operators, it isn't reasonable.
I'm not here to talk about possible workarounds, or to argue about whether a delay is an adequate workaround, or about whether the delay interval is long enough. That would miss the point.
I'm here to assert that snapd's philosophy is wrong, that it's design is broken, and that this makes snapd unfit for purpose.
Here are three scenarios:
1. A security vulnerability is found in some stable piece of software. A minimal fix that mitigates only this vulnerability (without changing other behavior) is written and backported to the stable release.
In this case, it makes sense to apply the targeted patch as quickly as possible. It's time sensitive, and the risk of unintended fallout is minimal. You can do this automatically with debian-security and unattended-upgrades. Even in industries with complex software compliance processes, there's usually a fast-track exception for targeted fixes to critical security vulnerabilities.
2. A feature was improved, added, or removed. It might improve your server's performance. It might improve performance for a different system that needs to talk to your server. It might reduce the load on the software publisher's server. It might change or remove behavior that you rely on, causing your system to break in obvious or not-so-obvious ways. It might introduce a new feature that you've been desperately waiting for.
Everybody wants these updates to happen.
But it takes time and resources to figure out what's changed, to figure out whether the change breaks your system, and then to implement and deploy a fix for your system that allows you to upgrade.
3. Adobe or Autodesk want to silently "upgrade" your perpetually licensed software to their new monthly SaaS version.
lol, fuck that, and fuck them.
----
The problem with this approach (for software publishers) is that writing and backporting targeted fixes is hard, boring, thankless work. Similarly, it can be difficult and expensive for a publisher to have to deal with multiple versions of their software existing at the same time. They might interact in conflicting ways, they might require lots of ugly, bug-prone version-checking in client software so that different feature sets can be handled correctly. The publisher may desperately want to turn off a legacy feature on their servers, but can't until the last stable releases that rely on it are properly deprecated.
But this is how stable software works. If you don't want to deal with these things, and want to insist that everyone needs to be on the latest version as quickly as possible, then your software isn't stable, and has no business running on a server.
Your emphasis is completely unwarranted here, since it's easy and trivial to install a snap without it being tied to the store such that it will never update:
then use the two commands given: Now certbot is installed as a snap, but software updates for the snap are disabled.Edit: I wouldn't recommend this, of course, because telling users to never update their Internet-connected software is a really bad idea, both for their security and for everyone else. More conventionally you'd want finer grained control over update schedules for production use (https://snapcraft.io/docs/keeping-snaps-up-to-date) rather than block updates entirely. Claiming that you cannot do it, with emphasis, though, is outright false and is an invalid claim only used by the haters. I bet you're now going to come back and tell me how the control isn't good enough somehow, but that doesn't matter. The point is that you can control updates, and you're being outright misleading by claiming otherwise.
The PPA for Certbot includes updates of a lot of OS packages, with with incompatibilities with other Ubuntu provided packages. I've spent hours tracking down and fixing the problems that have come from installing certbot via apt.
Usually, I'm much more a fan of native packages than snaps or similar, but in this case it makes a lot of sense.
Here is a link to the Github Issue: https://github.com/certbot/certbot/issues/5234#event-3691151...
(Edit: Adding issue link)
I formerly maintained this PPA before turning it over to some of the Ubuntu people. It's my understanding that certbot is going to be migrated out of universe soon and into the main Ubuntu archive. That will cause it to gain full support from Canonical. Either way, we will continue to support certbot in Debian (upstream of Ubuntu, if downstream from /actual/ upstream) indefinitely. The Ubuntu maintainers and I work closely on packaging.
I think you're mistaken. The Ubuntu Certbot packages are maintained and work fine. What you're referring to is the PPA, which is no longer required (since Ubuntu's distribution-shipped certbot packages have worked fine for a long time now).
Or, if not, can you point to specific issues with Ubuntu's own certbot packages please?
Host just needs to have ports 80 443 open and docker installed: https://github.com/rocdata/rocserver/blob/main/fabfile.py#L2...
PS: In case I can save anyone days of debugging, if you want to run a wordpress container using this setup, you'll need to do some manual config to convince wp it's running in HTTPS mode, see https://wordpress.org/support/article/administration-over-ss...
(link changed because only need SSL certs in prod -- so moved to a prod-specific docker-compose file)
If any of you have concerns specifically about certbot on Debian, please feel free to contact me directly at my username @debian.org.
But good to know it'll continue to be packaged (even though I didn't expect anything else from Debian). Thanks for your work!