It's normal that CVEs aren't filled directly after a disclosure. Takes some time till the database gets updated.
Not sure how that makes you concerned about responsible disclosure. As you can read in the bug report the apache security team was informed (actually it was one of their devs figuring out the details of the bug, I just saw it "in the wild", but at first was unable to reproduce it locally). I had also informed the distros mailing list a week in advance.
I'm aware responsible disclosure is a weakly defined term, but if you have any concerns that I did something irresponsible I'd like to know.
In my mind (biased) responsible disclosure includes ensuring that a respected database for tracking security issues has the issue described and publicly available either before or at the same time that a blog publicly discusses it. Otherwise the impression is that the blog is discussing a security issue before it's been addressed by interested parties.
That's just my opinion. Whether or not that's the "definition" of responsibility is of course up for discussion.
I've got two and three year old vulnerabilities I've disclosed which have CVE's which are still not populated. This is very common. No one waits for the CVE to be populated before going public.
The delay seems like it's caused by the people who update the database just not having the time to get it done. Or alternatively, just not prioritising it for the time they have.
Either way of looking at it, delaying release of vuln info to wait for the above to happen seems like it would let vulns keep in-the-wild-and-unknown for much longer.
That being said, if you're able to assist with or fix the not-enough-time-to-get-things-done problem, it wouldn't hurt. :)
One driver is popular software that depends on the .htaccess construct.
WordPress, for example. It can run on other servers, but their own docs call out the issue..."Since Nginx does not have .htaccess-type capability and WordPress cannot automatically modify the server configuration for you, it cannot generate the rewrite rules for you"
The quoted text is from their docs, and does not read that way. Some more:
"With Nginx there is no directory-level configuration file like Apache's .htaccess or IIS's web.config files. All configuration has to be done at the server level by an administrator, and WordPress cannot modify the configuration, like it can with Apache or IIS"
Almost every DO, vultr, Linode, etc guide I've seen for LAMP server config uses Apache. I also suspect apache, being the old guard, is the defacto standard in various appliances, routers, firewalls, IoT, etc devices. Its our generation's BIND. Its probably never going away, even if better competitors come and go.
As far as stats go, according to netcraft MS's IIS is leading with Apache second and nginx a distant third. Of course, this data isn't terribly reliable as netcraft can only see front ends, which may hide whats behind the load balancer, but IIS/Apache are still kings.
Its "killer feature" are htaccess files. It provides a possibility to give users a limited way to configure their virtual hosts, without giving them too much power. (Unfortunately there are some unfixed systemic security issues with that, namely symlink attacks...)
You can, and it's very unfortunate that the apache devs don't consider these things vulnerabilities.
(I reported a DoS via password hashes a while ago.)
Speed is one of the reasons to stay on apache. Using nginx to serve static and using php_mod vs fpm-php will be faster as long as you disable directory level .htaccess
Nginx should be faster in theory, but it's kind of hard to show in practice as the actual http parsing is just so much faster than anything else. You have to really tune them for for benchmark scenarios to see any kind of difference.
What your vendor supports and what you are most comfortable with configuring should decide which one you run.
PHP, as you say, is something else entirely. It's so much slower than your http server so the latter doesn't matter at all.
Not all the time... Before, when nginx was event and Apache was prefork or worker, yeah. But with the Event MPM, performance is pretty much neck-and-neck.
It's still a really nice web server. Easy to configure, incredibly reliable, twenty years of security fixes under its belt, endless docs/info available for it online. I've been meaning to learn nginx for a while, but with Apache working just fine, it doesn't feel very urgent.
Besides Apache and nginx, there aren't really any servers I'd trust for production use. Conceptually, I really like what Caddy is doing, but it's simply too young, not to mention the concerns that came up when Let's Encrypt had an outage back in May: https://github.com/mholt/caddy/issues/1680
The concerns about Caddy handling an ACME server outage are not really well founded these days, because:
1) Caddy has the most robust OCSP implementation of any web server. It caches staples locally and refreshes them halfway through the validity period so it can endure days-long OCSP responder outages.
2) Even if a certificate needs to be renewed while the ACME server is down, Caddy can endure 2-to-3-week-long outages of the ACME server because it renews 30 days out from expiration and tries twice per day until it succeeds, logging its actions along the way.
And because Caddy is written in Go, memory leaks like what Apache suffers are much less likely, if not impossible.
Two calls to OpenSSL and a few lines of shell script will fetch staples from LE whenever I want (ie more often than your defaults that have caused problems before) for HAProxy to serve. How's that for robust.
>2)...
great so you're as reliable as.. any other LE client that tries to renew before expiry, but without the ability to choose when to renew certs?
1) Caddy has never had OCSP outage problems. When LE's OCSP responders went down for a day a few months ago, Caddy was the ONLY server that kept sites online (unless nginx was explicitly configured for it; almost all Apache sites with it went down, including gnu.org).
2) Ours is the only LE client built directly into the web server, so "reliability" isn't really a comparable factor. By doing renewals automatically, Caddy's HTTPS implementation is more reliable and robust than doing it manually.
As someone who occasionally helps people with their web server/TLS/Let's Encrypt configuration, my experience is that there is a world of difference between a web server merely giving you the tools you'd need for a (relatively) sane OCSP and auto-renewal configuration, and one that takes care of everything. You should not need the amount of specialized knowledge you currently need to correctly configure OCSP on HAProxy or nginx, for example.
Failure to renew a certificate (for any number of reasons) and missing intermediate certificates are the most common issues, and both of these are essentially solved with a web server that supports ACME natively. The same applies to OCSP stapling - we're not going to achieve ubiquitous stapling support on the server side if every server admin needs to deploy a bash script calling to openssl because the native OCSP implementation in their web server is broken.
There is literally zero configuration required for HAProxy. You just store .ocsp files alongside the .pem's
Yes you need to fetch them regularly. I use the aforementioned shell script to do this.
Why do you people keep up this weird narrative that only caddy renews certificates automatically?
The native ocsp handling in HAProxy isn't broken: it just expects to have TLS material provided to it.
If someone setting up HAProxy on a server can't install a shell script calling two OpenSSL commands and setup cron to call it regularly, I don't think your big problem is "HAProxy doesn't handle ocsp". It's PEBKAC.
> There is literally zero configuration required for HAProxy. You just store .ocsp files alongside the .pem's
> Yes you need to fetch them regularly. I use the aforementioned shell script to do this.
It does sound like there's some configuration involved after all, namely the configuration of a cronjob, and perhaps adjustments to a bash script. This is something that the majority of server operators will not bother with, though these numbers might look slightly better for HAProxy since that's generally something you see used by more experienced sysadmins. Even with nginx, we're talking about one line of configuration for the actual stapling file - that's not the blocker, it should just work, by default. Fun fact: IIS does this.
> Why do you people keep up this weird narrative that only caddy renews certificates automatically?
I'm not sure who "you people" are, and I never made any such claim. What I'm saying is that web servers that support ACME natively leave a lot less room for the types of misconfigurations that I commonly see. I'm aware of an ACME module for apache and lua scripts for nginx which provide similar features (regarding renewal and certificates, not for the OCSP aspect), but they're not part of what you run by default when you use those web servers.
> The native ocsp handling in HAProxy isn't broken: it just expects to have TLS material provided to it.
I have yet to see you explain why this should be necessary. The web server has all the information it needs to do this. Other web servers can do this.
> If someone setting up HAProxy on a server can't install a shell script calling two OpenSSL commands and setup cron to call it regularly, I don't think your big problem is "HAProxy doesn't handle ocsp". It's PEBKAC.
This is like arguing MongoDB should continue to bind to all interfaces by default because everyone should have firewalls in place. Yes, they should, but it's never going to happen, so why can't we just have a safe default?
> It does sound like there's some configuration involved after all
How is "schedule a script to run @daily" or whatever schedule you want, at all match up to what you suggested is required:
>> You should not need the amount of specialized knowledge you currently need to correctly configure OCSP on HAProxy or nginx, for example.
If you're configuring a public facing server, a cronjob should not be considered "specialised knowledge".
> it should just work, by default
HAProxy and nginx do a fucking shit ton more than Caddy does. HAProxy doesn't really have a "default" you have to tell it what you want to do, specifically. This is a good thing.
> I'm not sure who "you people" are, and I never made any such claim.
>> By doing renewals automatically, Caddy's HTTPS implementation is more reliable and robust than doing it manually.
>> Failure to renew a certificate (for any number of reasons) and missing intermediate certificates are the most common issues,
You're both giving the impression that Caddy is able to automatically renew certificates, while no other software does this.
> I have yet to see you explain why this should be necessary.
Off the top of my head:
- Separation of concerns. I don't ask my mechanic to build me a car, or to drive me to the shops in it.
- Not every organisation uses LetsEncrypt certificates. Some will even use self-signed certificates.
- TCP or DNS load-balanced environments will want the exact same certificate & key on multiple servers.
- Caddy has already shown that trusting some "automagic" solution can bite you in the ass. Remember that time Caddy wouldn't even start, even though it had perfectly valid certificate material in it's cache, because LE was down?
> The web server has all the information it needs to do this.
HAProxy doesn't know the email address I'd want LetsEncrypt to use for notifications. It also shouldn't be agreeing to LE's terms on my behalf.
> This is like arguing MongoDB should continue to bind to all interfaces by default because everyone should have firewalls in place.
No, it isn't.
I'm sorry but you simply don't get the choice of arguing anything about how creating a fucking cronjob is "too hard" when Caddy doesn't even ship with a configuration to start as a service.
"I know I just had to google the shit out of how to create a systemd unit, but fuck it, it's better than writing a damned cron job for oscp staples!"
> If you're configuring a public facing server, a cronjob should not be considered "specialised knowledge".
Knowing how HAProxy's OCSP implementation works (i.e., by looking for a $cert.ocsp file) is the specialized knowledge. Odds are, you don't know about it and just won't use OCSP stapling. This is especially true since every web server does this differently.
> HAProxy and nginx do a fucking shit ton more than Caddy does. HAProxy doesn't really have a "default" you have to tell it what you want to do, specifically. This is a good thing.
I'm not arguing that it doesn't do a lot of things Caddy doesn't. I'm arguing for sane defaults. If I tell HAProxy to enable TLS and provide a certificate that contains an OCSP AIA extension, a sane default would be to use that information to manage OCSP stapling.
> You're both giving the impression that Caddy is able to automatically renew certificates, while no other software does this.
You're leaving out the part where I say these issues "[...] are essentially solved with a web server that supports ACME natively". I'm not referring to Caddy specifically. The web server doing this natively is far less error-prone. I have also mentioned alternatives.
> - Separation of concerns. I don't ask my mechanic to build me a car, or to drive me to the shops in it.
Where do you draw the line? Most deployments let their web servers handle TLS. OCSP stapling should be part of a modern TLS deployment. You could similarly argue that using HAProxy for TLS is wrong (and that you should rather put pound or stunnel in front of it).
> - Not every organisation uses LetsEncrypt certificates. Some will even use self-signed certificates.
That's fine; you can continue to manually configure certificates, even with Caddy, and I'm definitely not arguing that this option should be removed. ACME is also an open standard, so switching to other CAs can be as simple as configuring a different ACME server url.
> - TCP or DNS load-balanced environments will want the exact same certificate & key on multiple servers.
That's why you can replace the certificate and key storage in more complex environments[1].
> - Caddy has already shown that trusting some "automagic" solution can bite you in the ass. Remember that time Caddy wouldn't even start, even though it had perfectly valid certificate material in it's cache, because LE was down?
Software can break, what a surprise. You could similarly have a broken OCSP updater script that puts empty or expired OCSP responses in your stapling file because the check is missing or incomplete and break the site for your visitors. Or your ACME client could fail on renewal because of a backend change on Let's Encrypt's end that exposes a bug in it[2].
> HAProxy doesn't know the email address I'd want LetsEncrypt to use for notifications. It also shouldn't be agreeing to LE's terms on my behalf.
You can ... provide your email address? You keep arguing that this stuff is easy, surely adding a single line with your email address won't break the camel's back. Accepting the TOS can be solved as part of the installation or configuration process, much like ACME clients do it.
Anyway, you seem to be having some kind of problem with Caddy. I'm not interested in that discussion (which is to say I'm not interested in discussing this further, I think I've made my point) - my goal is to improve the user experience of server admins when they enable HTTPS, and I have no doubt that native ACME support and automated OCSP management, both with sane defaults, would both help HTTPS adoption and improve the quality of the TLS configuration of your average web server.
FYI, the issue a few months back wasn't "software breaks".
It worked exactly as they designed and implemented it.
The original issue was closed, before (as seems to be a pattern lately) they changed direction when they received so much negative feedback about their highly opinionated choices for the project.
To this day the same logic is still in caddy, but with a longer grace period. Try starting caddy offline with a cert that expires in less than 7 days.
I don't trust someone to develop a server if they think failing to start at all, is better than starting with close to expired or even expired certificates.
It's called the Ostrich Solution. You stick your head in the sand to ignore alternatives to buttress the argument that your solution is the sole solution to a problem. You then hope that no one does even the most rudimentary investigation, which is, unfortunately, getting more and more prevalent nowadays.
Lots of reasons. First of all, it is fast and reliable. The choice of MPMs allow for sys-admins to choose exactly how they want httpd to scale. Unparalleled RFC compliance. Load balancing with dynamic, runtime configuration. Failover. .htaccess file. Full support for FPM/FastCGI. Fully Open Source and community driven (not Open Core). mod_rewrite capability. Hundreds of modules for almost every and all situation. Most people simply take the "lazy" route and not even investigate httpd, instead relying on the FUD about much older versions (you still get, for example, the screed that "Apache forks off a new process for each request")... Apache httpd 2.4 is just as fast and reliable as any other web-server out there, nginx included. It's just not seen as the "cool and hip" web-server. I guess some people are more concerned about marketing.
Are you kidding? Apache is by far the most used web server for dynamic content-driven sites. It's "killer" feature is its robustness, in particular for (shared or otherwise) CGI hosting (and PHP hosting, of course). Outside the web app bubble, CGIs eg. per-request process spawning isn't a performance problem at all when you're using caching, which Apache httpd conveniently provides with mod_cache; mod_cache, unlike eg. Varnish and other dedicated caching solutions, has full RFC7324 caching with re-validation, not just "Expires:".
Apache httpd in many ways is a much more complete web server solution compared to, say, nginx (which I like as well). I'm prefering Apache httpd for the depth of available documentation alone, because everything is developed openly, C source is accessible and not hard to develop with, and because of the breadth of available options (certbot/letsencrypt/acme reference implementation on Apache, WebDAV, clustering etc. etc.).
It's pretty robust these days and is battle-hardened by the many attacks on it. Plus it has support for just about everything under the sun - that is, if someone has written something that extends or relies upon a web server, there will almost certainly be apache support.
(And I'm writing this as a dev who worked on a competing web server which IM-biased-O was vastly better :)
It's kept up to date and it works for everything I want it to work with. This idea that because it's old and not hip therefore it's use is questionable is kinda dumb. I was disappointed when OBSD wrote their own httpd that was initially filled with stupid bugs.
I use Apache specifically on shared web servers in order to run each customer's web site under a separate uid/gid (for security purposes). I use it on other $work web servers just for consistency/ease of management (performance isn't an issue).
Mostly I use it because, after probably 20 years or so, I'm quite comfortable with and accustomed to it.
That said, I have started putting a few of my own web sites on nginx just to try it out a bit and get some familiarity with it.
I'm astonished how many shared hosting provider still using apache 2.4.
Yes, there are a few things apache is remarkable about - the npm-itk engine could run as user processes, which was nice for shared hosting - but all the problems with it - slowloris, anyone? - simply outweights the benefits.
Shared hosting should be containerised or vm-ised instead and run on nginx/lighttpd/cherookee/etc + php-fpm 7.0 which will save you a lot of headaches and CPU cycles.
.htaccess can be just as much a benefit as a curse.
(Note: WordPress is completely nginx + php-fpm friendly.)
Lots of shared hosting uses tooling like cPanel shared hosting automation, which if I recall is still defaulting to Apache.
Nginx is surely the first choice if you're rolling your own shared hosting infrastructure, but most legacy commercial shared hosting automation frameworks still use Apache and customers don't care what the underlying tech is serving their bits over HTTP.
"all the problems" are really old, old news, long ago addressed and resolved. In fact, such statements are more FUD than anything else. Certainly you MUST know that Apache httpd supports PHP-FPM, for example.
The old problems are not solved and not resolved at all. I am working on a critical apache setup at the moment and the software is rigged with weird corner cases and bugs. This shit doesn't happen with haproxy and nginx.
You know you can use HAProxy with apache too right?
I never understand these arguments that basically pit Apache2 on its own/with modules against a completely modular arrangement of tls/cache/web/app as different layers.
If you use apache as the "web" in that setup, it needs relatively few modules loaded and when configured correctly can be blazing fast.
Apache is far from perfect, but among its ilk it is probably the one of the more robust and well tested. It's not as if you don't have to care about connection hogging attacks such as slowloris with other http servers.
The cycles saved on the nginx http parser, which is completely irrelevant to the performance of any real world web application, comes at the price of being hand optimized and opaque. You can't really argue for that type of performance engineering at the same time as you argue for solid security. The world generally needs more secure software, not more performant protocol parsers.
Do also note that nginx had a very similar data leak bug not even two months ago in the range header.
I was able to reproduce this issue on Apache 2.2.22 with a legitimate method:
<LIMIT GET>
Allow from x.y.z.v
Deny from all
</LIMIT>
for I in `seq 1 100`; do curl -sI -X OPTIONS http://myhost/ | grep ^Allow; done | uniq
Allow: ,GET,HEAD,POST,OPTIONS
Allow: GET,HEAD,POST,OPTIONS
Allow: ,GET,HEAD,POST,OPTIONS
Allow: GET,HEAD,POST,OPTIONS
Value returned in 'Allow' changes with every apache2 restart, but I haven't yet seen any parts of the config or something other than extraneous commans or 'allow' string.
After many repeated tests with multiple LIMIT sections in .htaccess (and without restarting apache2) i've managed to get a leak of part of config file:
For those who come straight to the comments, it should be noted that most Apache installs won't be affected and this is a relatively low-impact vulnerability, although for those who are affected it could be pretty bad.
The flaw only occurs if you have a "Limit" configuration that specifies an invalid HTTP method. For some unknown reason, this causes the leak. A "Limit" with a valid method, or no "Limit" at all, will not provoke the bug. I have to imagine that most Apache setups either don't configure this or configure it only with valid methods.
The article points out that shared hosting setups which allow individual users to customize settings through .htaccess would be vulnerable, since anyone on the machine could set an invalid configuration and then use the vulnerability to potentially get data from other users.
Obviously you should patch it when you can, but it is probably not time for most of us to panic.
I will leave here the link of our 2014 paper, "Support for Various HTTP Methods on the Web" (https://arxiv.org/pdf/1405.2330.pdf) for the reference. The issue is described in section 5.3 of the paper.
74 comments
[ 1.2 ms ] story [ 226 ms ] thread> CVE-2017-9798. -> https://nvd.nist.gov/vuln/detail/CVE-2017-9798
The CVE is still in RESERVED state. I'm frequently concerned about "responsible disclosure" in such cases.
Not sure how that makes you concerned about responsible disclosure. As you can read in the bug report the apache security team was informed (actually it was one of their devs figuring out the details of the bug, I just saw it "in the wild", but at first was unable to reproduce it locally). I had also informed the distros mailing list a week in advance.
I'm aware responsible disclosure is a weakly defined term, but if you have any concerns that I did something irresponsible I'd like to know.
That's just my opinion. Whether or not that's the "definition" of responsibility is of course up for discussion.
Either way of looking at it, delaying release of vuln info to wait for the above to happen seems like it would let vulns keep in-the-wild-and-unknown for much longer.
That being said, if you're able to assist with or fix the not-enough-time-to-get-things-done problem, it wouldn't hurt. :)
I'm really iterested on the whys (in addition to the obvious legacy reasons).
WordPress, for example. It can run on other servers, but their own docs call out the issue..."Since Nginx does not have .htaccess-type capability and WordPress cannot automatically modify the server configuration for you, it cannot generate the rewrite rules for you"
"With Nginx there is no directory-level configuration file like Apache's .htaccess or IIS's web.config files. All configuration has to be done at the server level by an administrator, and WordPress cannot modify the configuration, like it can with Apache or IIS"
What you are saying is, on it's own, true. It's also a solved problem for WP.
https://news.netcraft.com/archives/2017/07/20/july-2017-web-...
There are lots of legit reasons to run Apache:
- it works like a champ
- it's widely known
- it's got a big ecosystem (plugins, etc.)
- it has a fairly complete WebDAV implementation
- it's widely supported
TL;DR "legacy" sometimes means "good" and not just "old."
In this case legacy is either "I don't know anything else", "well it worked so far" or "meh".
As far as stats go, according to netcraft MS's IIS is leading with Apache second and nginx a distant third. Of course, this data isn't terribly reliable as netcraft can only see front ends, which may hide whats behind the load balancer, but IIS/Apache are still kings.
Its "killer feature" are htaccess files. It provides a possibility to give users a limited way to configure their virtual hosts, without giving them too much power. (Unfortunately there are some unfixed systemic security issues with that, namely symlink attacks...)
What your vendor supports and what you are most comfortable with configuring should decide which one you run.
PHP, as you say, is something else entirely. It's so much slower than your http server so the latter doesn't matter at all.
Besides Apache and nginx, there aren't really any servers I'd trust for production use. Conceptually, I really like what Caddy is doing, but it's simply too young, not to mention the concerns that came up when Let's Encrypt had an outage back in May: https://github.com/mholt/caddy/issues/1680
1) Caddy has the most robust OCSP implementation of any web server. It caches staples locally and refreshes them halfway through the validity period so it can endure days-long OCSP responder outages.
2) Even if a certificate needs to be renewed while the ACME server is down, Caddy can endure 2-to-3-week-long outages of the ACME server because it renews 30 days out from expiration and tries twice per day until it succeeds, logging its actions along the way.
And because Caddy is written in Go, memory leaks like what Apache suffers are much less likely, if not impossible.
(Source: I implemented it.)
Two calls to OpenSSL and a few lines of shell script will fetch staples from LE whenever I want (ie more often than your defaults that have caused problems before) for HAProxy to serve. How's that for robust.
>2)...
great so you're as reliable as.. any other LE client that tries to renew before expiry, but without the ability to choose when to renew certs?
2) Ours is the only LE client built directly into the web server, so "reliability" isn't really a comparable factor. By doing renewals automatically, Caddy's HTTPS implementation is more reliable and robust than doing it manually.
Ok fine, you want to pretend HAProxy isn't a thing that exists, we can do that.
> 2)
That isn't necessarily a good thing. Have you ever heard of separation of concerns?
You specifically mentioned how Caddy can manage 2-3 week outages, as if it's something unique to your software.
Every LE client that can be run from cron/a systemd timer renews certificates autagically.
So where's the manual part that you keep claiming is not "robust"?
Failure to renew a certificate (for any number of reasons) and missing intermediate certificates are the most common issues, and both of these are essentially solved with a web server that supports ACME natively. The same applies to OCSP stapling - we're not going to achieve ubiquitous stapling support on the server side if every server admin needs to deploy a bash script calling to openssl because the native OCSP implementation in their web server is broken.
Yes you need to fetch them regularly. I use the aforementioned shell script to do this.
Why do you people keep up this weird narrative that only caddy renews certificates automatically?
The native ocsp handling in HAProxy isn't broken: it just expects to have TLS material provided to it.
If someone setting up HAProxy on a server can't install a shell script calling two OpenSSL commands and setup cron to call it regularly, I don't think your big problem is "HAProxy doesn't handle ocsp". It's PEBKAC.
> Yes you need to fetch them regularly. I use the aforementioned shell script to do this.
It does sound like there's some configuration involved after all, namely the configuration of a cronjob, and perhaps adjustments to a bash script. This is something that the majority of server operators will not bother with, though these numbers might look slightly better for HAProxy since that's generally something you see used by more experienced sysadmins. Even with nginx, we're talking about one line of configuration for the actual stapling file - that's not the blocker, it should just work, by default. Fun fact: IIS does this.
> Why do you people keep up this weird narrative that only caddy renews certificates automatically?
I'm not sure who "you people" are, and I never made any such claim. What I'm saying is that web servers that support ACME natively leave a lot less room for the types of misconfigurations that I commonly see. I'm aware of an ACME module for apache and lua scripts for nginx which provide similar features (regarding renewal and certificates, not for the OCSP aspect), but they're not part of what you run by default when you use those web servers.
> The native ocsp handling in HAProxy isn't broken: it just expects to have TLS material provided to it.
I have yet to see you explain why this should be necessary. The web server has all the information it needs to do this. Other web servers can do this.
> If someone setting up HAProxy on a server can't install a shell script calling two OpenSSL commands and setup cron to call it regularly, I don't think your big problem is "HAProxy doesn't handle ocsp". It's PEBKAC.
This is like arguing MongoDB should continue to bind to all interfaces by default because everyone should have firewalls in place. Yes, they should, but it's never going to happen, so why can't we just have a safe default?
How is "schedule a script to run @daily" or whatever schedule you want, at all match up to what you suggested is required:
>> You should not need the amount of specialized knowledge you currently need to correctly configure OCSP on HAProxy or nginx, for example.
If you're configuring a public facing server, a cronjob should not be considered "specialised knowledge".
> it should just work, by default
HAProxy and nginx do a fucking shit ton more than Caddy does. HAProxy doesn't really have a "default" you have to tell it what you want to do, specifically. This is a good thing.
> I'm not sure who "you people" are, and I never made any such claim.
>> By doing renewals automatically, Caddy's HTTPS implementation is more reliable and robust than doing it manually.
>> Failure to renew a certificate (for any number of reasons) and missing intermediate certificates are the most common issues,
You're both giving the impression that Caddy is able to automatically renew certificates, while no other software does this.
> I have yet to see you explain why this should be necessary.
Off the top of my head:
- Separation of concerns. I don't ask my mechanic to build me a car, or to drive me to the shops in it.
- Not every organisation uses LetsEncrypt certificates. Some will even use self-signed certificates.
- TCP or DNS load-balanced environments will want the exact same certificate & key on multiple servers.
- Caddy has already shown that trusting some "automagic" solution can bite you in the ass. Remember that time Caddy wouldn't even start, even though it had perfectly valid certificate material in it's cache, because LE was down?
> The web server has all the information it needs to do this.
HAProxy doesn't know the email address I'd want LetsEncrypt to use for notifications. It also shouldn't be agreeing to LE's terms on my behalf.
> This is like arguing MongoDB should continue to bind to all interfaces by default because everyone should have firewalls in place.
No, it isn't.
I'm sorry but you simply don't get the choice of arguing anything about how creating a fucking cronjob is "too hard" when Caddy doesn't even ship with a configuration to start as a service.
"I know I just had to google the shit out of how to create a systemd unit, but fuck it, it's better than writing a damned cron job for oscp staples!"
Knowing how HAProxy's OCSP implementation works (i.e., by looking for a $cert.ocsp file) is the specialized knowledge. Odds are, you don't know about it and just won't use OCSP stapling. This is especially true since every web server does this differently.
> HAProxy and nginx do a fucking shit ton more than Caddy does. HAProxy doesn't really have a "default" you have to tell it what you want to do, specifically. This is a good thing.
I'm not arguing that it doesn't do a lot of things Caddy doesn't. I'm arguing for sane defaults. If I tell HAProxy to enable TLS and provide a certificate that contains an OCSP AIA extension, a sane default would be to use that information to manage OCSP stapling.
> You're both giving the impression that Caddy is able to automatically renew certificates, while no other software does this.
You're leaving out the part where I say these issues "[...] are essentially solved with a web server that supports ACME natively". I'm not referring to Caddy specifically. The web server doing this natively is far less error-prone. I have also mentioned alternatives.
> - Separation of concerns. I don't ask my mechanic to build me a car, or to drive me to the shops in it.
Where do you draw the line? Most deployments let their web servers handle TLS. OCSP stapling should be part of a modern TLS deployment. You could similarly argue that using HAProxy for TLS is wrong (and that you should rather put pound or stunnel in front of it).
> - Not every organisation uses LetsEncrypt certificates. Some will even use self-signed certificates.
That's fine; you can continue to manually configure certificates, even with Caddy, and I'm definitely not arguing that this option should be removed. ACME is also an open standard, so switching to other CAs can be as simple as configuring a different ACME server url.
> - TCP or DNS load-balanced environments will want the exact same certificate & key on multiple servers.
That's why you can replace the certificate and key storage in more complex environments[1].
> - Caddy has already shown that trusting some "automagic" solution can bite you in the ass. Remember that time Caddy wouldn't even start, even though it had perfectly valid certificate material in it's cache, because LE was down?
Software can break, what a surprise. You could similarly have a broken OCSP updater script that puts empty or expired OCSP responses in your stapling file because the check is missing or incomplete and break the site for your visitors. Or your ACME client could fail on renewal because of a backend change on Let's Encrypt's end that exposes a bug in it[2].
> HAProxy doesn't know the email address I'd want LetsEncrypt to use for notifications. It also shouldn't be agreeing to LE's terms on my behalf.
You can ... provide your email address? You keep arguing that this stuff is easy, surely adding a single line with your email address won't break the camel's back. Accepting the TOS can be solved as part of the installation or configuration process, much like ACME clients do it.
Anyway, you seem to be having some kind of problem with Caddy. I'm not interested in that discussion (which is to say I'm not interested in discussing this further, I think I've made my point) - my goal is to improve the user experience of server admins when they enable HTTPS, and I have no doubt that native ACME support and automated OCSP management, both with sane defaults, would both help HTTPS adoption and improve the quality of the TLS configuration of your average web server.
[1]: https://godoc.org/github.com/mholt/caddy/caddytls#...
FYI, the issue a few months back wasn't "software breaks".
It worked exactly as they designed and implemented it. The original issue was closed, before (as seems to be a pattern lately) they changed direction when they received so much negative feedback about their highly opinionated choices for the project.
To this day the same logic is still in caddy, but with a longer grace period. Try starting caddy offline with a cert that expires in less than 7 days.
I don't trust someone to develop a server if they think failing to start at all, is better than starting with close to expired or even expired certificates.
https://github.com/icing/mod_md
this is being folded into the official Apache httpd repo.
Apache httpd in many ways is a much more complete web server solution compared to, say, nginx (which I like as well). I'm prefering Apache httpd for the depth of available documentation alone, because everything is developed openly, C source is accessible and not hard to develop with, and because of the breadth of available options (certbot/letsencrypt/acme reference implementation on Apache, WebDAV, clustering etc. etc.).
(And I'm writing this as a dev who worked on a competing web server which IM-biased-O was vastly better :)
It's kept up to date and it works for everything I want it to work with. This idea that because it's old and not hip therefore it's use is questionable is kinda dumb. I was disappointed when OBSD wrote their own httpd that was initially filled with stupid bugs.
If it ain't broke don't fix it.
Mostly I use it because, after probably 20 years or so, I'm quite comfortable with and accustomed to it.
That said, I have started putting a few of my own web sites on nginx just to try it out a bit and get some familiarity with it.
Yes, there are a few things apache is remarkable about - the npm-itk engine could run as user processes, which was nice for shared hosting - but all the problems with it - slowloris, anyone? - simply outweights the benefits.
Shared hosting should be containerised or vm-ised instead and run on nginx/lighttpd/cherookee/etc + php-fpm 7.0 which will save you a lot of headaches and CPU cycles.
.htaccess can be just as much a benefit as a curse.
(Note: WordPress is completely nginx + php-fpm friendly.)
Nginx is surely the first choice if you're rolling your own shared hosting infrastructure, but most legacy commercial shared hosting automation frameworks still use Apache and customers don't care what the underlying tech is serving their bits over HTTP.
I never understand these arguments that basically pit Apache2 on its own/with modules against a completely modular arrangement of tls/cache/web/app as different layers.
If you use apache as the "web" in that setup, it needs relatively few modules loaded and when configured correctly can be blazing fast.
Apache is far from perfect, but among its ilk it is probably the one of the more robust and well tested. It's not as if you don't have to care about connection hogging attacks such as slowloris with other http servers.
The cycles saved on the nginx http parser, which is completely irrelevant to the performance of any real world web application, comes at the price of being hand optimized and opaque. You can't really argue for that type of performance engineering at the same time as you argue for solid security. The world generally needs more secure software, not more performant protocol parsers.
Do also note that nginx had a very similar data leak bug not even two months ago in the range header.
http://svn.apache.org/viewvc?view=revision&revision=1807754
The flaw only occurs if you have a "Limit" configuration that specifies an invalid HTTP method. For some unknown reason, this causes the leak. A "Limit" with a valid method, or no "Limit" at all, will not provoke the bug. I have to imagine that most Apache setups either don't configure this or configure it only with valid methods.
The article points out that shared hosting setups which allow individual users to customize settings through .htaccess would be vulnerable, since anyone on the machine could set an invalid configuration and then use the vulnerability to potentially get data from other users.
Obviously you should patch it when you can, but it is probably not time for most of us to panic.