HN Explorer

Apache HTTPd Server Developers Considered Harmful

18 points by joesuf4 ↗ HN
For the past 25 years, I have been the lead developer of the libapreq2 subproject within the Apache HTTPd Server Parent Project. The original idea of libapreq as a safe/performant HTML form and Cookie parsing library came out of a collaboration between Lincoln Stein and Doug MacEachern in the late 90s.

It was my vision back then to transform the library into a generic, non-Perl related C library that would support language bindings from other programming languages, which is why I pushed for the project to be homes under the HTTPd umbrella instead of the Apache-Perl project.

While this vision was wildly successful, with language bindings available for several languages like Perl, TCL, R, etc, ever since about 2010 its proven tragic for the existing user community consisting of all of them, not just Perl.

What happened? Philip Gollucci, a Perl/FreeBSD olleague of mine at the time, started agitating that we promote the project to be released from inside the HTTPd server itself. What Philip didn’t know very well back then was how utterly vapid and territorial that team had become, which would have meant having to collaborate with them directly on user-facing decisions about the code base.

In 2012, Philip got what he wanted and I stopped resisting, so he forked the existing project and copied the C library components into HTTPd core.

In 2016 I resigned from the Foundation en masse. You can guess the reasons.

In 2020 or so, Google’s Security Team took advantage of an alpha release of httpd 2.5 by fuzzing its 8 year old copy of apreq. It found a few hotspots that needed repair.

Instead of having the courtesy of reaching out to me, or anyone else involved in development of apreq, a junior engineer on the HTTPd team went about the business of “bug fixing” the vulnerabilities Google found. You can see a record of his trial and error work in every release since then.

But the coup de grace was the 2022 release of 2.17, wherein the rookie developer purposely introduced a fatal bug into the codebase, breaking a fifteen year old regression test.

If you are wondering how something with a broken regression test winds up on CPAN, you’ll have to look into how RELENG is done in the server project.

Long story short, they commented out the test and shipped it anyway, and called it a Security Release that fixed a vulnerability every prior release was susceptible to.

Why do I care now? Because I’m the sucker users reach out to for answers as a known subject matter expert.

This sucks, but I’m sorry to tell you that my days wearing the Superman cape at Apache ended 8 years ago.

23 comments

[ 2.3 ms ] story [ 67.2 ms ] thread
keikobadthebad
> In 2016 I resigned from the Foundation en masse.

"En masse":

All together and at the same time, in large numbers

https://dictionary.cambridge.org/dictionary/english/en-masse

joesuf4
Apologies if the meaning was unclear, but what it's meant to convey that I resigned ALL of my dozen-or-so positions from the org simultaneously.
hu3
For what is worth I understood your meaning immediately by applying the "most generous interpretation" principle.
slau
Might I suggest phrasing it exactly like that then? “I resigned all 12 positions I held at Apache because X” is a lot clearer to me.
fickle
or even further, naming some of the relevant positions (for example HTTP Server project management committee)
joesuf4
How peevish can you guys be?
joesuf4
1/ because I don’t remember exactly how many, 2/ because the record keeping at Apache was so haphazard at the time, half of the places I resigned from left me on the roster anyway (like the HTTPd server project itself).
kkielhofner
I’ll try to say this as respectfully and as kindly as possible. I’m genuinely trying to be helpful.

Seeing these kinds of stories and interacting with the type of people who have them brings to mind an old saying:

“If everyone you meet is an asshole, you’re the asshole”

Note that I’m not saying that’s the case here or insulting you, that’s just the saying.

This story is filled with what seems like (at best) “personality conflicts” and your very dim view of essentially everyone involved in it with the exception of yourself.

It even ends with you essentially calling yourself Superman.

I’m not sure what the point of this is (vent? warning?) but it doesn’t read like a sympathetic and even remotely balanced telling of events, that’s for sure.

It may be worth reflecting on what you may have contributed to this if your goal is reducing the chance of repeats.

foofie
It reads like OP has been holding a grudge for a decade or so, and today decided to shit on a project he contributed way back in the past out of sheer entitlement.

I'm not sure if that's the case, but that what's coming across. Small-minded pettiness.

joesuf4
It depends on whether you have any experience with F/OSS communities. If you don't , then 25 years is irrelevant, so yeah compressing it into 1 paragraph makes it seem petty.
foofie
> It depends on whether you have any experience with F/OSS communities.

It really doesn't. Awful behavior regardless.

joesuf4
You go with that cowboy.
joesuf4
It’s both. 2.17 was a fraudulent release, the release manager is now Chair of the PMC.

As I tell everyone else- don’t trust their releases.

joesuf4
As of Valentines Day, what existed in svn‘s apreq/trunk is kosher, because I’ve personally vetted it.

Nothing else is.

joesuf4
As I tell all my longstanding Apache friends, being involved with the foundation was transformational for me.

But at the same time, F/OSS has transformed into a loss leader for FAANGM monopolies, which coincides neatly with the timeline presented.

joesuf4
2.17 was released in Aug 2022. Here's the changes file: https://httpd.apache.org/apreq/docs/libapreq2/apreq_changes....

Here's the bugtraq event about it: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1018191

Does any of this make sense to you? Nobody who looks in the changes file has any idea the last three releases involved CVE issues. Not only are they purposefully obscured, the release itself contains undocumented bogus parser changes that were papered over during candidate voting.

joesuf4
What I would have done differently:

1/ report the Google Fuzz Report to the actual development team, to coordinate and collaborate on a solution.

2/ not take three whacks at a security release just to fix the same vulnerability

3/ not attempt to ship whimsical patches in a security release

4/ not patch the test suite to lie to voters about the viability of a release candidate.

5/ ship a hotfix immediately after users report the bugs on 2.17

fickle
> For the past 25 years, I have been the lead developer of the libapreq2 subproject within the Apache HTTPd Server Parent Project.

25 years is a long time for an imaginary title on a vanity project. I guess it helps to work 12.5 years on, 12.5 years off (no commits, no releases).

> In 2016 I resigned from the Foundation en masse. You can guess the reasons.

NPD?

joesuf4
Here's a list of sites who announce mod_apreq2 in their ServerTokens.

There are literally thousands of others who do not advertise the version number out of security concerns.

https://webtechsurvey.com/technology/mod_apreq2

joesuf4
Most customers are running it behind a proxy server as part of their middle-tier webapp stack, so this isn't even a tiny fraction of the existing user base.

And despite the decade-long warfare against its userbase by the community of glorified morticians at Apache HTTPd Server Project, it still has a growing customer-base!