93 comments

[ 878 ms ] story [ 4243 ms ] thread
When the missing funding of OpenSSL was discussed, it came up several times, that OpenSSH, while doing great, is quite underfunded, too. I am glad to see them getting some money.

What i can't really comment on myself, but am reading from the OpenBSD guys is, that the OpenSSL team does quite well with FIPS consulting and has no increased interest in improving the library.[0]

Even if those claims are not true, it would be nice to see several other TLS libraries (GnuTLS, LibreSSL etc.) getting sponsored to get some healthy competition. Maybe, they could even directly compete for shares of the funding by the Linux Foundation in some way.

[0]: http://www.openbsd.org/papers/bsdcan14-libressl/mgp00008.htm...

Straight from the horse's mouth[1]

> Also, the income they earn though their paid consulting work supports their unpaid work on OpenSSL, so by hiring OpenSSL team members you are not only solving your own problems but also helping to ensure the long term viability of the OpenSSL product.

They also on their website list hourly consulting starting at $250/hour. Neither of these describe how much they get out of this, but it seems reasonable to say that the "OpenSSL runs of $2k/year" line is disingenuous at best.

[1] http://www.openssl.org/support/consulting.html

"but it seems reasonable to say that the "OpenSSL runs of $2k/year" line is disingenuous at best."

Is it? The last contract listed on that page is 4 years old. Maybe they don't regularly get contracts.

Just give the money to the OpenBSD team. We saw with OpenSSH that they have a proven track record taking crappy security software and fixing it. Why does everyone have this aversion to giving the OpenBSD team the funding they deserve?

And "Theo's a dick" doesn't qualify as a valid reason to not fund real security development. For the work those guys have done improving the security infrastructure of every operating system (they lead, others followed), the entire team deserves to be well-off dicks. It's to me the ultimate highlight of OSS's funding problem. People make millions/billions of dollars off of this software, and nobody ever contributes any of that back to the shoulders they stood on to make that happen.

Personally I think two healthy forks of OpenSSL is far, far better than one. I also think that the OpenBSD team is going to take some time before it reaches it's goals, and more importantly before it's in a stable enough point where people can start working on porting their version to other platforms.

I don't think it makes sense to leave OpenSSL to wither while that happens, especially since it's an actively used product.

That being said I'm far, far more confident in the OpenBSD team than I am the OpenSSL one.

Actually if I'm giving someone a donation, charity, then whether they are or are not a dick is a perfectly valid part of the decision. And to me how you run a project is as important as the quality of the final result.
You don't have to have a reason to not donate to something, so the color of their shoes is also a perfectly valid part of the decision. The important question is whether it's a good reason. If you rank the style of his speech as a more important issue than having secure software (unless you think that the style of speech will negatively effect the software), I'd wonder how a single person's personality got so high on your list of priorities.

It sounds a bit like voting for a president because he's the guy you feel you'd most enjoy having a beer with: short-sighted.

If you think that Theo's abrasiveness will affect the success of LibreSSL as a project, then it is is relevant.

It is naive and shortsighted to dismiss non-technical reasons. Often they can be as important as the technical reasons, or even more.

> And "Theo's a dick" doesn't qualify as a valid reason to not fund real security development.

Yeah, but people who give money usually tend to see that as a valid reason.

Seems to have worked out for Torvalds and good chunks of the Web ecosystem.
Most of the "Linus being a dick" examples are him being a dick to developers, and several of those cases are "him being a dick to developers because they did something that affects users". That's the kind of thing that's good for PR more than bad for it, because when Linus rages at a developer for breaking things, a large portion of users think "Linus has my back".
And that's not the case with Theo?
Oh, c'mon, give it a rest. Linus, while not necessarily mild-mannered, "gets" people. Theo sounds a lot like a socially clueless ubergeek. They are both abrasive superficially, but Linus is aware of the "meta", or strategic, level in human interactions, and knows when to back off; Theo is blind to it. That much is obvious even from a cursory glance.
Theo's "outspokeness" is believed to be the reason that DARPA canceled a million dollar development grant.

https://en.wikipedia.org/wiki/Theo_de_Raadt#DARPA_funding_ca...

That reads less like Theo being a dick and more like the US making a farce of free speech in that instance.
You have a right to make yourself heard, not be shielded from criticism and other's exercising the right to free association.
(comment deleted)
Of course, but the reasoning here feels flawed. Person A doesn't approve of war, so we're going to cancel a university project grant to punish him? Not a wonderful way of running a grant program (or a government).
I think you mean "excuse". Anyone who's ever attempted to sell anything will tell you of customers who say "I would buy it if only you turned it upside-down and painted it blue" but don't cough up when you do. Similarly, every single person reading this uses SSH daily but will go "ah, uh, but Theo's a dick!" as their excuse not to cough up.
The reason I don't contribute to OpenSSH is because it is impossible to do so directly. I have no way to know how much, if any, of my contribution goes to OpenSSH when I have to donate to "the OpenBSD foundation".

If you feel better convincing yourself that this is not the case, feel free. But it is still a fact, and I do contribute financially to a lot of other projects.

That reminds me of TrueCrypt and the TrueCrypt Audit Project.
> Just give the money to the OpenBSD team

Unless OpenBSD decides to change, that is the way to fund OpenSSH. Currently you don't get to decide how your donations are spent by them, OpenSSH isn't it's own spinoff funded group.

Why does everyone have this aversion to giving the OpenBSD team the funding they deserve?

One reason may be that donations to the OpenBSD Foundation are not tax-deductible in the US.

Maybe I'm misremembering, but SSH wasn't crappy, it was just proprietary. (I'm no fan of proprietary software, but let's at least be honest here.)
you are correct, http://en.wikipedia.org/wiki/OpenSSH

"OpenSSH was created by the OpenBSD team as an alternative to the original SSH software by Tatu Ylönen, which is now proprietary software. Although source code is available for the original SSH, various restrictions are imposed on its use and distribution. OpenSSH was created as a fork of Björn Grönvall's OSSH that itself was a fork of Tatu Ylönen's original free SSH 1.2.12 release, which was the last one having a license suitable for forking."

OpenBSD is not auditing OpenSSL. They're substantially rewriting it. The net effect is hopefully similar, but it's a very different path to get there. Further, the refactor might introduce new bugs, and it can easily miss subtle bugs (we're talking about cryptography, which is not as easy to spot or to fix "accidentally" [which is part of OpenBSD's M.O.] as memory corruption).

"Theo's a dick" has nothing to do with why funds are being applied to audit and not to "just have OpenBSD rewrite everything".

Isn't there two sets of money going towards OpenSSL? One set is for hiring 2 full time developers (which seems like the money that should go towards OpenBSD) and then the other set is going to OCAP for a security audit.
I would trust Theo's team over the OpenSSL team any day of the week.
That's great, but neither LibreSSL nor the Linux Foundation open audit project has anything whatsoever to do with the OpenSSL team.
(comment deleted)
A security audit may also miss subtle bugs, and the proposed corrections may introduce new bugs.

A rewrite has the benefit that it will lead to manageable code, instead of the current mess. Clean code has less places where subtle bug can hide, that does not change just because you are doing cryptography.

Anyway, they should send money to both. Both are important, and those companies make so much money using free software, they shouldn't be choosing the projects with that fine granularity. The problem is that they won't, and as much as kyledrake does not like the answer, it's because of Theo. Yes, it's a stupid decision, but it does not make it less real.

Since it is not in fact possible to wave a magic wand and get everyone to run LibreSSL, which isn't even complete, the virtues of auditing the OpenSSL codebase seem pretty straightforward to me. How much have you donated to open audit projects? I've donated a bit, and I would be pissed to find out that my money was redirected from an audit to a rewrite.

I am not in the least bit interested in the Theo vs. Whoever drama subtext. This whole subthread strikes me as similar to any other thread about a charitable donation, where people come out of the woodwork to cast aspersions that money wasn't given to some worthier cause.

The vulnerabilities in OpenSSL seem to be symptoms of a wider problem, systemic problem, that comes from their project management.

It seems appropriate to point out that auditing and fixing the details is just patching symptoms -- and that many think a longer-term solution is needed.

In the context of a funded audit, this is about as meaningful to me as the complaint that OpenSSL is insecure because it's written in C --- which is also true.
Consider that OpenSSL is able to look at libressl and choose the good parts, preserving interoperability (.vs openbsd only).

The reasons to fund the audit, and two full time devs, are clear.

Except that the 'good parts' of LibreSSL consist largely of two things:

1. Removing a ton of broken, obsolete, or poorly-designed code 2. Fixing bugs and incorporating fixes from the official OpenSSL bug tracker

In short, the 'good parts' of LibreSSL are a team of people willing to devote significant time and effort to making the OpenSSL codebase secure, clean, portable, and trustworthy. That's something that the OpenSSL devs either didn't have the time or energy to do.

As for interoperability: OpenBSD's process has always been to write a single core with a single target (OpenBSD) and then to have a separate team take care of platform-specific issues. This allows the team doing security work on LibreSSL/OpenSSH to focus on the results by writing a single implementation, and the porting team can spend their time porting that implementation to other platforms (rather than having someone implement functionality while trying to be aware of how each platform handles things).

It's worked well for OpenSSH so far, so I don't see a problem with using the same pattern for LibreSSL.

> How much have you donated to open audit projects?

Personally I've never donated for auditing, but in my free time I do look at code myself.

Today I had to install a proxy server, so I downloaded the source to Apache's traffic server. Within five minutes I'd decided that the code contained enough problems that I couldn't recommend using it.

Now I've reported those bugs, and made a contribution..

> Since it is not in fact possible to wave a magic wand and get everyone to run LibreSSL

You say that, but it's not as though this is a fundamentally hard problem. How many servers are running nginx today vs. even a few years ago?

Moreover, the biggest problem sits with the biggest, and often most capable, internet companies. So having a reasonable path for them to a more secure TLS implementation, even if it meant hand rolling their own packages or building from source, is potentially a big win for the internet in general.

And, of course, eventually switching from openssl to libressl will become as easy as spending a couple minutes with a package manager.

The fact is that this industry has a track record of keeping up to date technologically. This isn't some corporate IT in the industry gulag saddled with bullshit LoB apps that depend on IE6 in perpetuity. We're talking about the most capable and savvy tech companies in the world. If there's a better option to be had, it'll see high adoption rates.

> You say that, but it's not as though this is a fundamentally hard problem. How many servers are running nginx today vs. even a few years ago?

How many years did it take to get there? How many servers are still running apache?

> And, of course, eventually switching from openssl to libressl will become as easy as spending a couple minutes with a package manager.

Eventually, maybe. That's no reason not to audit openssl right now, that people cannot switch - especially the ones that use platforms that the OpenBSD team decided to remove support for.

The comparison isn't terribly apt; for example, LibreSSL's intention is to be a drop-in replacement for OpenSSL; if nginx were a drop-in replacement for Apache I suspect we would see a lot more people using it than there are now.

As for auditing OpenSSL, the OpenBSD team has found so many problems, misdesigns, misfeatures, idiotic decisions, and bugs, that it seems a shame to repeat that work again. If I had to choose, I'd say give the OpenSSL name to the OpenBSD guys, let them take over the project officially, and let the folks behind the OpenSSL foundation handle things like platform-dependent code, FIPS compliance patches, etc.

From Beck's talk[1] they are not reviewing the cryptography code, merely cleaning up the programming. They admit they don't yet have cryptography experts reviewing if the algorithms/implementations are doing what they are supposed to be doing and that they are in fact secure.

[1] http://www.youtube.com/watch?v=GnBbhXBDmwU

I get what you're saying. But I think this is a case where simplification and rewrites is going to greatly enhance the ability to effectively audit the code. I will trust LibreSSL audits about a hojillion times more than OpenSSL audits at this point, just down to code complexity and surface area alone.

And there is no question about it they absolutely will introduce new bugs. But hopefully the increased scrutiny and easier to read code means that they are less likely to have extremely bad bugs latent in the codebase for years without anyone noticing, as has been the case with OpenSSL.

True, but many of the things OpenBSD are doing to their fork are making automated capture of errors easier. Their work getting rid of the layers of crud around the OS memory allocator alone mean that tools like valgrind and other profilers now have a fighting chance of actually being useful.
Is it possible to give them more granular donations? I want to support OpenSSH and LibreSSL, but I don't cae about OpenBSD.

The reason I don't donate to them is that I feel like most of my donation will be going towards something I don't care about.

Its one and the same, plus if you care about LibreSSL and OpenSSH than you need to care about OpenBSD because of the assumptions that code makes and what needs to added to the ported code to make it safe on other platforms.
Horse feathers.

Consider that ntp.org is being funded, not OpenBSD (and their inferior OpenNTP).

NTP is infrastructure. OpenBSD is not.

Consider that OpenSSL is being funded, not OpenBSD ( and their unusable at this point libressl project).

OpenSSL is infrastructure.

OpenSSH got some funding, because it is widely used. It is infrastructure, not 3-5 people scratching an itch in a project.

> Consider that OpenSSL is being funded, not OpenBSD ( and their unusable at this point libressl project).

> OpenSSH got some funding, because it is widely used. It is infrastructure, not 3-5 people scratching an itch in a project.

I'm not sure what to say as I think you missed my point by a mile. Giving money to OpenSSH or LibreSSL is giving money to OpenBSD, it is the same people and same foundation.

OpenSSH and, now, LibreSSL are project written to OpenBSD first and then ported. Ported platforms that don't have the libraries these projects expect (which are developed for OpenBSD) have those functions ported[1]. They are the same project with parts ported to other platforms.

I have no clue what the 3-5 people comment is even about.

1) with some exceptions such as generation of entropy since only the OS can do that well.

> Why does everyone have this aversion to giving the OpenBSD team the funding they deserve?

Given their work is avowdly OpenBSD-only and portability has to be added on in the case of e.g. OpenSSH it would seem a dead-end for anyone who wants broader platform support.

Strange characterization given OpenSSH is everywhere so it isn't a dead-end.
They've explicitly said that they are seeking to reduce the surface area for bugs by initially removing cross platform compatibility, and then exposing what primitives the OS needs to supply to make it possible to port. This seems like a much much saner approach to me than trying to work through every quirk in every operating system known to man in the one project itself. Any OS that doesn't implement any of the primitives can legally go and take OpenBSD's implementation and use it if necessary (and hopefully contribute money or code back to them).

I'm not sure why you brought up OpenSSH, since it's by far the most widely used SSh implementation in the world...

The point is that no one wants to fund a team who won't produce a final working product for the system they're running.

Linux Foundation wants things which run on Linux. Trying to figure out how you fund the BSD version + porting effort leads to a weird incentive scheme.

It is possible the OpenSSH funding, since it is done through the OpenBSD Foundation, could, at the Foundation's discretion, go toward LibreSSL, since it's the same group.
No it's not. Libressl is a different team; one that feels a fork was more appropriate than just fixing the problems in openssl.

IMHO, libressl is a mistake. It's splitting resources over something that needs to be as air-tight as possible. I'd much rather have 1 really really good ssl library that everyone uses instead of 2 so-so ones.

> No it's not. Libressl is a different team

OpenSSH and LibreSSL are both a part of OpenBSD. So when you donate to the OpenBSD Foundation, you are very much donating to one project.

> one that feels a fork was more appropriate than just fixing the problems in openssl

You can't start fixing things in other peoples' source tree just like that. I'm pretty sure nothing useful would've come out of it if the OpenBSD folk had sent half a million lines in diffs to OpenSSL; http://www.openbsd.org/papers/bsdcan14-libressl/mgp00026.htm...

> You can't start fixing things in other peoples' source tree just like that.

Yes, you can. It's called contributing to a project. If the "half million lines of diffs" were actually things needing fixing, then the upstream team would accept them. If they are not necessary changes (such as ripping out all windows compatibility), then no, they would reject such changes.

It will take years, maybe a decade before a new ssl library becomes the "default". OpenSSL has a lot of ground covered and a lot of history. Yes, it's common knowledge that libressl started before heartbleed, but the reasons for the project being started are mostly along the lines of:

1) We don't think upstream would take these changes

2) We don't like some aspects of the design philosophy

3) We can do it better.

All 3 reasons can be collapsed into a more focused effort to fix the already existing and very good ssl library; openssl.

> Yes, you can. It's called contributing to a project. If the "half million lines of diffs" were actually things needing fixing, then the upstream team would accept them. If they are not necessary changes (such as ripping out all windows compatibility), then no, they would reject such changes.

I take it you've never dealt with an inactive/apathetic upstream before? Just because someone is the steward of a project does not mean they should be. This is perhaps one of the most valid reasons to fork!

The LibreSSL team says that there were big problems on the tracker that languished for years, such as OpenSSL not working correctly when you disable their custom memory allocator. If the OpenSSL team can't deal with bug reports in a timely fashion, what makes you think they will bother reviewing and merging hundreds of thousands of lines of code?

Then you become the steward of the project and continue forward. Forking will introduce an untold number of new bugs, some of which may be worse than imagined. Right now, native libressl only works on bsd's, when openssl codebase works on many os's. There are ports being made, which will introduce more bugs.

Bugs being in a tracker for years is not uncommon.

Here's OpenSSH's tracker:

https://bugzilla.mindrot.org/buglist.cgi?bug_status=__open__...

331 bugs, a large majority of which are pre 2012.

This is not a sign of inactive/apathetic developers. It's a sign of big and old projects.

I have no doubt the OpenBSD folk are excited about this now... but 5 years from now? More? What's the long term viability of this project? Will they eventually put all OS's on an equal footing instead of *BSD's first and port to other OS's?

Forking was not the answer. The answer was to fix the perceived problems in OpenSSL and make it as solid as it can be. It's splitting talent and resources unnecessarily. Especially when the two projects are under the same umbrella (OpenBSD Foundation).

> Then you become the steward of the project and continue forward. ... The answer was to fix the perceived problems in OpenSSL and make it as solid as it can be. It's splitting talent and resources unnecessarily.

But that is what the fork is, OpenSSL with new stewards. What is your objection? That they are using a different name? That they decided to remove certain platforms which were a maintenance burden? That FIPS is broken by design and therefore isn't a priority? I imagine the OpenSSL team disagrees with the LibreSSL team on all of these issues. The only option was a fork.

> Right now, native libressl only works on bsd's, when openssl codebase works on many os's. There are ports being made, which will introduce more bugs.

One step backwards, two steps forward.

Most of these are related to portable openssh and not openssh. These are two very different teams.
> It will take years, maybe a decade before a new ssl library becomes the "default". OpenSSL has a lot of ground covered and a lot of history.

Why? The LibreSSL team are keeping the API backwards compatible, so it will literally be a drop-in replacement (once there are ports to other platforms). It should just require a recompile.

Since the barrier to entry will be fairly low, it is foreseeable that it could become popular quite quickly.

Anyone interested in details of LibreSSL development can watch the BSDCan talk at https://www.youtube.com/watch?v=oM6S7FEUfkU - there really are a LOT of instances of braindamaged code in OpenSSL and a radical repair is pretty much the only thing which will work.
I'd much rather have 1 really really good ssl library that everyone uses instead of 2 so-so ones.

That's reasonable, but those aren't the options at play here. Not only because GnuTLS is already a thing, but the chances of OpenSSL becoming really really good are questionable.

An OpenBSD guy gave a talk a few weeks back where he said that Heartbleed wasn't the reason for the split, it was the reason for digging into the code and realizing that OpenSSL under current leadership isn't capable of being a really really good option.

https://www.youtube.com/watch?v=GnBbhXBDmwU

"just fixing the problems in openssl."

That's what libressl is about. If you're in any doubt, please see this talk: https://www.youtube.com/watch?v=GnBbhXBDmwU

There is no doubt that is the intention.

The doubt is whether or not it is a good call to fork openssl instead of attempting to get changes into upstream that fix it and make it better, safer, more reliable.

Are we talking about OpenSSL that had bugs languishing for years? Yeah, good luck with that one. LibreSSL was the way to go and the OpenBSD folks are the ones I trust to do it.
Yup, just like all these OpenSSH bugs:

https://bugzilla.mindrot.org/buglist.cgi?bug_status=__open__...

Fork it now!

~~~

Seriously, stop buying into all the hype generated by heartbleed. Things will simmer down, and it's doubtful libressl will replace openssl anytime in the next 5 years as the standard default ssl lib for many things.

I do not buy into OpenSSL devs not wanting bugfixes.

Where are the public rejections/closures of submitted fixes? There aren't any. There are just assumptions that they wont take certain patches, or submitted patches waiting for review (how about you jump in and help review?).

I'm actually looking forward to seeing how the OpenSSL problem will deal with their own legacy code, compared to how the OpenBSD developers have handled it.

It seems that own of the only ways of dealing with the OpenSSL code is to strip out the code for a large number of, should we say "less used platforms". Is the OpenSSL developers willing to drop support for 16 bit Windows or OpenVMS?

Is the OpenSSL developers willing to drop support for 16 bit Windows or OpenVMS?

They either need to properly maintain it or drop it, and they don't have enough money to maintain it.

I just want to know who's still compiling against 16bit windows or OpenVMS. I know my world view isn't infinite, but those systems seem a bit out there.
Part of my job involves writing software on OpenVMS. We actually just recently ported something that needed OpenSSL and were happy to find an up-to-date version.
Out of curiosity (sorry if that's offtopic), but what kind of workload are you running?

Is there anything except resources that prevent you from moving to a more modern platform? Of course, "it works" is a valid argument there, too. But you seem to be writing new code, too.

No, nothing except resources is in the way of completion of a move to a more modern platform. It's coming gradually, but we can't drop everything for a year or two to devote all of our engineering resources to getting us there.

OpenVMS has some really good ideas baked into the OS that we've had to reimplement or find off-the-shelf solutions for our new platform (for example a distributed key-value store (called "logicals"), a job queue system, and a clustered filesystem) but nothing so earth-shattering that it would keep us on VMS.

The biggest downsides are expensive hardware (OpenVMS is designed around clusters of a few beefy boxes, rather than many commodity boxes), lack of community knowledge, and lack of new software available for the platform. (End of life is also looming: http://h71000.www7.hp.com/openvms/openvms_supportchart.html .)

What about BIND for DNS?
The DNS ecosystem is much more diverse. djbdns is considered to be the most secure, and there are a few other quality implementations. The root servers, for example, run a mixture of BIND and NSD, so no single bug can affect all of them.
I skimmed, but cannot seem to see which project is being supported when they say NTP.

When you support the OpenBSD Foundation you support:

- OpenBSD - OpenSSH - OpenBGPD - OpenNTPD - OpenSMTPD - LibreSSL

The wording makes me think that the initiative will be supporting something other than OpenNTPD

They're supporting 4 projects so far: OpenSSL, OpenSSH, NTPd, and an Open Crypto Audit Project (OCAP) audit of OpenSSL. The Network Time Protocol project is here: http://ntp.org/
How do code security audits actually work? Are various well-experienced people just combing through the code and trying to break it? Or is there a more formal process?
This depends on a couple of different things. The most important of which is "at what stage of development is the application? (i.e. how mature and well tested is this code)". Software Development Life Cycle (SDLC) processes are great when followed from the start. When they are applied long after the first 100k+ lines of code are written then its harder. A typical code audit for us (I do this professionally at http://leafsr.com) involves some threat modeling, attack surface enumeration, manual data-flow and taint analysis ("where does untrusted data come into this application and how is it handled") and finally just reading the code. Timing and scope will heavily influence how deep you can go. 1 week on OpenSSH will probably get you nothing, 6 weeks on OpenSSL will definitely get you something.

(edit: expanded on what is most important)

This is great news. NTP is one of the least appreciated OSS projects. Harlan and the rest of the ntp dev team are very helpful and deserve a lot of respect for keeping the clocks on time. I can only hope that increased ntp funding/awareness/development means that BitKeeper (not a typo) is finally replaced by git/mercurial.
(comment deleted)
If OpenSSL software foundation is a for profit operation, why are tech companies funding it(1) instead of LibreSSL?

1: http://arstechnica.com/information-technology/2014/04/tech-g....

The tech companies want OpenSSL to improve, and are willing to pay money; the OpenSSL guys will improve OpenSSL if paid money. What's the problem here?
The CII is not funding the OpenSSL Foundation; it is directly funding two OpenSSL developers, so they can work on whatever is best for OpenSSL, instead of whatever feature improvements contracted by the OpenSSL Foundation.

As a result, the people behind the OpenSSL Foundation are NOT taking a cut of the monies from the CII.

Just LibreSSL. Let OpenSSL die its deserved death. Portable LibreSSL will do wonders.
Having "Huawei" as one of the backers does not create confidance. Recent news shows that they had there hardware backdoored.

https://duckduckgo.com/?kh=1&q=Huawei&sites=www.schneier.com...

How is the NSA's ability to backdoor Huawei hardware relevant for Huawei's ability to provide money to help fund audits?

Presumably, the NSA hacking is a reason for Huawei to start caring a great deal more about investing in security.