And the summary is, separate "system" requirements/dependencies from those that packages depend on so that rebuilding a package doesn't break the system.
I think it is the opposite way around, so that rebuilding the system doesn't break a package.
Basically, they can switch system libraries or delete them entirely without the packages being affected since they will be using the non-private libraries from packages / ports.
I still can't understand why several BSDs so stubbornly cling to ports as a "solution" to package management. I ran into this problem, and variants of it, over and over again when trying to support FreeBSD in our products, in the past. Though pkgng may resolve some of my problems with package management on FreeBSD (it at least allows third party repositories, the lack of which was a big pain point for me in the past), I haven't tried it yet, and from what I can tell FreeBSD still favors ports, and several other BSDs are still using ports exclusively.
I've even seen this library mismatch happen on a couple of occasions in a fresh install, where installing a variety of ports leads to one or some of them being broken immediately after install because several packages called for different versions of libraries. And, the longer a BSD system is in service, the more likely some portupgrade, or installation of a new package, is to break some other thing on the system, requiring far reaching rebuilds (and sometimes, manual intervention, because the dependencies specified aren't sufficient to allow the system to automatically fix itself).
I begin to feel like a broken record saying this again and again, but: ports is a clever tool, but it is not a solution to the problem of package management.
There's so much to love about the BSDs, but lack of effective package management has always made it a non-starter for any purpose, for me. Effective package management is not esoteric or bleeding edge or experimental stuff, at this point. deb-based systems had apt 20+ years ago, RPM based systems had yum ~12 years ago. Slapping a band aid on the darned thing doesn't fix it.
And, so many conversations about packages devolve into ports proponents missing the point, and saying, "Well, you can install binary versions of ports, so it's fine." When the availability of binary packages is utterly orthogonal to the problems I'm talking about (though also a big deal, and building from source on a production system is silly and potentially dangerous).
So, I think I need to stop complaining about it, and start asking the right questions of BSD users. Why is ports such a fiercely defended thing when it has so many weaknesses compared to modern package managers? Does pkgng actually solve these problems (I can't really tell, it seems to be a meta-package system that still sits atop ports)?
Should be noted that OpenBSD has package tools and management much more along the lines of apt or yum. Ports on openbsd build into packages so you dont get a weirdly broken system if you do need to dip into ports for some reason (although the general recommendation is to use binary packages)
I don't think I've used ports on FreeBSD and have stuck with pkg. PC-BSD has a pretty extensive package management system[1], and and NetBSD's package management is used by multiple operating systems[2].
Ports isn't really used unless you have some pretty specific needs.
Library mismatches, dependency conflicts and progressive destabilization/entropy (system package configurations approaching global failure conditions like boot stalling as time passes and upgrades are made) were all frequent experiences I had on dpkg-based systems with the APT frontend that ultimately made me return to Slackware and its spartan package management where I have since experienced nothing of the sort.
OpenBSD has both binary packages and ports. Actually, the packages are built from ports, so if you want to customize a package, you checkout the ports tree, adjust the Makefile, add patches or whatever it is you need, and then you get a nice package that you can use on other OpenBSD installations (provided they are running the same version).
This is similar to what one does with, say, APT (apt-get {build-dep, source} <pkg>; modify; dpkg-builpackage ...).
When you get install a release, you set a package source (/etc/pkg.conf is more or less the equivalent of /etc/apt/*), and then you just pkg_add <pkg>. Package versions are fixed, pretty much the same way you get when you run debian stable.
There is one major difference though: OpenBSD is an OS, while Debian is a distribution. This means that debian will install the package files relatively to /, while OpenBSD will install them relatively to /usr/local.
It may sound stupid if you've only used Linux before, but it makes sense: files in /usr/bin, /usr/sbin,... are files that were shipped with the OS. What lives in /usr/local is what you installed on top of it.
Other than that, I don't see any major difference. You occasionally get some libraries that are out of sync with a package, but that happens on -current and it's usually that the package was not rebuilt yet on the mirror you're using. (Ever run Debian sid ? Well, it's the same thing)
What I'm saying is that OpenBSD uses ports effectively and in day to day usage I see no difference to APT-based distributions. I've never used it, but I hear very good things about pkgsrc (developed by NetBSD but it can also be used elsewhere. AFAIK, Joyent uses it and some people run it on Linux too). From what I can see on the homepage, it provides both sources and binaries.
Until pkgng was introduced, FreeBSD was lagging behind but from what I can tell, it seems the situation has improved.
What "weaknesses" related to ports do you have in mind ?
"What "weaknesses" related to ports do you have in mind ?"
The ones I've mentioned (and mentioned in the article we're commenting on):
- Library mismatches leading to broken packages, sometimes on a freshly installed system. I've always heard outright denial that this happens, but I've literally never installed a FreeBSD system that didn't end up with something broken by the time everything needed for a fully functioning web hosting system was installed. It's like there's Stockholm syndrome happening, or something.
- Poor dependency resolution, and weak validation tools, that requires manual intervention to prevent or repair those breakages which are bound to occur.
Other complaints I have about ports:
- No reasonable mechanism for offering third party package repositories.
- Default ports are usually anemic, missing very basic functionality, guaranteeing you have to build some things from source in order to get a really functional version of, say, Apache. Now, you have to make your own means of distributing these custom builds across your systems, without any reasonable means of setting up your own repository (you can duplicate the whole ports tree, and merge your changes, which is ridiculous, and the tools to help with this are weak).
Really, the biggest failings are in dependency specification and resolution. I don't know if it is a process failure (e.g., nobody is taking the time to specify things precisely enough to end up with a working package in every circumstance) or failure of the system (e.g., the system doesn't provide the means to specify things precisely enough to insure a functional system), or a combination of the two. But, the end result is that it is frightfully easy to end up with broken software, and often requires a lot of knowledge and human interaction to fix it.
Again, providing binaries is fine and all (and, I think, mandatory for large automated deployments), but is not the basis of the problems I've experienced with ports. The fragility of ports is my problem with it, not the surface level differences between how things are installed and updated.
> but I've literally never installed a FreeBSD system that didn't end up with something broken by the time everything needed for a fully functioning web hosting system was installed. It's like there's Stockholm syndrome happening, or something.
You must be doing something wrong if every FreeBSD install you do ends up broken.
Also before updating software in FreeBSD it's nice to take a ZFS snapshot before hand so you can roll back if something goes wrong, i've never had to rollback personally but it's still a nice precaution.
> - No reasonable mechanism for offering third party package repositories.
What do you mean? you can change pkg repos's by changing the configuration file. [1]
>- Library mismatches leading to broken packages, sometimes on a freshly installed system. I've always heard outright denial that this happens, but I've literally never installed a FreeBSD system that didn't end up with something broken by the time everything needed for a fully functioning web hosting system was installed. It's like there's Stockholm syndrome happening, or something.
I'm not questioning that is your experience, but I am curious what packages specifically, and what specifically broke. Because in my almost 20 years of using FreeBSD, I have literally never once had that happen.
Just generically saying "It always breaks" without examples is pretty difficult for anyone to defend or even begin to try to counter.
On FreeBSD 8 (I think), installing Postfix alongside several other packages that relied on OpenSSL led to Postfix breaking, because the OpenSSL version was updated but Postfix was not. This happened regardless of binary or source install. I don't remember the specifics, honestly, as it's been a while. But, it required manual intervention and patching the port to make it rebuild a functioning Postfix.
On some earlier FreeBSD version, I had a dependency hell situation arise with Apache. Again, some library got moved out from under it, and dependencies weren't sufficiently specific to allow it to be automatically fixed, even with a complete rebuild of all of the packages (which would be a ridiculous requirement that I would still complain about..but even that wouldn't work).
On the same system, saslauthd was utterly broken for months (maybe even years), whether installed from binary or source (there was an issue about it, nothing happened). It required me to manually patch it and update it to make it work.
Also, despite assertions to the contrary, I've found that "mixed" systems of binaries plus ports also often leads to brokenness (the port install will lead to a library being upgraded out from under the binary install software, requiring a port install from source to fix the problem). That should never happen. Random shit should never just stop working because you installed another package, and I've found it to happen at least once or twice on any FreeBSD system I've ever dealt with.
All of this is fuzzy memories. It's been a few years since I earnestly tried to support FreeBSD in our products, because the package management situation was such an ugly mess. Before that some of the kernel limits were problematic (like group membership was limited to 128 or 256 groups, which broke our permissions model for Apache back then, so we had to rebuild the kernel just to make our software functional). There's just been a lot of pain involved in making FreeBSD work for us, and I've spent more fruitless hours trying to make it work than any other system we support (and for a much smaller pool of paying customers...we can count the FreeBSD users on our fingers, vs. a few thousand Linux users). Despite a very low return on investment, I still dig in every few years and waste hours of my time fighting with shitty package management, because I really want to support it.
Anyway, let me be clear about this: If you have to manually patch or edit anything to get a functional installation, your package management tools are broken. In my experience, that's always been required. Admittedly, our stack of dependencies for our software is very large (Apache, MySQL, PostgreSQL, Postfix, Dovecot, Cyrus saslauthd, BIND, procmail, spamassassin, ClamAV, Perl and several modules, bash, awstats, modern Python and Ruby versions, and more that isn't coming to mind right now).
Can I make it work? Yes, I've done it a handful of times in our ten years of making Virtualmin as a commercial product. I honestly want to support FreeBSD, and always have wanted to support FreeBSD. xBSD users are almost universally extremely competent; they file useful bug reports, they're helpful when I have problems based on the limits of my knowledge of FreeBSD. But, it's always a massive pain in the ass, and when a new version rolls out, I find I have another stack of problems to work out manually. Most of those problems are related to package management failures.
Also, while xBSD users are often awesomely bright folks, they're also extremely defensive, to the point of it being difficult to address problems (like this one). We're talking about a post in the DragonflyBSD list that discusses this very problem, the exact thing I'm belly aching about, is a thing that is acknowledged as a longstanding issue in ports by the developers who have made some changes to try to address the problem. Why are we arguing about whether ...
>(Apache, MySQL, PostgreSQL, Postfix, Dovecot, Cyrus saslauthd, BIND, procmail, spamassassin, ClamAV, Perl and several modules, bash, awstats, modern Python and Ruby versions, and more that isn't coming to mind right now).
Outside of Postgres AND MySQL on the same box... your requirements are actually pretty standard, and something I always deploy. I guess I'll leave it to others to comment - but that's basically my standard install, so I'm at a loss. I've never had it break, thought perhaps you had some odd corner-case package that wasn't properly being vetted out. Unfortunately we can't rewind time - I'd strongly suggest you post the bugs on the bug tracker in the future, because your experience is definitely not something I'd consider normal.
>We're talking about a post in the DragonflyBSD list that discusses this very problem, the exact thing I'm belly aching about, is a thing that is acknowledged as a longstanding issue in ports by the developers who have made some changes to try to address the problem. Why are we arguing about whether it happens?
Why are we arguing about whether it happens? Because you didn't report it. When I asked for specifics, you should've provided a link to the FreeBSD bug tracker where you reported the issue. Instead, we're talking about it on a social media site 7 years after the fact, with vague details about the packages that caused the issue.
I literally don't have a horse in this race (I don't contribute to FreeBSD, I don't rely on it for my livelihood, I'm literally just a longtime luser) - but it honestly feels like you're attacking the project with unsubstantiated claims.
"Why are we arguing about whether it happens? Because you didn't report it."
On what do you base that assertion? (Actually, I didn't need to report most of the problems I ran into. They were already reported, but not fixed, when I came upon them. But, I've been an open source developer for 20+ years. I report shit.)
> - Library mismatches leading to broken packages, sometimes
> on a freshly installed system. I've always heard outright
> denial that this happens, but I've literally never installed
> a FreeBSD system that didn't end up with something broken by
> the time everything needed for a fully functioning web hosting
> system was installed. It's like there's Stockholm syndrome
> happening, or something.
I run into this less frequently than I did on rpm (CentOS, RHEL) distributions. "rpm hell" was a real thing, and still continues to a lesser degree. Try building an rpm for anything recent on an rpm distro and it is painful.
> - Poor dependency resolution, and weak validation tools, that requires
> manual intervention to prevent or repair those breakages which are
> bound to occur.
I haven't personally run into this in a long time.
> - No reasonable mechanism for offering third party package repositories.
pkgng does this. In fact, building entire custom package sets and repos is quite easy with poudriere and portshaker.
> - Default ports are usually anemic
I use fairly exotic options for some things, so I tend to build more packages than most people probably do. poudriere makes it easy to build sets with different options regularly. pkg makes creating your own repo quite easy too -- certainly moreso than setting up your own rpm repo with yum.
Ports used to have all those problems you outlined (prior to round about FreeBSD 9 probably) -- hell, you couldn't even reliably mix ports and packages! It was awful. Now ports simply build packages, and those get installed with pkgng. Much cleaner.
> So, I think I need to stop complaining about it, and start asking the right questions of BSD users. Why is ports such a fiercely defended thing when it has so many weaknesses compared to modern package managers?
First theres no dichotomy here. A person can use ports and pkgng. Secondly there's many nice thing about ports. If your organization has custom patches or compile options for common software, you can use the port system. Also I think a more important use of ports, is security fixes. When OpenSSL drops it's bug of the week you can patch openssl and rebuild the port tree. Then you can push the new, patched binaries into production. A sysadmin can do this all without relying on the package maintainer, ie no waiting on Redhat to push out new rpms. This can all be done in under an hour depending on the performance of the build server and automation processes in place.
> Does pkgng actually solve these problems (I can't really tell, it seems to be a meta-package system that still sits atop ports)?
Yes.
> and building from source on a production system is silly and potentially dangerous
Well of course you don't build directly on the production box. That's what a build server is for. You can use poudriere to build packages from the ports tree and then run your own package repo and have your production boxes pull from your private package repo.
"Also I think a more important use of ports, is security fixes. When OpenSSL drops it's bug of the week you can patch openssl and rebuild the port tree. Then you can push the new, patched binaries into production. A sysadmin can do this all without relying on the package maintainer, ie no waiting on Redhat to push out new rpms."
I can't remember the last time I was aware of a security problem before Red Hat (and CentOS) had rolled out a fixed package. I usually know about security issues when I see a slew of update notifications from my servers (even though I'm on several security related mailing lists).
Realistically, I rely on my OS vendor to provide my security fixes in a timely manner. If my systems security relied on me being able to apply patches manually in a timely manner, it would be a disaster waiting to happen. And, it is a big pile of labor I don't want on my plate.
Besides, nothing stops one from installing the source package (whether RPM or deb), adding the patch, rebuilding, and pushing to your local yum or apt repository. That workflow is entirely reasonable on Linux, as well (historically easier, since maintaining a third party repo was so much easier for yum, though apt is a bit more cantankerous; it looks like FreeBSD finally supports multiple repositories and you don't need to mirror the whole ports tree just to provide a handful of custom packages, which used to be the case). ports does not provide a unique feature here, and if it is required to manually patch software in order to be reasonably quick to address security issues, that's a bug not a feature.
> I can't remember the last time I was aware of a security problem before Red Hat (and CentOS) had rolled out a fixed package. I usually know about security issues when I see a slew of update notifications from my servers (even though I'm on several security related mailing lists).
Well looking at the last batch of OpenSSL security advisories, CVE-2015-3197 was reported and fixed by OpenSSL two days ago. [1]
Yet it looks like RedHat has yet to issue a fix. [2]
> Realistically, I rely on my OS vendor to provide my security fixes in a timely manner. If my systems security relied on me being able to apply patches manually in a timely manner, it would be a disaster waiting to happen. And, it is a big pile of labor I don't want on my plate.
It's not difficult to subscribe to your OS vendors security mailing list as well as the security mailing list for ALL the software you run in production.
So what? There is nothing unique about ports in this regard. If you want to manually patch your software, you can do it equally easily with deb/apt or rpm/yum/dnf.
I begin to feel like a broken record saying this again
and again, but: ports is a clever tool, but it is not
a solution to the problem of package management.
Of course the ports tree itself is just package build infrastructure. Those makefiles aren't a fully realized package management system. No one disputes that.
Prior to pkgng the answer to all of your problems (which basically boil down to wanting an apt- or yum-like user experience) was to create your own private binary package repo using tinderbox and manage installed packages using something like "portupgrade -PP" or portmaster. If you didn't need anything built with non-default options, you could use portupgrade/portmaster along with the public binary package repos, although I don't know how well maintained these were. If you didn't want to bother with tinderbox (and this is the route I took), you would use portupgrade or portmaster to handle all package management, with all packages built locally from source (ports tree). This latter approach worked well for me even though package building took a while, since all of my FreeBSD systems were servers and each only had a limited set of packages installed. Of course it doesn't scale. I was rather under the impression that most enterprise-scale FreeBSD installations went the tinderbox route.
So, I think I need to stop complaining about it, and
start asking the right questions of BSD users. Why is
ports such a fiercely defended thing when it has so
many weaknesses compared to modern package managers?
Does pkgng actually solve these problems (I can't
really tell, it seems to be a meta-package system
that still sits atop ports)?
I'm going to answer your second question first: use pkgng. Seriously. It's a complete binary package management system. It works just like apt or yum. The binary package repos are well maintained. Most everything people usually want is in there.
I can't speak for other BSD sysadmins, but I love being able to customize package builds to meet my needs. Sure, it's possible to build one's own debs or rpms or what have you, but I think poudriere makes maintaining a private package repo much simpler than something like Spacewalk. And in fact, that's how I do things nowadays. I maintain my own private package repo with everything built to my liking and updated my schedule.
This is great. A good example of why this is great, is openssl. Because ports/packages build with the base openssl, a security team supporting a release usually has to maintain an abi compatible version in base. If ports/packages simply used their a packaged of openssl, it would make it much easier for ports/packages to update in lockstep, as well as the base system to update on its own schedule (update to newer version and simply update any outdated api usages in base as well -- and for example to use libressl instead).
I think BSD's exporting so many libraries as part of the base install made much more sense in the time with small hard drives, but maybe not as much today.
Because ports/packages build with the base openssl,
a security team supporting a release usually has to
maintain an api compatible version in base.
Forcing ports to compile against the security/openssl port instead of the base version is pretty easy. You add the following knob to /etc/make.conf, to the port's Makefile, or to the make(1) command line:
WITH_OPENSSL_PORT=yes
The reason the security team has to maintain an API/ABI-compatible release of OpenSSL is because FreeBSD releases make that guarantee for all base system libraries---and the reason OpenSSL is in the base system in the first place is because other things in base like OpenSSH and Heimdal require it. This guarantee lets me build packages on FreeBSD 10.0 that work fine on FreeBSD 10.2 (and, generally, vice versa).
> Forcing ports to compile against the security/openssl port
> instead of the base version is pretty easy.
Yes. I do this currently, and it works great. But not everyone does, so the security team has to keep abi compatibility.
> The reason the security team has to maintain an API/ABI-compatible release
> of OpenSSL is because FreeBSD releases make that guarantee for all base
> system libraries---and the reason OpenSSL is in the base system in the
> first place is because other things in base like OpenSSH and Heimdal
> require it.
Head scratching That was...kind of my whole point. Anyway, openssl is planned to become private, so I just need to wait a bit (maybe Allan will get it into FreeBSD-11? see[1]).
27 comments
[ 3.1 ms ] story [ 66.2 ms ] threadAnd the summary is, separate "system" requirements/dependencies from those that packages depend on so that rebuilding a package doesn't break the system.
Basically, they can switch system libraries or delete them entirely without the packages being affected since they will be using the non-private libraries from packages / ports.
I've even seen this library mismatch happen on a couple of occasions in a fresh install, where installing a variety of ports leads to one or some of them being broken immediately after install because several packages called for different versions of libraries. And, the longer a BSD system is in service, the more likely some portupgrade, or installation of a new package, is to break some other thing on the system, requiring far reaching rebuilds (and sometimes, manual intervention, because the dependencies specified aren't sufficient to allow the system to automatically fix itself).
I begin to feel like a broken record saying this again and again, but: ports is a clever tool, but it is not a solution to the problem of package management.
There's so much to love about the BSDs, but lack of effective package management has always made it a non-starter for any purpose, for me. Effective package management is not esoteric or bleeding edge or experimental stuff, at this point. deb-based systems had apt 20+ years ago, RPM based systems had yum ~12 years ago. Slapping a band aid on the darned thing doesn't fix it.
And, so many conversations about packages devolve into ports proponents missing the point, and saying, "Well, you can install binary versions of ports, so it's fine." When the availability of binary packages is utterly orthogonal to the problems I'm talking about (though also a big deal, and building from source on a production system is silly and potentially dangerous).
So, I think I need to stop complaining about it, and start asking the right questions of BSD users. Why is ports such a fiercely defended thing when it has so many weaknesses compared to modern package managers? Does pkgng actually solve these problems (I can't really tell, it seems to be a meta-package system that still sits atop ports)?
Ports isn't really used unless you have some pretty specific needs.
1) http://web.pcbsd.org/doc-archive/10.1.2/html/update.html
2) http://www.netbsd.org/docs/software/packages.html
This is likely a YMMV issue.
This is similar to what one does with, say, APT (apt-get {build-dep, source} <pkg>; modify; dpkg-builpackage ...).
When you get install a release, you set a package source (/etc/pkg.conf is more or less the equivalent of /etc/apt/*), and then you just pkg_add <pkg>. Package versions are fixed, pretty much the same way you get when you run debian stable.
There is one major difference though: OpenBSD is an OS, while Debian is a distribution. This means that debian will install the package files relatively to /, while OpenBSD will install them relatively to /usr/local.
It may sound stupid if you've only used Linux before, but it makes sense: files in /usr/bin, /usr/sbin,... are files that were shipped with the OS. What lives in /usr/local is what you installed on top of it.
Other than that, I don't see any major difference. You occasionally get some libraries that are out of sync with a package, but that happens on -current and it's usually that the package was not rebuilt yet on the mirror you're using. (Ever run Debian sid ? Well, it's the same thing)
What I'm saying is that OpenBSD uses ports effectively and in day to day usage I see no difference to APT-based distributions. I've never used it, but I hear very good things about pkgsrc (developed by NetBSD but it can also be used elsewhere. AFAIK, Joyent uses it and some people run it on Linux too). From what I can see on the homepage, it provides both sources and binaries.
Until pkgng was introduced, FreeBSD was lagging behind but from what I can tell, it seems the situation has improved.
What "weaknesses" related to ports do you have in mind ?
The ones I've mentioned (and mentioned in the article we're commenting on):
- Library mismatches leading to broken packages, sometimes on a freshly installed system. I've always heard outright denial that this happens, but I've literally never installed a FreeBSD system that didn't end up with something broken by the time everything needed for a fully functioning web hosting system was installed. It's like there's Stockholm syndrome happening, or something.
- Poor dependency resolution, and weak validation tools, that requires manual intervention to prevent or repair those breakages which are bound to occur.
Other complaints I have about ports:
- No reasonable mechanism for offering third party package repositories.
- Default ports are usually anemic, missing very basic functionality, guaranteeing you have to build some things from source in order to get a really functional version of, say, Apache. Now, you have to make your own means of distributing these custom builds across your systems, without any reasonable means of setting up your own repository (you can duplicate the whole ports tree, and merge your changes, which is ridiculous, and the tools to help with this are weak).
Really, the biggest failings are in dependency specification and resolution. I don't know if it is a process failure (e.g., nobody is taking the time to specify things precisely enough to end up with a working package in every circumstance) or failure of the system (e.g., the system doesn't provide the means to specify things precisely enough to insure a functional system), or a combination of the two. But, the end result is that it is frightfully easy to end up with broken software, and often requires a lot of knowledge and human interaction to fix it.
Again, providing binaries is fine and all (and, I think, mandatory for large automated deployments), but is not the basis of the problems I've experienced with ports. The fragility of ports is my problem with it, not the surface level differences between how things are installed and updated.
You must be doing something wrong if every FreeBSD install you do ends up broken.
Also before updating software in FreeBSD it's nice to take a ZFS snapshot before hand so you can roll back if something goes wrong, i've never had to rollback personally but it's still a nice precaution.
> - No reasonable mechanism for offering third party package repositories.
What do you mean? you can change pkg repos's by changing the configuration file. [1]
[1]: https://www.freebsd.org/cgi/man.cgi?query=pkg.conf&sektion=5...
That is new in FreeBSD 10, and did not apply to the versions I was working with. I assume it is part of pkgng?
I'm not questioning that is your experience, but I am curious what packages specifically, and what specifically broke. Because in my almost 20 years of using FreeBSD, I have literally never once had that happen.
Just generically saying "It always breaks" without examples is pretty difficult for anyone to defend or even begin to try to counter.
On FreeBSD 8 (I think), installing Postfix alongside several other packages that relied on OpenSSL led to Postfix breaking, because the OpenSSL version was updated but Postfix was not. This happened regardless of binary or source install. I don't remember the specifics, honestly, as it's been a while. But, it required manual intervention and patching the port to make it rebuild a functioning Postfix.
On some earlier FreeBSD version, I had a dependency hell situation arise with Apache. Again, some library got moved out from under it, and dependencies weren't sufficiently specific to allow it to be automatically fixed, even with a complete rebuild of all of the packages (which would be a ridiculous requirement that I would still complain about..but even that wouldn't work).
On the same system, saslauthd was utterly broken for months (maybe even years), whether installed from binary or source (there was an issue about it, nothing happened). It required me to manually patch it and update it to make it work.
Also, despite assertions to the contrary, I've found that "mixed" systems of binaries plus ports also often leads to brokenness (the port install will lead to a library being upgraded out from under the binary install software, requiring a port install from source to fix the problem). That should never happen. Random shit should never just stop working because you installed another package, and I've found it to happen at least once or twice on any FreeBSD system I've ever dealt with.
All of this is fuzzy memories. It's been a few years since I earnestly tried to support FreeBSD in our products, because the package management situation was such an ugly mess. Before that some of the kernel limits were problematic (like group membership was limited to 128 or 256 groups, which broke our permissions model for Apache back then, so we had to rebuild the kernel just to make our software functional). There's just been a lot of pain involved in making FreeBSD work for us, and I've spent more fruitless hours trying to make it work than any other system we support (and for a much smaller pool of paying customers...we can count the FreeBSD users on our fingers, vs. a few thousand Linux users). Despite a very low return on investment, I still dig in every few years and waste hours of my time fighting with shitty package management, because I really want to support it.
Anyway, let me be clear about this: If you have to manually patch or edit anything to get a functional installation, your package management tools are broken. In my experience, that's always been required. Admittedly, our stack of dependencies for our software is very large (Apache, MySQL, PostgreSQL, Postfix, Dovecot, Cyrus saslauthd, BIND, procmail, spamassassin, ClamAV, Perl and several modules, bash, awstats, modern Python and Ruby versions, and more that isn't coming to mind right now).
Can I make it work? Yes, I've done it a handful of times in our ten years of making Virtualmin as a commercial product. I honestly want to support FreeBSD, and always have wanted to support FreeBSD. xBSD users are almost universally extremely competent; they file useful bug reports, they're helpful when I have problems based on the limits of my knowledge of FreeBSD. But, it's always a massive pain in the ass, and when a new version rolls out, I find I have another stack of problems to work out manually. Most of those problems are related to package management failures.
Also, while xBSD users are often awesomely bright folks, they're also extremely defensive, to the point of it being difficult to address problems (like this one). We're talking about a post in the DragonflyBSD list that discusses this very problem, the exact thing I'm belly aching about, is a thing that is acknowledged as a longstanding issue in ports by the developers who have made some changes to try to address the problem. Why are we arguing about whether ...
Outside of Postgres AND MySQL on the same box... your requirements are actually pretty standard, and something I always deploy. I guess I'll leave it to others to comment - but that's basically my standard install, so I'm at a loss. I've never had it break, thought perhaps you had some odd corner-case package that wasn't properly being vetted out. Unfortunately we can't rewind time - I'd strongly suggest you post the bugs on the bug tracker in the future, because your experience is definitely not something I'd consider normal.
>We're talking about a post in the DragonflyBSD list that discusses this very problem, the exact thing I'm belly aching about, is a thing that is acknowledged as a longstanding issue in ports by the developers who have made some changes to try to address the problem. Why are we arguing about whether it happens?
Why are we arguing about whether it happens? Because you didn't report it. When I asked for specifics, you should've provided a link to the FreeBSD bug tracker where you reported the issue. Instead, we're talking about it on a social media site 7 years after the fact, with vague details about the packages that caused the issue.
I literally don't have a horse in this race (I don't contribute to FreeBSD, I don't rely on it for my livelihood, I'm literally just a longtime luser) - but it honestly feels like you're attacking the project with unsubstantiated claims.
On what do you base that assertion? (Actually, I didn't need to report most of the problems I ran into. They were already reported, but not fixed, when I came upon them. But, I've been an open source developer for 20+ years. I report shit.)
Ports used to have all those problems you outlined (prior to round about FreeBSD 9 probably) -- hell, you couldn't even reliably mix ports and packages! It was awful. Now ports simply build packages, and those get installed with pkgng. Much cleaner.
First theres no dichotomy here. A person can use ports and pkgng. Secondly there's many nice thing about ports. If your organization has custom patches or compile options for common software, you can use the port system. Also I think a more important use of ports, is security fixes. When OpenSSL drops it's bug of the week you can patch openssl and rebuild the port tree. Then you can push the new, patched binaries into production. A sysadmin can do this all without relying on the package maintainer, ie no waiting on Redhat to push out new rpms. This can all be done in under an hour depending on the performance of the build server and automation processes in place.
> Does pkgng actually solve these problems (I can't really tell, it seems to be a meta-package system that still sits atop ports)?
Yes.
> and building from source on a production system is silly and potentially dangerous
Well of course you don't build directly on the production box. That's what a build server is for. You can use poudriere to build packages from the ports tree and then run your own package repo and have your production boxes pull from your private package repo.
I can't remember the last time I was aware of a security problem before Red Hat (and CentOS) had rolled out a fixed package. I usually know about security issues when I see a slew of update notifications from my servers (even though I'm on several security related mailing lists).
Realistically, I rely on my OS vendor to provide my security fixes in a timely manner. If my systems security relied on me being able to apply patches manually in a timely manner, it would be a disaster waiting to happen. And, it is a big pile of labor I don't want on my plate.
Besides, nothing stops one from installing the source package (whether RPM or deb), adding the patch, rebuilding, and pushing to your local yum or apt repository. That workflow is entirely reasonable on Linux, as well (historically easier, since maintaining a third party repo was so much easier for yum, though apt is a bit more cantankerous; it looks like FreeBSD finally supports multiple repositories and you don't need to mirror the whole ports tree just to provide a handful of custom packages, which used to be the case). ports does not provide a unique feature here, and if it is required to manually patch software in order to be reasonably quick to address security issues, that's a bug not a feature.
Well looking at the last batch of OpenSSL security advisories, CVE-2015-3197 was reported and fixed by OpenSSL two days ago. [1]
Yet it looks like RedHat has yet to issue a fix. [2]
> Realistically, I rely on my OS vendor to provide my security fixes in a timely manner. If my systems security relied on me being able to apply patches manually in a timely manner, it would be a disaster waiting to happen. And, it is a big pile of labor I don't want on my plate.
It's not difficult to subscribe to your OS vendors security mailing list as well as the security mailing list for ALL the software you run in production.
[1]: https://www.openssl.org/news/vulnerabilities.html#y2016
[2]: https://access.redhat.com/security/cve/cve-2015-3197
Prior to pkgng the answer to all of your problems (which basically boil down to wanting an apt- or yum-like user experience) was to create your own private binary package repo using tinderbox and manage installed packages using something like "portupgrade -PP" or portmaster. If you didn't need anything built with non-default options, you could use portupgrade/portmaster along with the public binary package repos, although I don't know how well maintained these were. If you didn't want to bother with tinderbox (and this is the route I took), you would use portupgrade or portmaster to handle all package management, with all packages built locally from source (ports tree). This latter approach worked well for me even though package building took a while, since all of my FreeBSD systems were servers and each only had a limited set of packages installed. Of course it doesn't scale. I was rather under the impression that most enterprise-scale FreeBSD installations went the tinderbox route.
I'm going to answer your second question first: use pkgng. Seriously. It's a complete binary package management system. It works just like apt or yum. The binary package repos are well maintained. Most everything people usually want is in there.I can't speak for other BSD sysadmins, but I love being able to customize package builds to meet my needs. Sure, it's possible to build one's own debs or rpms or what have you, but I think poudriere makes maintaining a private package repo much simpler than something like Spacewalk. And in fact, that's how I do things nowadays. I maintain my own private package repo with everything built to my liking and updated my schedule.
But I guess PC-BSD could be considered a distro.
I think BSD's exporting so many libraries as part of the base install made much more sense in the time with small hard drives, but maybe not as much today.
[1]: https://wiki.freebsd.org/OpenSSL/Base