Hadn't heard about Rocky Linux since the initial announcement, but this is really great to see. 10 year LTS is pretty great, especially coming from Amazon Linux where you're lucky for three years...
I'm super excited about this because it's being done in tandem with the release of Peridot, Rocky's internal tooling to build and maintain distro forks.
Is anyone aware of a decent reference comparing say Alma and Rocky?
I imagine the 'product' is essentially the same, but policies or procedures may differ - and these can be significant! It's what draws me to the Fedora project so much.
Asking as someone who wasn't really that bothered by CentOS Stream. I've seen it as basically rolling point releases, which wouldn't be a problem for me personally
My info might be slightly outdated cause last I looked into it was a few months ago, but I would guess it's not terribly different. Yes both are essentially the same product. Main differences from user perspective are reliability of package servers and speed of updates, plus community experience if you get involved there (friendly tip: I suggest you stay the hell away from the cess pool that is Reddit. That whole site probably needs to die in a fire)
*Alma Linux:*
- Works closer with upstream (actually sends patches and participates with dev on CentOS)
- Backed by a for-profit company (which I consider a good thing for longevity and reliability, but of course there's a risk of profit and community incentives misalignment)
- Fastest on updates when upstream releases
- Friendly and appreciative (generally speaking) toward Red Hat (without which none of their business would be possible)
*Rocky Linux*
- "Community" owned, but backed by some corporate money
- Very volunteer-run. Have some paid employees working on but many volunteers
- Hostile (and sometimes hateful) toward Red Hat
I mostly use CentOS Stream though and that has been rock solid. I use Alma on my router and on a prod database server that will be nearly impossible to rebuild that I migrated to Alma from Cent, but CentOS feels better to me now than it did before changing to Stream.
What are you going to do when support for CentOS Stream 8 ends in 2024? Alma Linux 8 is supported until 2029 at least - for me it's a no brainer to use Alma for something like a database server.
Red Hat does support in-place upgrades for RHEL via Leapp, and the process around this has been improving. Granted I know in the past this process was... sub-optimal and restrictive. It is true that CentOS has not supported this. I am hoping we can achieve a "dnf system-upgrade" process on RHEL, but we'll see how that goes.
We still recommend having strong backup, rebuild, and automation strategies regardless. I personally would rather spin up a side system, ensure everything works, and nuke the original. That would apply to either test a in-place upgrade or a from-scratch build.
I’ve never heard good things about the experience of using leapp and it always seems like an unloved project.
Upgrading major versions of RHEL is akin to jumping 5+ versions of Fedora at once and I don’t think it could ever really work well without waaaaay too much work that probably isn’t worth it.
The Alma Linux “elevate” project seems to be attempting that using leapp as a framework but also allowing migration between RHEL derivatives so we’ll see how that pans out I guess.
Sorry for the late reply. based on your username I suspect we are kindred spirits :-D
by 2024, for things still in service I will most likely upgrade it to CentOS Stream 9. by about that time I'm usually running in to annoying things about the older OS and I want updated tools and such, which is enough incentive to get the rebuild done.
The box that will be extremely hard to rebuild I already moved to Alma, where it will stay hopefully until EOL. There is one box I moved to Stream that I may just move to Alma to squeeze a few more years out of it in anticipation of it retiring. Overall though I really hate being on extended support (past the initial period). The farther in it gets the more headaches pop up. I got hosed by deprecations of SSL and old boxes that suddenly stopped working (because they were using protocols deemed too old/insecure by modern browsers). That was an absolute nightmare because it was instant prod outage and we had no time to plan.
Why should I use this over, say, Ubuntu or Linux Mint? What’s the unique selling point? And what’s the target audience besides Linux enthusiasts who still distrohop?
In case you don't intend distrohopping you use this for being "an open-source enterprise operating system designed to be 100% bug-for-bug compatible with Red Hat Enterprise Linux®."
So its actually for people trying to replace CentOS which RedHat decided to drop :)
It's a server distro (unlike Mint) in the Red Hat family (unlike Ubuntu). As a server-oriented distribution it has options for installing with a bare minimum of packages, including no GUI, and typically has longer support cycles. As a Red Hat derivative, it generally features more recent versions of core Linux userland things like systemd, since Red Hat employs a lot of the developers of those features. In comparison to Ubuntu it also has far less snap packages, which some people like because snaps have various problems in their current implementation.
The target audience is people who used to use Centos, before Red Hat ended Centos 8 years early and moved it to a rolling release schedule. Its primary competitors in the server-focused Red Hat derivatives are Red Hat Enterprise Linux itself, which is paid; Centos Stream, which is rolling release; or Alma Linux, which is, currently, basically the same distro due to how recent the Centos debacle was, but could diverge in the future.
It's off-brand RHEL. There's some server software that isn't compatible with any other distro. So if you wanted to run it, you either had to pay for a RHEL license, or run it on CentOS, which was the original official off-brand version of RHEL.
Then CentOS was killed by IBM/RedHat and reborn as the beta distro for the next RHEL release. Rocky Linux and Alma Linux were created to fill this niche. For example, you can run your production server on RHEL, but run your test and dev machines on Rocky while being reasonably sure that there won't be any compatibility issues.
Driver compatibility with redhat. Niche hardware like stack bypass network cards, packet capture cards, fpga cards will tend to have drivers issued for redhat, but not always for anything else. By using rocky/alma, you improve your odds to dodge hassle adapting the driver packs to other linux distros.
Net booting. Distros vary in how awkward it is to get stripped setups set up for net booting. My experience is that the redhat world distros are easier for this than the debian world.
I've setup 100s of PXE boot servers, including CentOS, Rocky, and Ubuntu over the last 10 years or so. I've not noticed any difference between RHEL derived and Debian derived. On the redhat side they are called kickstart files, and on the debian side pre-seed. Both are a bit wonky and hard to debug before you have a full boot, but both seem to work fine.
It's been close to 1.5 years. I've been using both since the beginning and can see pretty well how quickly both systems pick up updates. But let's just look at release delays (in days since the official RHEL is shipped):
ver Alm Rocky
8.4 8 34
8.5 3 6
8.6 2 6
9.0 9 58
Not seeing any patterns here?
One of them is being done by a team that's been shipping another Linux distribution for a decade and has the whole process streamlined and automated. The other started by loud release announcements, creating Slack groups and marketing materials, only then going for solving the technical stuff. I think I've made my choice pretty much right then and there.
Seeing how Rocky guys behaved towards the community (like their refusal to go to a popular Linux podcast unless the host was willing to forego any comparisons with other Linux distributions), and these release delays proved that.
It's amusing that you're presenting Alma as the veterans, seeing as how several founding members of Rocky were responsible for starting CentOS in the first place.
But you're right, they're both RHEL clones, so it's only worth differentiating based on externalities. Rocky is backed by industry veterans and part of the 9.0 delay was so they could dogfood Peridot. Alma is backed by a web company who spent the majority of the past couple years Valley-washing their Russian origins. A while back I watched their CEO beg their executive team to cut ties with Russian media sites. Rockey had a community governance model first, they had a distro-dedicated SecureBoot solution first (Alma 'borrowed' CloudLinux's), and so forth. If your metric is 'get package releases to my AWS fleet first' then Alma is winning. For all the rest of the provisioning and longevity issues, Rocky is the winner. It's all a matter of priorities.
Alma is not as RHEL-compatible as rocky, they use subkeys for signing whereas rocky just uses their signing keys. This is enough that I can’t use alma for building my rpm’s using mock.
I'd love to see a technical writeup of the differences. If Rocky genuinely were better, a 5-week delay vs Alma would probably be ok, at least for the original release, as long as the dot releases are faster. No one wants security vulnerabilities delayed by 5 weeks.
This is not correct. The issue is trying to use Alma 8’s mock chroot on a CentOS 7 host. The (older) versions of yum and rpm there don’t support subkeys.
This would be closer to a bug-for-bug compatibility issue, as a result of an implementation change on the infra side of things. It's not due to the OS which is bug-for-bug compatible with its origin, RHEL 8.
While it would be nice if EL7 stacks are considered by the rebuild distributions, it's not a requirement and they are free to use features supported by the platforms they're building.
Well, I can do a mock -r almalinux-8-x86_64 bootstrap build on RHEL7. (I can't remember whether there are currently any problems maintaining for RHEL8 on RHEL7, but it has been impossible to maintain packages for RHEL N+1 on RHEL N in the past. I just moved to a RHEL8 VM rather than prat about with 7 more.)
> several founding members of Rocky were responsible for starting CentOS in the first place
Hum, that's a subjective view of the history of "CentOS" told by Greg Kurtzer...
In practical terms, Greg just helped setup the legal structure around the distribution that Rocky McGough created.
In the 20 years history of CentOS, Greg Kurtzer stayed 2 years at the beginning, doing mainly PR/legal/community stuff, then left when Lance Davis inherited the ownership of the project after Rocky committed suicide.
I'm not sure anyone is calling Greg a "CentOS founding member" except himself, everywhere.
The CentOS project itself called Greg a founding member in a blog post interviewing him in 2019, well before the Red Hat/CentOS events in December 2020.
If you have some references to back your assertions, I would very much like to see them. You're not the first person I've seen assert that Greg shouldn't be called a founder of CentOS, but I've yet to see that backed up.
Also, you seem to minimize the non-technical, especially legal, work around setting up and maintaining a distribution, but I don't understand why. In terms of legal work, trademarks need to be registered, non-profits need to be set up to fund critical items like build infrastructure, and more. Each of these seems like exceptionally valuable, if not critical, to the success of a distribution.
That CentOS blog post is an interview with Greg. The source for Greg being the founder is...himself. That post was supposed to be the first in a series of interviews with people who were around in the early days of CentOS, which is mentioned in the second to last paragraph. Those interviews happened, but they were never published because the interviewees disagreed with the premise set forth in the first post about Greg being the founder. It was decided that publishing additional interviews with conflicting claims would not be a good look for the project. All other articles referring to Greg as the founder use this blog post or Greg's own claims elsewhere as the source. Greg is the root source for all claims of himself being the founder. None of the other early CentOS folks have backed up his claims, despite him asking them to. There are even people in the Rocky project that have asked him to stop claiming this. Greg's response is typically to post archive links that show 1) he started the cAos Foundation and 2) CentOS started in the cAos Foundation. These are facts that no one is disputing. What is disputed is whether his level of involvement in CentOS itself merits declaring himself the founder many years after the fact.
It would frighten me having Kurtzer doing legalities, given his bizarre legal theory on copyright specifically (no longer recorded in the issues of a deleted github repo for me to reference).
Sorry, I don't know why I missed off the sentence. The statement was that putting something under the Berkeley licence used by Singularity made it public domain, i.e. they could strip my employer's copyright, specifically. No answer to whether that was speaking for LBL legal.
Sorry, I don't remember the specific instance of what happened there, but specifically Singularity was never technically "public domain". It was always copyrighted and licensed (firstly with the DOE/LBL license based on the 3-clause BSD license, then just BSD3).
Was the issue on the project copyright, or was it the copyright in the source files?
While I have some experience in open source and copyright, I'm certainly no expert nor do I claim to be. Luckily I have people I can rely on for advice and guidance (e.g. back then, it was probably LBL/UCOP's Tech Transfer), but sometimes things still get mucked up.
Sorry if there was a miscommunication or an error on my part back then.
Lance did get control of the project, but not due to Rocky's death.
He got it because after he suggested the name (CentOS) and I approved it and it was then accepted by the others, he promised to hand over the domain name which he was squatting on to the Caos Foundation (a 501(c)3).
Jump forward, Lance and the Centos web team had Red Hat trademark violations on the centos.org website and they alienated Red Hat's legal team enough for them to come after me personally. I believed in Red Hat and we needed to correct our actions, so I reached out to Lance and the web team to rectify the situation. My requests were ignored as they didn't care that they were doing a disservice to Red Hat. I forced the situation as much as I could, which was purely political as the Caos Foundation didn't have access or control over the domain. After the PNALV fiasco, matters got worse, and Lance was able to control the project because he retained controlled the domain.
This was how Lance inherited the ownership of the project, well over a year after Rocky passed away. It was a matter of board manipulation (he convinced the Caos Foundation board that it was okay for him to own the domain). Note, he continued being a bad actor even afterwords when he went AWOL and was pocketing donations personally when the developers had to write him a public open letter to hand over the domain.
What I said is not my subjective view of history, it is corroborated fact which was never challenged or questioned until I founded Rocky Linux.
The right question to ask now... Why are some people spreading rumors trying to rewrite history and slander me (and thus Rocky).
> The right question to ask now... Why are some people spreading rumors trying to rewrite history and slander me (and thus Rocky).
Nobody here or anywhere is trying to slander Rocky, or yourself. And everything I said (or others said) cannot be called a "rumor". Everything said is factually correct, there is just a disagreement on the interpretation of these facts.
I will be honest with you, every time I read something where you claim to be a "founder of CentOS", it irritates me, for a number of reasons.
1/ CentOS was not created in a short time. Ideas, source code and history were spread and shared over multiple years and from multiple projects, I tend to consider that "founders" are "recognized as", not "claimed to be". Yet you are very loud about shouting everywhere your "founder" status to promote your Linux distribution. All in all, this status of yours is the sole selling point of Rocky Linux.
2/ There is, IMHO, a difference between "being there during the foundation" and "being a founder". I value the work you did with Caos, but to me being there in 2002 does not make you a "founder".
3/ There is a meaning in the status of founder, it implies some legacy on the future success of the project. Bill gates is a legitimate founder of MSFT because even though he's not there anymore, he did put the company on a track for what it is today.
I do not agree that you have a legacy claim on CentOS' future success. You were let go of the org after 2 years (I know you see it as the board having been "manipulated" by Lance).
TLDR: I do not contest your contribution to CentOS in the early days. But I think calling yourself a "founder" is misleading, and is a borderline dishonest attempt at reaping the success of an org that earned 99% of its respect long after you were let go.
> Nobody here or anywhere is trying to slander Rocky, or yourself. And everything I said (or others said) cannot be called a "rumor".
Did you even read what you wrote? The entire thing is slanderous and based on rumors and you have provided ZERO facts or citations. Others have even pointed you to some links, and if you were really curious, the facts are easily found, go look before you post inaccurate information and perpetuate rumors.
Another thing, I don't appreciate being attacked anonymously when all I'm trying to do is help the community by filling a much needed pain point.
> Everything said is factually correct, there is just a disagreement on the interpretation of these facts.
Actually, what you said is not factually correct and you've provided zero proof. Justify your allegation that me being a founder of CentOS is subjective or you just look like a hater and a troll.
> 1/ CentOS was not created in a short time. Ideas, source code and history were spread and shared over multiple years and from multiple projects, I tend to consider that "founders" are "recognized as", not "claimed to be". Yet you are very loud about shouting everywhere your "founder" status to promote your Linux distribution.
First off, CentOS was absolutely founded in a short time. It's all clearly documented on the Caos email list and easily corroborated by archive.org, but that point is neither here nor there. I led the project from inception to the point where it was literally a household name. If you don't think my contribution was valuable, then you are entitled to that perspective.
> All in all, this status of yours is the sole selling point of Rocky Linux.
Nope, wrong, I never said that and I don't like anyone putting words in my mouth.
The background I have from early days of CentOS is entirely valid and what we did was amazing, but there were of course things we could have done better. Based on this experience, myself and the team leads/board were able to make decisions for the betterment of the project and recapitulate what CentOS did right, and reevaluate what could have been done better.
The Rocky team has done remarkably amazing to bring Rocky Linux out to the community and our growth demonstrates we solved a major pain point for the community. To marginalize all of that because my role with CentOS "irritates" you is just childish.
> 2/ There is, IMHO, a difference between "being there during the foundation" and "being a founder". I value the work you did with Caos, but to me being there in 2002 does not make you a "founder".
That point is accurate, there were a lot of people who were "there during the foundation", and that doesn't make them "founders".
I created the Caos Foundation, I incorporated it, managed it, converted it to a 501(c)3 which then became the origin of CentOS.
Red Hat EOLed the freely available RHL and pivoted it to RHEL thus adversely affecting many people in the Linux and open source community. We (Rocky, Lance, Russ, and myself -- the primary CentOS Founders) decided to rebuild the sources of RHEL with Rocky taking the first stab at it with CentOS3 (which was the current version of RHEL). That was the moment of Founding CentOS. Its initial name was "Caos EL"[1] and it wasn't until a bit later did Lance suggest the name CentOS to me and I, as the project lead, approved it first, then it went to group vote. Rocky was the person who announced the name change from Caos-EL to CentOS on the Caos email list[2] along with letting the world know he was about 99% done with it (this is right before he passed away). Then I continued to lead the CentOS project for another 18'ish months.
> 3/ There is a meaning in the status of founder, it implies some legacy on the future success of the project. Bill gates is a legitimate founder of MSFT because even though he's not there anymore, he did ...
I wonder what is the issue if the company "borrowed" its own stuff to use in a different context. Alma always looked like the more serious operation and, if nothing else, the company supplying kernelcare might be considered useful, even if it's proprietary, unfortunately.
> several founding members of Rocky were responsible for starting CentOS in the first place.
The "several" part is false. Greg is the only one with any claim at all to the early days of CentOS. He created the Caos Foundation, which is where CentOS was started. He believes that makes him the CentOS founder, despite others disputing that claim. None of the other early CentOS folks are working on Rocky.
Faster like Springdale? This is an endurance race. CentOS is the survivor of several clones.
"The best reason we have is our speed. If we assume all RHEL clones are equal in terms of software, people tend to then weigh speed and community size/support very heavily. We get new packages out very very quickly because nearly all the rebuilds can be automated. We had PUIAS 6 out over a month before CentOS 6 came out. The same is true of minor revisions - of CentOS, SL, and PUIAS, we had a 5.8 release out first." - "IAmA Developer for the PUIAS Linux distribution - AMA"
Rocky made it clear they have a new build system which lead to their delay. I think it's called peridot. I would imagine that it's not easy to start fresh in two places at once. Alma has an advantage with a process from CloudLinux already in place and now they've rebranded that build system as Alma and so on. Credit to Alma for keeping up and doing their thing; hopefully they'll be fully decoupled from CL in the future.
CentOS had historically fell behind in release times, but we all of a sudden want to paint others in a bad light for being behind. Did everyone forget about CentOS 7 having average of 30 days delay behind each point release? 7.4 being the most at 43 days. What about CentOS 6.0, with 242 days?
As for their behavior, it takes two to tango. I usually like to thank carlwshill and "conan_kudo" (who should pick a better name since he thinks he's from the anime his picture is from) for being true stars in the open source community and really bringing out not only the best in others, but the best in themselves day in and day out. I'm not sure how anyone can put up with them, regardless of which community you're in (CentOS/Fedora EPEL/others), but who am I to judge, I'm just a user.
I don't have exceptional experience with either Alma or Rocky, but I've been administering enterprise Red Hat and CentOS for years.
1.5 years in enterprise time is hardly any time at all. Heck, it takes that time to approve a budget in some enterprises. Yeah, time-to-release is an important metric, but software compatibility and industry support is really the magic sauce.
In my industry, we'll be using current CentOS 7 installs until EOL, and watch all derivatives with interest over the long term - given they provide anything over current RH ecosystem (RHEL, CentOS Stream, Fedora).
You're not wrong, but CentOS (from inception) has always been really slow to release. When RHEL 7 dropped it took months to get the first CentOS build. CentOS having the same found, it isn't terribly surprising to me to see them be a little slower. When Alma launched, speed of updates was a specific goal of theirs because CentOS had been so painful in that area.
Most of the time I think it's fine. Mainly it hurts when there are security updates you need.
There's reasons why it takes time. The perspective that it's "just produce rebranded RPMs" really undersells even something as significant as the amount of server power required to recompile all the packages. You couldn't get the packages until the distribution upstream had released, so no way to get ahead of the build time. You just had to suck it up at release time.
You used to be able to track the build process for CentOS when it was a RHEL clone, see how many packages were left to go as the days crawled past.
Things are a little better with the way that development happens more recently. 6 was a massive delay for distributions because RedHat had overhauled a lot around the build process and distributions needed to completely overhaul their stuff too, in ways that weren't that obvious.
> The perspective that it's "just produce rebranded RPMs" really undersells even something as significant as the amount of server power required to recompile all the packages.
Just wanted to clarify, I agree with you completely. I didn't anywhere perpetuate the unfair "it's just produce rebranded RPMs" (which is actually accurate, but trivializes the significant process involved in rebranding and rebuilding, hence IMO misleading). It's a significant amount of work. The time it takes is not unreasonable given the effort involved. That said, the amount of time was painful. I've never complained about it because I'm grateful for anything and everything they do, and I'm not complaining about it now. I'm recognizing the facts around speed of delivery, but in no way suggesting they aren't for a good reason.
Alma Linux has always been faster for releases since the beginning. I believe their first release was 8.3 and was very ahead of Rocky and they've maintained that pace to this day. Security updates are within a day or two, point releases are within 7 days and I think the 9.0 release was less than a month.
This is probably because the project was initially founded by Cloud Linux, which I believe already had the expertise to do RedHat clones and basically donated the setup - whereas Rocky started from scratch from what I can tell.
They were pretty open that the 9 release too longer because they were building out a new build system and infrastructure to support it and future releases.
I’d also say what really counts is time to release critical security updates which they seem to be very on top of.
> Note that AlmaLinux released 9.0 about 40 days ago. They're also significantly faster with releasing minor updates (including releases like 8.6).
'Faster' is not a recommendation for an OS developer, IMHO; I don't need operating systems faster - in fact, I like to give them time to mature in other people's hands and to test them myself. The only things I want ASAP are security fixes that are critical to my systems.
If Alma sells themselves on 'faster' (do they?), that would be a significant concern. Why would they do that? Why create a perverse incentive for themselves?
Why is this a concern?
If Alma and Rocky are essential identical, speed is the only/main differentiator, and it matters. If an urgent vulnerability gets patched in RHEL, and I'm using Rocky or Alma, I want the one that gets patched sooner. Right now, that's Alma.
Would you rather have `.rpm`s build by someone on the command line using `rpmbuild` or `mock` or something built by a CI with proper bootstraping and "nearer" to reproduceable builds. Also, by cutting corners on the CI, you risk introducing mild ABI problems which wont crash, but causes instabilities and potential security vulnerabilities. If they say they needed time to do it properly, I respect that.
if that's the case, why in your example is someone running rpmbuild doing it so much faster than an automated CI system.
If your reasoning is that it will be faster in the future. Fine. But I doubt it.
Criticizing Alma because it was first sounds like FUD.
I don't know how RHEL packages are built exactly, since I'm not an employee, but mock is what the Fedora infrastructure runs under koji for builds. I don't see how "a CI" can make the process "nearer to reproduceable builds", whatever that means. I'm not aware or serious efforts on reproducible builds for Fedora/EL, in contrast to Debian, though there has been talk of it. You obviously should never use rpmbuild for binary rpms outside a chroot.
The Alma infrastructure appears to be in their github space, though I don't know anything about it. What's wrong with it?
For the record, I am not saying something is wrong with Alma. I am saying if Rocky says they need time to get it right, then I see this as a good sign.
About `mock`, the problem is the ABI. If you don't build the packages in the "perfect" order, the ABI degrades over time. For example, some libraries might accidentally add something in the middle of a struct. The API is 100% compatible. It will also run without any warning, but all pointers in the application using the libraries provided by those packages will now have an offset. A boolean might now point to in integer or something like this. If you don't have the tooling to detect this and don't have the tooling to ensure you build packages in the right order (and rebuild when needed), then you will eventually get some of these problems. Mostly on point releases. To solve this, the "trivial" way is to follow the RHEL build ordering, which requires some tools. The "correct" way is to use `libabigail`, `libsolv`, `libdnf` and other binary tooling and keep track of these things.
There are more of these little papercuts left and right you get when you build a RHEL clone. You can always cut corners and manually build everything, but you will payback the time you save in outages. RedHat has the test suite, the clones only have a small part of it, they have to be extra careful.
>> I am saying if Rocky says they need time to get it right, then I see this as a good sign.
I've worked in software engineering for a long time. Sometimes delays are a good thing; A sign of waiting for quality. And sometimes they're a sign that things went wrong. That poor decisions were made, or implementation was slow due to junior people etc..
Interpreting 'need time to get this right' as a positive, and worse, as somehow a negative on Alma, who thrashed them to the finish line with an identical product, doesn't make sense.
So, the claim is that RHEL has broken packaging which doesn't reflect ABI changes, and somehow that means you have to reverse engineer its build mechanism from srpms to accommodate it being non-deterministic somehow? Care to give an example? I don't remember ever seeing one. (Of course package maintainers for EPEL, for instance, should use abipkgdiff. I don't remember what the status of automating that is in Fedora.)
The Alma Linux guys seemed happy to engage with the community, for example by coming on the Linux Unplugged podcast and discussing the (at the time) future governance structure of Alma. Rocky Linux rather agressively declined to come onto that same podcast iirc.
What's the thought process behind buying commercial support for a free, binary-compatible RHEL clone, instead of just paying for Red Hat support? Is the calculation that the once-in-a-blue-moon call to a commercial Rocky Linux support vendor when your server catches fire would still be cheaper than the cost of a normal RHEL license?
Disclaimer: I'm a former Red Hat employee, but only speak for myself. I'm also going to speak very frankly.
You got it. From the customer side support from them is cheaper than Red Hat's (sometimes much cheaper). From the supplier side, they don't have to bear much (or any) of the cost of development (beyond infra hosting and re-branding), so they can beat Red Hat's price very easily. I personally find that gross and unethical, but that's just my opinion.
Yes, this is a good point to make explicitly, thank you. The support is from 3rd parties. Those 3rd parties are probably kicking some funding back, but I don't think Rocky is doing anything unethical by offering their rebuild or by accepting that money.
I think overall Rocky is a net positive for the world and for Red Hat.
No, it's the same. I think the amount of upstream contributions (i.e. patches) that the entity sends offsets some of the ethicality deficit. At some point it would even go positive if they send enough upstream fixes.
Obviously this is entirely my opinion :-D
There's also an impossible-to-measure factor in the form of eco-system benefit though. For example, I would never have paid for RHEL had I not entered into the eco-system through Fedora and CentOS. So while RH didn't make money from my CentOS usage, it did eventually make them money because I bought RHEL later when it was worth it. I don't know how you would calculate that, but it does offset ethicality deficit somewhat as well
hi - thank you for speaking plainly to a large tech audience. Isnt there some "market correction" due though, overall, since OSS and Linux have become so central, so deeply performant, while the engineers and other "community" repeatedly get zero money.. Although the point of simply cloning and re-selling the work of RedHat, perhaps with support claims, might look bad, we overall have to allow some growth for non-centralized players right?
Yes, I agree there's value in decentralization. And to clarify, I don't think there are is anything unethical about offering a RHEL clone like Rocky and Alma do. I think that's a net positive for everyone, even Red Hat. My beef is more with the people that sell support which directly undermines Red Hat and ultimately hurts all eco-system users because it means less development, less QA, etc.
That said market competition in general is a good thing, and I don't doubt for a minute that Red Hat prices would be a lot higher without the competition. It's a complex equation that's impossible to calculate since the inputs are immeasurable and in many cases theoretical.
You're forgetting to mention the part where Rocky Linux is not just a rebrand of RHEL, it's a revival of CentOS, which IBM killed off in what's effectively a bait-and-switch, forcing customers to go through the painful/expensive migration process to another distro, or the less-painful but still expensive migration process to RHEL.
Rocky Linux is a shining example of both a free market and the open source community working to the benefit consumers. I don't see how that's unethical.
Sadly, the decision to kill CentOS was entirely Red Hat's. The self-appointed 'community managers' decided that the community didn't actually want CentOS, they wanted a free version of RHEL-Beta called "CentOS Stream."
Nobody said it was a community choice. Red Hat's own layered products (OKD, RDO, oVirt) needed to be based on a distribution that leads RHEL rather than one that trails it, and that's why CentOS Stream was born.
Dropping CentOS Linux was a completely different thing and one should also acknowledge that there are two very different parts of the CentOS community.
Those that simply needed a free RHEL, didn't have any benefit from CentOS Stream. However, their usecase is filled by Alma/Rocky.
Downstream CentOS distributions however only got benefits from CentOS Stream. There are many private ones, for example Facebook runs on a CentOS derivative, but the most prominent example is Alma itself, which existed (IIRC with another name) even when CentOS Linux existed.
And to be honest, only the latter are really part of the community. Downloading an ISO doesn't make you part of the community. I myself used CentOS Linux on a small EC2 VM but I didn't consider myself to be part of their community (I have since switched to Amazon Linux, for what it's worth).
So all that Red Hat did was basically restructure their collaboration with downstream distros. On one hand they enabled those distros to collaborate even more to RHEL development, which is now public (including individual patches to the kernel, if you remember the circa 2011 kerfuffle). On the other hand release rebuilds are entirely in the hand of the community.
Now, I am not saying everything was perfect. The announcement sucked in many ways, and there still isn't a good solution to use RHEL container images on public CI. People inside Red Hat (including me) will all tell you the same. However, it's intellectually dishonest to ignore that there was and is a CentOS world that goes beyond "I need free Linux and I don't/cannot use Debian", and Red Hat has been very receptive to the needs of that world.
> CentOS world that goes beyond "I need free Linux and I don't/cannot use Debian", and Red Hat has been very receptive to the needs of that world.
There really wasn't a CentOS world beyond that, because it wasn't a true community distribution. Red Hat never attempted to meaningful involve the community, all board members or whatever pretend org CentOS has are RH employees. There were never any kind of community elections or anything of the sort.
There totally was. Facebook had been running CentOS long before Stream, and had several RPM backports to CentOS 7 on GitHub. And guess what, they love CentOS Stream. The default desktop distros for Facebook developers are Fedora and Stream.
Honestly if all you know about CentOS is "download ISOs and report bugs that with some luck will be forwarded to RH Bugzilla", you don't know anything about the CentOS community.
Again: downloading ISOs doesn't make you part of a community.
Facebook is the exact example the self-appointed CentOS leadership used. So you're either part of that cohort, or you're really drinking their koolaide.
Well, it's the classic "We don't need support until we need support" perspective from the CFO or CIO or whomever that's looking to cut costs. I've had clients who always had some excuse for why their mission-critical production box still couldn't justify the cost of a proper RHEL license ("No! If build my PROD box on RHEL, then my DR box will have to be RHEL, and my Test box will have to be RHEL! So you see it's really three licenses I'd have to buy!") and always insisted on CentOS. But despite needing everything to work correctly 100% of the time and mandating several levels of redundancy, they were curiously okay with being up the creek if they ran into an OS problem. Go figure.
FWIW, clients I dealt with were universally thrilled with RedHat support, the ones who had it. I had one tell me over the phone once, "Oh yeah! Red Hat is amazing! If an issue gets assigned to an engineer, it will be resolved by that same engineer. Unlike you guys."
absolutely, I saw that quite a bit as well. I was really glad to see Red Hat change their policy so that non-prod machines are free now. That makes it a lot more affordable for people doing CI/CD with staging and dev environments. The old model disincentivized good practices.
I appreciate your candor, but what makes this "gross and unethical" exactly? RedHat itself makes money by charging for support on work made by thousands of other coders outside the company (i.e. Linus et al). It's hypocritical if you ask me.
Good point, there's some hypocrisy at play here. I think overall though the "taking" done by RH is on the whole much less than most of the 3rd party support vendors. But my original language was a little harsher than I really feel.
Red Hat funds an enormous amount of the development of projects (like the kernel), and makes Fedora one of the best (IMHO the best) distros for personal computing, so is one of the top contributors to the community at large. Their support also actively fixes bugs and sends them upstream.
Contrast that with many of the cheaper 3rd party supports, who rarely if ever send contributions (beyond bug reports, which are sometimes a positive contribution, but frequently are net drain because the bug reports don't contain enough info to be reproducible or actionable). They also don't do much or any development.
Of course this is a broad stereotype. I'm sure you can find 3rd party support providers that do contribute to the community.
> RedHat itself makes money by charging for support on work made by thousands of other coders outside the company
Linux is not developed by thousands of volunteers on their free time in their garage.
At least 85% of commits to the Linux kernel are from corporates (AMD, Intel, Broadcom, RedHat, etc). Linux is free and open source code, but very largely written by people who are paid to do so, by companies that need to somehow make a profit.
RedHat amounts to roughly 20% of these corporate commits, and that's counting in the share of other corporates that frankly mainly only contribute work for their own drivers.
I see paying for RHEL as being 50% for support / 50% for sponsoring Linux development as a whole.
I'd say it's a two-sided sword. There's no question RH employs/founds large parts of Linux development, with only Suse being remotely as involved (maybe historically). OTOH, RH has pushed "innovations" such as systemd purely in their own interest, fragmenting a once-strong and user-centric F/OSS Unix community also including the BSDs into a Linux-only cloud slavedom. Plus, it was IBM/RH who cancelled the CentOS roadmap (after having bought-out the CentOS project and community); they can't now expect to be treated as trusted bona-fide Linux steward or something.
IBM had nothing to do with the CentOS decision. It was long time Red Hat people who made the decision. I don't agree with everything about the decision, but I don't think it's as bad as most people say it is[1].
You are definitely right that RH has pushed things in their own interest, but if those things don't offer value to the broader community, then the community won't adopt them. Red Hat can't force Debian or Ubuntu or Arch to adopt anything. They can push it through Fedora Cent and RHEL, but that's it. The other distros adopted systemd because it offered benefits/improvements over existing things like Upstart. I like firewalld, but that's a good example of something that is only on RH despite RH pushing it. If systemd was really such a negative, then you'd see distros like Devuan take off. A frequent criticism is things like, "Red Hat made Gnome dependent on systemd" which isn't wrong, but they didn't just do it because they could. There were real benefits there.
I think the reality of life is that there will always be people who want things to change, and those who don't want the change. To succeed you have to find a balance.
Also important to remember when decrying "fragmentation" (which I decry also btw), in a massive heterogeneous community like open source, you're gonna have users who have completely different needs and use cases, and both are valid. The beauty of it is the code is open and free, so people can serve niche use cases as well as standard.
While I agree with you, it is fair to say RH has an unusual amount of leverage with respect to forcing things. They directly control a lot of big ticket projects, and have powerful leadership positions in others. They can coordinate major changes across the board and the momentum they can throw behind some decisions can certainly exert a LOT of pressure. This isn't necessarily a bad thing though - as you note it can be good to have a leader, but it's also something that can be detrimental too.
> I don't agree with everything about the decision, but I don't think it's as bad as most people say it is[1].
It was bad because when I chose CentOS for servers at work I thought I could count on letting them live their lives for about 10 years without worrying about them. I have a lot of things to do and taking care of those servers is a very low priority.
Thank God there was a script to convert my CentOSes to Rocky and that it went great, otherwise it would have taken time I don't have to reinstall those servers.
Yes I will 100% grant on that one. Pulling the plug so early on CentOS was low and I don't think they should have done that.
The whole "we never put an EOL date on it" excuse was really weak. It looks to nearly everyone like just an unintentional omission, not a legal hedge. It ended up working out well like you said thanks to Alma and Rocky, but the panic and uncertainty it caused was wholly unnecessary.
They also created quay.io and fragmented docker global public repo by asking sponsored FOSS project to publish ONLY on quay.
I really dislike redhat and won't touch anything related with them.
I really don't understand why companies would pay thousands per server for support. I'm working with Linux systems since 2 decades and never needed to pay a cent in licenses or support.
Red hat didn't create quay, they acquired it. But even if they had made it, I think that's an important and healthy thing to have. You could call it fragmentation, but I see it as taking on a monopoly by offering choice. Docker fully supports alternative registries, so I wouldn't consider that fragmentation either. Remind me a lot of when Apache was the only http server, and nginx came out. Some people said fragmentation, but most said, "cool, we have a choice and some market competition, without which development tends to stagnate"
There are things to dislike about red hat, but quay is one I would praise them for instead.
and RH doesn't have to bear much of the cost of the development of Linux, GNU, and many other products. This is the spirit of OS.. If you think that's unethical, you should find another employer ;)
edit: ah.. former employee.. so that part was already done :)
> RH doesn't have to bear much of the cost of the development of Linux, GNU, and many other products.
I disagree. RH is one of the top contributors to many of the major projects that make up the distro. Also the process of building/maintaining a distro is itself enormous. The 3rd parties have none of that expense so they can undercut the cost easily. In the end it hurts the whole ecosystem, while benefiting a select few.
If this hurts the whole ecosystem, all community developed distributions must have been hurting it, too, by your bizarre interpretation of free software and/or arrogant take on ethics.
Explain how anything you've said so far in this sub thread can't be summed up to "red hat good, everyone else leeches".
Yep, definitely not illegal, although IMHO there things that are legal but still unethical. And the line of ethicality is highly subjective. There is sort of a limiter in place in that if it got too bad RH could kill it pretty quickly by not publishing all the SRPMs. Although if they did that, I think the ecosystem would fall apart. I would bail.
Part of my irritation is definitely a bad taste in my mouth still from Oracle.
I think (or at least I hope) that anyone who buys a support contract from a third-party company for a third-party distribution understands that they are not getting the same level of support that they would from Red Hat.
There is lots of room in the enterprise sector for third-party support. This is, for example, what almost all consultants effectively are. The third-party vendor can solve a lot of problems that the customer may not have the experience to deal with and can be well worth the price paid. But at the end of the day if there's an actual bug all the way upstream in RHEL, only RHEL can (permanently) fix that.
I've dealt with Red Hat in the past and one good thing about their support is that if you have a particularly thorny problem or a genuine bug, you will often eventually end up talking directly to someone who is either wrote the code or sits next to the person who did.
You make some great points, thanks (side note: I love HN for conversations exactly like this one). You've won me over somewhat. As long as the support vendor isn't misrepresenting what they offer and how it differs from Red Hat, it doesn't seem nearly as unethical as it felt initially. I've only had experience with a handful of vendors, but all of them marketed themselves as "same support as Red Hat, 1/4 to 1/2 the price." I find that gross, but if it were "we'll help you setup and configure your machine" rather than "use us instead of Red Hat" I don't really have an issue with that.
Nobody ever got fired for hiring red hat for red hat support. To the other post’s point those may not be the contracts red hat wants. They want big enterprise where there is 0 reason to not go red hat.
It goes further than that. If you file a bug on Red Hat's bugzilla with or without a support contract, you will quite often get a response if you put the effort into producing a detailed report. No guarantee, sometimes just others with the same issue, but still pretty good. You will also find Red Hat employees on the mailing lists for the OSS projects they contribute or depend on who actively participate in conversations there.
If you have a support contract though and open a case, the level and quality of support is usually very high.
I've been involved with companies buying from IBM for over 20 years. Since the Lotus Notes days I've always been impressed by their support chasing down hairy issues. Like, their tier1 would escalate to T3 when mostly needed. Compare to Google where robots lock you out for days. Expensive but never had a client be mad about it.
This isn't my experience at all. I've personally had three paid support cases attached to Bugzilla issues. Two of them were for bugs that had one-line fixes that were already committed upstream, along with test cases, and that I just needed Red Hat to backport. They sat for about 6 months before I got any response other than the automated monthly "our engineers are working on the Bugzilla." The third was for a problem that existed upstream too. It was opened over 2 years ago and still isn't fixed.
The components for the first two were NSS (Network Security Services, not Name Service Switch) and GCC, and the third was Kerberos. I don't have the BZ number handy right now.
For the first two it's probably that they were considered not bad enough for an asynchronous release. With CentOS Stream, you would have a better understanding of when the patch was committed and more certainty of whether it would be fixed in the next minor (6-months cadence) release.
Feel free to write to me at pbonzini@redhat.com, I don't work on Kerberos myself but I know people in the team.
I don't think it's fair to be that harsh.
Ultimately Red Hat incurs costs from engineering RHEL. They fund that through support contracts. So obviously a company that doesn't incur these costs can sell support contracts for cheap.
While it's legal, I agree there are grey areas around the ethics. Of course it's OSS, so you can't have expectations, but just imagine if all RHEL customers switched to 3rd party support. Red Hat would stop existing. Then what would these 3rd party support companies do? They'd go under too, as they'd have nothing to support.
I'm just using the exact same words used by the OP...
No one asked them to do that and put it in open source, if they are not happy they can go to closed source or different license... Like Microsoft... Maybe the new OP employer? That would make a lot of sense when I see his attitude
Unethical would be if the companies selling the support then turn around and use their (one) Red Hat license whenever there's a real difficult problem.
But the vast majority of "support" for Linux isn't engineer-level, it's likely config and setup. Which is where both Red Hat and others try to make their money.
Out of curiosity, what sort of problems would customers typically run into?
My very naive understanding is that it’s just the OS and then only even a distribution (just to emphasize- very naive), what would they need apart from some networking and to run some software on top that needs so much support?
It really runs the gamut, from bugs in included software to "why are we dropping packets" (usually misconfiguration) to an update broke our existing config or an update failed to install, etc. I saw one ticket that was "missing build dependencies" where they couldn't build their own code because of a package difference, and while Red Hat didn't fix their code, they did give them some helpful pointers.
Former consultant here: There are a lot of entities out there that aren't really a good fit for Red Hat's service offerings, but who may still need the system. As others have said, Red Hat is a great provider if you may need support from someone deep in the bowels of the exact code you're having a problem with.
Having consultants out there using CentOS/Rocky/whatever gets these "not a good fit" customers burden off Red Hat, which is probably an advantage to them. I operated what amounts to a "phone a Linux friend" service for ~18 years, and that sort of help just doesn't really fit into Red Hat's offering.
It's nice in many ways to be able to disconnect from the licensing model and just be able to deploy Linux boxes, but still get help when you need it. I recall one conversation about Red Hat licensing related to a machine I was installing for a client named "The Fedora Project" that went like this:
"I assume you'd like me to put Red Hat on there?" "Yes." "Can you provide me with a license key to use on it?" "Uhhh. Just go ahead and install CentOS".
RedHat licenses are just too expensive by sticker price. Beancounters haunt us for deploying RedHat because "the Windows license for that box would have been cheaper in our licensing model". That there is support included which we never use doesn't matter to them, the usual suggestion being "buy support for one box, test everything on that one, and open a support case for that one box, replicate the solution everywhere".
If you want to fix this, make a RedHat license be significantly cheaper than the equivalent Windows product. Charge for support by ticket/case and only support licensed boxes. You'll earn a lot more because it'll look cheaper to the beancounters.
Did CentOS not do the same? RH bought them, right? Then changed it to a rolling release experiment. I guess that was a talent acquisition? So RH benefited off the previous attempt to do this.
Also RH packages open source project and also does not necessarily gives back to every project.
The right to help your neighbor is important to open source. I find Rocky does just that.
Yes, IBM/Redhat decided to make CentOS less attractive to those who need a RHEL work alike. This was done on very short notice, leaving people with the uncomfortable decision to A) pay for RHEL B) tolerate a rolling release (not directly related to RHEL 8 or C) switch to another OS.
Rocky and Alma popped up for folks that want a bug for bug compatible RHEL OS.
Live by the sword, die by the sword. Red Hat can develop a closed-source OS and sue people who copy it, but then no one would use their product. If one of your major selling points is open source, not to mention benefiting from all of the non-RH developed code that is in RHEL, you can't complain about the perfectly predictable consequences of that.
RH publishes all their sources (including the MIT/BSD/etc ones that they don't have to publish). They're the very opposite of "closed source". I don't know of them suing anybody. And nobody is complaining about people using the code, or about the existence of Rocky or Alma or any other rebuild distros.
You are confused. Read my comment again. I’m just saying, if you don’t want people cloning your software and competing with you for support contracts, then don’t make it open source. It’s absurd to make your product open source, use open source as a marketing point, incorporate millions upon millions of lines of other peoples code into your product, and then complain when people do exactly what your open source license gives them the right to do.
And you literally said:
>I personally find that gross and unethical, but that's just my opinion.
Your notion of ethics is unequivocally and objectively wrong. Red Hat derives massive value up to and including its very existence and every single penny that passes through its hands from participation in the larger open source ecosystem that comes with the obligation to share their work not only with random bob hacker but with people like Rocky Linux, Alma, and even folks like Oracle.
You cannot acknowledge that they owe everything based on willing participation in an ecosystem where they wherein they openly made a promise and follow that up with the idea that its immoral for anyone to rely on such a promise. It's logically and morally inconsistent.
Gross is just an emotive label you have slapped on after incorrectly concluding that such actions are unethical.
You can't argue a position by linking to a Wikipedia article about a logical fallacy.
It cannot be unethical to rely on a freely given promise that the other side has derived substantial value from.
Red Hat received the entire basis of their business by virtue of their side of that promise they cannot and indeed haven't tried to withhold performance of their obligation in that regard.
Under what ethical theory do you propose that creating downstream distros is unethical? Is it likewise unethical for Ubuntu to exist? If your answer differs why?
Redhat builds off of a mountain of open source software that they didn't write. To call it gross and unethical is blatantly hypocritical. If Redhat doesn't want others to use software they create in certain ways, then they should change the license of the OS. Oh, that's right, they can't, because it's not all their software.
I've often seen internal IT/ops departments build their own packages for CentOS (pre Stream days) and do their own support. It's not the right call for every organization, but in that case, the Red Hat support does little.
I tried to speak to Redhat about getting support for servers that didn't fit into their annual per-server subscription model. They didn't even reply to multiple emails.
Redhat sales are now firmly set in the IBM enterprise model. I'm sure it works for them, but it didn't work for us as a potential customer moving from Centos.
It's not about the money. It's about not having to put up with subscription-manager. (I often use the free clones for this reason at work on my sandbox VMs, even though my employer has a paid Red Hat subscription.)
I find it interesting that there's a need for Rocky to exist at all. IBM basically killed Centos, which was the unofficial Red Hat release that came into existence when Red Hat stopped providing free versions of Red Hat which Red Hat then grand fathered in as a thing they actively supported. That whole process was just weird. I did not encounter a single project that actually used Red Hat after that happened: world + dog used Centos at the time. That seemed like a horrible process to put up artificial hurdles for people to transition to Red Hat. Killing Centos and then having Rocky and Alm emerge immediately after makes this even less likely. As a business strategy this just seems pretty misguided.
These days, I don't deal with the whole red hat ecosystem at all (why would I?). Software ships in docker form these days and I'll typically either deploy it on whatever my cloud provider uses without really messing with that directly or setting up a simple Ubuntu server with Docker using things like Ansible. My daily driver is a Manjaro laptop currently. I like it because I get current versions of everything I need as opposed to old stuff like Gnome 40. Do people actually put Rocky on workstations? My impression is that that whole ecosystem is more about servers and that Fedora is the preferred thing for people in that ecosystem on desktops.
RedHat benefit is support. You can sue them if something goes wrong ;)
Good engineering is using on local what is used on servers. RHEL definitely is about servers foremost. There are many industries where RHEL is a must. Amazon Linux is derivative of RHEL.
Good point about containers.
Oracle Linux 9 also dropped a week ago, and their support is available if increasingly expensive.
The problem with both of these is app support certifications. Microsoft SQL Server (for example) is solely supported on RHEL, although it's obvious that they build on CentOS.
To me it just looks like they fragmented their own market for no good reason and drove a lot of business to competitors competing with them with their own product. Before Centos, the upsell for Red Hat support was pretty easy: lots of companies were already using it anyway so twisting their arms a little for some support money was probably easy.
Now, not so much. Do they even know who uses Alm or Rocky or Amazon Linux or Oracle Linux? Do these companies share their CRM data with IBM/Red Hat. Probably not. And of course both are in the business of providing support and consultancy themselves. IBM does the hard work and those companies reap the benefits. Very noble but not very smart.
The whole setup is weird. The only reason for these distributions to exist at all is that IBM/Red Hat refuses to distribute their own software directly. They are still doing the work and providing it for free in source code form (not optional) which is why this works at all. But they've created this whole side hustle that is cutting them off from their own user base. How is that in any way a smart business strategy? All they'd have to do to fix it is roll back to just providing Red Hat to people that want it for free and then up-sell them on support contracts. That's business they aren't currently getting.
Not to rain on this parade, but what specifically is the target user for Centos/Rocky these days?
I've been a Linux user for more than 20 years. I absolutely saw the value of Mandrake/RedHat/Debian for years. Boring, stable. Got it. Those are important qualities.
And Centos gave a free, easy to acquire and use taste of said stability.
But today... everything is so different. I run rolling release, have a billion random services running at any time because it's a dev machine, and... nothing really ever breaks. I get kernel vulnerability patches in a couple days, vs waiting weeks for some hacked together 2.6.8294 backport.
Even if that weren't the case, between snapshotting filesystems and containerized everything, it just feels like such cautious, controlled OS software is a thing of the past.
As always, happy to be told I'm incorrect and/or hear use cases.
The target audience is the machines that host your vms, these need to have a stable environment for a long time(of course patched for security when needed).
About 18 years ago I wrote an internal intranet application for someone in PHP (yuck). This runs one of their core business processes. Every 3 years we do a review of the system, replace the hardware and update the software. This is exactly where CentOS variants sit for me. The last two upgrade cycles, the computer has not been down at all in the interim. It has literally been up for 3 years. The only changes have been to support later versions of Postgres and PHP and to move the backups to S3 rather than tape.
Why not the cloud? The company is in a rural location with a shitty internet connection and 50 users. When initially provisioned there was no internet connection at all!
There’s a lot of stuff which is in the territory of “if it’s not broken, don’t fix it” and that’s where these distributions sit. That may cover industrial, office, line of business apps, temporarily connected or fully offline systems. Or just stuff you don’t want to piss around with.
Also never underestimate the power of having stuff you don’t have to touch or even go near. When was the last time you got paid monthly to do something once every three years?
> Also never underestimate the power of having stuff you don’t have to touch or even go near. When was the last time you got paid monthly to do something once every three years?
This is not the reality for 99,9% of the world. If it is online, you need to patch security vulneralibities.
Because your system is not immune to them.
I'm not saying it isn't patched. It is just not bleeding edge or changing rapidly therefore the tradeoffs are different.
There's stuff out there that is nearing relic status that is still up and running and not patched or supported. Some of it isn't even touching a network so that's fine.
It's what you get support for. At my job we run several third party applications that are targeted at RHEL/RHEL-clones. Sure, they might run on other distros, but if you need the vendors support, you better be running RHEL/CentOS/Alma/Rocky.
Supercomputing centers and all the servers are prime candidates, where the hardware and software stack never change for the lifetime of the computer (which is around 7-8 years at least).
Do note that what users can use on that, and what system admins use is a completely different stack. Users get an update almost every 3 months. The stability is more for the hardware drivers and things like that, where things are still very iffy because a lot of hardware is actually latest hardware, so not all bugs have been sorted. Infiniband for example is not stable and mainstream at all.
Ubuntu will go through two LTS releases in that time, while you can continue with a single RHEL/CentOS release.
Supercomputing... Note that Summit changed from RHEL7 to RHEL8 mis-stream. (IBM insisted the system I work on had to do the same on similar hardware because the RHEL 7.9 distribution I could see didn't exist. It allowed more recent CUDA, but that wasn't the rationale.)
I don't know what "system admins use is a completely different stack. Users get an update almost every 3 months" is about. Infiniband is fine, and typically quite up-to-date in RHEL, but you end up having to deal with the Mellanox nightmare, at least for Nvidia GPU support; roll on RoCM. That area is a good demonstration of the practical evil of proprietary software.
If you want to make a static build with an ancient glib (so it runs everywhere) but you want to use a recent compiler then CentOS is the straightforward option. It provides modern toolchains over an ancient base system
I think this is how you make Appimage files that run on all distros (maybe someone can correct me if I'm wrong)
There are entire industries that cannot and will not use rolling releases. That can either be regulatory requirements, or it can be environment where you need/want to be sure that the platform is identical to a reference system.
The best examples are perhaps hospitals, and medical equipment. I previously worked with a hospital where the software is only certified to run on RedHat, and where the system is very tightly controlled. A patch can only be deployed in production, if it has parsed testing and staging environments.
I've also read about research projects, where different versions of glibc, would yield different results. In this case I will continue to argue that this is pretty bad for reproducibility and more of a problem with the people doing the science and not underlying operation system.
Errors, bugs, security holes do sometimes make it into rolling releases, and in some environments regressions are catastrophic or deadly.
Personally I tend opposite world view compared to you. Rolling release are great, for a very small niche, such as desktops for office workers or developers in certain industries.
Rocky Linux is self is a great project, because it allows you to have an environment identical, or close to identical, to the RedHat installations that will eventually run test, staging, QA and production environments, but without the cost. I had been skeptical that Rocky Linux would deliver, but I'm very please and happy that they do.
I want to tell you that you are incorrect even I upvoted your comment :). There are industries which cannot use rolling distros for whatever reason. I switched jobs at the beginning of this year from a cloud-application development where everything could work on rolling releases and more or less lived on the bleeding edge to developing and provisioning a cloud platform (not everything can work on a public cloud: energy companies, essential telecommunication, many healthcare providers don't want to do that etc). Two worlds so close together physically but totally different philosophies.
We don't use CentOS or Rocky. Our customers pay for RedHat or SLES. But I can see that someone wants RHEL and don't want to pay for the license and don't want to switch to Ubuntu/Debian or OpenSUSE either. I don't understand that, they are not that expensive for what they are but ... I am a software developer by profession, not an accountant. (I even saw industries paying for Windows Servers or HP UX, the prices were pretty eye-watering; the Linux-based options are either cheaper or at the same level as MS Win which for multi-million Euro company is not THAT much and even can pay off when you need to call support.)
For me, it's about having a stable platform for something; not all (or even most) components in a software system benefit at all from following the latest releases, and what "stable" distributions give me is the ability to keep those components up-to-date security-wise very easily and without having to worry that everything could explode due to major changes when I run "dnf update" (breakage does happens, but it's rare). I can have hundreds of hosts auto-patching themselves and still sleep soundly.
Something also needs to run your Kubernetes, and that can't always be AWS.
I've been happy with Oracle Linux for free at home usage. With a well-established, stable, consistent timed releases. Is there any reason why should I switch to Rocky (or Alma?)
177 comments
[ 0.21 ms ] story [ 239 ms ] threadhttps://github.com/rocky-linux/peridot
I imagine the 'product' is essentially the same, but policies or procedures may differ - and these can be significant! It's what draws me to the Fedora project so much.
Asking as someone who wasn't really that bothered by CentOS Stream. I've seen it as basically rolling point releases, which wouldn't be a problem for me personally
My info might be slightly outdated cause last I looked into it was a few months ago, but I would guess it's not terribly different. Yes both are essentially the same product. Main differences from user perspective are reliability of package servers and speed of updates, plus community experience if you get involved there (friendly tip: I suggest you stay the hell away from the cess pool that is Reddit. That whole site probably needs to die in a fire)
*Alma Linux:*
- Works closer with upstream (actually sends patches and participates with dev on CentOS)
- Backed by a for-profit company (which I consider a good thing for longevity and reliability, but of course there's a risk of profit and community incentives misalignment)
- Fastest on updates when upstream releases
- Friendly and appreciative (generally speaking) toward Red Hat (without which none of their business would be possible)
*Rocky Linux*
- "Community" owned, but backed by some corporate money
- Very volunteer-run. Have some paid employees working on but many volunteers
- Hostile (and sometimes hateful) toward Red Hat
I mostly use CentOS Stream though and that has been rock solid. I use Alma on my router and on a prod database server that will be nearly impossible to rebuild that I migrated to Alma from Cent, but CentOS feels better to me now than it did before changing to Stream.
10 years of unpaid support has always been an anomaly that no other LTS distro provided.
RHEL 7 -> RHEL 8: https://access.redhat.com/documentation/en-us/red_hat_enterp...
RHEL 8 -> RHEL 9: https://access.redhat.com/documentation/en-us/red_hat_enterp...
We still recommend having strong backup, rebuild, and automation strategies regardless. I personally would rather spin up a side system, ensure everything works, and nuke the original. That would apply to either test a in-place upgrade or a from-scratch build.
Upgrading major versions of RHEL is akin to jumping 5+ versions of Fedora at once and I don’t think it could ever really work well without waaaaay too much work that probably isn’t worth it.
The Alma Linux “elevate” project seems to be attempting that using leapp as a framework but also allowing migration between RHEL derivatives so we’ll see how that pans out I guess.
Red Hat also has a project called 'convert2rhel' which i imagine is similiar to elevate.
by 2024, for things still in service I will most likely upgrade it to CentOS Stream 9. by about that time I'm usually running in to annoying things about the older OS and I want updated tools and such, which is enough incentive to get the rebuild done.
The box that will be extremely hard to rebuild I already moved to Alma, where it will stay hopefully until EOL. There is one box I moved to Stream that I may just move to Alma to squeeze a few more years out of it in anticipation of it retiring. Overall though I really hate being on extended support (past the initial period). The farther in it gets the more headaches pop up. I got hosed by deprecations of SSL and old boxes that suddenly stopped working (because they were using protocols deemed too old/insecure by modern browsers). That was an absolute nightmare because it was instant prod outage and we had no time to plan.
So its actually for people trying to replace CentOS which RedHat decided to drop :)
The target audience is people who used to use Centos, before Red Hat ended Centos 8 years early and moved it to a rolling release schedule. Its primary competitors in the server-focused Red Hat derivatives are Red Hat Enterprise Linux itself, which is paid; Centos Stream, which is rolling release; or Alma Linux, which is, currently, basically the same distro due to how recent the Centos debacle was, but could diverge in the future.
Then CentOS was killed by IBM/RedHat and reborn as the beta distro for the next RHEL release. Rocky Linux and Alma Linux were created to fill this niche. For example, you can run your production server on RHEL, but run your test and dev machines on Rocky while being reasonably sure that there won't be any compatibility issues.
Net booting. Distros vary in how awkward it is to get stripped setups set up for net booting. My experience is that the redhat world distros are easier for this than the debian world.
On top of that, I like their attitude a lot more compared to what I've seen from Rocky's developers (regarding community interactions and such stuff).
https://lwn.net/Articles/896438/
https://en.wikipedia.org/wiki/AlmaLinux#Releases
One of them is being done by a team that's been shipping another Linux distribution for a decade and has the whole process streamlined and automated. The other started by loud release announcements, creating Slack groups and marketing materials, only then going for solving the technical stuff. I think I've made my choice pretty much right then and there.
Seeing how Rocky guys behaved towards the community (like their refusal to go to a popular Linux podcast unless the host was willing to forego any comparisons with other Linux distributions), and these release delays proved that.
But you're right, they're both RHEL clones, so it's only worth differentiating based on externalities. Rocky is backed by industry veterans and part of the 9.0 delay was so they could dogfood Peridot. Alma is backed by a web company who spent the majority of the past couple years Valley-washing their Russian origins. A while back I watched their CEO beg their executive team to cut ties with Russian media sites. Rockey had a community governance model first, they had a distro-dedicated SecureBoot solution first (Alma 'borrowed' CloudLinux's), and so forth. If your metric is 'get package releases to my AWS fleet first' then Alma is winning. For all the rest of the provisioning and longevity issues, Rocky is the winner. It's all a matter of priorities.
As a user of CentOS looking for a replacement, Alma and Rocky should be 100% identical. The _only_ difference is delay after RHEL launches.
There is a Red Hat BZ issue here: https://bugzilla.redhat.com/show_bug.cgi?id=2017069
https://access.redhat.com/articles/rhel-abi-compatibility
https://access.redhat.com/articles/rhel8-abi-compatibility
https://access.redhat.com/articles/rhel9-abi-compatibility
This would be closer to a bug-for-bug compatibility issue, as a result of an implementation change on the infra side of things. It's not due to the OS which is bug-for-bug compatible with its origin, RHEL 8.
While it would be nice if EL7 stacks are considered by the rebuild distributions, it's not a requirement and they are free to use features supported by the platforms they're building.
Hum, that's a subjective view of the history of "CentOS" told by Greg Kurtzer...
In practical terms, Greg just helped setup the legal structure around the distribution that Rocky McGough created.
In the 20 years history of CentOS, Greg Kurtzer stayed 2 years at the beginning, doing mainly PR/legal/community stuff, then left when Lance Davis inherited the ownership of the project after Rocky committed suicide.
I'm not sure anyone is calling Greg a "CentOS founding member" except himself, everywhere.
https://blog.centos.org/2019/03/greg-kurtzer-centos-founder/
For what it's worth, The Register refers to Greg as a CentOS founder in their first Rocky article:
https://www.theregister.com/2020/12/10/rocky_linux
If you have some references to back your assertions, I would very much like to see them. You're not the first person I've seen assert that Greg shouldn't be called a founder of CentOS, but I've yet to see that backed up.
Also, you seem to minimize the non-technical, especially legal, work around setting up and maintaining a distribution, but I don't understand why. In terms of legal work, trademarks need to be registered, non-profits need to be set up to fund critical items like build infrastructure, and more. Each of these seems like exceptionally valuable, if not critical, to the success of a distribution.
Why are you not considering PR/legal/community stuff as being worthy?
Thanks.
Was the issue on the project copyright, or was it the copyright in the source files?
While I have some experience in open source and copyright, I'm certainly no expert nor do I claim to be. Luckily I have people I can rely on for advice and guidance (e.g. back then, it was probably LBL/UCOP's Tech Transfer), but sometimes things still get mucked up.
Sorry if there was a miscommunication or an error on my part back then.
Heck, that's the very stuff nobody wants to take care of, kudos to him he wanted to spend his time on these.
He got it because after he suggested the name (CentOS) and I approved it and it was then accepted by the others, he promised to hand over the domain name which he was squatting on to the Caos Foundation (a 501(c)3).
Jump forward, Lance and the Centos web team had Red Hat trademark violations on the centos.org website and they alienated Red Hat's legal team enough for them to come after me personally. I believed in Red Hat and we needed to correct our actions, so I reached out to Lance and the web team to rectify the situation. My requests were ignored as they didn't care that they were doing a disservice to Red Hat. I forced the situation as much as I could, which was purely political as the Caos Foundation didn't have access or control over the domain. After the PNALV fiasco, matters got worse, and Lance was able to control the project because he retained controlled the domain.
This was how Lance inherited the ownership of the project, well over a year after Rocky passed away. It was a matter of board manipulation (he convinced the Caos Foundation board that it was okay for him to own the domain). Note, he continued being a bad actor even afterwords when he went AWOL and was pocketing donations personally when the developers had to write him a public open letter to hand over the domain.
What I said is not my subjective view of history, it is corroborated fact which was never challenged or questioned until I founded Rocky Linux.
The right question to ask now... Why are some people spreading rumors trying to rewrite history and slander me (and thus Rocky).
Nobody here or anywhere is trying to slander Rocky, or yourself. And everything I said (or others said) cannot be called a "rumor". Everything said is factually correct, there is just a disagreement on the interpretation of these facts.
I will be honest with you, every time I read something where you claim to be a "founder of CentOS", it irritates me, for a number of reasons.
1/ CentOS was not created in a short time. Ideas, source code and history were spread and shared over multiple years and from multiple projects, I tend to consider that "founders" are "recognized as", not "claimed to be". Yet you are very loud about shouting everywhere your "founder" status to promote your Linux distribution. All in all, this status of yours is the sole selling point of Rocky Linux.
2/ There is, IMHO, a difference between "being there during the foundation" and "being a founder". I value the work you did with Caos, but to me being there in 2002 does not make you a "founder".
3/ There is a meaning in the status of founder, it implies some legacy on the future success of the project. Bill gates is a legitimate founder of MSFT because even though he's not there anymore, he did put the company on a track for what it is today.
I do not agree that you have a legacy claim on CentOS' future success. You were let go of the org after 2 years (I know you see it as the board having been "manipulated" by Lance).
TLDR: I do not contest your contribution to CentOS in the early days. But I think calling yourself a "founder" is misleading, and is a borderline dishonest attempt at reaping the success of an org that earned 99% of its respect long after you were let go.
Did you even read what you wrote? The entire thing is slanderous and based on rumors and you have provided ZERO facts or citations. Others have even pointed you to some links, and if you were really curious, the facts are easily found, go look before you post inaccurate information and perpetuate rumors.
Another thing, I don't appreciate being attacked anonymously when all I'm trying to do is help the community by filling a much needed pain point.
> Everything said is factually correct, there is just a disagreement on the interpretation of these facts.
Actually, what you said is not factually correct and you've provided zero proof. Justify your allegation that me being a founder of CentOS is subjective or you just look like a hater and a troll.
> 1/ CentOS was not created in a short time. Ideas, source code and history were spread and shared over multiple years and from multiple projects, I tend to consider that "founders" are "recognized as", not "claimed to be". Yet you are very loud about shouting everywhere your "founder" status to promote your Linux distribution.
First off, CentOS was absolutely founded in a short time. It's all clearly documented on the Caos email list and easily corroborated by archive.org, but that point is neither here nor there. I led the project from inception to the point where it was literally a household name. If you don't think my contribution was valuable, then you are entitled to that perspective.
> All in all, this status of yours is the sole selling point of Rocky Linux.
Nope, wrong, I never said that and I don't like anyone putting words in my mouth.
The background I have from early days of CentOS is entirely valid and what we did was amazing, but there were of course things we could have done better. Based on this experience, myself and the team leads/board were able to make decisions for the betterment of the project and recapitulate what CentOS did right, and reevaluate what could have been done better.
The Rocky team has done remarkably amazing to bring Rocky Linux out to the community and our growth demonstrates we solved a major pain point for the community. To marginalize all of that because my role with CentOS "irritates" you is just childish.
> 2/ There is, IMHO, a difference between "being there during the foundation" and "being a founder". I value the work you did with Caos, but to me being there in 2002 does not make you a "founder".
That point is accurate, there were a lot of people who were "there during the foundation", and that doesn't make them "founders".
I created the Caos Foundation, I incorporated it, managed it, converted it to a 501(c)3 which then became the origin of CentOS.
Red Hat EOLed the freely available RHL and pivoted it to RHEL thus adversely affecting many people in the Linux and open source community. We (Rocky, Lance, Russ, and myself -- the primary CentOS Founders) decided to rebuild the sources of RHEL with Rocky taking the first stab at it with CentOS3 (which was the current version of RHEL). That was the moment of Founding CentOS. Its initial name was "Caos EL"[1] and it wasn't until a bit later did Lance suggest the name CentOS to me and I, as the project lead, approved it first, then it went to group vote. Rocky was the person who announced the name change from Caos-EL to CentOS on the Caos email list[2] along with letting the world know he was about 99% done with it (this is right before he passed away). Then I continued to lead the CentOS project for another 18'ish months.
> 3/ There is a meaning in the status of founder, it implies some legacy on the future success of the project. Bill gates is a legitimate founder of MSFT because even though he's not there anymore, he did ...
The "several" part is false. Greg is the only one with any claim at all to the early days of CentOS. He created the Caos Foundation, which is where CentOS was started. He believes that makes him the CentOS founder, despite others disputing that claim. None of the other early CentOS folks are working on Rocky.
"The best reason we have is our speed. If we assume all RHEL clones are equal in terms of software, people tend to then weigh speed and community size/support very heavily. We get new packages out very very quickly because nearly all the rebuilds can be automated. We had PUIAS 6 out over a month before CentOS 6 came out. The same is true of minor revisions - of CentOS, SL, and PUIAS, we had a 5.8 release out first." - "IAmA Developer for the PUIAS Linux distribution - AMA"
CentOS had historically fell behind in release times, but we all of a sudden want to paint others in a bad light for being behind. Did everyone forget about CentOS 7 having average of 30 days delay behind each point release? 7.4 being the most at 43 days. What about CentOS 6.0, with 242 days?
As for their behavior, it takes two to tango. I usually like to thank carlwshill and "conan_kudo" (who should pick a better name since he thinks he's from the anime his picture is from) for being true stars in the open source community and really bringing out not only the best in others, but the best in themselves day in and day out. I'm not sure how anyone can put up with them, regardless of which community you're in (CentOS/Fedora EPEL/others), but who am I to judge, I'm just a user.
1.5 years in enterprise time is hardly any time at all. Heck, it takes that time to approve a budget in some enterprises. Yeah, time-to-release is an important metric, but software compatibility and industry support is really the magic sauce.
In my industry, we'll be using current CentOS 7 installs until EOL, and watch all derivatives with interest over the long term - given they provide anything over current RH ecosystem (RHEL, CentOS Stream, Fedora).
Most of the time I think it's fine. Mainly it hurts when there are security updates you need.
You used to be able to track the build process for CentOS when it was a RHEL clone, see how many packages were left to go as the days crawled past.
Things are a little better with the way that development happens more recently. 6 was a massive delay for distributions because RedHat had overhauled a lot around the build process and distributions needed to completely overhaul their stuff too, in ways that weren't that obvious.
Just wanted to clarify, I agree with you completely. I didn't anywhere perpetuate the unfair "it's just produce rebranded RPMs" (which is actually accurate, but trivializes the significant process involved in rebranding and rebuilding, hence IMO misleading). It's a significant amount of work. The time it takes is not unreasonable given the effort involved. That said, the amount of time was painful. I've never complained about it because I'm grateful for anything and everything they do, and I'm not complaining about it now. I'm recognizing the facts around speed of delivery, but in no way suggesting they aren't for a good reason.
This is probably because the project was initially founded by Cloud Linux, which I believe already had the expertise to do RedHat clones and basically donated the setup - whereas Rocky started from scratch from what I can tell.
I’d also say what really counts is time to release critical security updates which they seem to be very on top of.
'Faster' is not a recommendation for an OS developer, IMHO; I don't need operating systems faster - in fact, I like to give them time to mature in other people's hands and to test them myself. The only things I want ASAP are security fixes that are critical to my systems.
If Alma sells themselves on 'faster' (do they?), that would be a significant concern. Why would they do that? Why create a perverse incentive for themselves?
The Alma infrastructure appears to be in their github space, though I don't know anything about it. What's wrong with it?
About `mock`, the problem is the ABI. If you don't build the packages in the "perfect" order, the ABI degrades over time. For example, some libraries might accidentally add something in the middle of a struct. The API is 100% compatible. It will also run without any warning, but all pointers in the application using the libraries provided by those packages will now have an offset. A boolean might now point to in integer or something like this. If you don't have the tooling to detect this and don't have the tooling to ensure you build packages in the right order (and rebuild when needed), then you will eventually get some of these problems. Mostly on point releases. To solve this, the "trivial" way is to follow the RHEL build ordering, which requires some tools. The "correct" way is to use `libabigail`, `libsolv`, `libdnf` and other binary tooling and keep track of these things.
There are more of these little papercuts left and right you get when you build a RHEL clone. You can always cut corners and manually build everything, but you will payback the time you save in outages. RedHat has the test suite, the clones only have a small part of it, they have to be extra careful.
I've worked in software engineering for a long time. Sometimes delays are a good thing; A sign of waiting for quality. And sometimes they're a sign that things went wrong. That poor decisions were made, or implementation was slow due to junior people etc..
Interpreting 'need time to get this right' as a positive, and worse, as somehow a negative on Alma, who thrashed them to the finish line with an identical product, doesn't make sense.
Could you elaborate?
There were multiple times that the media and podcasters would try to pin the two projects against each other, perhaps for ratings..?
Not to take the drama and flare out of the story, but we just didn't want to engage in that.
You got it. From the customer side support from them is cheaper than Red Hat's (sometimes much cheaper). From the supplier side, they don't have to bear much (or any) of the cost of development (beyond infra hosting and re-branding), so they can beat Red Hat's price very easily. I personally find that gross and unethical, but that's just my opinion.
I think overall Rocky is a net positive for the world and for Red Hat.
Obviously this is entirely my opinion :-D
There's also an impossible-to-measure factor in the form of eco-system benefit though. For example, I would never have paid for RHEL had I not entered into the eco-system through Fedora and CentOS. So while RH didn't make money from my CentOS usage, it did eventually make them money because I bought RHEL later when it was worth it. I don't know how you would calculate that, but it does offset ethicality deficit somewhat as well
That said market competition in general is a good thing, and I don't doubt for a minute that Red Hat prices would be a lot higher without the competition. It's a complex equation that's impossible to calculate since the inputs are immeasurable and in many cases theoretical.
Rocky Linux is a shining example of both a free market and the open source community working to the benefit consumers. I don't see how that's unethical.
Dropping CentOS Linux was a completely different thing and one should also acknowledge that there are two very different parts of the CentOS community.
Those that simply needed a free RHEL, didn't have any benefit from CentOS Stream. However, their usecase is filled by Alma/Rocky.
Downstream CentOS distributions however only got benefits from CentOS Stream. There are many private ones, for example Facebook runs on a CentOS derivative, but the most prominent example is Alma itself, which existed (IIRC with another name) even when CentOS Linux existed.
And to be honest, only the latter are really part of the community. Downloading an ISO doesn't make you part of the community. I myself used CentOS Linux on a small EC2 VM but I didn't consider myself to be part of their community (I have since switched to Amazon Linux, for what it's worth).
So all that Red Hat did was basically restructure their collaboration with downstream distros. On one hand they enabled those distros to collaborate even more to RHEL development, which is now public (including individual patches to the kernel, if you remember the circa 2011 kerfuffle). On the other hand release rebuilds are entirely in the hand of the community.
Now, I am not saying everything was perfect. The announcement sucked in many ways, and there still isn't a good solution to use RHEL container images on public CI. People inside Red Hat (including me) will all tell you the same. However, it's intellectually dishonest to ignore that there was and is a CentOS world that goes beyond "I need free Linux and I don't/cannot use Debian", and Red Hat has been very receptive to the needs of that world.
There really wasn't a CentOS world beyond that, because it wasn't a true community distribution. Red Hat never attempted to meaningful involve the community, all board members or whatever pretend org CentOS has are RH employees. There were never any kind of community elections or anything of the sort.
Red Hat killed CentOS.
Honestly if all you know about CentOS is "download ISOs and report bugs that with some luck will be forwarded to RH Bugzilla", you don't know anything about the CentOS community.
Again: downloading ISOs doesn't make you part of a community.
FWIW, clients I dealt with were universally thrilled with RedHat support, the ones who had it. I had one tell me over the phone once, "Oh yeah! Red Hat is amazing! If an issue gets assigned to an engineer, it will be resolved by that same engineer. Unlike you guys."
Red Hat funds an enormous amount of the development of projects (like the kernel), and makes Fedora one of the best (IMHO the best) distros for personal computing, so is one of the top contributors to the community at large. Their support also actively fixes bugs and sends them upstream.
Contrast that with many of the cheaper 3rd party supports, who rarely if ever send contributions (beyond bug reports, which are sometimes a positive contribution, but frequently are net drain because the bug reports don't contain enough info to be reproducible or actionable). They also don't do much or any development.
Of course this is a broad stereotype. I'm sure you can find 3rd party support providers that do contribute to the community.
Linux is not developed by thousands of volunteers on their free time in their garage.
At least 85% of commits to the Linux kernel are from corporates (AMD, Intel, Broadcom, RedHat, etc). Linux is free and open source code, but very largely written by people who are paid to do so, by companies that need to somehow make a profit.
RedHat amounts to roughly 20% of these corporate commits, and that's counting in the share of other corporates that frankly mainly only contribute work for their own drivers.
I see paying for RHEL as being 50% for support / 50% for sponsoring Linux development as a whole.
You are definitely right that RH has pushed things in their own interest, but if those things don't offer value to the broader community, then the community won't adopt them. Red Hat can't force Debian or Ubuntu or Arch to adopt anything. They can push it through Fedora Cent and RHEL, but that's it. The other distros adopted systemd because it offered benefits/improvements over existing things like Upstart. I like firewalld, but that's a good example of something that is only on RH despite RH pushing it. If systemd was really such a negative, then you'd see distros like Devuan take off. A frequent criticism is things like, "Red Hat made Gnome dependent on systemd" which isn't wrong, but they didn't just do it because they could. There were real benefits there.
I think the reality of life is that there will always be people who want things to change, and those who don't want the change. To succeed you have to find a balance.
Also important to remember when decrying "fragmentation" (which I decry also btw), in a massive heterogeneous community like open source, you're gonna have users who have completely different needs and use cases, and both are valid. The beauty of it is the code is open and free, so people can serve niche use cases as well as standard.
[1]: Query string gets you past the "monthly limit" paywall that sometimes pops up: https://freedomben.medium.com/centos-is-not-dead-please-stop...
It was bad because when I chose CentOS for servers at work I thought I could count on letting them live their lives for about 10 years without worrying about them. I have a lot of things to do and taking care of those servers is a very low priority.
Thank God there was a script to convert my CentOSes to Rocky and that it went great, otherwise it would have taken time I don't have to reinstall those servers.
The whole "we never put an EOL date on it" excuse was really weak. It looks to nearly everyone like just an unintentional omission, not a legal hedge. It ended up working out well like you said thanks to Alma and Rocky, but the panic and uncertainty it caused was wholly unnecessary.
There are things to dislike about red hat, but quay is one I would praise them for instead.
edit: ah.. former employee.. so that part was already done :)
I disagree. RH is one of the top contributors to many of the major projects that make up the distro. Also the process of building/maintaining a distro is itself enormous. The 3rd parties have none of that expense so they can undercut the cost easily. In the end it hurts the whole ecosystem, while benefiting a select few.
If this hurts the whole ecosystem, all community developed distributions must have been hurting it, too, by your bizarre interpretation of free software and/or arrogant take on ethics.
Explain how anything you've said so far in this sub thread can't be summed up to "red hat good, everyone else leeches".
Where does the code come from ? All in the game yo.
There is lots of room in the enterprise sector for third-party support. This is, for example, what almost all consultants effectively are. The third-party vendor can solve a lot of problems that the customer may not have the experience to deal with and can be well worth the price paid. But at the end of the day if there's an actual bug all the way upstream in RHEL, only RHEL can (permanently) fix that.
I've dealt with Red Hat in the past and one good thing about their support is that if you have a particularly thorny problem or a genuine bug, you will often eventually end up talking directly to someone who is either wrote the code or sits next to the person who did.
If you have a support contract though and open a case, the level and quality of support is usually very high.
Feel free to write to me at pbonzini@redhat.com, I don't work on Kerberos myself but I know people in the team.
CentOS/Rocky has better ROI is better when stuff doesn’t get fixed on something you are paying money for.
No one asked them to do that and put it in open source, if they are not happy they can go to closed source or different license... Like Microsoft... Maybe the new OP employer? That would make a lot of sense when I see his attitude
But the vast majority of "support" for Linux isn't engineer-level, it's likely config and setup. Which is where both Red Hat and others try to make their money.
My very naive understanding is that it’s just the OS and then only even a distribution (just to emphasize- very naive), what would they need apart from some networking and to run some software on top that needs so much support?
Having consultants out there using CentOS/Rocky/whatever gets these "not a good fit" customers burden off Red Hat, which is probably an advantage to them. I operated what amounts to a "phone a Linux friend" service for ~18 years, and that sort of help just doesn't really fit into Red Hat's offering.
It's nice in many ways to be able to disconnect from the licensing model and just be able to deploy Linux boxes, but still get help when you need it. I recall one conversation about Red Hat licensing related to a machine I was installing for a client named "The Fedora Project" that went like this:
"I assume you'd like me to put Red Hat on there?" "Yes." "Can you provide me with a license key to use on it?" "Uhhh. Just go ahead and install CentOS".
If you want to fix this, make a RedHat license be significantly cheaper than the equivalent Windows product. Charge for support by ticket/case and only support licensed boxes. You'll earn a lot more because it'll look cheaper to the beancounters.
Also RH packages open source project and also does not necessarily gives back to every project.
The right to help your neighbor is important to open source. I find Rocky does just that.
Rocky and Alma popped up for folks that want a bug for bug compatible RHEL OS.
RH publishes all their sources (including the MIT/BSD/etc ones that they don't have to publish). They're the very opposite of "closed source". I don't know of them suing anybody. And nobody is complaining about people using the code, or about the existence of Rocky or Alma or any other rebuild distros.
And you literally said:
>I personally find that gross and unethical, but that's just my opinion.
You cannot acknowledge that they owe everything based on willing participation in an ecosystem where they wherein they openly made a promise and follow that up with the idea that its immoral for anyone to rely on such a promise. It's logically and morally inconsistent.
Gross is just an emotive label you have slapped on after incorrectly concluding that such actions are unethical.
[1]: https://en.wikipedia.org/wiki/Straw_man
It cannot be unethical to rely on a freely given promise that the other side has derived substantial value from.
Red Hat received the entire basis of their business by virtue of their side of that promise they cannot and indeed haven't tried to withhold performance of their obligation in that regard.
Under what ethical theory do you propose that creating downstream distros is unethical? Is it likewise unethical for Ubuntu to exist? If your answer differs why?
Redhat sales are now firmly set in the IBM enterprise model. I'm sure it works for them, but it didn't work for us as a potential customer moving from Centos.
https://www.redhat.com/en/store/linux-platforms
Glad to see this release from Rocky!
Rocky Linux 8.4 GA - https://news.ycombinator.com/item?id=27579297 - June 2021 (24 comments)
Rocky Linux 8.4 RC1 - https://news.ycombinator.com/item?id=27568087 - June 2021 (64 comments)
Rocky Linux releases its first release candidate - https://news.ycombinator.com/item?id=27304012 - May 2021 (116 comments)
Rocky Linux: A CentOS replacement by the CentOS founder - https://news.ycombinator.com/item?id=25445725 - Dec 2020 (533 comments)
Rocky Linux: Community enterprise OS compatible with RHEL - https://news.ycombinator.com/item?id=25358739 - Dec 2020 (16 comments)
Original CentOS founder intends to create new fork of RHEL - https://news.ycombinator.com/item?id=25354811 - Dec 2020 (8 comments)
As for OpenSuse, official support from things like Docker is pretty rare.
These days, I don't deal with the whole red hat ecosystem at all (why would I?). Software ships in docker form these days and I'll typically either deploy it on whatever my cloud provider uses without really messing with that directly or setting up a simple Ubuntu server with Docker using things like Ansible. My daily driver is a Manjaro laptop currently. I like it because I get current versions of everything I need as opposed to old stuff like Gnome 40. Do people actually put Rocky on workstations? My impression is that that whole ecosystem is more about servers and that Fedora is the preferred thing for people in that ecosystem on desktops.
https://www.suse.com/c/suse-liberty-linux/
Oracle Linux 9 also dropped a week ago, and their support is available if increasingly expensive.
The problem with both of these is app support certifications. Microsoft SQL Server (for example) is solely supported on RHEL, although it's obvious that they build on CentOS.
Now, not so much. Do they even know who uses Alm or Rocky or Amazon Linux or Oracle Linux? Do these companies share their CRM data with IBM/Red Hat. Probably not. And of course both are in the business of providing support and consultancy themselves. IBM does the hard work and those companies reap the benefits. Very noble but not very smart.
The whole setup is weird. The only reason for these distributions to exist at all is that IBM/Red Hat refuses to distribute their own software directly. They are still doing the work and providing it for free in source code form (not optional) which is why this works at all. But they've created this whole side hustle that is cutting them off from their own user base. How is that in any way a smart business strategy? All they'd have to do to fix it is roll back to just providing Red Hat to people that want it for free and then up-sell them on support contracts. That's business they aren't currently getting.
I've been a Linux user for more than 20 years. I absolutely saw the value of Mandrake/RedHat/Debian for years. Boring, stable. Got it. Those are important qualities. And Centos gave a free, easy to acquire and use taste of said stability.
But today... everything is so different. I run rolling release, have a billion random services running at any time because it's a dev machine, and... nothing really ever breaks. I get kernel vulnerability patches in a couple days, vs waiting weeks for some hacked together 2.6.8294 backport.
Even if that weren't the case, between snapshotting filesystems and containerized everything, it just feels like such cautious, controlled OS software is a thing of the past.
As always, happy to be told I'm incorrect and/or hear use cases.
Why not the cloud? The company is in a rural location with a shitty internet connection and 50 users. When initially provisioned there was no internet connection at all!
There’s a lot of stuff which is in the territory of “if it’s not broken, don’t fix it” and that’s where these distributions sit. That may cover industrial, office, line of business apps, temporarily connected or fully offline systems. Or just stuff you don’t want to piss around with.
Also never underestimate the power of having stuff you don’t have to touch or even go near. When was the last time you got paid monthly to do something once every three years?
This is not the reality for 99,9% of the world. If it is online, you need to patch security vulneralibities. Because your system is not immune to them.
There's stuff out there that is nearing relic status that is still up and running and not patched or supported. Some of it isn't even touching a network so that's fine.
Do note that what users can use on that, and what system admins use is a completely different stack. Users get an update almost every 3 months. The stability is more for the hardware drivers and things like that, where things are still very iffy because a lot of hardware is actually latest hardware, so not all bugs have been sorted. Infiniband for example is not stable and mainstream at all.
Ubuntu will go through two LTS releases in that time, while you can continue with a single RHEL/CentOS release.
I don't know what "system admins use is a completely different stack. Users get an update almost every 3 months" is about. Infiniband is fine, and typically quite up-to-date in RHEL, but you end up having to deal with the Mellanox nightmare, at least for Nvidia GPU support; roll on RoCM. That area is a good demonstration of the practical evil of proprietary software.
I think this is how you make Appimage files that run on all distros (maybe someone can correct me if I'm wrong)
The best examples are perhaps hospitals, and medical equipment. I previously worked with a hospital where the software is only certified to run on RedHat, and where the system is very tightly controlled. A patch can only be deployed in production, if it has parsed testing and staging environments.
I've also read about research projects, where different versions of glibc, would yield different results. In this case I will continue to argue that this is pretty bad for reproducibility and more of a problem with the people doing the science and not underlying operation system.
Errors, bugs, security holes do sometimes make it into rolling releases, and in some environments regressions are catastrophic or deadly.
Personally I tend opposite world view compared to you. Rolling release are great, for a very small niche, such as desktops for office workers or developers in certain industries.
Rocky Linux is self is a great project, because it allows you to have an environment identical, or close to identical, to the RedHat installations that will eventually run test, staging, QA and production environments, but without the cost. I had been skeptical that Rocky Linux would deliver, but I'm very please and happy that they do.
We don't use CentOS or Rocky. Our customers pay for RedHat or SLES. But I can see that someone wants RHEL and don't want to pay for the license and don't want to switch to Ubuntu/Debian or OpenSUSE either. I don't understand that, they are not that expensive for what they are but ... I am a software developer by profession, not an accountant. (I even saw industries paying for Windows Servers or HP UX, the prices were pretty eye-watering; the Linux-based options are either cheaper or at the same level as MS Win which for multi-million Euro company is not THAT much and even can pay off when you need to call support.)
Something also needs to run your Kubernetes, and that can't always be AWS.
Use a proper distro