(Playing devil's advocate a bit): But is that not the premise and promise of Linux/*BSD - to provide a viable and free alternative to commercial Unix?
Semi-related: now that RedHat is an IBM entity, with the attendant deprecation of CentOS, I fund myself lumping RH into the "fully commercial Unix" mental category.
It's not even remotely the same. It's much closer to a rolling release distribution than a stable one. Redhat is unable to guarantee that the same binary will work on Centos 8 in a few months compared to now. That renders Centos utterly unusable for stable environments.
Also, you can't use Centos anymore for test environments. You now have to go with Redhat and the entire licensing shebang when you init your test vms.
It's not as if they'd suddenly dump breaking changes on CentOS users. Instead of there being a CentOS 8.1, 8.2 etc. there is just a CentOS 8. Similar to how you can enable the 'debian-updates' repository in Debian.
>It's not even remotely the same. It's much closer to a rolling release distribution than a stable one
No it isn't. CentOS Stream will never make a change which would be out of place for RHEL to make, which means ABI and APIs (and version numbers for the most part) remain stable for the entire lifetime of the distro.
That is wholly incomparable with something like Debian Testing or Arch or Fedora Rawhide which "roll" through major updates all the time.
Because it is fully comercialised? There isn't even a free personal version of rhel.
The source is open but so were many of the one commercial unixes. In fact some customers in the early days of Unix used to change their codebase a lot.
I don't think commercial and open source are mutually exclusive. Not the OP though so perhaps they meant something else.
Personally I don't think open source would have to be free as in beer but I don't like it when big tech takes too big a role in it. Eventually it'll just become another cash cow.
Not to mention the binary compatible clone that this company chose to use. Wtf is this dude on about? Rocky couldn’t exist if RHEL weren’t open source. Idk where people come up with this stuff.
> They were using CentOS for no cost previously and have switched to another no cost RHEL rebuild (Rocky).
Since they involved CIQ, I'm pretty sure they are now paying support for Rocky, which, if you read the article, was clear they didn't mind paying for. It was the per-cpu subscription that was not realistic in their business model.
That is the way I read this, and Rakuten is a 1.6 trillion yen company. I wonder if they give back to the community ? I am fairly sure they do not, but this based upon the tone of the article :(
The issue with Redhat in my experience is not having to pay money for it (when I worked with Redhat it was for the government who had bought some huge bulk license and had no issues with paying that, still cheaper than many other licenses), it is that the license servers, etc always have broken while CentOS just have worked. More moving parts which can break. The act of checking licenses makes the product worse than the free one. And I have had similar issues with OpenShift.
Edit: This reminded me of another story. I once had a long fight with Gitlab's support due to an issue with their billing. They failed to fix it so now we run on the community edition which just works.
My preferens for free is due to there often being bugs in the billing/license checking.
Absolutely this. Even if you've written off the cost of the licenses, you're still better off avoiding their licensing system entirely. Red Hat seems to have engineered a situation where their paid product is worse than the free product, even ignoring the price tag.
Red Hat's business model was to both support and profit from open source. That is not an easy path, but they were mostly successful, at least until they were swallowed by IBM. If IBM manages to destroy Red Hat's value proposition, this opens a business opportunity for a company that takes a less antagonistic approach to its customers.
It looks like Amin is confusing "open source" with "free" (as in free beer).
He seems to be pissed off that Red Hat discontinued the free CentOS distribution. And instead of paying for Red Hat, it goes to Rocky Linux, a free fork of Red Hat, just like CentOS. It is fine (at least legally), but the paid Red Hat distribution is no less open-source than Rocky Linux, in fact, the latter exists because Red Hat is open-source.
That's why I think this article particularly ironic, it says that Red Hat is not "true" open-source and then made a move that is only possible because Red Hat is open-source. It didn't even go to something like Debian, which is the most "free/libre/open-source" of the major distros, it went to the clone of the distro they criticize.
When people say they want "true opensource", they usually mean they want to be the bosses and dictate how the product is being developed to the people who are actually doing all the work and giving all the funding.
He is not a stupid person, he knows what he's talking about. Of course he doesn't wan to pay for RHEL licenses, and he explains why the licensing model used by RH now is a bad choice for a business with a high number of nodes.
Which is why proper open source should be GPL, with a commercial license, so that the likes of him pay for the work of others that make his business possible to start with.
Tell that to the MySQL folks. They sold it to Sun for a billion dollars, then made Larry Ellison pay for its development when he finally realised buying it with Sun wouldn’t allow him to kill it.
It was a way to reply “has to pay for work of others” but which work ? And which others ?
Cause currently Redhat is owner of Ansible and make money with it, while me as a contributor never got any penny they get from Ansible.
So why rakuten should pay Redhat for work not getting paid by Redhat? See the point ? Who is parasiting the ecosystem here ?
He isn’t stupid, but he’s either uninformed, or he’s lying because he’s dead wrong about their licensing model. You don’t pay per core. You pay per physical machine (of any size), or per 2 VMs (of any size). If you’re heavily virtualizing, you buy a VDC license at an MSRP of $4k USD, and you run all the VMs you want on a single hypervisor. If you’ve got more than a couple hundred machines, the VDC licensing is probably your best option. Alternatively, use RH’s virtualization solution, and pay only for the OpenShift licenses.
He doesn’t want to pay for the OS at all, he just wants to pay for support. That’s fine, but he’s still factually incorrect in what he’s said.
As someone that uses RHEL coz of some of our customers need it, I can get their point.
RHEL might be open source but it is structured (obviously) to sell the support license. You can't "just" install DRBD like on every other distro, you need to get the storage whatever the fuck pack, or get the packages from EPEL. In Debian DRBD module just comes with kernel, need to install it on RHEL etc.
It's annoying to manage even as paid customer and wholly worse experience than just installing Debian.
For some of us, using an older but way more tested and more stable version is equally important. CentOS stream moves the release point to (RHEL+1), which sometimes break things in unexpected ways.
This is why we also migrating to Rocky on such systems.
Looks like our environments are different. The environment we use is a descendant of Scientific Linux, so we always worked in (RHEL-1) environments. So pinning to RHEL is not helping us, again.
You can have a single RHEL8 reference (not sure how hard is to get the full RHEL BOM) to base your versions from and use the CentOS repos as source for the packages themselves.
My CentOS boxes are pulling straight from 9 stream. I had an issue with Grafana today I’m still figuring out.
Why Rocky? Why trust the guy that sold to RHEL once already? Why not Alma that has a stated commitment to never sell to a private corporation? Alma also has a stated turn around time for security patches of 2-3 business days, something I have not seen from Rocky.
If we decide we can’t afford 100% RHEL licensing, Alma was my choice, but I see so many choosing Rocky, and I’m genuinely curious why people are doing that.
Either appear to be acceptable choices. Looking at their respective websites, they both make similiar commitments to remaining independent. From what I found, the Rocky website actually makes the commitment in stronger terms than the Alma website.
The change had nothing to do with IBM. I know people view this with skepticism, but it's true.
The primary purpose of CentOS Stream is this: to allow a real community to be built around CentOS, where partners and users can contribute. With CentOS if you encountered a bug your only option was to file it against RHEL (not CentOS), wait for someone at Red Hat to get to it, then wait for RHEL to pick it up and release it. You would be very lucky for that whole process to complete within 6 months and if you're not a customer you probably won't be able to test a hotfix on your own systems.
Whereas with Stream: partners and downstream developers can file issues directly against Stream, and then contribute fixes directly to Stream, where they get QA'd in public and are added directly to the future release of RHEL. So Oracle Linux, Rocky, Alma and anyone else can actually participate in the process and contribute back instead of having to just take whatever RH throws over the wall.
However, the messaging and timelines around all of this were handled... not well, and some of the acrimony around that is reasonable even if the changes make sense.
Wasn't that the purpose of Fedora? An up to date distro where new technologies can tested, community and partner feedback, etc.
In any case, I get the justification for CentOS stream, but changing CentOS 8's EoL from 2029, to Jan 1st 2020 ... after it shipped is definitely a good way to alienate your customers and community.
Fedora is where new major versions of RHEL (think 7, 8, 9) come from. Within a major version of RHEL, there was no public upstream under the previous model, and now that public upstream is CentOS Stream.
Fedora moves fast enough that by the time a RHEL release comes out, Fedora has moved far enough to not be very relevant as an upstream.
People think it’s even remotely close to Fedora because it was initially communicated as an upstream, rolling distro (which Fedora isn’t, anyway, so …), and they never read any of the press or forum posts after day one.
Even if Stream had versions, it’s still only tracking about a minor release ahead of RHEL. If you trusted RHEL’s QE process before, there’s little reason to stop trusting it. I really don’t think much has changed in terms of usage outside of some infrequently used software, but you have to account for that when managing them on a RHEL system, too, so I still don’t really understand the problem.
Another upside of Alma is it is backed by the CloudLinux guys who have been around for a long time with a solid reputation in the Linux/hosting industry—- and I believe you can also obtain a support package for a relative pittance (obviously it’s not RHELcorp-level support, but if you require/need support, it’s something at least).
We have a couple thousand Alma 8 in production and it’s been smooth sailing.
Rocky is more visible, I don't see that as a bad thing, not like it spends many resources. It also helps partners decide to support them, in particular Mellanox now Rocky-8.
Your environment is that important but never considered actually using RHEL (or Ubuntu ESM)?
> sometimes break things in unexpected ways
I have faith on RH to keep CentOS stream to be quite stable now that it runs on its own brand.
I can't understand how people had super faith on CentOS when it was run by a community which even had someone lock up the organization's fund/asset at some point.
One day I'd like to know whether sales were really hurt or it's like 'we've been downloaded 1 million times our single is 5usd, we could have made 5musd' we'... Well no.
People think that we use CentOS because we’re cheap. The reality is the software we use is certified against it.
We’re a supercomputing center. We’re federated. Everybody is using CentOS. So, we also use CentOS on our pets sometimes because it reduces administration load.
We’re using it for years. We can patch it if we need it. We can submit patches if we need it. We use CentOS as a multinational community, which is completely different from “Hey, look! Free RHEL!”.
We’ll see where we gonna evolve. Stream? Rocky? Alma? It’s will be a collective decision. For some pets we migrated to Rocky, and we’re happy, for now.
Price sensitivity is a thing and I'm quite sure I don't get my money's worth with RHEL prices especially when they made us run around explaining our use case to their business people and them telling us months later they're not interested in the end and 'just buy and use openstack'.
You're talking about a community problem (which most community have at some time and I don't believe for a second that companies don't have these kinds of events or worse) when people are talking about what they liked and actually observed to be good for them in centos. Yes, work should be paid. True also that some things can bring more customers without being profitable by themselves. Yes people are allowed to be angry about a bait and switch.
One can be sad and angry that a service that one used to rely on is discontinued or made very expensive.
People were just free riding a community effort off of a commercial product that costs a lot of money and one can be sad when things stopped the way they wanted but angry?
Besides, do all those people complaining have projects that are very sensitive to stability that CentOS stream is going to ruin? That is quite hard to believe.
Not sure if it was even a bait when people just chose to use it and CentOS never locked you up in using it.
You can think of it as RHEL tracks CentOS and has it's point releases spun out of that. There is no reason to think CentOS stream will break more often than RHEL does, because, if that happens, it's Red Hat shooting its own foot.
Open source does not mean that you can just install it. I understand that for majority of open source projects that’s true, but definitions are definitions. May be we should invent better terms like non-commercial free software and commercial free software and set some expectations for the first category.
But still Rocky Linux is possible only because Red Hat provides sources for its distribution. There’s no Rocky Windows.
> He seems to be pissed off that Red Hat discontinued the free CentOS distribution.
Just to clarify, CentOS is not discontinued. Basically all that is discontinued is CentOS "point" releases. There is CentOS stream 9, it gets updates, there will be CentOS stream 10, it will get updates, there will be overlap between CentOS stream 9 and 10 being supported. Sure the CentOS stream won't be exactly equivalent to RHEL point releases, CentOS stream 9 won't be RHEL 9.1 - but RHEL 9.1 will be cut from CentOS stream 9 AFAIU.
And to further add, a lot of the major Red Hat backed projects switched to using Centos steam as the officially supported redhat lineage. Ceph and OpenStack Kolla are both good, major examples.
CERN was a big user of CentOS. After redhat changed CentOS8 EOL, CERN was allowed to use RHEL. Last time I checked, they suggested others in high energy physics community to use Rocky or Alma if RHEL is not an option.
Well, I checked again. From CERN Oct 28, 2022 Linux Future Committee meeting [1]:
- Several bugs have been encountered in Stream, which were not present in RHEL.
- Some user communities had previously expressed concerns on the feasibility of using Stream in a production setting. This feeling has also recently been expressed internally by some IT groups.
- CERN is evaluating our original recommendation [0] of utilising CentOS Stream as the default Linux Operating System.
Something still exists that is called CentOS, but as an upstream to rather than a downstream from RH, it’s not equivalent or even similar to what CentOS used to be.
To me it is not that critical what the specific organizational structure is that relates to CentOS Stream and it's relation to RHEL. What matters more is how it affects me as someone using it.
CentOS was never RHEL, it was though very close, CentOS Stream is also very close. Whenever I used RHEL or CentOS before CentOS Stream I always updated to the latest versions of packages for the major release version, if I use CentOS stream I will still get updates to the latest versions of packages for the major release version. Maybe I get them a bit earlier than RHEL, but this to me is somewhat irrelevant. If I want reproducibility there are still plenty of ways to get it, including RHEL UBI.
UBI is fine for deployment. But in a shop where development workstations are all rhel, it was nice to know that the package versions available for 7.9 matched for both rhel and centos. It isn't that centos stream has no value, its just it isn't the same value. Now rocky is. Our business already made the switch for anything it was using centos for.
> CentOS was never RHEL, it was though very close, CentOS Stream is also very close.
It was close enough, Stream is almost as close and for some people it isn't close enough when they need bit for bit compatibility for the software they are going to run
What is such a big difference as not to keep using it?
You seem to believe a community supported distro was as good as a corporate supported distro and put such a high faith in it but now somehow it's all corrupt when RH supports it putting it as an upstream?
If your usage is that critical, why not buy RHEL (or Ubuntu extended support) for 10 years support and stop freaking out like your Linux days have come to an end?
Stream is inherently less well-tested than the normal stable releases. I'm not aware of any specific bugs that have come up in it, but it get updates before red hat is willing to push those updates to people who have a support contract.
Stream also notably has a much shorter lifecycle, with support for 8 Stream ending in 2024, compared to 2029 for rhel/rocky/alma.
To me the main value of using rhel-based distros is the great reliability and long support life. Stream compromises on both of those compared to what normal centos offered, and rocky and alma currently offer.
You are. The charts are confusing. At the bottom it says full support ends May 31, 2027.
Red Hat made some strange decisions with the way they’re handling Stream vs RHEL, plus the communication was abysmal, and they changed models mid-cycle. Be mad at them for that all you want, the rest was overblown.
This company could have just used Stream 9 if he wanted a free OS. Since he’s virtualizing, he could have bought hypervisor licenses for $4k a piece that allow you to run as many VMs as your hypervisor can handle. He could have used OpenShift for virtualization and container orchestration, and just paid for the OpenShift licensing. If you use OpenShift, you can run all the RHEL VMs and UBIs you’d like at no extra cost. What you never do is pay per core. There isn’t a single licensing model from RH that forces you to do that. If you license individually, each license supports one bare metal machine, or two VMs. This guy is full of shit. He’s just mad everything isn’t $0.
I don't think Stream compromises in any significant way on reliability. They test it rigorously. The short support window is a bit of a bummer, though.
That said, I'm running CentOS Stream 9 right now. It's supported until 2027. I generally like to keep my operating systems relatively up to date, so a roughly 5 year support window is adequate for me.
5 years sounds good enough.
Ubuntu does that too with LTS unless you go extended 10 years support.
People freaking out because their free candy is no longer as tasty as one could hope, it looks quite funny they have zero willingness to pay for such a long support effort.
Every package that gets pushed into Stream has gone through the entire QA process for being included in RHEL. While it is of course possible that problems could be noticed later on, it's not all that likely.
CentOS as ‘a useful distribution for what 95% of what CentOS users were using it for’ has been discontinued. We used to use it, and we just can’t anymore - several years before the promised support period was up…
My reading of this is that Rakuten used to use a combination of RHEL and CentOS. Now that CentOS as a "down stream" equivalent of RHEL is dead (its selling point), they have reassessed their platform and gone with Rocky.
Rocky is "down stream" of RHEL, CentOS 8+ is "up stream" of RHEL (since 2020, and what Fedora has always been).
CIQ sell support contracts for Rocky linux, so rather than having a combination of RHEL (with support) and CentOS. They now just have Rocky with an optional support contract.
Linux is open source, they are aloud to do this, thats the point. They are not leaching. Red Hat, by ending CentOS as a "down stream" equivalent, have brought this on themselves.
There will be a lot of other users of RHEL and CentOS who are going through the same process right now as CentOS 7 support ends next year. We will see more companies making the same choice.
I’d bet that’s accurate. The CEO very clearly did not understand the licensing models because you don’t pay per core for RHEL licenses; this isn’t Oracle. VDC licenses are cheap: MSRP is $4k USD per hypervisor with premium support, and I’m sure you can negotiate a discount at a certain size. You can probably game VDC licensing if you really wanted (I’m not endorsing the idea of stealing from Red Hat, and to be clear, my company isn’t interested in getting sued to save what amounts to pocket change for them). Or you could switch to OpenShift for virtualization (because who isn’t at least _considering_ dumping VMWare now that Broadcom owns them?) and container management, and run all the RHEL VMs and UBIs your heart desires.
Let me quote the article: “I called him and… said ‘listen, I want to give you a shot and I'm willing to work with you’. So we worked with him for 13 months to build our real-time kernel to meet complex workloads with one condition: It has to be open source and it has to derive its characteristics from a large community,”
Rakuten worked with the founder of Rocky Linux for 13 months to develop real-time kernel. I don't know what kind of deal Rakuten promised to Rocky Linux (founder) and its community, but Rakuten contribute by open-sourced the work to community. That's enough for me. That's the spirit of open source.
It reads to me as though they outsourced their performance tuning. There are tons of tweaks you make to it for specific environments. I’m not sure what they would have done that any experienced engineer could not have done.
The fact is that most users of open source care more about not paying than freedom or openness. The same factor is why people go for “free” surveillanceware or fishing-hook-ware.
The people who actually understand and care about freedom is a small niche.
I would agree 100%, but this is not 90s Red Hat, it's IBM now.
It seems like the classical "Here's our new pricing and for being a long time customer we throw in a bottle of vaseline too". Which technically has nothing to do with open source but Red Hat ( like every company in this space ) created an "ecosystem" and uses the word "open source" VERY liberally when selling you stuff. Some of their opensource software need some paid service or "maintenance" contract to work or have any value.
I'm sure Red Hat/IBM is ( mostly ) covered on the legal side, but the sausage is made in a very different way.
Like what? You’re paying for engineering, which provides stability, and support. There’s nothing from Red Hat that isn’t upstream for free. They even open source companies that they purchase (like 3scale). The base OS has multiple clones (Alma, Rocky, etc.), Satellite comes from Foreman, AAP2 comes from Ansible and AWX, IdM from FreeIPA, OpenShift from OKD, and the list goes on. The only thing I can think of that doesn’t have an upstream equivalent Insights, but that’s because it uses RedHat’s ML systems, and connecting your systems to theirs costs money. Insights-core is open source, but may not provide much value since you can’t connect to RH’s backend.
> uses the word "open source" VERY liberally when selling you stuff.
RedHat has consistently been the most aggressive supporter of Open Source out of any for-profit companies. Not only do they contribute to open source projects, they consistently release their own code/projects as open source early on, which could easily be kept proprietary (if they chose) and used as a competitive advantage over others.
This is true though as RH will terminate your contract if you redistribute their binaries. So they use a contractual backdoor to undermine GPL. And before people talk about how GPL covers source but not binaries, it covers binaries. In fact, all of CentOS, Rocky, Alma, etc. is a massive waste of everyone's time, energy, land, and electricity. Something akin to crypto mining. Everyone could have used RHEL, but we need this complicated indirection for no reason.
It covers binaries all right, the way I understand this RH is very aggressively pursues this as a trademark violation (which it shouldn't be unless it's deployed to third party that can give the blame to RH).
It's kinda scary that there are so few mainstream Linux distros that people are willing to trust on their servers. Debian, RHEL, that's about it. And only one of those is commonly supported by major vendors like Dell, HPE and EMC.
What do you mean? Ubuntu is hugely popular (i recall some cloud provider saying it's the most popular distro they host), and so is SUSE in certain parts of the world (like Germany).
I don't recall having problems with Dell server-related packages on Debian, even if their own crapware (OpenManage Essentials and then Enterprise, legitimately one of the worst pieces of software I've ever had the displeasure of using) for managing iDRACs and firmware came as a CentOS-based appliance.
Well first of all Ubuntu is Debian based, so choosing one over the other is mostly semantics. And secondly, even if you can use Dell software, it's not supported. EMC for example will not support you if you don't use supported distros.
It's still a frighteningly low number of distros considering we used to have an abundance of distros to hop around on for desktop use. Distrowatch.com for example, but for all those distros, when it comes to servers we narrow ourselves down to less than a handful.
> EMC for example will not support you if you don't use supported distros.
Support you how and with what? What does your storage vendor have to do with your server's distros? Even the distro running on the server itself doesn't matter, really, because it's not like Dell/HP are writing distro-specific userspace drivers.
>What does your storage vendor have to do with your server's distros?
Operating systems have a hardware compatibility list (HCL). If your product is not on the HCL for the operating system vendor and you have a problem--at least one somehow plausibly related to the interaction between your product and the server--neither vendor is obligated to support you. (They may well try but there's no guarantee.)
Many people, myself included, don’t trust Debian. They patch too much sometimes for dubious reasons and have a poor trail record when it comes to introducing bugs. I wouldn’t use it for anything work related.
I have seen Gentoo used as a base to deploy binary packages veted by the org in the past. Gentoo provides tooling to do that and it worked fine.
By whom? Because I can assure you that it’s not true.
Debian patches a lot for completely bonker reasons. They do it to support architectures nobody uses. They do it to remove part of packages for purely ideological motives. They do it to split packages into multiple parts making work harder for upstream when bugs are found. They do it to integrate with their own subpar tools. Sometimes they introduce major bugs along the way like the infamous OpenSSL one.
I understand people liking the fact that Debian is community maintained but the security track record is simply not there.
Verizon, GitHub, Godaddy, Cisco, Google, NASA. If you don't like a debian package you can roll your own which is dead simple. I've never worked for a company that is 100% reliant on OS provided packages. Seems like it is true...
Are you sure all those companies use Debian? I thought Verizon and GoDaddy were using CentOS. Also Google's Debian is customized might not really be appropriate to still call it Debian.
>On May 13th, 2008 the Debian project announced that Luciano Bello found an interesting vulnerability in the OpenSSL package they were distributing. The bug in question was caused by the removal of the following line of code from md_rand.c
> MD_Update(&m,buf,j);
> [ .. ]
> MD_Update(&m,buf,j); /* purify complains */
>These lines were removed because they caused the Valgrind and Purify tools to produce warnings about the use of uninitialized data in any code that was linked to OpenSSL. You can see one such report to the OpenSSL team here. Removing this code has the side effect of crippling the seeding process for the OpenSSL PRNG. Instead of mixing in random data for the initial seed, the only “random” value that was used was the current process ID. On the Linux platform, the default maximum process ID is 32,768, resulting in a very small number of seed values being used for all PRNG operations.
The most security focused uh? the distro where a maintainer modifies code he doesn't understand because valgrind complains about it? In one of the most important packages of a linux distribution? No.
Anyone who has had memories of this event would treat anything stamped with debian's name as nuclear waste.
Debian has had many conflicts with upstream projects for their relentlessly idiotic patching. Firefox once prohibited Debian from using the FF trademarked name and debian in turn renamed firefox "Iceweasel" in their distribution to continue shipping their tainted version.
I'd trust any of these : RHEL/Fedora, openSUSE, Gentoo and Arch over debian any day.
Ubuntu is the largest deployment footprint, then debian, then RHEL.
I wonder where Centos users went, given it is not the same anymore, but it used to run all of facebook at some point.
> And it took a long time to get Rocky Linux to where Rakuten Mobile needed it. “To get Rocky Linux to deliver on the same performance of Red Hat for us was a 13-month process. It was not trivial… there are so many tweaks you have to do… and we took all of these tweaks we've done and we gave them back to the [Rocky Linux] community,” said Amin.
Sounds like they worked with the developer of Rocky Linux for 13 months to get the realtime kernel where it needed to be, and I would assume the author did not work for free and was compensated by Rakuten. "We decided to move to fully open source software and paid the author to customize it for us and released our customizations as open source" doesn't sounds very leachy to me.
That may actually depend on which version of RHEL and in which decade the kernel was released... But yeah, that's more to blame the org that won't update and not on RH to provide weird backport kernels for a decade.
The interesting thing is that pretty much every single RHEL customer kept CentOS on the side as an "alternative". Over the past few years a lot of those alternatives actually became production systems (support became an app thing with Docker, not an OS thing), and now _of course_ Rocky is a better option, due to the lower cost and community backing (or just moving the hosts to Ubuntu/Debian).
Is it leeching if they're paying the founder of CentOS and creator of Rocky Linux to help them build out the system to suit their needs and then contributing that code back upstream?
I mean, I guess he could have been working for them for free for thirteen months, but that seems rather unlikely.
It is. Rocky Linux is not in the same space like RHEL. They cant dictate the market prices.
I very much doubt Rocky is a sustainable business model. Hell even RHEL was not...
This is the aftermath of the enterprise/FOSS (pay for support/whatever model that never really worked) war on prices and we are basically going back to the era for paying for software and some people and businesses dont like it.
I don't see how this is leeching. They did pay for the work to get the distribution ready for their needs (unless they didn't, in which case Greg Kurter must be a saint), that work ended up back upstream.
Is your argument solely that Rocky Linux didn't extract the maximum amount of money they could from their client, or is there something else I'm missing here?
Also, let's not forget that he also said that he's fine with paying for support - just not CPU cores (which also does imply the RT kernel work was paid for). Paying for support is categorically not leeching.
> I shouldn't be forced to pay subscription fees… I have no problem to pay support [fees]
The last time Red Hat saw an annual revenue decline was 1999. Cash Flow grew more or less in tandem with revenue. I don't know what the future would have looked like for Red Hat, but the other enterprise Linux companies seem to be doing well.
CentOS used to a recompile of Redhat, guaranteed binary compatible, with a 10 year EoL.
AFTER CentOS came out they changed it from 2029 to Jan 1st 2020. Now CentOS stream is not a recompile, but has less tested changes BEFORE RHEL, and is now a rolling release with a 5 year EoL.
So for those that need compatibility with RHEL, that's broken.
> To get Rocky Linux to deliver on the same performance of Red Hat for us was a 13-month process. It was not trivial…
I wonder why is that, if Rocky is repackaging RHEL source performance should be identical, no?
Article mentions realtime kernel. It seems they were working with RH on custom kernel (that's the support contract he mentioned), but started to be asked to pay for RHEL license to use it, which understandably pisses him off
1) I am not sure Rocky were actually re-packaging the RT stuff originally, they've got it now though.
2) RT is pretty finicky to tune correctly for your use case at the best times, and especially for use cases as exacting as the 5G RAN. I assume there is a consulting aspect to this that they are paying the CIQ folks for, but probably based on man hours or a flat fee rather than the per socket/node subscription they might have paid Red Hat.
Corporate support of a community-led RHEL compatible OS seems like an obviously good thing, and I'm surprised that so many people are worried about Red Hat's welfare? If someone doesn't feel that the costs of RHEL licensing outweigh the benefits then that's a reasonable choice for them to make, and contributing money to an existing community project that others can benefit from is way better than them doing it in-house. Open source isn't about getting everything for free, but it is about getting to choose to pay someone who fits your needs better rather than being tied to a specific vendor.
Fedora has a lot more churn and is more bleeding edge. Cent OS Stream is quite stable and is kind of like a thoroughly tested pre-release of the next version of RedHat.
Commits into CentOS Stream go through testing and become RHEL.
Those commits are mostly from Red Hat developers, because all RHEL devel now happens in Stream first, but iiuc technically anyone could submit a PR, and maybe CentOS Stream community members do that from time to time.
Fedora is entirely separate packages from CentOS/EL. Fedora is much closer to upstream projects. Fedora has shorter lifecycle, no API/ABI guarantee across releases, moves fasts and breaks more things, etc.
CentOS Stream also provides a way for the communities of various clones to contribute back. Previously if you found a bug all you could do was report it against RHEL (not even CentOS) and wait for a RH engineer to get to it, then wait again for it to be released, before you'd have a chance of even testing the fix on your own systems.
Every fourth version of Fedora gets forked into the new version of CentOS Stream. Then for these rest of its lifecycle it will be developed independently. Every change in this version will go into the corresponding RHEL version after existing for afew months.
The numbered version of CentOS Stream is only "upstream" of the same numbered Enterprise Linux version. RHEL and CentOS 9 are both forked from Fedora 34, for example.
It'd be such a relief if online Linux threads focused mostly on the technology itself and not on the various personalities historically associated with it.
I've played the OS switcheroo game previously, it's wasteful and expensive. At this point I won't touch anything RHEL based. I have no doubts someone at IBM is working on making RHEL incompatible with other clones.
Just to add some ”inside baseball” to this thread, I work for Rakuten Symphony so can give my take from my perspective, be they right or wrong. The open discussion and different points of view is what is so healthy if something wants to get stronger.
We are breaking apart the traditional monolith deployments in radio base stations, that are deployed at a very large distributed scale. For example we are deploying 30,000 edge nodes to do real time processing of radio baseband signalling, 23,000 of them in the the next 10 months. Now these workloads have more in common with high performance computing than they do with running standard business application systems, so we are aware we have to modify the kernel to make it cope with the real time requirements. We are removing the monolithic model because we know we can reduce the costs and increase the deployment optionality. And the existing enterprise OS licensing model was prohibitively expensive at this scale. And then our open source choices were changed.
We are happy to contribute our real time learnings back to a community since that helps us help the community to help us. What we have done is not for everybody, and others need to make their own choices for their own reasons. But we believe it is having those choices is what creates a healthy industry. And we no longer had the choices we wanted, so we want to contribute to making them for others.
175 comments
[ 2.8 ms ] story [ 244 ms ] threadThey were using CentOS for no cost previously and have switched to another no cost RHEL rebuild (Rocky).
Semi-related: now that RedHat is an IBM entity, with the attendant deprecation of CentOS, I fund myself lumping RH into the "fully commercial Unix" mental category.
CentOS is still out there. It's just that it does no longer follow RHEL on every step, but is ahead of it all the time.
Also, you can't use Centos anymore for test environments. You now have to go with Redhat and the entire licensing shebang when you init your test vms.
No it isn't. CentOS Stream will never make a change which would be out of place for RHEL to make, which means ABI and APIs (and version numbers for the most part) remain stable for the entire lifetime of the distro.
That is wholly incomparable with something like Debian Testing or Arch or Fedora Rawhide which "roll" through major updates all the time.
The source is open but so were many of the one commercial unixes. In fact some customers in the early days of Unix used to change their codebase a lot.
I don't think commercial and open source are mutually exclusive. Not the OP though so perhaps they meant something else.
Personally I don't think open source would have to be free as in beer but I don't like it when big tech takes too big a role in it. Eventually it'll just become another cash cow.
Incorrect.
https://developers.redhat.com/articles/faqs-no-cost-red-hat-...
Since they involved CIQ, I'm pretty sure they are now paying support for Rocky, which, if you read the article, was clear they didn't mind paying for. It was the per-cpu subscription that was not realistic in their business model.
They're in the "Backed By" section of https://rockylinux.org/about/ , so I'm going with "yes".
Giving back to the community could be both more or less than paying cash.
https://news.ycombinator.com/item?id=33406027
This seems simply another case of using something as the backbone of a business and demanding it to be free (as in beer).
Edit: This reminded me of another story. I once had a long fight with Gitlab's support due to an issue with their billing. They failed to fix it so now we run on the community edition which just works.
My preferens for free is due to there often being bugs in the billing/license checking.
Red Hat's business model was to both support and profit from open source. That is not an easy path, but they were mostly successful, at least until they were swallowed by IBM. If IBM manages to destroy Red Hat's value proposition, this opens a business opportunity for a company that takes a less antagonistic approach to its customers.
Is Debian-Testing any less "true Open Source" than Debian-Stable?
He seems to be pissed off that Red Hat discontinued the free CentOS distribution. And instead of paying for Red Hat, it goes to Rocky Linux, a free fork of Red Hat, just like CentOS. It is fine (at least legally), but the paid Red Hat distribution is no less open-source than Rocky Linux, in fact, the latter exists because Red Hat is open-source.
That's why I think this article particularly ironic, it says that Red Hat is not "true" open-source and then made a move that is only possible because Red Hat is open-source. It didn't even go to something like Debian, which is the most "free/libre/open-source" of the major distros, it went to the clone of the distro they criticize.
So why rakuten should pay Redhat for work not getting paid by Redhat? See the point ? Who is parasiting the ecosystem here ?
If they were using non-copyleft licenses then RHEL packages would be closed.
He doesn’t want to pay for the OS at all, he just wants to pay for support. That’s fine, but he’s still factually incorrect in what he’s said.
RHEL might be open source but it is structured (obviously) to sell the support license. You can't "just" install DRBD like on every other distro, you need to get the storage whatever the fuck pack, or get the packages from EPEL. In Debian DRBD module just comes with kernel, need to install it on RHEL etc.
It's annoying to manage even as paid customer and wholly worse experience than just installing Debian.
All that is mostly solved with CentOS. I really don't get the people so upset about it.
This is why we also migrating to Rocky on such systems.
And that's for my pets. The cattle is blissfully unaware of anything different.
My CentOS boxes are pulling straight from 9 stream. I had an issue with Grafana today I’m still figuring out.
If we decide we can’t afford 100% RHEL licensing, Alma was my choice, but I see so many choosing Rocky, and I’m genuinely curious why people are doing that.
The primary purpose of CentOS Stream is this: to allow a real community to be built around CentOS, where partners and users can contribute. With CentOS if you encountered a bug your only option was to file it against RHEL (not CentOS), wait for someone at Red Hat to get to it, then wait for RHEL to pick it up and release it. You would be very lucky for that whole process to complete within 6 months and if you're not a customer you probably won't be able to test a hotfix on your own systems.
Whereas with Stream: partners and downstream developers can file issues directly against Stream, and then contribute fixes directly to Stream, where they get QA'd in public and are added directly to the future release of RHEL. So Oracle Linux, Rocky, Alma and anyone else can actually participate in the process and contribute back instead of having to just take whatever RH throws over the wall.
However, the messaging and timelines around all of this were handled... not well, and some of the acrimony around that is reasonable even if the changes make sense.
In any case, I get the justification for CentOS stream, but changing CentOS 8's EoL from 2029, to Jan 1st 2020 ... after it shipped is definitely a good way to alienate your customers and community.
Fedora moves fast enough that by the time a RHEL release comes out, Fedora has moved far enough to not be very relevant as an upstream.
Even if Stream had versions, it’s still only tracking about a minor release ahead of RHEL. If you trusted RHEL’s QE process before, there’s little reason to stop trusting it. I really don’t think much has changed in terms of usage outside of some infrequently used software, but you have to account for that when managing them on a RHEL system, too, so I still don’t really understand the problem.
We have a couple thousand Alma 8 in production and it’s been smooth sailing.
> sometimes break things in unexpected ways
I have faith on RH to keep CentOS stream to be quite stable now that it runs on its own brand.
I can't understand how people had super faith on CentOS when it was run by a community which even had someone lock up the organization's fund/asset at some point.
People believe that CentOS was the same RHEL for free. Bit to bit. May be with changed logos.
So people believed that they’ll get rock stable system for free by using CentOS. They believed it so much that it hurted RHEL sales.
None of those beliefs were correct. But lost sales were real.
We’re a supercomputing center. We’re federated. Everybody is using CentOS. So, we also use CentOS on our pets sometimes because it reduces administration load.
We’re using it for years. We can patch it if we need it. We can submit patches if we need it. We use CentOS as a multinational community, which is completely different from “Hey, look! Free RHEL!”.
We’ll see where we gonna evolve. Stream? Rocky? Alma? It’s will be a collective decision. For some pets we migrated to Rocky, and we’re happy, for now.
Being founded by former CentOS founder also helped. MySQL guys found Maria, CentOS founder started Rocky. That’s OK in my book.
You're talking about a community problem (which most community have at some time and I don't believe for a second that companies don't have these kinds of events or worse) when people are talking about what they liked and actually observed to be good for them in centos. Yes, work should be paid. True also that some things can bring more customers without being profitable by themselves. Yes people are allowed to be angry about a bait and switch.
One can be sad and angry that a service that one used to rely on is discontinued or made very expensive.
Besides, do all those people complaining have projects that are very sensitive to stability that CentOS stream is going to ruin? That is quite hard to believe.
Not sure if it was even a bait when people just chose to use it and CentOS never locked you up in using it.
Most of the whining sounds very childish.
But still Rocky Linux is possible only because Red Hat provides sources for its distribution. There’s no Rocky Windows.
Just to clarify, CentOS is not discontinued. Basically all that is discontinued is CentOS "point" releases. There is CentOS stream 9, it gets updates, there will be CentOS stream 10, it will get updates, there will be overlap between CentOS stream 9 and 10 being supported. Sure the CentOS stream won't be exactly equivalent to RHEL point releases, CentOS stream 9 won't be RHEL 9.1 - but RHEL 9.1 will be cut from CentOS stream 9 AFAIU.
"Which are in large parts RH/IBM open source projects, too"
?
https://mobile.twitter.com/passthejoe/status/144693144711837...
- Several bugs have been encountered in Stream, which were not present in RHEL.
- Some user communities had previously expressed concerns on the feasibility of using Stream in a production setting. This feeling has also recently been expressed internally by some IT groups.
- CERN is evaluating our original recommendation [0] of utilising CentOS Stream as the default Linux Operating System.
[0] https://linux.web.cern.ch/#update-on-centos-linux-strategy
[1] https://indico.cern.ch/category/13390/
CentOS was never RHEL, it was though very close, CentOS Stream is also very close. Whenever I used RHEL or CentOS before CentOS Stream I always updated to the latest versions of packages for the major release version, if I use CentOS stream I will still get updates to the latest versions of packages for the major release version. Maybe I get them a bit earlier than RHEL, but this to me is somewhat irrelevant. If I want reproducibility there are still plenty of ways to get it, including RHEL UBI.
It was close enough, Stream is almost as close and for some people it isn't close enough when they need bit for bit compatibility for the software they are going to run
You seem to believe a community supported distro was as good as a corporate supported distro and put such a high faith in it but now somehow it's all corrupt when RH supports it putting it as an upstream?
If your usage is that critical, why not buy RHEL (or Ubuntu extended support) for 10 years support and stop freaking out like your Linux days have come to an end?
Stream also notably has a much shorter lifecycle, with support for 8 Stream ending in 2024, compared to 2029 for rhel/rocky/alma.
To me the main value of using rhel-based distros is the great reliability and long support life. Stream compromises on both of those compared to what normal centos offered, and rocky and alma currently offer.
https://www.centos.org/centos-stream/
Red Hat made some strange decisions with the way they’re handling Stream vs RHEL, plus the communication was abysmal, and they changed models mid-cycle. Be mad at them for that all you want, the rest was overblown.
This company could have just used Stream 9 if he wanted a free OS. Since he’s virtualizing, he could have bought hypervisor licenses for $4k a piece that allow you to run as many VMs as your hypervisor can handle. He could have used OpenShift for virtualization and container orchestration, and just paid for the OpenShift licensing. If you use OpenShift, you can run all the RHEL VMs and UBIs you’d like at no extra cost. What you never do is pay per core. There isn’t a single licensing model from RH that forces you to do that. If you license individually, each license supports one bare metal machine, or two VMs. This guy is full of shit. He’s just mad everything isn’t $0.
That said, I'm running CentOS Stream 9 right now. It's supported until 2027. I generally like to keep my operating systems relatively up to date, so a roughly 5 year support window is adequate for me.
People freaking out because their free candy is no longer as tasty as one could hope, it looks quite funny they have zero willingness to pay for such a long support effort.
Rocky is "down stream" of RHEL, CentOS 8+ is "up stream" of RHEL (since 2020, and what Fedora has always been).
CIQ sell support contracts for Rocky linux, so rather than having a combination of RHEL (with support) and CentOS. They now just have Rocky with an optional support contract.
Linux is open source, they are aloud to do this, thats the point. They are not leaching. Red Hat, by ending CentOS as a "down stream" equivalent, have brought this on themselves.
There will be a lot of other users of RHEL and CentOS who are going through the same process right now as CentOS 7 support ends next year. We will see more companies making the same choice.
Let me quote the article: “I called him and… said ‘listen, I want to give you a shot and I'm willing to work with you’. So we worked with him for 13 months to build our real-time kernel to meet complex workloads with one condition: It has to be open source and it has to derive its characteristics from a large community,”
Rakuten worked with the founder of Rocky Linux for 13 months to develop real-time kernel. I don't know what kind of deal Rakuten promised to Rocky Linux (founder) and its community, but Rakuten contribute by open-sourced the work to community. That's enough for me. That's the spirit of open source.
The people who actually understand and care about freedom is a small niche.
It seems like the classical "Here's our new pricing and for being a long time customer we throw in a bottle of vaseline too". Which technically has nothing to do with open source but Red Hat ( like every company in this space ) created an "ecosystem" and uses the word "open source" VERY liberally when selling you stuff. Some of their opensource software need some paid service or "maintenance" contract to work or have any value.
I'm sure Red Hat/IBM is ( mostly ) covered on the legal side, but the sausage is made in a very different way.
RedHat has consistently been the most aggressive supporter of Open Source out of any for-profit companies. Not only do they contribute to open source projects, they consistently release their own code/projects as open source early on, which could easily be kept proprietary (if they chose) and used as a competitive advantage over others.
This is true though as RH will terminate your contract if you redistribute their binaries. So they use a contractual backdoor to undermine GPL. And before people talk about how GPL covers source but not binaries, it covers binaries. In fact, all of CentOS, Rocky, Alma, etc. is a massive waste of everyone's time, energy, land, and electricity. Something akin to crypto mining. Everyone could have used RHEL, but we need this complicated indirection for no reason.
I don't recall having problems with Dell server-related packages on Debian, even if their own crapware (OpenManage Essentials and then Enterprise, legitimately one of the worst pieces of software I've ever had the displeasure of using) for managing iDRACs and firmware came as a CentOS-based appliance.
It's still a frighteningly low number of distros considering we used to have an abundance of distros to hop around on for desktop use. Distrowatch.com for example, but for all those distros, when it comes to servers we narrow ourselves down to less than a handful.
Support you how and with what? What does your storage vendor have to do with your server's distros? Even the distro running on the server itself doesn't matter, really, because it's not like Dell/HP are writing distro-specific userspace drivers.
Operating systems have a hardware compatibility list (HCL). If your product is not on the HCL for the operating system vendor and you have a problem--at least one somehow plausibly related to the interaction between your product and the server--neither vendor is obligated to support you. (They may well try but there's no guarantee.)
I don't care if DistroWatch shows bunch of random newbie friendly distro in the upper ranking, they're not corporate supported.
Debian doesn't do clean 2 years LTS release cycle nor has 10 years extended support.
I have seen Gentoo used as a base to deploy binary packages veted by the org in the past. Gentoo provides tooling to do that and it worked fine.
Oracle also still sell a distribution.
ArchLinux, something I would never use professionally but enjoy on my machine. Packages are mostly vanilla and are updated quickly.
The last software company I worked for used CentOS.
Debian patches a lot for completely bonker reasons. They do it to support architectures nobody uses. They do it to remove part of packages for purely ideological motives. They do it to split packages into multiple parts making work harder for upstream when bugs are found. They do it to integrate with their own subpar tools. Sometimes they introduce major bugs along the way like the infamous OpenSSL one.
I understand people liking the fact that Debian is community maintained but the security track record is simply not there.
>On May 13th, 2008 the Debian project announced that Luciano Bello found an interesting vulnerability in the OpenSSL package they were distributing. The bug in question was caused by the removal of the following line of code from md_rand.c
> MD_Update(&m,buf,j); > [ .. ] > MD_Update(&m,buf,j); /* purify complains */ >These lines were removed because they caused the Valgrind and Purify tools to produce warnings about the use of uninitialized data in any code that was linked to OpenSSL. You can see one such report to the OpenSSL team here. Removing this code has the side effect of crippling the seeding process for the OpenSSL PRNG. Instead of mixing in random data for the initial seed, the only “random” value that was used was the current process ID. On the Linux platform, the default maximum process ID is 32,768, resulting in a very small number of seed values being used for all PRNG operations.
The most security focused uh? the distro where a maintainer modifies code he doesn't understand because valgrind complains about it? In one of the most important packages of a linux distribution? No. Anyone who has had memories of this event would treat anything stamped with debian's name as nuclear waste.
Debian has had many conflicts with upstream projects for their relentlessly idiotic patching. Firefox once prohibited Debian from using the FF trademarked name and debian in turn renamed firefox "Iceweasel" in their distribution to continue shipping their tainted version.
I'd trust any of these : RHEL/Fedora, openSUSE, Gentoo and Arch over debian any day.
To be fair, the maintenir had asked about the change on the project mailing list and was given the go ahead.
I too don’t like the general attitude of Debian towards patching but it’s an issue of the whole project. There is no need to single out someone.
It sounds like Rakuten is paying CIQ instead of Red Hat on basis of a price increase from Red Hat.
Sounds like they worked with the developer of Rocky Linux for 13 months to get the realtime kernel where it needed to be, and I would assume the author did not work for free and was compensated by Rakuten. "We decided to move to fully open source software and paid the author to customize it for us and released our customizations as open source" doesn't sounds very leachy to me.
I mean, companies should. I have a couple ;-)
But they are loud and spend most of their time powered down.
I mean, I guess he could have been working for them for free for thirteen months, but that seems rather unlikely.
I very much doubt Rocky is a sustainable business model. Hell even RHEL was not... This is the aftermath of the enterprise/FOSS (pay for support/whatever model that never really worked) war on prices and we are basically going back to the era for paying for software and some people and businesses dont like it.
Is your argument solely that Rocky Linux didn't extract the maximum amount of money they could from their client, or is there something else I'm missing here?
> I shouldn't be forced to pay subscription fees… I have no problem to pay support [fees]
AFTER CentOS came out they changed it from 2029 to Jan 1st 2020. Now CentOS stream is not a recompile, but has less tested changes BEFORE RHEL, and is now a rolling release with a 5 year EoL.
So for those that need compatibility with RHEL, that's broken.
I wonder why is that, if Rocky is repackaging RHEL source performance should be identical, no?
Article mentions realtime kernel. It seems they were working with RH on custom kernel (that's the support contract he mentioned), but started to be asked to pay for RHEL license to use it, which understandably pisses him off
Would you be mostly guessing for things like build flags and optimizations? Could something as simple as package build order affect performance?
1) I am not sure Rocky were actually re-packaging the RT stuff originally, they've got it now though.
2) RT is pretty finicky to tune correctly for your use case at the best times, and especially for use cases as exacting as the 5G RAN. I assume there is a consulting aspect to this that they are paying the CIQ folks for, but probably based on man hours or a flat fee rather than the per socket/node subscription they might have paid Red Hat.
Those commits are mostly from Red Hat developers, because all RHEL devel now happens in Stream first, but iiuc technically anyone could submit a PR, and maybe CentOS Stream community members do that from time to time.
Fedora is entirely separate packages from CentOS/EL. Fedora is much closer to upstream projects. Fedora has shorter lifecycle, no API/ABI guarantee across releases, moves fasts and breaks more things, etc.
The answer is simple, Debian. True freedom.
We are breaking apart the traditional monolith deployments in radio base stations, that are deployed at a very large distributed scale. For example we are deploying 30,000 edge nodes to do real time processing of radio baseband signalling, 23,000 of them in the the next 10 months. Now these workloads have more in common with high performance computing than they do with running standard business application systems, so we are aware we have to modify the kernel to make it cope with the real time requirements. We are removing the monolithic model because we know we can reduce the costs and increase the deployment optionality. And the existing enterprise OS licensing model was prohibitively expensive at this scale. And then our open source choices were changed. We are happy to contribute our real time learnings back to a community since that helps us help the community to help us. What we have done is not for everybody, and others need to make their own choices for their own reasons. But we believe it is having those choices is what creates a healthy industry. And we no longer had the choices we wanted, so we want to contribute to making them for others.