> Under Red Hat control, CentOS no longer modeled RHEL production code but was marketed as a beta—a preview of what was coming down the pipe. Eventually, the company sunsetted the project entirely, and as a result, new copycats emerged.
I thought CentOS was downstream of RHEL. Doesn't this describe what CentOS Stream became?
That very brief segment of the article pretty much collapsed a decade of time into two sentences, but yes it appears to be denoting the transition from CentOS [Linux] to CentOS Stream.
Right, but I assume the first sentence refers to CentOS and the second sentence of "sunsetting" referring to the end of CentOS and the start of CentOS Stream. After all, CentOS became a part of Red Hat back in 2014 and stopped in 2021.
In other words, how was downstream CentOS considered a beta or "a preview of what was coming down the pipe"?
I suspect that AmazonLinux (also originally downstream of RHEL) caused Red Hat to reconsider this. I’m not sure why exactly. Perhaps Red Hat/IBM compete with Amazon in some space, or perhaps they felt that Amazon should release their patches.
The CentOS Stream transition was a shift from the "throw it over the wall" model of open source like Android to a more collaborative approach.
Example: As a user of (original) CentOS, you discover a bug. Where do you report that bug? The Red Hat Enterprise Linux bug tracker, because CentOS is just a clone and the community has no real ability to fix things. Then what? You wait until a Red Hat employee gets around to looking at it, and then wait until the next patch release which might be up to 6 months away.
Now repeat with the new model.
As a user of either CentOS Stream or one of the RHEL clones such as Alma Linux, Rocky Linux, Oracle Linux, etc. you discover a bug. Where do you report that bug? The CentOS Stream bug tracker, upstream for RHEL and thus also upstream for all RHEL clones. CentOS Stream is a community project, so the community (including yourself, and including maintainers for Alma / Rocky / Oracle Linux) can actively contribute to and drive the fix onwards to completion and test it afterwards, wheras before it would sit in internal nightly builds for a few months.
Both the community and Red Hat benefit from aligning incentives that way. However, I understand why there was a lot of frustration about the way it was announced and implemented. It could have been handled better.
disclosure: I work for Red Hat, but not on anything related to RHEL or CentOS Stream.
People used CentOS because it was extremely stable. It was the safest dsitribution to pick for a long-term support version of something, e.g. if you are distributing hardware to customers and you know you will have to maintain support for that hardware for 5-10 years. Companies that want this don't really care about community bugfixing and all that. They're happy with security patches so that their products look good in security audits, and will apply their own patches for other problems if they hurt their own product.
CentOS Stream is the exact opposite of all of that, with no upside at all for this model. It will push numerous companies away from IBM RedHat entirely. Especially as it is widely seen as the start of IBM creating a bigger moat around RHEL, so there is little confidence that future clones of RHEL won't see a similar fate one way or another. And while finding a bug and not being sure who to report it to is bad, finding out that your Linux distro was buried by your maintainers half-way through the lifecycle and won't get any new updates would be disastrous.
It cannot change because of how copyright works. If you mean dropping CentOS Stream, that of course can happen but then all the Alma Linux folks have to do is go back to how CentOS was developed in 2010.
It's also not that Red Hat saw something wrong with CentOS Linux. To put it simply, between 2014 and 2019 Red Hat learnt that the company needed a free RHEL upstream more than they needed a free RHEL downstream. CentOS Stream is all about sharing participation to the CentOS community so that everybody does what they need and they don't have to ask mom Red Hat. Just read https://www.theregister.com/2021/07/09/centos_stream_greg_ku... ("Greg Kurtzer: Red Hat did the right thing and the new scenario is better than the old").
In fact, a lot of effort went towards making it possible to ship RHEL's upstream as a rolling release. Consider that ten years ago you had to ask specifically, months in advance, about updating a package in a RHEL minor release. In 2010, a quality engineer and I spent a few weeks working on grep just to make sure that the bugs were fixed in Fedora before RHEL 6 forked, because otherwise we'd be stuck with those bugs for years. These days I could just open a merge request on CentOS Stream and ask to update to a newer version of grep.
Disclaimer: I work for Red Hat, but not as a spokesperson
I may be off, but hey. As I understood it at the time, the reason wasn’t “OMG there are folks not paying us for the bits” but rather “we have this number of engineers, they can work either on CentOS Linux or on Stream” and Stream is more of a benefit to Red Hat as an upstream for RHEL.
What kind of Stockholm Syndrome is required for a user to believe that a transition from "bug-compatible with FIPS-certified stable releases" to "no-promises best-effort ABI stability" is desireable because now we get to do our own tech support?
I'll never in my life understand this. It's like Red Hat took the time they should have used to talk to their customers and spent it writing a Guide To Enjoying Eating Shit to lecture everyone about how great things are now.
I'd respect the company a lot more if they just said "we bought all this CentOS stuff and it turned out to be a money pit so we're throwing it out and making it a part of the beta testing process" instead of this relentless gaslighting that anyone on Earth wanted this.
> The first company to successfully commercialize open source software at scale
It depends on the definition of scale, or the company. Cygnus started in 1989 and was doing well, it merged with Red Hat around 1999.
> The company generates way more value than it captures and has a low rake out of necessity—none of the software it creates is proprietary.
There are expenses and revenues, and companies like Red Hat rake in revenues - which are more than expenses. So did MySQL and other companies.
> It’s under constant threat of being disrupted or undermined by other open source providers.
Back before cloud got big, I worked for companies that paid for HP/Dell service agreements and RHEL service agreements. If we were a smaller company we may have gone with CentOS or something, but we weren't.
The thing not mentioned is not only that people can and do use things like Debian instead, but that RHEL gained from Linux and Debian as well. So the benefits flow both ways.
> open core model...MongoDB...Elastic
This is another model which can be done. Three years ago, Red Hat sold for more than twice what MongoDB's current market cap is.
> The Red Hat model won’t be repeated.
It's worked for a number of companies in the past, some of whom were purchased by Red Hat. I don't see any reasons why it won't happen again, although it's not every day that a Red Hat is founded. No reasons are given why this will never happen again - all the reasons given were things Red Hat, MySQL, Cygnus etc. had to deal with.
> Support and services models don’t achieve hypergrowth
Red Hat sold for $34 billion. If that is not enough hypergrowth for some VC or founder, then they will go with another model.
> Support and services companies are uninvestable. Annual revenues are low and nonrecurring, and margins are tight. VCs are looking for 80%+ gross margins and hockey stick growth.
I don't know how revenues are nonrecurring - I worked for a number of companies that paid service contracts for RHEL and other Red Hat products alongside our HP/Dell service contracts.
Also - Intel, Netscape, Greylock and Benchmark did invest in Red Hat.
Also, companies like Red Hat and MySQL did have hockey stick-like growth.
For a company following the Red Hat model that needs to raise money from VCs, these thoughts might be applicable. But for companies following the Red Hat model that are not losing money, they don't really have to worry about what VCs think of this.
From my limited existence, one of my older employers paid Red Hat a lot of money for a fairly limited/short engagement with 1-2 of their software engineers and 1 of their solutions engineers, and then paid tens of thousands per year for I think 2 production support contracts for their software for years after.
At the same time, I could see how they could have gotten pinched recently if they were unwilling to pay more to retain talent and are now getting squeezed from the other side by companies moving away from their products to reduce costs.
I remember back when Microsoft used to be anti open source too. They would release super straw man type articles about how proprietary software was actually cheaper with lower ToS and other poppycock.
Weird to see a startup making such simplistic, wrong statements to promote their business rather than to protect an entrenched business.
I find it hilarious they are saying VCs wont invest in redhat style business because it's growth is not "hyper".
These same VCs who are so fundamentally incompetent at the basics of capitalism, like risk management 101 of interest rates, that they had to get bailed out by the government, by the FDIC randomly deciding to rewrite the entire rule book that the rest of us normies have to live by.
I don't see it that way. They were bribed with venture debt and mortgages to ignore the tail risk of uninsured deposits. Probably many VCs would have made the same decision in retrospect.
It is no secret that VC's are not interested in productive enterprises which produce a reliable and fair return on investment through revenue-generating activities.
As you point out, They are obviously interested in concentration of vast amounts of wealth, which requires a completely different set of tactics. Cornering markets and establishing monopolies, regulatory capture, pump and dumps and other forms of grift and large-scale antisocial behaviour.
It should be completely obvious to any onlooker that they have no interest in innovation or technology :)
They might but unless they also come on a supported OS with all the certifications checkboxes checked it won’t even get its foot in the door. It helps if that other software is built on as much of the already supported and maintained set of base libraries.
You can have CIS, FIPS, and a whole host of other compliance without Red Hat, arguably just as easily.
The reality is containers and container-first OSs are killing RHEL, you no longer need a breadth of OS packages or a fully capable OS. Just docker or podman. The only place RHEL has a stranglehold is with COTS apps, but those are all moving to containers too. I can't think of any widely used container that uses RHEL/CentOS or Fedora for the base image, that was a huge miss on RH's part.
Eh, OpenShift is too opinionated to be valuable for the companies that can afford it, and way too expensive for those that might benefit from it vs a cloud based PaaS. OpenShift's biggest problem has always been that it continues to diverge just enough from vanilla k8s to require "OpenShift specific" shit for everything.
I don't see OpenShift getting the same COTS lift as RHEL, so in my mind it's still going to be a net loss for RH long term as that revenue leaves RHEL and doesn't go to OCP.
This isn’t true at all, openshift is a conformant distro of k8s. You are not required to use any
OpenShift specific shit, but what it adds is very nice.
So containers run on air I take it? No, RHEL isn't being killed, but you're not going to see it as base image for your typical GitHub project. In the enterprise though, where security is tight, no one is going to build containers based on Alpine which is maintained by what, 10 people? Their customers are banks, governments, etc. which will happily pay for what they provide.
Alpine as a base image is perfectly fine; it is a tiny base system after all.
RH base image is also relatively small (compared to full RHEL system). It is also completely useless, as you cannot dnf install anything while building the image, if you do not have access to RHEL repos. If you have to bring in your own packages, you might use Alpine just as well.
I woould also contest your claim about "happily pay". No, the payment goes after the sales folks do their job, which isn't easy either. There's nothing "happily" about it.
Also that statement is kinda silly "everyone would switch", very broad and sweeping.
That's way easier said than done to convince our clients that "oh yeah, we're using not-RHEL now". We'd have to guarantee some self support to them in that case, in order to alleviate their fears.
Red Hat early on invested in understanding the needs of the different industry verticals they were trying to sell into and made sure their products could meet things such as compliance and audit requirements of differing sectors. They developed guides on how to deploy their products in a manner which met those requirements and their professional services engineers understood not just Red Hat products but the specific needs of differing sectors. This is how I saw them winning enterprise customers over larger established players in the early 2000s. Even today most opensource businesses don't do this to the extent Red Hat was in the early 2000s.
Moreover, I remember Red Hat sales reps back in the 2000s to be incredibly technical and knowledgable compared to every other vendor's sales reps. Every time I had a specific question regarding technical implementation or even compliance laws in my region, the sales rep knew the answer or forwarded me immediately to someone who did. They were very organized.
This is correct. They still do this. They are the only ones. They go beyond just "hey, I know of a database that could handle that size" to "why are you storing this here? this is PII, this is analytical data, this is business data, here's a proposal for doing this with open source at scale". It's not just "buy our product". They go above and beyond the needs of the product to the needs of the customer, like you said.
Other “open source” companies are always focused on modern startup stuff. Too much fake it till you make it.
They have a unique value proposition in that they are an OS, so they are everywhere by default. It makes sense to have a .gov, banking, healthcare, etc teams who grok whatever the issues are in those verticals.
Red Hat worked because it was a time (the height of the dot-com bubble) when open source operating systems were something of a Wild West - rapidly becoming an industry standard, but not well understood by existing businesses. Red Hat provided guidance for that transition.
Confluent was arguably also able to do this for a similar reason - at the time they launched, the Kafka model was becoming vital, but was also novel and difficult to manage in production, and its enterprise services and support were critical for adoption. (They do have a managed cloud now, but their enterprise services were 73% of their revenue in 2021 and 61% in 2022: https://investors.confluent.io/node/8466/html ).
But it's hard to run what is essentially a professional services firm and popularize an open-source product at the same time... unless your open source product is already rapidly becoming paradigm-defining, and you're the people with the unique know-how to position yourself as the saviors for that space.
Confluent also moved early on the Confluent Community Licence for Kafka Connect connectors they developed to prevent other service providers being able to provide the service that Confluent did.
Side note on Confluent’s products. Thanks for that link - the “Disaggregation of Revenue” table where the 73% and 61% numbers came from shows that Confluent Cloud revenue increased from $94M in 2021 to $211M in 2022, which was more than Confluent Platform’s revenue growth. If those growth rates continue then Confluent Cloud will very soon be the majority of their revenue.
I always take those numbers with a big pinch of salt. There is a wave, in the management world, that wants every business to be "Cloud” (i.e. SaaS), because of the (largely correct) assumption that it enables better rent-seeking and long-term lock-in. So every vendor is officially busy "pivoting to a subscription model" and wants to show growth in that area to investors, so they file everything they can under "cloud" divisions.
1. Customer support
2. Certifications
3. Vendor support
I don't know another distribution of Linux that provides all these things to the extent Red Hat does. You don't use Red Hat because you want to, you use it because it's the only choice.
Any business could do exactly the same thing, and do it better than Red Hat. But there's really not much reason for a customer to move away from Red Hat if they're already on it, and it would take you a hell of a long time and a dumptruck full of capital to match them on the above.
Open Core is not a business model, just like Open Source isn't a business model. A business model is promising to fix a customer's problem, certify their software, and get vendors to support them. That alone will get you a yearly renewing contract for a couple million that you then upsell over time.
When you run enterprise product X on Red Hat version Y, that vendor will do less fingerpointing. They may even work with Red Hat in the backend to deal with bugs, etc.
If you go put that on Debian or whatever, they’ll make you reproduce on RHEL.
This becomes a big deal when you have RHEL, VMWare, and other vendors in the mix. If you ignore the certification, the app vendor or VMWare will just shrug at you.
>A business model is promising to fix a customer's problem, certify their software, and get vendors to support them.
This can also be described as assuming liability.
Fact of the matter is people don't want to assume liability themselves. If they can pay to get someone else to assume that liability (which Red Hat is happy to do) they will.
The rest of the FOSS ecosystem on the other hand is all about assuming liability yourself. No warranties expressed nor implied, no support, no nothing. For a business that is simply not palatable.
This is also why businesses happily pay tons of monies to Microsoft, Adobe, IBM, etc. for licenses and support. They assume liability, liability the businesses don't want to assume.
"Assume" is a different word, it means implicitly accept. There is no such thing as an implicit clause in any contract.
Definitely willing to be shown otherwise, but I've never seen any vendor accept liability for there product beyond "you can have your money back, we'll take our software back".
4. We have all the in-house support staff we need and only want to use the self-support license, but since we're a service provider we were forced into the CCSP program where each license costs a lot more.
All service providers have to be in the CCSP program, nothing wrong with that. And we do use it for clients. But it kinda sucks that even our in-house stuff has to cost the same. Discourages use of RHEL in-house.
If you're the DoD and you need to run above TS-SCI stuff and somehow Windows isn't a good fit and you don't have any spare fresh-faced kids stealing your docs to impress their virtual friends or leak to newspapers, then you run RHEL on Dell servers flown over UT and Dallas stadiums with all of the S&S fees.
I don't know another distribution of Linux that provides all these things to the extent Red Hat
Oracle would technically be another one. They will even sell you Oracle Hardware (or Oracle cloud) to run your Oracle Linux. Of course there are many good reasons not to want to use Oracle for this.
SUSE are probably the closest company to following Red Hat's business model (with a more European/German focus). The other ones are the cloud vendors who are getting around to realising now that without certification there are many government orgs, large companies, the military etc who simply cannot use their cloud services.
Why would they? Managed services were never part of the Red Hat business model. They've added some cloud stuff as of late, but in general you don't go to Red Hat to buy or lease hardware. And I don't think Red Hat ever had any serious interest in selling you any either.
It's a shame your number one reason is customer support, when at least in my experience it has been woefully lacking.
Despite having a reasonably large contract with them at work, we had multiple support tickets go without response for months, with at least one that I am aware of being left open for almost 2 years before being closed as "won't fix"
Noting that its only my personal experience I am sure plenty of others have had the opposite experience, however I have found canonical support to be vastly superior as as such I would never chose RH if support was the primary thing I was after.
That sounds like an account management issue. You need to know how to manage and work with vendors more often than not it’s not the size of the account that matters but how you engage with them.
to be clear when I say no response, I am talking they got consistent updates that essentially said "we have no update on this issue", not them simply never putting anything on them.
also like it or not, by paying for support people have a right to expect that the company being paid will be engaging with you and the request you are putting into there system, not that you will have to find a person to chase down because your tickets are being ignored.
I think it would be a great benefit to our country if governments started phasing out the purchase of any proprietary software.
If you want to make software for the government, it must be public domain.
In the long run we'd all have lower taxes, more efficient government, and better software for all to use.
The downside would be some people in the 1% would no longer earn billions in fees from the rest of us for "licensing" software laden with technical debt to the government.
Proprietary often has a lower total cost of ownership than the comparable loss of productivity and loss of support compared to open source software.
Do you have a good example of a company that provides support comparable to Microsoft's www.office.com (word, excel, powerpoint, outlook, teams, and one note). Note that this is a web instance which means that the cost of installing and managing software updates on various computers is reduced.
The lower taxes is not guaranteed at all when looking at the total cost of ownership.
With open source software, this often means increasing the staffing (because there's no corresponding org that will do the support with the necessary SLA) which also means that there isn't necessarily more efficient government.
The "better software for all to use" - switching to LibreOffice (or one of its forks) doesn't mean that the open source developers are going to prioritize the issues from Smith County, KS when there's an issue that impacts them.
To go down this path of "lets have government use open source" it is first necessary to get the open source developers to want to make their product competitive for government use compared to the proprietary ones that are out there and for companies to step up and provide similar SLAs that Microsoft and others offer.
That is something that can happen now, but isn't because there isn't enough interest from the FOSS community to do that for that sector. Trying to push the public sector to do it would be painful, expensive, and inefficient and not result in any better software.
> Proprietary often has a lower total cost of ownership
I think you are right in the short run, but I think in the long run TCO is far higher with proprietary software. Because you are sinking money into a lost cause and delaying the cost of a future expensive migration from the closed source to the open source.
People and organizations will always want to mold their software to their domain, and for a while and for the right price, that works. But proprietary companies sweep technical debt under the rug, talent leaves, they start to slow down, and eventually the software slows down the organization considerably. It becomes harder for the organization to hire and train people on the proprietary stack. I recently spent a week working alongside federal employees and almost without exception everyone hated their tech stack, attributed a double digit percentage of their productivity lost to it, and would often build things using open source on their own machines to get work done.
Eventually the org switches to a more open source stack, but the process to extract themselves is costly. I think that cost might be greater than the cost of just starting with open source in the first place.
It seems almost like a natural law that all proprietary software is eventually replaced by open source alternatives.
> Do you have a good example of a company that provides support comparable to Microsoft's www.office.com
Not as a package, no. For each individual item, there are alternatives, but I'll agree there is a missing offering for a fully self hosted alternative to Google Workspaces or Microsoft Office 365. If I'm not mistaken, Collabra Online is trying to be that, but the execution seems to leave something to be desired.
> The "better software for all to use" - switching to LibreOffice (or one of its forks) doesn't mean that the open source developers are going to prioritize the issues from Smith County, KS when there's an issue that impacts them.
I wonder if ChatGPT could influence things here. I wonder if support agents from ChatGPT trained on loads of open source content and code will turn out to be more effective than human support agents for proprietary stacks, and you might be able to build an open Office365 or Google Workspace competitor.
Anyway, just doing some back of the envelope math, I estimate governments in the USA spend $20B - $50B a year on proprietary software[1]. It certainly gives a lot to open source too, but I believe if more of that money was diverted to open source we'd all be better off in the long run. Not a slam dunk case, I know, I'm at the early stages of this line of thought. And of course, even if you could prove that open source would have a much lower TCO, it would be a protracted battle to go up against the folks with a lot of money on the line to deny that truth.
[1] According to this site (https://www.itdashboard.gov/itportfoliodashboard) it looks like total unclassified Federal Government IT spending is ~$84B a year. It looks like Palantir alone gets $1B a year from the government. The latest information I can find on Oracle is that 25% of their revenue comes from US Gov (so say ~$12B per year). Microsoft is perhaps $5 - $25B? Not sure.
Money isn't diverted to open source - its spent on staffing (both in house and consultancies) needed to support the open source software.
Hiring Deloitte or Accenture or another consultancy to provide the support for an open source piece of software (be it directly or "we need to have people with these skills") is still extracting money from the org and I can assure you that the consultancy isn't contributing back to FOSS.
As it is, Microsoft is a known quantity with a support SLA that meets the necessary requirements both for hosting of data and for response time. The cost of Microsoft licenses and support is less than the cost of hiring additional staff or bringing in a consultancy.
A budget item with a fixed number in it is much preferable for budgetary allocation than open ended numbers typically found from consultancies.
The support of open source software is often more costly for the public sector than going with a big tech solution - even when including the cost of licensing.
Until the cost of support for open source solutions comes down to where the proprietary options are, it is often not something that is considered.
RedHat is an example of a company that is doing it right - offering the support that the public sector desires with a consistent budget line item.
If you can stand up a PostgreSQL instance that is able to compete with the Oracle performance and support (e.g. backup restore failed - call support and have someone on the phone helping now) for less than the cost of the Oracle licensing you will have people beating down your door to sign up.
Same thing with Microsoft.
For the federal government to do this on their own is much more costly. It goes back to the consultancies to implement, migrate, and maintain. Trying to staff up to be competitive to be able to do it in house is also quite costly (note the pay difference between public sector and private and then multiply that difference as an ongoing expense by the number of employees needed).
And so this returns back to the FOSS community to address the needs of the public sector and create the needed companies that are competitive to be able to provide that ongoing support.
ChatGPT will not write an office.com competitor this decade... or next.
I think your assessment of the current situation is probably accurate.
My point is it's suboptimal, and the U.S. Federal Government could change things, as the biggest buyer of tech in the world. If they said "everything has to be public domain", it would be a seismic event. I think we would all be better off if they did this (probably with the exception of <1% who would lose monopoly profits).
Nit: I think you misread my ChatGPT comment. I said ChatGPT could be a competitor to office.com's _support_ agents, not that ChatGPT could build office.com.
Expected to see canonical model get mentioned in the article, combination of proprietary online services with opensource products.
Well we know all of debate around it and the peculiarities of canonical, but I feel like it worths mentioning.
Is Canonical making money now? Last I heard (though that was a good few years ago) they were still net negative after many years of making Ubuntu successful.
If you are looking for "Investable Hyppergrowth" Open Core is a great choice. If you want to maximize the value project proviedes to the universe sticking to Open Source is better choice
modern redhat --and this cannot be stressed enough-- is nothing like old redhat despite IBM's fervent insistance after the acquisition they would remain hands off.
The hat killed Centos because some IBM midlevel manager needed to make quota for a bonus, and they didnt stop to realize just how unsuccessful this tactic was for Oracle/MySQL.
moderen redhat is carved into about 40 seperate repositories and channels for packages designed to fleece you right down to the last RPM. need Ansible? sure it comes with redhat 9. need posix acl support from galaxy? that's another repo you didnt pay for. same goes for about a hundred other packages.
redhat might have had great tech support in the past, but put a ticket in now and you're in IBM levels of "prove it" hell with SOSReport zips and zoom screenshares until your entry level support guy gives up the ghost and calls a developer, who tells you the bug cant/wont be fixed because libraries are hard or a rebase is impossible or any other number of boilerplate excuses why this ancient 4.x kernel based distro cant just source its solutions from fedora like they said they do
and the flagships? forget it. Yeah Quay exists but try to implement it and youll find cool stuff like an administrator account defined in configuration but not created. once created in the admin panel youll have to go back and disable new user creation so you can control user creation at all, and the whole thing just feels rickety and poorly designed from the get-go. none of this is mentioned in the docs overtly, but is buried in Redhats bloated docs pages. just buy a quay hosted whatever from the hat instead.
Openshift is equally appalling as its a massive ecosystem Redhats done virtually everything to control and nothing to distribute or competently document. its painfully obvious IBM wants you to hire a consultant to build and deploy it...There is an open-source offering, but it cant be easily rolled into an airgap installation and it suffers the same level of byzantine and poorly written documentation as Quay. K3S/RKE2 easily beat it for nearly every use case.
Then theres the docker fiasco which unsurprisingly neither of these juggernauts wanted to deal with the others bullshit so now dockers broken in all but the most monied instances of IBM hardware. redhat became so absolutely resolute about podman being "everything you can do i can do too" that it has a fucking compose option to translate docker compose into podman because people like compose but rewriting your appstack to pods when you dont want to be arsed with k8s is sort of a non-starter.
> The hat killed Centos because some IBM midlevel manager needed to make quota for a bonus, and they didnt stop to realize just how unsuccessful this tactic was for Oracle/MySQL.
RHEL clones were always going to exist, this is and was obvious, and there are more of them now than ever. That's not the point. The point is that now there is a way for RHEL clones, including ones from competitors like Oracle, have some way of contributing back to RHEL, whereas beforehand if you were a CentOS user your only possible course of action was to file a bug in the RHEL bug tracker and wait for a Red Hat employee to prioritize it.
The business benefit is also a technical benefit - the ability to fix bugs and add features is no longer strictly limited by the amount of engineering capacity Red Hat can contribute. Companies like Facebook use CentOS Stream at scale internally and make contributions that ultimately end up in RHEL, users and vendors have a target they can integrate against early without needing gated betas, etc.
I work at Red Hat for the last eight years, long before IBM had anything to do with us, and I cannot stress enough how much IBM is *nowhere* to be seen in not just our day to day engineering jobs, but pretty much not even periodically. I know nobody at IBM, I have no email addresses of anyone at IBM, nothing we get in our inbox (and we get a lot) ever has anything to do with "IBM" in it, I have no links or access to any IBM-specific intranets or resources of any kind. They truly have stuck to their promise of not changing Red Hat; our support system is the same, we still respond to bugs, ask for SOS reports, and customer support people do collaborative screenshares, just as we were always doing, there's nothing different about any of that. There have been no top-down changes or decrees or anything I'm aware of that would change how we do any of that stuff.
as far as Podman it's all I use for all of my server environments and I like it a lot better than Docker. It's simpler without all the weird "Dockerish" things getting in the way. Not having a monolithic server hang the whole machine due to a misbehaving container is well worth it.
Well, there's two very different views of reality.. And one of them appears to be from a very unhappy customer.
Is it possible that different IBM Depts - or IBM regions, are selling Redhat Software/Support without much actual involvement of the Redhat business? Like the might have done in their traditional reseller/consulting/support role?
TBH I actually think it's likely - sales and account execs can probably just tickbox add RH stuff to their package deals, copy&paste the slide deck - and post sale the customer support channel will be the general enterprise support one.
IBM are probably very deliberately channelling all first level support through there anyway, same as for everything else - so core Redhat support staff might only get looped in for 2nd and 3rd level issues for those accounts, if at all.
It's quite easy to see that effectively creating two sales/support channels depending on whether you "bought from Redhat" or "packaged from IBM" - but equally it's unlikely to be presented or perceived that way, until it's too late.
I believe you when you say they've been hands off - it's surprising, but I can also see how it would make this kind of situation _worse_, because they're still effectively treating redhat as an "external" provider they resell.
Not that fully integrating the business would make it better of course, probably just different worse ( just based on my experience of corporate mergers, not IBM specific... ).
> as far as Podman it's all I use for all of my server environments and I like it a lot better than Docker. It's simpler without all the weird "Dockerish" things getting in the way. Not having a monolithic server hang the whole machine due to a misbehaving container is well worth it.
Podman-compose was feature-lacking and beta even when podman already had its 3rd major release, while the communitcation was that "Podman is a drop-in replacement for Docker". It just wasn't a drop-in at all for all compose users. People either stopped considering Podman or enablded the socket daemon, so that they are still able to use docker-compose. But this goes against the design ideas podman had.
Feels as half-baked as Ingnition, which also still feels very bare-bone, as if there's no financial interest to enhance it further.
The model is quite simple and it will work for others aswell, as long as onyone provide the software or core for free and for customized solution they will come to you.
I was at Red Hat very early on. Red Hat has been remarkably consistent throughout its journey once it graduated from selling boxes of software at book stores (which was a necessary step). Red Hat only astook venture money once it was ready to go public and never before. The philosophy of Red Hat was to "make the pie bigger" and still largely is, therefore it supports open source first before anything else. This works because it provides product-market fit at scale and by nature. The thought that "all" Red Hat sells is software support, certifications, and subscriptions to updates is so short sighted and misses the point.
Red Hat created an industry and all of the successful enterprise Linux distros have followed its model to some extent or another (Canonical, SUSE, etc). You don't need "open core" (which is a misnomer if there ever was one) if you continue down a path that creates and maintains communities of software authors out there in the world that will constantly feed value back into the ecosystem.
A virtuous cycle of community maintained software alongside a thriving and growing business is still achievable, no one seems to have the guts to do it.
I find Redhat jumped the shark when it started selling its distribution with different licensing options that let you use different number of CPU cores.
Per-core licensing is more or less limited to JBoss stuff and the likes. Stuff like RHEL, OpenShift Enterprise, etc. is typically licensed by socket-pair.
Partly it is, but there wasn't much hope to convert those to RHEL. The plan was:
1) to offload the production of the releases (effectively what is now Alma Linux). Because there's so much use of CentOS, it was pretty much a given that somebody would pick up the slack. Most of those companies weren't running stock CentOS anymore, they had their own downstream repositories
2) to give more insight into the making of RHEL minor release to CentOS SIGs and to companies that run CentOS derivatives
Facebook for example does not use the RHEL kernel and was one of the early adopters of CentOS Stream even before CentOS Linux 8 was terminated.
> Red Hat created an industry and all of the successful enterprise Linux distros have followed its model to some extent or another (Canonical, SUSE, etc)
IBM created that industry, by lifting Linux into their supported platforms portfolio.
and the first Enterprise Linux ever was suse linux enterprise for s/390, followed by x86 releases at about the time RH released their first RHEL.
They deserve credit but while IBM did spend lots of money on Linux around 2000, and that helped legitimize Linux. I feel that RH has always been the leader in this industry, and RH Linux was well entrenched in the enterprise long before IBMs big push. So IBM helped but they did not create it, in my view.
I would disagree with RHEL being entrenched by the time of IBMs big push. At the time, you were still more likely to see commercial Unix than Linux at large companies.
Red Hat becoming entrenched wasn't a guarantee in my mind until IBM made that big push. If you could point to a single event that made it likely Linux would take over, without a shadow of a doubt it would be the blessing it received from IBM. There were an enormous number of bigcorps that made initial investment in Linux after that.
IBM did not create that industry, though they certainly helped it along.
Red Hat didn't create the industry single-handed, either. They deserve a lot of credit but Linux succeeded because it was a group effort where there was a chance for a lot of players to make money (and opportunity to displace others).
IBM did a 1bn USD investment creating the market for Linux based enterprise infrastructure, "vendor independent" on all their server products (PPC, s390 itanic and once available x68 based servers), with all IBM software products (db/2, websphere, ...) available "certified" on sles, RHEL and some Asian localized Linux variants.
this created an immense Push on HP, dell, Dec, sgi, ... to follow suit
and oracle, sap, Software AG, anything serious.
this push and commitment b IBM was.so.relevant because it established Linux as a serious platform with enterprise level SLAs and mutual commitments for the full stack.
the 4d chess was to establish a solid consulting business for "consolidation" on the "single platform"
and it paid off immensely for IBM --- and for most others.
given the IBM market position around 2000 this was pivotal and prophetic.
To this day every IBM customer I've ever contracted for has also been a Red Hat customer. That even applies for mainframe shops. Sales reps and support and account managers at IBM and Red Hat had eachother on speed dial a decade ago since it was so common to work a ticket that involved both companies products, or sell a bundle that would include both companies software and/or hardware, etc.
My only question is what the hell took IBM so long in making the decision to buy them out. I've never been less surprised by a big acquisition than I was with IBM and Red Hat. The only surprising thing was that nobody outside of that space saw it coming.
>Getting contracts for development work with large customers can be difficult since most organizations already have preferred vendors for software development. Developing features under a support model gives you no entitlement to the technology. And, if you’re good at what you do, you’ll eventually put yourself out of a job since customers can use it without help
But there are a large number of outsourced software development companies, some of them publicly traded- Cognizant/Infosys/Wipro types, plus Accenture, plus the new generation of Eastern European ones like EPAM and Grid Dynamics. Government-y ones like CGI. Oldschool companies offering consultancies like EY/Deloitte, etc. Hovering in the background is Toptal, which is technically different but I would imagine the bulk of their revenue comes from enterprise outsourced dev contracts. All of these companies are making this business model work somehow? And there are lots of them too. So it can't be that bad of a way to make money (yes I understand that these are not VC-investable businesses)
The biggest reason why we don’t see support and services companies achieving hyper-growth is that they are uninvestable. Annual revenues are low and nonrecurring, and margins are tight. VCs are looking for 80%+ gross margins and hockey stick growth.
In other words, Red Hat's model doesn't work for Wall St and VCs looking to make quick easy money riding out the next tech 'unicorn'. Red Hat has survived and thrived for 30 years now, while most modern VC backed tech companies are lucky to make it past 3. The infiltration of Silicon Valley VC vultures was one of the worst things to happen to the tech industry and why we rarely see sustainable startups like Red Hat anymore.
I don't think this can be overstated enough. The incentives for vc-backed companies these days don't align with this model, hence the reason it's not more widely in use.
And a very good argument for taxing that money out of their hands and investing it into productive enterprises that spur actual innovation and gainful employment.
It's almost like building a company slowly, within your means, and focusing on improving the product, is the wrong way to do it these days. Better go with some BS that's flashy, make big claims and promises, and try to get some of that sweet VC money!
You know what works just like the Red Hat model, but yields way better returns? The AWS model. Just take all these tech-tested market-tested open source projects, slap on a new name and bill them by the hour.
I'd be curious to know the revenue breakdown of all their various products as a percentage of the total. I wonder if all those products you reference bring in much $$ compared to the bread and butter ones that they started off with - EC2, S3, etc
One might argue that Oracle also made this model work.
Not sure about their revenue between different divisions, but they finished open-sourcing even the last proprietary part of their JDK, making OpenJDK the reference implementation, plus they finance 90+% of all its commits. They make some revenue from selling OracleJDK licenses (and even most of those are free if you stay on the latest version) which includes support. Many of their customers are on older Java versions though, for example their Java 8 support is valid for something like 2030, so mostly government, big enterprises.
(For some reason though, there is just endless bullshit around Java, even though the exact same model/license is true for the Linux kernel)
The funny thing about Oracle haters is how they ignore that Oracle and IBM stand by Java since the early days, they collaborated with Sun on the Network Computer project, were one of the first SQL vendors to include the JRE on the database engine, provide a GUI based on Swing, JSF framework, JVM implementation, IDE, and so on.
They also ignore the fact that Sun also charged money for support, JDK was only available as free beer with source on request, the whole Apache drama with the TCK, and access to legacy versions required a Sun developer account with license.
Or that they were the ones introducing the separation of having UNIX for users, and an additional license for an UNIX SDK, which is what made people rush to improve GCC, until then mostly ignored project.
I think that most Oracle haters hate them for their obscenely expensive and often idiosyncratic database product, as well as their legal division being larger than their engineering division.
And speaking of databases, the idea of running custom Java code inside the database seems like an abuse of the database. How many people actually use this feature, and how many of them like it (as opposed to having made a bad technical decision 20 years ago)?
Plenty, this is no different from running C and Perl, which Oracle did before, what PostgreSQL allows for (which Rust stable support being cheered over here last week), or running .NET on SQL Server.
When performance matters, stored procedures are the way to go, not wasting network traffic and CPU cycles on the client, As it so happens, these additional runtimes are great and safer way to extend PL/SQL, pg/SQL, T-SQL,... than writing C extensions.
> When performance matters, stored procedures are the way to go, not wasting network traffic and CPU cycles on the client, As it so happens, these additional runtimes are great and safer way to extend PL/SQL, pg/SQL, T-SQL,... than writing C extensions.
While I agree in principle, in practice to me it feels like the tooling just isn't there.
Testing and debugging is a bit more difficult than with other languages (e.g. even using breakpoints and stepping through code), things like logging typically don't have pleasant implementations (e.g. log shipping), the discoverability and even version control of the code also tend to be worse, among other things. That's even before you get into building around the particular runtime that you're provided with, trying to get a grip on dependency/package management, automated CI deploys, rollbacks, monitoring/health checks, local development environments and so on.
My experiences might be the opposite of some folks, but I recall working on a system where most of the logic was implemented in the database packages and something like Java was used just as a glorified templating solution to serve a webapp. The performance was great, but actually working with the codebase was an utter nightmare, so it's not worth it in my opinion. That's like choosing to write a webapp in Assembly just because it's faster.
Do you debug stored procedures?
47% Never
44% Rarely
9% Frequently
Do you have tests in your database?
70% No
15% I don't know
14% Yes
Do you keep your database scripts in a version control system?
54% Yes
37% No
9% I don't know
Do you write comments for the database objects?
49% No
27% Yes, for many types of objects
24% Yes, only for tables
If half the people don't debug their code, three quarters don't test their code, almost half don't use version control and about half don't bother writing comments of any sort, that's the kind of code that I don't want to be working with and would advise others against going for that approach. While we can talk about the fact that these things can be done, the fact that they're not, is evidence enough that the community just isn't there yet.
Use databases for what they're good at (including some in-database processing, like reporting), but don't try to do everything in them.
Let's not blame databases for lack of skills or interest in how to use them properly.
Oracle, Microsoft and IBM provide the same kind of IDEs, graphical debugging and source control as any other programming language.
It is this lack of skills that comes up with fads like NoSQL.
By the way, there are similar results related to debugging in other languages, where most can't do better than printf debugging, don't know unit tests, profilers or static analysers.
The outcome of bootcamps or CS degrees without sound engineering practices, while people label themselves "engineer".
> Let's not blame databases for lack of skills or interest in how to use them properly.
We can (and should) explore the causes for it, sure, but that doesn't change the reality that if you join a project that uses a certain approach, it isn't guaranteed to be using the best possible practices, but rather whatever is popular and easy to do in the industry, unless you're very selective about where you work.
> By the way, there are similar results related to debugging in other languages, where most can't do better than printf debugging, don't know unit tests, profilers or static analysers.
I'd say that this is true to some degree and is also a reflection of either poor tooling, or lack of interest. For example, command line debuggers with arcane keybinds will be harder for the average developer to learn and use effectively, than just clicking on the line they want to stop at in a JetBrains (or similar) IDE and just clicking a custom run button that will launch their entire project in debug mode. The same goes for being able to run either your entire test suite or a particular test by just clicking a button in the source file, helpfully shown by a good IDE.
Things get worse when you want to test the integration with an actual data source (like an external API or a database), because in some cases you'll have to mock so much of it that you won't be testing anything remotely close to the real thing, or will have to deal with bootstrapping an instance of the API (if you can even self-host the full thing) or a real database, which will be really good from how truthful your tests are for real world use cases, but will need certain configuration and resources for setting it all up. Sometimes you can get away with something close enough, like using an in-memory database behind an ORM, but some of those abstractions end up leaky. Even worse if you want to test your integration against cloud services.
Static analysis tools are not without their issues either: something like SonarQube is good theoretically, but will have you struggling against setting up the actual scanner (including mundane stuff like source file encoding) on your CI server, setting up separate configurations if you ever want to run it against your local codebase and will be hard to configure in regards to what should or shouldn't actually trip up the analysis and throw warnings at you, because not all of the recommendations will be even viable for your framework and how it expects code to be written.
> It is this lack of skills that comes up with fads like NoSQL. ... The outcome of bootcamps or CS degrees without sound engineering practices, while people label themselves "engineer".
Does it mean that we shouldn't do these things? No, but it definitely means that we shouldn't just wave our hands around and suggest that it's just an issue of education, when actually trying to use the current technologies is often like banging your head against a wall. Use what works well and causes the least headaches, be open to eventually trying new things as the ecosystem and tooling improve, but don't stray too far from what others do successfully for now either.
From what I've seen, the things that have absolutely improved are schema versioning solutions (and thus, versioned DB migrations) and the ability to run database instances locally for development (in throwaway containers), so that you can test breaking migrations with believable seeded data before it ever needs to run on a shared environment. Codegen still could be better (e.g. generating Java/.NET/... entity code for an ORM in a schema-first approach), but some forwards/backwards engineering has been around for a decent amount of time at least, when dealing with models (for example, in MySQL Workbench, though pgAdmin is still lagging behind there). There are even tools for easier development of APIs, like Hasura, PostGraphile, PostgREST and so on, though the adoption there varies.
RedHat is the employer of many Linux developers, including maintainers for GNU software etc., with SuSE and Canonical distant followers (at least it used to be like that).
So I guess that makes them both a very significant contributor as well as deserved receiver of financial resources towards F/OSS. OTOH, apart from adaptions to ever-changing hardware, you could say that the times where Linux innovations are actually serving end users is long, long behind us, and anything RH pushed, such as systemd, gnome, and in particular podman (to take away from Docker ie what RH sees as their market, and theirs alone) etc. is basically zero sum software created out of a self-serving interest of selling support contracts by definition. That Linux isn't pro consumer is also evident by it being used for clouds and for a privacy-invading mobile O/S.
Linux was nice and gave a direction to a developer community as long as it was chasing commercial Unix/POSIX with strong precedents and blueprints what to implement. Now it's just an IBM division.
People that weren't there in the begining don't get it that Linux only really took off around the UNIX offerings of the early 2000's, when Oracle, Compaq, Intel, IBM money started be pumped into Linux in some form.
The thing with UNIX/POSIX clones, is that there is hardly any inovation beyond adding new drivers and kernel features, as POSIX doesn't have anything else to offer.
1. For me until proven otherwise, RedHat have an OpenCore model, the CentOS shift is too recent to be meaningful. My company literally panicked about their CentOS servers because they didn't want a rolling distro and so decided to switch to something else free for low risk VM (worse I'm afraid) and pay RHEL support for critical applications.
2.There are a shitload of successful support company for real OpenSource product, the author just choose to ignore them. Maybe because they are mostly smaller and "uninvestable". Just look a the support advertised options for PostgreSQL in North-America only... https://www.postgresql.org/support/professional_support/nort...
They switched to Oracle Linux... and for the love of humankind don't ask me why...
(At least they had the sense to switch to REHL for critical servers)
Now on the app side we are left with some vendors that don't support installs on OracleLinux (because who would switch to that... again don't ask me why).
221 comments
[ 2.0 ms ] story [ 341 ms ] threadI thought CentOS was downstream of RHEL. Doesn't this describe what CentOS Stream became?
In other words, how was downstream CentOS considered a beta or "a preview of what was coming down the pipe"?
That segment of the article in particular is very poorly written, and seems to aim for sensationalism rather than accuracy.
To answer your question directly: it wasn't.
I suspect that AmazonLinux (also originally downstream of RHEL) caused Red Hat to reconsider this. I’m not sure why exactly. Perhaps Red Hat/IBM compete with Amazon in some space, or perhaps they felt that Amazon should release their patches.
Example: As a user of (original) CentOS, you discover a bug. Where do you report that bug? The Red Hat Enterprise Linux bug tracker, because CentOS is just a clone and the community has no real ability to fix things. Then what? You wait until a Red Hat employee gets around to looking at it, and then wait until the next patch release which might be up to 6 months away.
Now repeat with the new model.
As a user of either CentOS Stream or one of the RHEL clones such as Alma Linux, Rocky Linux, Oracle Linux, etc. you discover a bug. Where do you report that bug? The CentOS Stream bug tracker, upstream for RHEL and thus also upstream for all RHEL clones. CentOS Stream is a community project, so the community (including yourself, and including maintainers for Alma / Rocky / Oracle Linux) can actively contribute to and drive the fix onwards to completion and test it afterwards, wheras before it would sit in internal nightly builds for a few months.
Both the community and Red Hat benefit from aligning incentives that way. However, I understand why there was a lot of frustration about the way it was announced and implemented. It could have been handled better.
disclosure: I work for Red Hat, but not on anything related to RHEL or CentOS Stream.
Also, Stream support period is much shorter than the corresponding major release (e.g. Stream 8 till May 2024, RHEL 8 till 2029).
CentOS Stream is the exact opposite of all of that, with no upside at all for this model. It will push numerous companies away from IBM RedHat entirely. Especially as it is widely seen as the start of IBM creating a bigger moat around RHEL, so there is little confidence that future clones of RHEL won't see a similar fate one way or another. And while finding a bug and not being sure who to report it to is bad, finding out that your Linux distro was buried by your maintainers half-way through the lifecycle and won't get any new updates would be disastrous.
It's also not that Red Hat saw something wrong with CentOS Linux. To put it simply, between 2014 and 2019 Red Hat learnt that the company needed a free RHEL upstream more than they needed a free RHEL downstream. CentOS Stream is all about sharing participation to the CentOS community so that everybody does what they need and they don't have to ask mom Red Hat. Just read https://www.theregister.com/2021/07/09/centos_stream_greg_ku... ("Greg Kurtzer: Red Hat did the right thing and the new scenario is better than the old").
In fact, a lot of effort went towards making it possible to ship RHEL's upstream as a rolling release. Consider that ten years ago you had to ask specifically, months in advance, about updating a package in a RHEL minor release. In 2010, a quality engineer and I spent a few weeks working on grep just to make sure that the bugs were fixed in Fedora before RHEL 6 forked, because otherwise we'd be stuck with those bugs for years. These days I could just open a merge request on CentOS Stream and ask to update to a newer version of grep.
I may be off, but hey. As I understood it at the time, the reason wasn’t “OMG there are folks not paying us for the bits” but rather “we have this number of engineers, they can work either on CentOS Linux or on Stream” and Stream is more of a benefit to Red Hat as an upstream for RHEL.
I'll never in my life understand this. It's like Red Hat took the time they should have used to talk to their customers and spent it writing a Guide To Enjoying Eating Shit to lecture everyone about how great things are now.
I'd respect the company a lot more if they just said "we bought all this CentOS stuff and it turned out to be a money pit so we're throwing it out and making it a part of the beta testing process" instead of this relentless gaslighting that anyone on Earth wanted this.
It depends on the definition of scale, or the company. Cygnus started in 1989 and was doing well, it merged with Red Hat around 1999.
> The company generates way more value than it captures and has a low rake out of necessity—none of the software it creates is proprietary.
There are expenses and revenues, and companies like Red Hat rake in revenues - which are more than expenses. So did MySQL and other companies.
> It’s under constant threat of being disrupted or undermined by other open source providers.
Back before cloud got big, I worked for companies that paid for HP/Dell service agreements and RHEL service agreements. If we were a smaller company we may have gone with CentOS or something, but we weren't.
The thing not mentioned is not only that people can and do use things like Debian instead, but that RHEL gained from Linux and Debian as well. So the benefits flow both ways.
> open core model...MongoDB...Elastic
This is another model which can be done. Three years ago, Red Hat sold for more than twice what MongoDB's current market cap is.
> The Red Hat model won’t be repeated.
It's worked for a number of companies in the past, some of whom were purchased by Red Hat. I don't see any reasons why it won't happen again, although it's not every day that a Red Hat is founded. No reasons are given why this will never happen again - all the reasons given were things Red Hat, MySQL, Cygnus etc. had to deal with.
> Support and services models don’t achieve hypergrowth
Red Hat sold for $34 billion. If that is not enough hypergrowth for some VC or founder, then they will go with another model.
> Support and services companies are uninvestable. Annual revenues are low and nonrecurring, and margins are tight. VCs are looking for 80%+ gross margins and hockey stick growth.
I don't know how revenues are nonrecurring - I worked for a number of companies that paid service contracts for RHEL and other Red Hat products alongside our HP/Dell service contracts.
Also - Intel, Netscape, Greylock and Benchmark did invest in Red Hat.
Also, companies like Red Hat and MySQL did have hockey stick-like growth.
For a company following the Red Hat model that needs to raise money from VCs, these thoughts might be applicable. But for companies following the Red Hat model that are not losing money, they don't really have to worry about what VCs think of this.
At the same time, I could see how they could have gotten pinched recently if they were unwilling to pay more to retain talent and are now getting squeezed from the other side by companies moving away from their products to reduce costs.
There are dozens of "open source business models" and as this article shows the "red hat model" isn't even one thing.
Weird to see a startup making such simplistic, wrong statements to promote their business rather than to protect an entrenched business.
These same VCs who are so fundamentally incompetent at the basics of capitalism, like risk management 101 of interest rates, that they had to get bailed out by the government, by the FDIC randomly deciding to rewrite the entire rule book that the rest of us normies have to live by.
As you point out, They are obviously interested in concentration of vast amounts of wealth, which requires a completely different set of tactics. Cornering markets and establishing monopolies, regulatory capture, pump and dumps and other forms of grift and large-scale antisocial behaviour.
It should be completely obvious to any onlooker that they have no interest in innovation or technology :)
Not if they need a long term supported and certified OS with various compliance checkmarks associated with it.
The reality is containers and container-first OSs are killing RHEL, you no longer need a breadth of OS packages or a fully capable OS. Just docker or podman. The only place RHEL has a stranglehold is with COTS apps, but those are all moving to containers too. I can't think of any widely used container that uses RHEL/CentOS or Fedora for the base image, that was a huge miss on RH's part.
I don't see OpenShift getting the same COTS lift as RHEL, so in my mind it's still going to be a net loss for RH long term as that revenue leaves RHEL and doesn't go to OCP.
:shockedpikachuface:
Alpine as a base image is perfectly fine; it is a tiny base system after all.
RH base image is also relatively small (compared to full RHEL system). It is also completely useless, as you cannot dnf install anything while building the image, if you do not have access to RHEL repos. If you have to bring in your own packages, you might use Alpine just as well.
I woould also contest your claim about "happily pay". No, the payment goes after the sales folks do their job, which isn't easy either. There's nothing "happily" about it.
That's way easier said than done to convince our clients that "oh yeah, we're using not-RHEL now". We'd have to guarantee some self support to them in that case, in order to alleviate their fears.
Moreover, I remember Red Hat sales reps back in the 2000s to be incredibly technical and knowledgable compared to every other vendor's sales reps. Every time I had a specific question regarding technical implementation or even compliance laws in my region, the sales rep knew the answer or forwarded me immediately to someone who did. They were very organized.
They have a unique value proposition in that they are an OS, so they are everywhere by default. It makes sense to have a .gov, banking, healthcare, etc teams who grok whatever the issues are in those verticals.
Linux Kernel, Various JBoss Software (Keycloak is cool), OpenShift, OpenStack, Kubernetes, and more [1]
[1] https://redhatofficial.github.io/#!/main
Confluent was arguably also able to do this for a similar reason - at the time they launched, the Kafka model was becoming vital, but was also novel and difficult to manage in production, and its enterprise services and support were critical for adoption. (They do have a managed cloud now, but their enterprise services were 73% of their revenue in 2021 and 61% in 2022: https://investors.confluent.io/node/8466/html ).
But it's hard to run what is essentially a professional services firm and popularize an open-source product at the same time... unless your open source product is already rapidly becoming paradigm-defining, and you're the people with the unique know-how to position yourself as the saviors for that space.
Any business could do exactly the same thing, and do it better than Red Hat. But there's really not much reason for a customer to move away from Red Hat if they're already on it, and it would take you a hell of a long time and a dumptruck full of capital to match them on the above.
Open Core is not a business model, just like Open Source isn't a business model. A business model is promising to fix a customer's problem, certify their software, and get vendors to support them. That alone will get you a yearly renewing contract for a couple million that you then upsell over time.
If you go put that on Debian or whatever, they’ll make you reproduce on RHEL.
This becomes a big deal when you have RHEL, VMWare, and other vendors in the mix. If you ignore the certification, the app vendor or VMWare will just shrug at you.
This can also be described as assuming liability.
Fact of the matter is people don't want to assume liability themselves. If they can pay to get someone else to assume that liability (which Red Hat is happy to do) they will.
The rest of the FOSS ecosystem on the other hand is all about assuming liability yourself. No warranties expressed nor implied, no support, no nothing. For a business that is simply not palatable.
This is also why businesses happily pay tons of monies to Microsoft, Adobe, IBM, etc. for licenses and support. They assume liability, liability the businesses don't want to assume.
"Assume" is a different word, it means implicitly accept. There is no such thing as an implicit clause in any contract.
Definitely willing to be shown otherwise, but I've never seen any vendor accept liability for there product beyond "you can have your money back, we'll take our software back".
That's a HUGE deal for multinationals, and virtually nobody else offers that.
Oracle would technically be another one. They will even sell you Oracle Hardware (or Oracle cloud) to run your Oracle Linux. Of course there are many good reasons not to want to use Oracle for this.
https://docs.openshift.com/
Despite having a reasonably large contract with them at work, we had multiple support tickets go without response for months, with at least one that I am aware of being left open for almost 2 years before being closed as "won't fix"
Noting that its only my personal experience I am sure plenty of others have had the opposite experience, however I have found canonical support to be vastly superior as as such I would never chose RH if support was the primary thing I was after.
also like it or not, by paying for support people have a right to expect that the company being paid will be engaging with you and the request you are putting into there system, not that you will have to find a person to chase down because your tickets are being ignored.
If you want to make software for the government, it must be public domain.
In the long run we'd all have lower taxes, more efficient government, and better software for all to use.
The downside would be some people in the 1% would no longer earn billions in fees from the rest of us for "licensing" software laden with technical debt to the government.
Do you have a good example of a company that provides support comparable to Microsoft's www.office.com (word, excel, powerpoint, outlook, teams, and one note). Note that this is a web instance which means that the cost of installing and managing software updates on various computers is reduced.
The lower taxes is not guaranteed at all when looking at the total cost of ownership.
With open source software, this often means increasing the staffing (because there's no corresponding org that will do the support with the necessary SLA) which also means that there isn't necessarily more efficient government.
The "better software for all to use" - switching to LibreOffice (or one of its forks) doesn't mean that the open source developers are going to prioritize the issues from Smith County, KS when there's an issue that impacts them.
To go down this path of "lets have government use open source" it is first necessary to get the open source developers to want to make their product competitive for government use compared to the proprietary ones that are out there and for companies to step up and provide similar SLAs that Microsoft and others offer.
That is something that can happen now, but isn't because there isn't enough interest from the FOSS community to do that for that sector. Trying to push the public sector to do it would be painful, expensive, and inefficient and not result in any better software.
I think you are right in the short run, but I think in the long run TCO is far higher with proprietary software. Because you are sinking money into a lost cause and delaying the cost of a future expensive migration from the closed source to the open source.
People and organizations will always want to mold their software to their domain, and for a while and for the right price, that works. But proprietary companies sweep technical debt under the rug, talent leaves, they start to slow down, and eventually the software slows down the organization considerably. It becomes harder for the organization to hire and train people on the proprietary stack. I recently spent a week working alongside federal employees and almost without exception everyone hated their tech stack, attributed a double digit percentage of their productivity lost to it, and would often build things using open source on their own machines to get work done.
Eventually the org switches to a more open source stack, but the process to extract themselves is costly. I think that cost might be greater than the cost of just starting with open source in the first place.
It seems almost like a natural law that all proprietary software is eventually replaced by open source alternatives.
> Do you have a good example of a company that provides support comparable to Microsoft's www.office.com
Not as a package, no. For each individual item, there are alternatives, but I'll agree there is a missing offering for a fully self hosted alternative to Google Workspaces or Microsoft Office 365. If I'm not mistaken, Collabra Online is trying to be that, but the execution seems to leave something to be desired.
> The "better software for all to use" - switching to LibreOffice (or one of its forks) doesn't mean that the open source developers are going to prioritize the issues from Smith County, KS when there's an issue that impacts them.
I wonder if ChatGPT could influence things here. I wonder if support agents from ChatGPT trained on loads of open source content and code will turn out to be more effective than human support agents for proprietary stacks, and you might be able to build an open Office365 or Google Workspace competitor.
Anyway, just doing some back of the envelope math, I estimate governments in the USA spend $20B - $50B a year on proprietary software[1]. It certainly gives a lot to open source too, but I believe if more of that money was diverted to open source we'd all be better off in the long run. Not a slam dunk case, I know, I'm at the early stages of this line of thought. And of course, even if you could prove that open source would have a much lower TCO, it would be a protracted battle to go up against the folks with a lot of money on the line to deny that truth.
[1] According to this site (https://www.itdashboard.gov/itportfoliodashboard) it looks like total unclassified Federal Government IT spending is ~$84B a year. It looks like Palantir alone gets $1B a year from the government. The latest information I can find on Oracle is that 25% of their revenue comes from US Gov (so say ~$12B per year). Microsoft is perhaps $5 - $25B? Not sure.
Hiring Deloitte or Accenture or another consultancy to provide the support for an open source piece of software (be it directly or "we need to have people with these skills") is still extracting money from the org and I can assure you that the consultancy isn't contributing back to FOSS.
As it is, Microsoft is a known quantity with a support SLA that meets the necessary requirements both for hosting of data and for response time. The cost of Microsoft licenses and support is less than the cost of hiring additional staff or bringing in a consultancy.
A budget item with a fixed number in it is much preferable for budgetary allocation than open ended numbers typically found from consultancies.
The support of open source software is often more costly for the public sector than going with a big tech solution - even when including the cost of licensing.
Until the cost of support for open source solutions comes down to where the proprietary options are, it is often not something that is considered.
RedHat is an example of a company that is doing it right - offering the support that the public sector desires with a consistent budget line item.
If you can stand up a PostgreSQL instance that is able to compete with the Oracle performance and support (e.g. backup restore failed - call support and have someone on the phone helping now) for less than the cost of the Oracle licensing you will have people beating down your door to sign up.
Same thing with Microsoft.
For the federal government to do this on their own is much more costly. It goes back to the consultancies to implement, migrate, and maintain. Trying to staff up to be competitive to be able to do it in house is also quite costly (note the pay difference between public sector and private and then multiply that difference as an ongoing expense by the number of employees needed).
And so this returns back to the FOSS community to address the needs of the public sector and create the needed companies that are competitive to be able to provide that ongoing support.
ChatGPT will not write an office.com competitor this decade... or next.
My point is it's suboptimal, and the U.S. Federal Government could change things, as the biggest buyer of tech in the world. If they said "everything has to be public domain", it would be a seismic event. I think we would all be better off if they did this (probably with the exception of <1% who would lose monopoly profits).
Nit: I think you misread my ChatGPT comment. I said ChatGPT could be a competitor to office.com's _support_ agents, not that ChatGPT could build office.com.
The hat killed Centos because some IBM midlevel manager needed to make quota for a bonus, and they didnt stop to realize just how unsuccessful this tactic was for Oracle/MySQL.
moderen redhat is carved into about 40 seperate repositories and channels for packages designed to fleece you right down to the last RPM. need Ansible? sure it comes with redhat 9. need posix acl support from galaxy? that's another repo you didnt pay for. same goes for about a hundred other packages.
redhat might have had great tech support in the past, but put a ticket in now and you're in IBM levels of "prove it" hell with SOSReport zips and zoom screenshares until your entry level support guy gives up the ghost and calls a developer, who tells you the bug cant/wont be fixed because libraries are hard or a rebase is impossible or any other number of boilerplate excuses why this ancient 4.x kernel based distro cant just source its solutions from fedora like they said they do
and the flagships? forget it. Yeah Quay exists but try to implement it and youll find cool stuff like an administrator account defined in configuration but not created. once created in the admin panel youll have to go back and disable new user creation so you can control user creation at all, and the whole thing just feels rickety and poorly designed from the get-go. none of this is mentioned in the docs overtly, but is buried in Redhats bloated docs pages. just buy a quay hosted whatever from the hat instead.
Openshift is equally appalling as its a massive ecosystem Redhats done virtually everything to control and nothing to distribute or competently document. its painfully obvious IBM wants you to hire a consultant to build and deploy it...There is an open-source offering, but it cant be easily rolled into an airgap installation and it suffers the same level of byzantine and poorly written documentation as Quay. K3S/RKE2 easily beat it for nearly every use case.
Then theres the docker fiasco which unsurprisingly neither of these juggernauts wanted to deal with the others bullshit so now dockers broken in all but the most monied instances of IBM hardware. redhat became so absolutely resolute about podman being "everything you can do i can do too" that it has a fucking compose option to translate docker compose into podman because people like compose but rewriting your appstack to pods when you dont want to be arsed with k8s is sort of a non-starter.
I cannot stress enough how untrue this is. See my comment https://news.ycombinator.com/item?id=35586311
RHEL clones were always going to exist, this is and was obvious, and there are more of them now than ever. That's not the point. The point is that now there is a way for RHEL clones, including ones from competitors like Oracle, have some way of contributing back to RHEL, whereas beforehand if you were a CentOS user your only possible course of action was to file a bug in the RHEL bug tracker and wait for a Red Hat employee to prioritize it.
The business benefit is also a technical benefit - the ability to fix bugs and add features is no longer strictly limited by the amount of engineering capacity Red Hat can contribute. Companies like Facebook use CentOS Stream at scale internally and make contributions that ultimately end up in RHEL, users and vendors have a target they can integrate against early without needing gated betas, etc.
I work at Red Hat for the last eight years, long before IBM had anything to do with us, and I cannot stress enough how much IBM is *nowhere* to be seen in not just our day to day engineering jobs, but pretty much not even periodically. I know nobody at IBM, I have no email addresses of anyone at IBM, nothing we get in our inbox (and we get a lot) ever has anything to do with "IBM" in it, I have no links or access to any IBM-specific intranets or resources of any kind. They truly have stuck to their promise of not changing Red Hat; our support system is the same, we still respond to bugs, ask for SOS reports, and customer support people do collaborative screenshares, just as we were always doing, there's nothing different about any of that. There have been no top-down changes or decrees or anything I'm aware of that would change how we do any of that stuff.
as far as Podman it's all I use for all of my server environments and I like it a lot better than Docker. It's simpler without all the weird "Dockerish" things getting in the way. Not having a monolithic server hang the whole machine due to a misbehaving container is well worth it.
Is it possible that different IBM Depts - or IBM regions, are selling Redhat Software/Support without much actual involvement of the Redhat business? Like the might have done in their traditional reseller/consulting/support role?
TBH I actually think it's likely - sales and account execs can probably just tickbox add RH stuff to their package deals, copy&paste the slide deck - and post sale the customer support channel will be the general enterprise support one. IBM are probably very deliberately channelling all first level support through there anyway, same as for everything else - so core Redhat support staff might only get looped in for 2nd and 3rd level issues for those accounts, if at all.
It's quite easy to see that effectively creating two sales/support channels depending on whether you "bought from Redhat" or "packaged from IBM" - but equally it's unlikely to be presented or perceived that way, until it's too late.
I believe you when you say they've been hands off - it's surprising, but I can also see how it would make this kind of situation _worse_, because they're still effectively treating redhat as an "external" provider they resell.
Not that fully integrating the business would make it better of course, probably just different worse ( just based on my experience of corporate mergers, not IBM specific... ).
Podman-compose was feature-lacking and beta even when podman already had its 3rd major release, while the communitcation was that "Podman is a drop-in replacement for Docker". It just wasn't a drop-in at all for all compose users. People either stopped considering Podman or enablded the socket daemon, so that they are still able to use docker-compose. But this goes against the design ideas podman had.
Feels as half-baked as Ingnition, which also still feels very bare-bone, as if there's no financial interest to enhance it further.
Red Hat created an industry and all of the successful enterprise Linux distros have followed its model to some extent or another (Canonical, SUSE, etc). You don't need "open core" (which is a misnomer if there ever was one) if you continue down a path that creates and maintains communities of software authors out there in the world that will constantly feed value back into the ecosystem.
A virtuous cycle of community maintained software alongside a thriving and growing business is still achievable, no one seems to have the guts to do it.
(edited for typo)
What are some per-CPU offers from them? (Because that's what I'd expect from Oracle, and hardly from RH.)
Cell phone manufacturers and social media companies run collectively around 100M CentOS servers that I know if directly.
https://www.redhat.com/en/blog/fastest-road-centos-linux-red...
1) to offload the production of the releases (effectively what is now Alma Linux). Because there's so much use of CentOS, it was pretty much a given that somebody would pick up the slack. Most of those companies weren't running stock CentOS anymore, they had their own downstream repositories
2) to give more insight into the making of RHEL minor release to CentOS SIGs and to companies that run CentOS derivatives
Facebook for example does not use the RHEL kernel and was one of the early adopters of CentOS Stream even before CentOS Linux 8 was terminated.
There is a difference between CentOS and CentOS Stream
https://blog.centos.org/2020/12/future-is-centos-stream/
> Red Hat created an industry and all of the successful enterprise Linux distros have followed its model to some extent or another (Canonical, SUSE, etc)
IBM created that industry, by lifting Linux into their supported platforms portfolio.
and the first Enterprise Linux ever was suse linux enterprise for s/390, followed by x86 releases at about the time RH released their first RHEL.
Red Hat becoming entrenched wasn't a guarantee in my mind until IBM made that big push. If you could point to a single event that made it likely Linux would take over, without a shadow of a doubt it would be the blessing it received from IBM. There were an enormous number of bigcorps that made initial investment in Linux after that.
Red Hat didn't create the industry single-handed, either. They deserve a lot of credit but Linux succeeded because it was a group effort where there was a chance for a lot of players to make money (and opportunity to displace others).
this created an immense Push on HP, dell, Dec, sgi, ... to follow suit
and oracle, sap, Software AG, anything serious.
this push and commitment b IBM was.so.relevant because it established Linux as a serious platform with enterprise level SLAs and mutual commitments for the full stack.
the 4d chess was to establish a solid consulting business for "consolidation" on the "single platform"
and it paid off immensely for IBM --- and for most others.
given the IBM market position around 2000 this was pivotal and prophetic.
My only question is what the hell took IBM so long in making the decision to buy them out. I've never been less surprised by a big acquisition than I was with IBM and Red Hat. The only surprising thing was that nobody outside of that space saw it coming.
But there are a large number of outsourced software development companies, some of them publicly traded- Cognizant/Infosys/Wipro types, plus Accenture, plus the new generation of Eastern European ones like EPAM and Grid Dynamics. Government-y ones like CGI. Oldschool companies offering consultancies like EY/Deloitte, etc. Hovering in the background is Toptal, which is technically different but I would imagine the bulk of their revenue comes from enterprise outsourced dev contracts. All of these companies are making this business model work somehow? And there are lots of them too. So it can't be that bad of a way to make money (yes I understand that these are not VC-investable businesses)
In other words, Red Hat's model doesn't work for Wall St and VCs looking to make quick easy money riding out the next tech 'unicorn'. Red Hat has survived and thrived for 30 years now, while most modern VC backed tech companies are lucky to make it past 3. The infiltration of Silicon Valley VC vultures was one of the worst things to happen to the tech industry and why we rarely see sustainable startups like Red Hat anymore.
No, they worked for me too - not an enterprise -
Not sure about their revenue between different divisions, but they finished open-sourcing even the last proprietary part of their JDK, making OpenJDK the reference implementation, plus they finance 90+% of all its commits. They make some revenue from selling OracleJDK licenses (and even most of those are free if you stay on the latest version) which includes support. Many of their customers are on older Java versions though, for example their Java 8 support is valid for something like 2030, so mostly government, big enterprises.
(For some reason though, there is just endless bullshit around Java, even though the exact same model/license is true for the Linux kernel)
They also ignore the fact that Sun also charged money for support, JDK was only available as free beer with source on request, the whole Apache drama with the TCK, and access to legacy versions required a Sun developer account with license.
Or that they were the ones introducing the separation of having UNIX for users, and an additional license for an UNIX SDK, which is what made people rush to improve GCC, until then mostly ignored project.
And speaking of databases, the idea of running custom Java code inside the database seems like an abuse of the database. How many people actually use this feature, and how many of them like it (as opposed to having made a bad technical decision 20 years ago)?
When performance matters, stored procedures are the way to go, not wasting network traffic and CPU cycles on the client, As it so happens, these additional runtimes are great and safer way to extend PL/SQL, pg/SQL, T-SQL,... than writing C extensions.
While I agree in principle, in practice to me it feels like the tooling just isn't there.
Testing and debugging is a bit more difficult than with other languages (e.g. even using breakpoints and stepping through code), things like logging typically don't have pleasant implementations (e.g. log shipping), the discoverability and even version control of the code also tend to be worse, among other things. That's even before you get into building around the particular runtime that you're provided with, trying to get a grip on dependency/package management, automated CI deploys, rollbacks, monitoring/health checks, local development environments and so on.
My experiences might be the opposite of some folks, but I recall working on a system where most of the logic was implemented in the database packages and something like Java was used just as a glorified templating solution to serve a webapp. The performance was great, but actually working with the codebase was an utter nightmare, so it's not worth it in my opinion. That's like choosing to write a webapp in Assembly just because it's faster.
For example, look at the JetBrains survey, where they asked some questions about databases: https://www.jetbrains.com/lp/devecosystem-2021/databases/
If half the people don't debug their code, three quarters don't test their code, almost half don't use version control and about half don't bother writing comments of any sort, that's the kind of code that I don't want to be working with and would advise others against going for that approach. While we can talk about the fact that these things can be done, the fact that they're not, is evidence enough that the community just isn't there yet.Use databases for what they're good at (including some in-database processing, like reporting), but don't try to do everything in them.
Oracle, Microsoft and IBM provide the same kind of IDEs, graphical debugging and source control as any other programming language.
It is this lack of skills that comes up with fads like NoSQL.
By the way, there are similar results related to debugging in other languages, where most can't do better than printf debugging, don't know unit tests, profilers or static analysers.
The outcome of bootcamps or CS degrees without sound engineering practices, while people label themselves "engineer".
We can (and should) explore the causes for it, sure, but that doesn't change the reality that if you join a project that uses a certain approach, it isn't guaranteed to be using the best possible practices, but rather whatever is popular and easy to do in the industry, unless you're very selective about where you work.
> By the way, there are similar results related to debugging in other languages, where most can't do better than printf debugging, don't know unit tests, profilers or static analysers.
I'd say that this is true to some degree and is also a reflection of either poor tooling, or lack of interest. For example, command line debuggers with arcane keybinds will be harder for the average developer to learn and use effectively, than just clicking on the line they want to stop at in a JetBrains (or similar) IDE and just clicking a custom run button that will launch their entire project in debug mode. The same goes for being able to run either your entire test suite or a particular test by just clicking a button in the source file, helpfully shown by a good IDE.
Things get worse when you want to test the integration with an actual data source (like an external API or a database), because in some cases you'll have to mock so much of it that you won't be testing anything remotely close to the real thing, or will have to deal with bootstrapping an instance of the API (if you can even self-host the full thing) or a real database, which will be really good from how truthful your tests are for real world use cases, but will need certain configuration and resources for setting it all up. Sometimes you can get away with something close enough, like using an in-memory database behind an ORM, but some of those abstractions end up leaky. Even worse if you want to test your integration against cloud services.
Static analysis tools are not without their issues either: something like SonarQube is good theoretically, but will have you struggling against setting up the actual scanner (including mundane stuff like source file encoding) on your CI server, setting up separate configurations if you ever want to run it against your local codebase and will be hard to configure in regards to what should or shouldn't actually trip up the analysis and throw warnings at you, because not all of the recommendations will be even viable for your framework and how it expects code to be written.
> It is this lack of skills that comes up with fads like NoSQL. ... The outcome of bootcamps or CS degrees without sound engineering practices, while people label themselves "engineer".
Does it mean that we shouldn't do these things? No, but it definitely means that we shouldn't just wave our hands around and suggest that it's just an issue of education, when actually trying to use the current technologies is often like banging your head against a wall. Use what works well and causes the least headaches, be open to eventually trying new things as the ecosystem and tooling improve, but don't stray too far from what others do successfully for now either.
From what I've seen, the things that have absolutely improved are schema versioning solutions (and thus, versioned DB migrations) and the ability to run database instances locally for development (in throwaway containers), so that you can test breaking migrations with believable seeded data before it ever needs to run on a shared environment. Codegen still could be better (e.g. generating Java/.NET/... entity code for an ORM in a schema-first approach), but some forwards/backwards engineering has been around for a decent amount of time at least, when dealing with models (for example, in MySQL Workbench, though pgAdmin is still lagging behind there). There are even tools for easier development of APIs, like Hasura, PostGraphile, PostgREST and so on, though the adoption there varies.
Edit: oh, an...
So I guess that makes them both a very significant contributor as well as deserved receiver of financial resources towards F/OSS. OTOH, apart from adaptions to ever-changing hardware, you could say that the times where Linux innovations are actually serving end users is long, long behind us, and anything RH pushed, such as systemd, gnome, and in particular podman (to take away from Docker ie what RH sees as their market, and theirs alone) etc. is basically zero sum software created out of a self-serving interest of selling support contracts by definition. That Linux isn't pro consumer is also evident by it being used for clouds and for a privacy-invading mobile O/S.
Linux was nice and gave a direction to a developer community as long as it was chasing commercial Unix/POSIX with strong precedents and blueprints what to implement. Now it's just an IBM division.
People that weren't there in the begining don't get it that Linux only really took off around the UNIX offerings of the early 2000's, when Oracle, Compaq, Intel, IBM money started be pumped into Linux in some form.
The thing with UNIX/POSIX clones, is that there is hardly any inovation beyond adding new drivers and kernel features, as POSIX doesn't have anything else to offer.
1. For me until proven otherwise, RedHat have an OpenCore model, the CentOS shift is too recent to be meaningful. My company literally panicked about their CentOS servers because they didn't want a rolling distro and so decided to switch to something else free for low risk VM (worse I'm afraid) and pay RHEL support for critical applications.
2.There are a shitload of successful support company for real OpenSource product, the author just choose to ignore them. Maybe because they are mostly smaller and "uninvestable". Just look a the support advertised options for PostgreSQL in North-America only... https://www.postgresql.org/support/professional_support/nort...
(At least they had the sense to switch to REHL for critical servers)
Now on the app side we are left with some vendors that don't support installs on OracleLinux (because who would switch to that... again don't ask me why).