Strangely, the author makes a good point. Oracle or not, sticking closer to the upstream kernel is not a bad way to manage an “enterprise” kernel. Maybe we’ve all been too willing to sit back and accept how RHEL does kernels as “the way”.
I would expect that tracking the Linux kernel master just 4 weeks behind is going to cause more cases where, combined with enterprise software of typical quality, you'd need vendor support than with the RH way of changing as little as possible for as long as possible. And support is exactly what Oracle sells.
To be clear, I only used Oracle Linux briefly while playing with the free tier of Oracle Cloud.
Edit 2: my bad, I didn't read the "behind the tip of the LTS tree" part carefully. So, the difference is that Oracle tries to ship all LTS kernel updates, while RH tries to cherry-pick critical security and bug fixes only?
Red Hat doesn't change as little as possible, at least 30% of the changes to Linux make it to RHEL.
Most of the RHEL kernel is a few months behind upstream despite the old version. The next minor release of RHEL, to be released in November, probably will have features up to 6.3 for many subsystems and bugfixes up to 6.5 for example.
Interesting, thank you for the details. This means that in some way, RHEL is ahead of Oracle Linux in terms of kernels? I don't see an UEK version tracking Linux 6.1 at all, AFAIK.
It's true that Red Hat does pull in changes from upstream for several subsystems of Linux. It's genuinely a frankenkernel mixed with code from 6.3, 6.1, etc.
But you're still beholden to what the Red Hat maintainers are pulling in and focusing on. It's still not a general follow upstream wholesale.
Maybe, but it's not obvious. There are plenty of novel vulnerabilities in interfaces like io_uring and so forth. It's not that those features are bad, I'm just saying that there are tradeoffs to always getting the new stuff.
Maybe the compromise solution is to use newer kernels but keep certain features turned off until they "bake" properly.
But what stops Oracle from just pulling a Red Hat when people switch over to Oracle Linux? Oracle hasn't exactly proven itself to be trustworthy over the years and that's putting it lightly.
I think we need a fully open source alternative to RHEL not bound to any company. Something akin to the Debian project that can serve as an upstream reference distro and repository.
> I think we need a fully open source alternative to RHEL not bound to any company
I believe the problem is not that there wouldn’t be open source alternatives but that that’s not what enterprise wants. Enterprise wants a company behind the distribution.
Of course, think about the relationship between Ubuntu and Debian. The idea is the upstream would be a fully open-source project that vendors build off of downstream. This way we can prevent vendors from subjugating a distribution like with Red Hat and CentOS and if a vendor pulls out of the market or goes out of business it doesn't take down the ecosystem with it.
At the same time it is not that convincing point because in lot of cases the value from freezing is pretty small for kernel compared to other software, because kernels strong "do not break userspace" attitude. Version freezing is far more valuable for various other projects that do not take backwards compatibility that seriously.
I thought the Linux kernel had a strong “don’t break userspace” attitude, but it was a free for all in kernel space. If you’re developing kernel modules or have custom hardware with drivers, I could see having a backported kernel as being a major problem for support and development.
My (spectator's) understanding is that ABI compatibility is not a priority for kernelspace in Linux, but there is care taken for backwards compatibility in other ways (like at an API level). You're just expected to recompile your out-of-tree kernel modules when there's a new kernel version.
If you're using proprietary hardware drivers which target RHEL, you need stable kernel.
Also kernel updates breaking drivers is a thing.
So there's definitely value with frozen kernel version. And its almost the same value as frozen software. It just works and it won't break after update.
The same company that just two weeks ago was sending out unsolicited emails asking companies about Java licensing - a friendly shakedown for using the official JRE...
Frankly, I don't get it Oracle. I would say the priorities are backwards - JRE free, Oracle Linux paid would make more objective sense (especially because the JRE used to be free... and there's a lot of free JREs out there.)
Also, don't be stupid enough to download the VirtualBox Extensions package at work; or your work will probably get an email from a licensing agent.
> "As you know people, as you learn about things, you realize that these generalizations we have are, virtually to a generalization, false. Well, except for this one, as it turns out. What you think of Oracle, is even truer than you think it is. There has been no entity in human history with less complexity or nuance to it than Oracle. And I gotta say, as someone who has seen that complexity for my entire life, it's very hard to get used to that idea. It's like, 'surely this is more complicated!' but it's like: Wow, this is really simple! This company is very straightforward, in its defense. This company is about one man, his alter-ego, and what he wants to inflict upon humanity -- that's it! ...Ship mediocrity, inflict misery, lie our asses off, screw our customers, and make a whole shitload of money. Yeah... you talk to Oracle, it's like, 'no, we don't fucking make dreams happen -- we make money!' ...You need to think of Larry Ellison the way you think of a lawnmower. You don't anthropomorphize your lawnmower, the lawnmower just mows the lawn, you stick your hand in there and it'll chop it off, the end. You don't think 'oh, the lawnmower hates me' -- lawnmower doesn't give a shit about you, lawnmower can't hate you. Don't anthropomorphize the lawnmower. Don't fall into that trap about Oracle."
So, that rant actually predates Oracle v. Google. (For a bit of a history of my LISA talk that you quoted, and then my added color on Oracle v. Google see [0].) Oracle has shown everyone over and over again who they are; it shows how deep the mistrust in Red Hat has become that Oracle seems benign by comparison.
There is another problem that wasn't covered in the article. The 10+ years of stability leads to behaviors and outcomes that remind me of the long-lived SSL certificate problem. Updating is done so infrequently that the "how?" is forgotten. As the 10 year support limit approaches, most of the old team members who did it last time are gone, tech debt is through the roof, few people know where everything is or how to build it, and so on. Enterprise Linux "stability" enables all sorts of bad behavior if your company is inclined that way.
LetsEncrypt did us a huge favor by forcing automation vs having the guy who knows how to update the SSL certs every 4.9 years and left 6 months ago. I'd like to see the RHEL stability model go away too and force people to complete their automation and solve the problems of being able to rebuilding on demand - and actually doing it.
(I know, most HN folk are well disciplined but there are a lot of corporate cultures that are not.)
Don't you end up with the same problem with the automation that has been running fine for 5 years, then suddenly breaks? And the person that set it up is either gone, or has no clue how they did it 5 years ago.
Every 6 months? That seems like a pretty long window for tribal knowledge to get lost. Is 6 months arbitrary or is there some reasoning behind that cadence?
That's an amazing amount of effort expended on something that provides exactly zero revenue. I understand the concept, but I've never been fortunate enough to work in a business where that was practical.
LE by default successfully ran 2 months prior. 2 months and 5 years are two completely different worlds in terms of bit rot. That and there are many generic tutorials and scripts and knowledgeable devs for configuring LE fresh.
Before LE almost no one automated SSL cert refresh. Depending on your SSL cert vendor you couldn't automate things even if you wanted to. It's not that the automation ran fine for five years, it's that you'd be lucky if the manual process last done 5 years ago was even documented.
SSLMate is about as old as LE, they both started around the same time.
Recently saw a (thankfully not mission-critical) old k8s cluster fall down with absurd incompatibilities between node versions, cluster versions, and cert-manager versions - all of which only support upgrades one version at a time. Even infrastructure-as-code doesn’t save you if you need to upgrade something but don’t have the time and expertise (and esoteric changelog knowledge!) to reliably upgrade everything else.
> I'd like to see the RHEL stability model go away too and force people to complete their automation and solve the problems of being able to rebuilding on demand - and actually doing it.
Whenever there's a new distribution release it invariably breaks a bunch of things with the automation and you spend more time massaging your playbook so it works again than it would have taken to do it by hand
systemd threw the biggest wrench, by far, in my automation workflows (this was before containers came to dominate a lot of the landscape, so everything was managed with init scripts). I like it now, but it also broke things quite frequently in the early days, and there was a looong period of time when you had to shim software to work with systemd.
But even still, things like snaps, the way Debian handles system Python, and various little changes that have an outsize effect on automated deployment do cause a good amount of churn with automation.
Yup, enterprise Linux insulates you from unneeded change (in the business context). For most companies, systemd will have no impact on the bottom line vs sysvinit vs whatever.
However, paying an extra engineer to sort through all the changes possibly will.
On the other hand, there's some interesting trends like monokernels and minimal OS images that leverage services running off-machine instead of expecting so many local services removing some of the complexity/volatility (DNS, SMTP, federated login)
> Whenever there's a new distribution release it invariably breaks a bunch of things with the automation and you spend more time massaging your playbook so it works again than it would have taken to do it by hand
The rolling-release life is that things break constantly, during each week’s upgrade, but only a little bit at a time (and hopefully in staging). I don’t know if this is better for system administration, necessarily, but if you’re used to a stable-release dynamic of heavy discrete breakage and piles of backported patches, then you might be imagining the same scale of breakage every upgrade, which is isn’t the experience at all. So don’t discard the rolling-release option because of this preconception.
> The difference is: If there is a weekly breakage on a weekly update, delaying with it is just part of the process and timed in.
That entirely depends on your operations model. There's a difference between, say, a nuclear power plant and a colo web hosting shop. With the latter, sure, no problem risking "minor" weekly breakage. With the former, I'd much rather have scheduled, heavily tested and carefully monitored maintenance windows.
And HN tends to underestimate the number of places like the former exist. Backbones of global finance and telecom, industrial facilities of all kinds, etc.
So how about a more down-to-earth example. Medical imaging.
Changes to graphics drivers, can, do, and have impacted how things like MRI results get rendered by software. It's going to have at least some networking with the rest of the hospital and difficult to completely airgap, but at the same time you cannot update it willy nilly with the latest and greatest uncertified drivers.
That's precisely where "enterprise" distros are sometimes necessary.
At $dayjob, PACS devices use a separate VLAN and can only talk to their servers. Although to be fair, most of the problems with that system are down to the frontend using outdated js/asp/Java code that edge/chrome doesn't support anymore.
The hospital there also isn't the organisation doing the development and shouldn't really care about the OS the appliance uses.
The vendor (Siemens or whoever) should make updating the OS part of their development cycle and certify accordingly. If drivers are relevant part of the product there, they have to take enough ownership on the drivers to ensure they do the right thing.
Even this is probably an over exaggerated example. Although to your point with thousands of hospitals, medical centers and imaging centers in the US there are enough to consider it a common occurrence at a community level.
Another example would be systems setup for CAD. Certain levels of CAD require certified video drivers in order to unsure tolerances are met. Error introduced at this level destroys product and possibly people.
It is the countless places that exist at some level between your two extremes of the nuclear power plant and the colo web host that need Enterprise Linux.
My point when I mentioned rolling-release was that I’m very unsure whether they need Enterprise Linux and a bout of desperate firefighting every several years or a rolling-release distro and a small but respected team with a staging environment and a steady supply of handheld fire extinguishers.
I could be convinced the former were the answer in many cases, but I’ve never seen the argument for that move beyond a bare proclamation like you’ve just made. It may also be that the argument for this is so mired in the particulars of a situation that it essentially can’t be articulated, but for me that’s mutually exclusive with it applying to a broad, vaguely defined classes of deployments like you’ve just done. (A complex argument is bound to produce an intricately shaped class.)
My (admittedly theoretical) fear in my initial comment was that the LTS people read about kernel updates every month, think of the breakage their LTS encounters on every kernel update, and nope out. Yet without the humongous pile of backported patches the LTS requires kernel updates are the most benign thing in the world if you can afford the reboots—a decade of them with literally not a single issue can and does happen.
"I’ve never seen the argument for that move beyond a bare proclamation like you’ve just made."
Do you work in an industry that requires a specific service level to be maintained? How many hours per day? What does downtime cost? What does redundancy cost?
We have situations that cost millions of dollars per hour when downtime occurs. Now how do you protect those systems?
A definition of Hubris is not believing something exists because you haven't seen it before.
Our tasks as Systems Administrators is not to make a career out of updating systems for maintenance patches. Our task is to design systems to minimize updates and then redesign systems to eliminate wasted user interaction and downtime.
I think it was an analogy. If you don't do a thing for a long time (updating SSL certificates, updating a Linux system to a major new version), the knowledge of how the systems were maintained/built gets lost. If you have an automated, repeatable process that moves with the times, it is more likely that the process is codified (either in documentation or in infrastructure as data) and easy to repeat.
Likewise, it's been suggested that the 19.6-year (1024-week) GPS epoch is pessimal. Rollover is infrequent enough to be ignored, but frequent enough to actually happen and cause problems.
Folks who know such things better than I do, have suggested that it would've been far better at like a 64-week rollover (or just chop it to 52 and leave part of the code space unused), that way everyone would have to have a plan for it. Nobody could claim they don't expect their hardware to be in use 64 weeks in the future therefore they can ignore rollover.
Funnily enough, I had the Unix epoch time question come up with a customer (who makes very long-lived pseudo-embedded systems) come up in discussion last week.
>> I'd like to see the RHEL stability model go away too and force people to complete their automation and solve the problems of being able to rebuilding on demand - and actually doing it.
So how could this be accomplished?
A Nix-OS style approach coupled with an immutable OS core?
It is much harder to offer stability guarantees than to just publish updates in a rolling release fashion. And yet big organizations pay big money for that stability or the support to poke an enterprise software provider to get stuff working for their needs.
>> ArchLinux-style rather thsn NixOS style. Just roll the updates when they are ready into your very own test, int, acc, and finally prod.
The issue is that a rolling-release approach does not have stability guarantees and forces everything to be upgrading all the time.
This does not work very well if you have specialized hardware or scientific equipment. If the drivers for your lab equipment work with a given release of an enterprise linux, you can't just jump on the next release until you have working drivers ready.
The same is true if you are working with some enterprise software which is only certified to work with a given release of an enterprise linux. Would you really want to run business critical software on a version of the operating system which is not (yet) supported by the vendor?
All those hours hunting for the reasons why something suddenly stopped working every two or three weeks need to be paid. So maintenance cost for Linux servers would either skyrocket or no updates would ever be done for years. There are very good reasons why rolling releases in infrastructure are basically a no-go.
Same here, I ran Arch on Hetzner Cloud and find it superior in many subtle ways.
Rolling updates mean that I have to carefully choose what to install in order to keep the maintenance costs down. This has the side effect of reducing the attack surface.
I also manually review updates, which means that I keep up with the news in OS land.
I have to reboot once in a while because of updates, which means that I test resilience of my infrastructure.
By comparison, RedHat stack at work feels creepled and ancient.
You need a RedHat account (read: subscription) for pretty much everything, even the most basic documentation or downloads and yet my bugs in their bugzilla linger for months with none even trying to reproduce, let alone fix.
Every single time I tried using arch, I had nothing but problems with updates. I wasted a lot of time hunting down documentation to things that broke, like mailing, logging, the DE, bluetooth, you name it. Changes that get taken care of by default in other distros. I had some very nasty surprises while using arch. My stable Ubuntu or Debian installs didn't even have a single glitch in the exact same timeframe.
Depends, I think. I remember you being able to finagle a passthrough of devices, the underlying software can do that with little issue and once passed through it shouldn't be an issue, but I vaguely remember there being some notion of that being dissuaded. Mostly because of the increased attack surface, I think. Though that was a few years ago now.
Qubes OS doesn't really solve much in regards to stability and work required to update in a professional setting though, I'd say.
Passing through a device is more secure than bare-metal, yes, but I meant it as the Qubes OS project themselves dissuading the notion. And doing so decreases security for the whole system, which is why they advice against it, because you've now given a potential bridge back to the main system. Not that there's been many exploits for that yet, but if such systems were more common, there would be.
Regardless, even with the ease of virtual machines, Qubes OS doesn't really solve problems involved in professional management, nor should it be considered for that given the overhead of the system. It's a neat system, and for general purpose use it's pretty cool, but for stability and work required to update a system, it really doesn't.
Sure, it makes general use-cases pretty easy to update, but those aren't really much of a problem if you've set up a PXE server and have a base image together with default configuration. The issues occur when you're updating servers, which is what I was talking about, because the end-user isn't that important in this context.
Whether you have it running in a hypervisor or bare metal, it all comes back to properly configured backups. Qubes OS doesn't solve this, nor is it meant to. It increases security at the cost of convenience and complexity. Nor does it solve stability in a professional environment, because while it does give you an isolated OS per application or stack of applications, you've now increased your maintenance surface. Servers need to be updated, within a reasonable time frame of ones being provided, and general computing often requires more of that too.
And at the end of the day, while I do like Qubes OS and do like virtual machines, they're not the be all, end all, in regards to security. Exploits exist, and as with all things, the more common they become, the more will be made.
I do still hope for a system like Qubes OS in the future, just not Qubes OS.
> I'd like to see the RHEL stability model go away too and force people to complete their automation and solve the problems of being able to rebuilding on demand - and actually doing it.
In this model, what happens when the next Python2->Python3 breaking change comes along?
Using whatever Python your distribution needed is bad practice. Own your application environment, there are plenty of ways to do this, such as Nix and Docker, which make your Python environments reproducible across systems.
Also, Enterprise Linus is one of the reasons (definitely not the only) that the migration took such a long time. Too many enterprise shops that stuck with Python 2 because it's the lazy thing to do. The tech debt grows every year you don't move with the ecosystem.
>Using whatever Python your distribution needed is bad practice. Own your application environment, there are plenty of ways to do this, such as Nix and Docker, which make your Python environments reproducible across systems.
How far down does "own your application environment" extend? How about libc? What is the role of the underlying OS?
>> How far down does "own your application environment" extend?
It depends on the needs of your application.
>> How about libc?
If you need to make sure the underlying libc has what you need, you must either bring your own libc or have sufficient feature test macros and adapters to account for possible differences.
>> What is the role of the underlying OS?
It depends on what the application requires. What operating system features, if any, do you require? Do you have any timing or scheduling requirements that are sensitive for your application? Do you need real-time responsiveness?
How does the operating system handle failure scenarios? What guarantees, if any, does it make when hardware fails? Is it okay for your application to crash if a portion of the computer's memory or disk borks?
I wonder if there would be a market for an enterprise-grade server microkernel OS. It's not the 90s anymore - Nintendo and QNX are shipping tens of millions of microkernel installs every year; and hardware is fast enough that choosing correctness and security over speed is a valid tradeoff. Maybe if I win the lottery...
Kaspersky recently developed their own proprietary microkernel OS. AFAIK they target it for IoT, but kernel is kernel, probably could be used with ordinary servers as well.
Main issue is drivers, of course. It's hard to beat Linux. It contains open source drivers and server vendors usually target Linux and Windows with their driver efforts.
Not necessarily if you build them right. Nintendo’s Switch is a true microkernel and, if it cost 200% performance, there’s no way it would be viable on a 2015 Tegra X1. The 200% thing is kind of a myth that doesn’t apply to modern practice - now it’s more like 10%.
As for 10% security - it’s more than 10%. Take my same example, the Switch. No bugs have been found to launch unapproved software in the last 4 years. There’s always the Secure Boot bug by NVIDIA in earlier consoles, but not even a WebKit bug will get you homebrew on a Switch. Kind of a big deal…
Another example of this would be Microsoft’s experiments with what would happen if an OS was built with all apps running in managed code - no compiled apps. Performance cost? They got it down to just 7% (though, admittedly, Midori never shipped, but it did host Bing in a few countries for a few years.)
That is a good point, however I've not heard of too many cases where organizations intentionally skip RHEL releases. Systems that are being actively developed do regularly upgrade through each RHEL release, and the 10 year support just lets them be lazy about how quickly they do so. The only systems I see intentionally riding out the 10+ year support are deprecated systems that are already announced to sunset by the time RHEL support ends.
The five year reign of RHEL7 was too long and did result in the very issues you bring up, but the ~3 year duration of RHEL 5,6 & 8 was short enough to avoid problems due to attrition in enterprise settings (unlike startups which have higher turnover, and not counting bus factors of one - no release cycle can't solve that).
And like others have pointed out, automation doesn't help as much when moving between releases. We have everything configuration controlled with kickstart and ansible and/or docker, and it is great for reproducibility within a release cycle, but it doesn't save much time or knowledge between releases. And Ubuntu is even worse in that regard despite having a shorter release cycle.
It's one of the things that ground Yahoo to a halt. We spent years migrating from RHEL-4 to 6, then RHEL-6 to RHEL-7, and by the time the projects were pretty much complete, the next sunset was approaching. My cynicism comes from seeing the bad things that "Enterprise Linux" enabled there.
Admittedly, Yahoo was an extreme case. It never solved the really building problem - the culture from the early days was to compile, ship and forget. Once a RHEL-6 package was pushed to our dist/yinst system (packages), it would never be rebuilt unless it was 1) necessary, or 2) It was time to try and figure out how to build it on RHEL-7.
A lot of effort was spent in the later years to try and address this (by burning the old tech stack to the ground), but the culture was pervasive for the longest time. If 10-year-RHEL didn't exist we would have been forced to address the building processes.
If it's hard or error prone, then do it frequently until you get the process nailed down.
>That is a good point, however I've not heard of too many cases where organizations intentionally skip RHEL releases. Systems that are being actively developed do regularly upgrade through each RHEL release, and the 10 year support just lets them be lazy about how quickly they do so.
Industry specific - but finance world, we still had straggling RHEL5 machines up until a year or two ago and still have a bunch of RHEL6 machines and have basically NO RHEL9. The vast majority of the machines are all sitting on RHEL7/8.
In 2017 I led a (painful) project to migrate from RHEL5 to Ubuntu 16. Since then, it has been pretty easy to go to 18 then 20, soon 22. The previous migration was in 2010 and was from RedHat 6 (not RHEL) to RHEL5. These projects were for projects that are "ghost deprecated" in that not much time is spent in talking about them but they are critically important to the business, as profit centers and cost drivers, but not the new flashy stuff. So we saved $10s of millions of dollars in hardware savings mostly due to the improved schedulers in 4.0.x series of kernels compared to 2.6.28 kernel. The same image became the basis of the containerized version that was rolled out a bit later.
Oh my sweet summer child. There are plenty of enterprises still on RHEL 6, paying extra to keep patches coming. I bet there are still some on RHEL 5. Large, global companies. And of course there are many environments where software and hardware stay frozen for years. Telecoms, factory floors, automotive, finance, disconnected edge devices of all types... they test the absolute crap out of those systems, and pay for every certification availabl, and then leave them running for a decade.
The ability to stay abreast of major version updates is a wonderful attribute of many environments, but definitely far from all.
Years back (late 1990s) I worked with some mainframe people. They bragged that everything is hot fixable on the fly so they can apply security updates or replace broken hardware without rebooting. Then they admitted they schedule a reboot every 6 months anyway. Turns out the redundant backup power supply failed in at the same time and one hot patch was not applied to startup scripts and it took a week to figure out what was missing so the system booted again. By rebooting every 6 months they remember everything and so can get the system back up.
I probably have some details wrong in the story above. I worked with those people, but never on the mainframe. I think the point stands though, if you don't do something often it can't be done.
This doesn't surprise me. Mainframes aren't just about never failing; they have a whole culture, including ops, around providing availability in ways that actually work.
BLISS is a typeless word-oriented language like BCPL, so I am surprised to see it in this list. Also I am amused to hear VAX/VMS described as a mainframe operating system.
The string handling functions in chapter 20 require the programmer to pass around separate pointer and length arguments, which is not what I would call proper strings.
I've also heard of teams that shut down the mainframe for an hour during a time change. It's an easy way to avoid application issues for a small amount of downtime.
We used to do this on several hpux servers at $dayjob. However 95% of those servers have long since been decommissioned, and the remaining server didn't actually need it to begin with. (It was really anything that had an oddball database that needed it)
Think of all the places Linux runs. Planes, trains, and automobiles. Medical equipment. So many other places. Many places that don't have readily available network access. Yet, many "enterprises" need support here.
If a medical device or train works but needs support for years and years, should someone be constantly updating Linux? What about the software that runs on Linux and is tested there?
Considering just the modern cloud environment really limits where enterprise Linux runs and is useful. And, where there are calls for really long support contracts.
> If a medical device or train works but needs support for years and years, should someone be constantly updating Linux? What about the software that runs on Linux and is tested there?
Yes. If you have embedded software in the field and it is running on hardware that has not reached its EOL, then you absolutely should be fixing bugs and vulnerabilities, doubly so when that hardware is attached to any kind of network, and triply so when the software talks to some kind of cloud services.
For most products, the customer should have the power decide when that hardware reaches EOL. In other words, it should be illegal (and severely punished) to disable or downgrade devices remotely, whether by abdicating the responsibility to maintain their software or by shutting down network services that those devices require to operate fully.
At the very least, that would prevent the proliferation of pervasive networking features that have no business communicating with anyone except their owners.
> If you have embedded software in the field and it is running on hardware that has not reached its EOL, then you absolutely should be fixing bugs and vulnerabilities, doubly so when that hardware is attached to any kind of network, and triply so when the software talks to some kind of cloud services.
Talk to Qualcomm and NXP and MediaTek and Broadcom and STMicroelectronics and....
Seriously, most of this is completely out of the product engineers' control. We can't update the hardware because our vendors (ALL of our vendors) control the kernel selection, almost never upstream, and never keep the board support packages up to date. Never.
IME the only exceptions to that rule are RaspberryPi (kinda sorta), AMD, and Intel. I only recently managed to finally extricate all my projects from Linux 2.6 which I considered a minor miracle. In 2023.
Doesn't matter if the manufacturer provides updates for embedded software if the customer doesn't want to install those updates because they'd have to incur the cost and downtime of retesting and updating their upstream software to address any incompatibilities caused by those fixes. It's more common than you'd think.
> Yes. If you have embedded software in the field and it is running on hardware that has not reached its EOL, then you absolutely should be fixing bugs and vulnerabilities
Ok, I'll be the unpopular person here.
Some bugs, even security ones: are ok.
I know that a very significant amount of software is mature these days, but sometimes upgrading causes different bugs, which are either harder to diagnose or even potentially more deadly.
I work in gamedev though, releases are tradeoffs with which bugs we accept.
It is very frequently the case that fixing one bug will incur several other bugs, it's just a case of understanding if they're worse or not.
For example: I don't care that my TV will have a 1/100 crash when launching the settings. I will just launch settings again-
Coincidentally: having everything constantly updating and internet connected is counter-intuitive. I used to spend entire evenings waiting for my PS4 to install new software on itself, but the PS2 (which contained many bugs!) worked much more often.
> For most products, the customer should have the power decide when that hardware reaches EOL. In other words, it should be illegal (and severely punished) to disable or downgrade devices remotely, whether by abdicating the responsibility to maintain their software or by shutting down network services that those devices require to operate fully.
I mostly agree, but the severe punishment for not keeping software or services updated and running should be to release as much of the software as you own and a full hardware spec, well before you turn off the lights. People have to be allowed to get out of a business.
> whether by abdicating the responsibility to maintain their software
If you bought a product and the vendor said it will be supported for X years with updates — that’s all you get — X years. If you want to keep using a product for X+1 years, that’s fine, but that’s on you. I agree with your sentiment that hardware shouldn’t be disabled, but I don’t expect security updates past EOL. You can’t expect a business to keep supporting a product longer than they originally said they would (without compensation).
But these are enterprise products… support contracts last a long time. More than enough time to migrate (or past the time when you should be migrating). That’s part of why the long term support exists in the first place.
Absolutely spot on here, there's no reason for the obsession HN (and the wider internet) has with upgrades in these kinds of environments. These are not environments where you can tolerate unplanned downtime, this isn't a silly web app running in us-east.
It's easier to do an update with a single security fix rather than an update that rolls in a ton of new functionality that ends up breaking your device. Seen this time and time again with OS/dependency update.
In my experience updates are not forgotten. Even automated.
Upgrades however are a different story. The major version changes require a ton of testing and manual massaging. This is why enterprises like to have that infrequently. For the systems that are easier you can still choose to follow the releases quickly.
Because security patches are being back ported is usually not a real issue.
To ensure that those who possess the certificate, still control the domain.
The main issue is that certificates should really be automated by every web server by default. At least for those with public IP addresses. There are servers like Caddy which implemented it, but it should be basic feature that just works without any additional configuration.
Long term support allows bad behavior but I think it's still useful to reduce the amount of feature/breaking changes happening to software.
Those problems can also be mitigated with mandatory environment rebuilds which is trivial for a lot of setups with infrastructure as code.
At the extreme end, you have Kubernetes/CNCF where 6 months go by and you're many versions behind with a huge changelog of breaking changes you have to fix first. Stable APIs and stable ABIs are very useful here (which enterprise Linux provides).
Some time ago I was listening a guy talking about (operational) risk management in financial industry. One of his main points was that the systems in financial organizations should not be completely bug-free and automated. Because when something eventually happens, if there is no-one who has had to fix issues in the systems regularly, there is nobody around who can fix the system efficiently.
An example of argument that belongs to a weird class of arguments you at the same time want to agree and disagree.
> There is another problem that wasn't covered in the article. The 10+ years of stability leads to behaviors and outcomes that remind me of the long-lived SSL certificate problem. Updating is done so infrequently that the "how?" is forgotten. As the 10 year support limit approaches, most of the old team members who did it last time are gone, tech debt is through the roof, few people know where everything is or how to build it, and so on.
This is known as the out-of-the-loop performance problem. [1]
We truly live in amazing times! We have language models that sound human and internet from space, but never bothered to schedule that script for updating TLS certs. Or put it in version control for that matter.
Probably more of an ass-covering situation: bugs are known in older software and not doing something about a known risk is obviously bad. otoh not mitigating an unknown and unidentified risk (bugs in new software) is ok.
See. I used to think similarly. Newer software must be more secure, because we definitely know about all these problems and how to deal with them.
What that doesn’t consider is that the vast majority of programmers simply don’t care to keep up with security concerns. And that’s why two of the most dangerous, easy to exploit, easy to stop attacks out there (SQL injection, XSS) are still among the most prevalent.
As long as features are growing, the claims that new is more secure is difficult to believe.
that's why the whole rewrite in $languageX is so annoyingly frustrating when touted as fixing so many issues just because it uses $languageX. How many bugs not present in original version get introduce is always swept under the rug of the $languageX evangelists
I could imagine an updated take on “Enterprise Distro”, in order to take advantage of the constantly shifting sands, being by and large defined & differentiated by some fantastical gauntlet of automated V&V pipelines: big piles of test suites, fuzzers, prop tests, trace-based testing, etc. etc.
More or less setting up a rigorous de facto compliance regime and making the arms race for enterprise distros be about who can establish the best regime and best gauntlet automation.
If you have to run an "Enterprise Linux", you are probably required by regulations or compliance to have a support agreement in place. Whether it's RedHat or Oracle or something else, security is going take third place to compliance and stability.
Not necessarily. RHEL-clones are ubiquitous in high-performance computing. Clusters and super-computers are hard to setup, consist of hundreds or thousands of nodes, and are used by scientists who are not programmers by trade. The RHEL-clone model works well for them, as they can't expect their users to keep code up to date with changes in the OS, they are used by known, mostly-trusted users, and the systems they run on usually stay fixed their 5-10 year lifespan. Having a single supported OS for the entire lifetime of the system saves a lot of work.
"regulations" or "compliance" aren't the only reasons to value stability. If you have a $2 billion facility or a $200 million piece of industrial equipment, you're going to place value on stability regardless of what the regulations say.
What's missing is an example of a case where there is a potentially relevant exploit:
Many times, even Debian has avoided being hit by a vulnerability, just due to a good wack of issues being in fresh code changes.
And many times, even when the vulnerability has been a long existing one: RHEL is pretty focused on defense-in-depth, through compiler flag choices, SELinux setups, etc.
What is Oracle going to do with the userland/RPM compat with RHEL? It's wonderful that they try to ship every fix from the LTS kernel tree and that UEK users already don't have an expectation of a 100% identical kernel. But you still need userland compatibility, which will be much harder to ensure given the sheer number of RPM packages out there.
The Linux kernel has a pretty strict "don't break userland" policy, and if it does it's a bug, so I wouldn't expect using a newer kernel to be a problem.
No, I meant to ask how Oracle is going to ensure that they deliver 10s of thousands RPM packages to be sufficiently compatible with those that RHEL ships (following the rhelgate or whatever we should call it).
> Rolling release distributions like OpenSUSE Tumbleweed follow upstream much more closely while still maintaining stability through thorough automated testing.
That is different kind of stability. No rolling release distro offers API (nevermind ABI) stability the way rhel etc do, which is the big selling point. You can install updates on enterprise distros without needing to worry too much about your application breaking because someone decided to have some fun with the API.
>This collection of software will remain locked at its specific version throughout the lifespan of that Enterprise Linux distribution release – which is often 10 years or more.
Is this 10 year assumption true?
Are people running 10 year old versions of operating systems today in non-safety or non-high security aspects?
10 years ago today is when the following were roughly released:
Absolutely. RHEL 7 (and downstreams) has maintenance support for another year, so there are many systems that administrators and now looking at upgrading. Speculatively, Red Hat timed the source availability change strategically to coincide with this epoch and force these administrators into a tight spot and hopefully get more subscriptions.
Should these systems have been upgraded sooner? One year has usually allowed plenty of time for testing and rollout of the next major release.
They want to install security patches, and not change anything else. 8 years ago was Red Hat 6.7 and 7.0.
Windows Server 2012 R2 and 8.1 were 10 years ago.
I think there is something much more fundamental going on...
1. Maintenance is expensive.
2. Maintenance of living systems is even more so.
3. People hate spending money on maintenance.
4. You reap what you sow.
We can't even get bridge or road maintenance done correctly in the US [0]. Why should we think that IT systems would be any different when we just don't have that culture?
This is not a Linux problem - it's a culture problem (and probably goes well beyond the US).
Yeah. Us programmers' minds keep looking for a 'solution', but this is one of those solutionless paradoxes modern technology brings. If technology is developing and improving, it will need maintainance. If it's mature and stable, it ideally won't need maintaince or there will be enough knowledgeable workers who know how to do it, and their knowledge won't get obsolete in a short time. Unfortunately, and fortunately, technology is improving.
Talk to the CFOs and CTOs. They fund the hardware refresh cycles and have bonuses attached to regular appeasement of shareholders, the market as well as auditors and controllers.
The initiative begets the spend. The spend and roadmap stretches 10-12 years, chassis fans and PSUs have MTBF that conveniently coincide with Moore’s law and the dark arts of finance, where ownership is good but only to that boundary.
How is Kernel live patching as an approach for reaching the middle ground for enterprise applications? Amazon linux 2023 makes use of it.
Note how “Ksplice was the first project for live patching the Linux kernel; however, ksplice was sold to Oracle and eventually changed to a closed-source tool.”
Okay, that's fair, but do you kind of see how a shiny new domain name & blog with exactly one post and an anonymous author saying you can trust them because they're not affiliated with Oracle...
Can you see how that appeal for trust hasn't been built on firm ground?
Sure, I suppose. I've never been called a sockpuppet before so I'm simultaneously insulted but also...honored in an odd way?
Anyway, I'm not sure what else you'd have me do here. We all have to start somewhere. I put my thoughts out there under something I directly control and I'd rather people focus on the central thesis of the thoughts being proposed over resorting to personal attacks.
I'll continue posting small essays on relevant topics under the domain.
I didn't mean to say you were wrong in what you chose to write, but also that it wasn't wrong for someone to be highly skeptical of its authenticity. Write what you want and people will take it as they will, I just wanted to point out that how the GP comment took it wasn't without some basis.
As to the merits of your thesis, I'll outline my understanding of it & my thoughts in response:
1) You're proposing OEL as an alternative for RHEL (and other such offerings)
2) People might object to it based on Oracle's reputation (well deserved)
3) People should put that repulsion aside though because...
4) Oracle really is letting people use it for free so users don't have to enter into a $ relationship with this predatory company
However:
People choose RHEL (or any paid enterprise distro) primarily because of the support contracts that guarantee service. This undermines your appeal in #3 & #4 above because it requires the exact thing you're saying users can avoid to overcome their repulsion.
Also, if enough people start using this distro then I have no doubt that Oracle will at some point switch things up and try to monetize that userbase under threat of crippling legal bills.
I say all of this as someone who has actually been deposed by lawyers in a lawsuit with Oracle, where events leading up to the lawsuit were very much a bait-and-switch. I won't bog down this comment with the details but if you're interested in the exact nature of that bait & switch let me know and I'll comment in reply-- it may change your mind (or not) about trusting Oracle not to do something like that with OEL
>I didn’t see a particular appeal for trust in the OP
It's implicit within the statement that they don't have a conflict of interest. Trust is precisely what conflicts of interest are about, where their presence often (maybe not always) introduces the need to be more skeptical of the person's motives, i.e., less trust.
You deleted your other comment replying to me, I'm not sure why-- there was nothing wrong with questioning how I initially chose to respond, so I'll paste what I was writing back to you on that topic here:
Your original comment said 1) you didn't see an appeal for trust and 2) went on to make broader statements that trust isn't necessarily required when considering an opinion (on it's merits, independently of high/low trust, was my interpretation of what you wrote)
#2 Does not modify #1 in a way that would prohibit fair treatment of #1 when considering it alone. #1 is a straightforward question about the content of the article.
I didn't choose to address #2 because I agree, trust is not a requirement when considering someone's reasoning. In agreeing with you, there wasn't much for me to add.
I deleted my first response, because on further reflection it seemed you were more intent on defending your position, than having a friendly dialog.
In friendly dialog, it's quite common to make agreements equally clear to disagreements.
In addition, ghosting your conversation partner's agreed to points is indistinguishable from ignoring points you simply don't have a good retort to. If you were profusely nodding in agreement while reading parts of my comment, that was unfortunately invisible to me.
In addition, my original post had the phrase "particular appeal for trust", followed by the very relevant sentence "Not mentioning anything about non association would have seemed weirder." You also ghosted that in your response.
So you simply ignored my entire larger implication that it's not worth talking about the trustworthiness of the author of the OP, since there's no tangible evidence one way or the other.
That seemed to be what we were having, up until this reply from you where you seem to accuse me of conversing in bad faith or rudeness. My apologies if that was not your intent and I am reading too much into this accusation of ghosting. I even took the time, after seeing your reply before deletion and trying to address its contents, to still reply elsewhere to let you know, in essence, I found the deleted comment to be valid and deserved a reply of its own. Far from ghosting you, I un-ghosted your ghosting of me! :) Okay, now I'm having a bit of fun with the "ghosting" idea because I really do view this as a friendly dialogue as well, and I hope you take it in the spirit intended. So let me explain myself further, both in my original original comment and in reply to what you've written here.
1) My original comment: I merely chimed in to reply to the author of the linked article, who felt personally attacked by the previous comment by someone else, to give a perspective of why someone might be skeptical of their motivations. As you say there is no tangible evidence either way but the circumstances of their post-- new domain, anonymous, first-post... those do provide grounds for an inference that the blogger might not be authentic. I am being very careful to say might, because it is that benefit of the doubt that the harsh comment directly accusing sock puppetry failed to give to the author, an accusation I think was over stated and inflammatory. I attempted to temper that harshness with a less accusatory view, and thereby begin engaging the author-- if they chose-- in a more level headed conversation. They kindly took up that implicit invitation and I tried to provide a more thoughtful reply regarding the contents of their blog post than the original accuser did.
2) You replied to this by addressing only part of my comment-- that about the appeal for trust-- rather than address my underlying claim about why a person might be skeptical of the motives of the author. You ghosted me on that point! (again I jest) I don't see anything wrong with you choosing to only address the portion you disagreed with-- if I was mistaken in reading an "appeal" into the author's post then why not discuss that mistake before moving on? As you say, it's a dialogue, there's no need to assume I considered it ended and wouldn't engage in more depth on that or other points if you were willing & continued to reply.
3) Now I'll explain why I didn't consider the "...would have seemed weirder" context necessary to explicitly address when replying to you: We were then discussing whether or not an appeal had been made, and that is a question that is wholly separate from the question of it being weird to not make such an appeal. The appeal for trust can be there (or not) regardless of whether or not its absence would would be strange. Its existence is independent from the issue of it being expected or normal to include it. "...weirder" may be relevant in the broader conversation regarding the pragmatics involved (I use that in the linguistic sense of meaning) and logical inferences that might reasonably be made or not made by the author claiming a lack of conflicts of interest.
Side note: I hope you don't see the above dissection as adversarial. I am dissecting my own responses to explain them further, and dissecting yours in an effort to put what you wrote into my own words, in my own understanding, to make it clear what my understanding was, in case I did not accurately interpret your intended meaning. I enjoy questions of pragmatics & meaning in discourse, intended message vs. received message. In that vein, we've opened up 3 topics much different than the author's enterprise linux argument. I think we can leave that to the side as we pursue broader questions of meaning and trust. I'll be a little more comprehensive in this reply because we arrived at a point where you believed that my...
First of all: Thank you for taking considerable time to compose your thoughtful in-depth response.
I honestly didn’t think, that my first point had ignored anything in your first post about trust, but I see now that I misinterpreted your intent in that post. And so I happily concede that I inadvertently “ghosted” that intent (I’m glad you appreciated the weird sense of enjoyment I had in appropriating that term into an uncommon context - and that you ran with it in your response!).
Your point about mentioning the less charitable route in my thoughts is valid, yet in my defence, I typed up that post articulating why I had deleted the other post. So I shared what were originally intended to remain private thoughts, not to attack, but to explain the post deletion. In my mind, deleting that post had been the more charitable thing to do. But when you took it upon yourself to resurrect it in some way, I thought it would be more charitable to explain the deletion very honestly - including my up to then private thoughts - rather than to stay silent and making the ghosting permanent.
Sharing those previously private thoughts was intended as an act of charity, much in the sense that spouses or friends explain to each other why they were angry, rather than keeping it to themselves. And since this forum doesn’t have private chatting, our exchange is now ingestible by all of those present and future ML/AI bots for better and worse :-)
Human (and pretty much any) communication is indeed a long and deep topic that much smarter minds than mine have not “solved”. Suffice it to say, that it’s a miracle that it ever works at all.
And in this specific case - somewhat counterintuitively - our disagreement also unearthed underlying common ground in carrying the conversation into philosophical meta-topics far removed from the original RH/IBM and Oracle.
On the old Slashdot we probably would have been buried in off-topic ratings. Here on HN (for unrelated reasons) we’re buried under a long flagged and dead post. Douglas Adams would have been able to spin that fact into an entire chapter, if not into a full book!
Thanks again for your investment of time and emotional energy. I appreciate that very much and also look forward to future exchanges! Warmest regards from this anonymous coward!
This whole model is in decline. Most companies have moved to the cloud and are using kubernetes. If you're running servers a datacenter then you might use enterprise linux. You are probably using some kind of virtualization like ESX or Nutanix. You might be using k8s there too. The age of running your own linux servers is ending in a way.
Most unicorn companies with an average half-life measured in months, maybe. Most companies worldwide across all industries, probably not. The pendulum will swing the other way sooner or later anyway as it always does.
> Oracle has also publicly continued to commit to keeping “…the binaries and source code for that distribution publicly and freely available”. At this point their distribution has been around for 17 years, and they’ve never tried to restrict access to their source code in that time.
I’ll be impressed when they open source Oracle DB and let Microsoft fork it and sell support for it.
It’s pretty easy to commit to “open source” a copy of RHEL. Especially when linux isn’t core to their business but a bait and switch to get their customers on Oracle’s platform.
For me was when I was trying to incorporate Redhat, Intel and Mirantis as a secure platform for on-prem-cloud as a service offering - and we were setting up the environment with RH, Intel and Mirantis (I was the Mirantis tech PM) - and it was a nightmare of IBM-esque beurocracy.
it reminded me of the nightmares of installing equipment in an IBM/Intel DC in the 90s...
Too fn many non-technical stakes in an undefined landscape and RH attempting to "feel enterprise" with Intel - and it was a disaster. (two FN months to get IP allocations????)
Yeah - I signed off on redhat way before this - but this is what made me hate RHEL.
They got too smarmy, just as LinuxCare did...
But to answer the Q -- They attempted to get 'too enterprisey with it' and emulate those who they were previously trying to take down.
We use Oracle Linux where I work because we used to use Solaris. At one point it looked suspiciously like Redhat had explicitly removed support for some Oracle hardware from their kernel and the UEK saved us. Besides the kernel they also offer a few other extras and optional newer versions like dtrace, support for some extra filesystems and newer KVM. I get the impression that the fact that they're underdogs in the Linux market helps to keep them honest. Much of their key staff are clearly open source advocates even if Larry only cares about money. And while the Oracle support is not up to the standards of the old Sun support it really isn't a bad choice. And it'd be easy to switch away if we ever need to.
>Redhat had explicitly removed support for some Oracle hardware
Sure, if you already had Oracle as a hardware vendor then the cat is out of the bag, barn doors open w/ horse gone. After that, using their distro isn't going to increase your attack surface much, the killer is already inside the house.
>Oracle Linux... set aside your automatic (and justified) repulsion
I can't for two reasons:
1) The only way I avoid the repulsive side of Oracle in this is if I don't care about having a reliable guaranteed enterprise support contract. No one is going to have more expertise in how this distro is built & functions than Oracle, and that's the no-go zone.
2) It's not just about the current status of the project but also the future and what happens if this author is successful and lots of people switch over to Oracle Linux. Nothing stops them from switching things up at short notice (unless I pay them for a service guarantee!) and pulling the rug out from underneath the whole thing. And the more that people adopt it as their distro of choice, the more likely Oracle is to look around for some way to monetize it at gun-point (well, crippling legal bills at least)
Lets not forget as well that as pointed out in this article OEL is essentially just a play at undercutting RHEL and is the main reason (imho) that RH/IBM took the steps they have over the last few months is because of OEL undercutting them (I think it had VERY VERY little to do with Rocky/Alma)
I think that one of the biggest problems for my use case is license management for things outside the data center. With windows I can buy a pro license and have it attached to the hardware. I never have to worry about tracking or managing the license. With RHEL I have to make sure that the computer is registered and logged in to the license correctly. There is a lot of extra work for things like developer workstations or instrument controllers etc. If I could just buy a license of a major version of RHEL for $200-$500 and have it locked to the hardware like windows that would relieve a major administrative burden.
As a former hobbyist that is now working a lot with RHEL as a sysadmin, I was really surprised to learn how little advantage there actually is with buying Red Hat Support.
You are always limited to their opinionated decisions on what your deployment should look like* (or risk losing support), but at the same time, the support you actually get is next to none.
If I can't fix it myself, it ain't gonna be fixed.
At this point, I don't know what we are paying for anyways.
*As an example, more recent versions of RHEL only allow the use of NetworkManager for permanent network configuration. In a production hypervisor system, NM is completely unsuitable in my opinion. It's full of footguns and that will bite us at some point
> At this point, I don't know what we are paying for anyways.
You get to shift your blame on someone else. It's a commercial "covering your behind as a service", that you just blame $vendor when things go bad. It's the game big corps play, sometimes called "compliance".
Erm, enterprise Linux systems backport major new features all the time. The version number of some software means very little. This article's whole premise is flawed.
232 comments
[ 34.8 ms ] story [ 2774 ms ] threadI did not see that twist coming.
Strangely, the author makes a good point. Oracle or not, sticking closer to the upstream kernel is not a bad way to manage an “enterprise” kernel. Maybe we’ve all been too willing to sit back and accept how RHEL does kernels as “the way”.
Linus' "security problems are just bugs" approach has been worrisome.
To be clear, I only used Oracle Linux briefly while playing with the free tier of Oracle Cloud.
Edit: sorry, but I don't get it, please break it down for me. Latest UEK 7 is based off 5.15 LTS ( https://docs.oracle.com/en/operating-systems/uek/ ), that puts them at branching off a kernel released on 2021-10-31 (according to https://en.wikipedia.org/wiki/Linux_kernel_version_history#R... ). How is that even remotely 4 weeks behind master?!
Edit 2: my bad, I didn't read the "behind the tip of the LTS tree" part carefully. So, the difference is that Oracle tries to ship all LTS kernel updates, while RH tries to cherry-pick critical security and bug fixes only?
Most of the RHEL kernel is a few months behind upstream despite the old version. The next minor release of RHEL, to be released in November, probably will have features up to 6.3 for many subsystems and bugfixes up to 6.5 for example.
But you're still beholden to what the Red Hat maintainers are pulling in and focusing on. It's still not a general follow upstream wholesale.
You can see and track what UEK is doing by looking at their Github: https://github.com/oracle/linux-uek/commits/uek7/u1
Aren't they?
This is probably a lot better than expecting it to be possible to maintain a secure release with a level of compatibility guarantee for 10+ years.
Maybe the compromise solution is to use newer kernels but keep certain features turned off until they "bake" properly.
I think we need a fully open source alternative to RHEL not bound to any company. Something akin to the Debian project that can serve as an upstream reference distro and repository.
I believe the problem is not that there wouldn’t be open source alternatives but that that’s not what enterprise wants. Enterprise wants a company behind the distribution.
Or am I the only one?
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...
Also kernel updates breaking drivers is a thing.
So there's definitely value with frozen kernel version. And its almost the same value as frozen software. It just works and it won't break after update.
https://news.ycombinator.com/item?id=36599118
Frankly, I don't get it Oracle. I would say the priorities are backwards - JRE free, Oracle Linux paid would make more objective sense (especially because the JRE used to be free... and there's a lot of free JREs out there.)
Also, don't be stupid enough to download the VirtualBox Extensions package at work; or your work will probably get an email from a licensing agent.
There might be different alignments, but you simply cannot write stuff like this and get away with it if any of your superiors disagree.
That page literally admits Oracle is an asshole company
Place I used to work blocked the virtualbox home page because exactly this had happened.
https://www.youtube.com/watch?v=-zRN7XLCRhc&t=1981s
> "As you know people, as you learn about things, you realize that these generalizations we have are, virtually to a generalization, false. Well, except for this one, as it turns out. What you think of Oracle, is even truer than you think it is. There has been no entity in human history with less complexity or nuance to it than Oracle. And I gotta say, as someone who has seen that complexity for my entire life, it's very hard to get used to that idea. It's like, 'surely this is more complicated!' but it's like: Wow, this is really simple! This company is very straightforward, in its defense. This company is about one man, his alter-ego, and what he wants to inflict upon humanity -- that's it! ...Ship mediocrity, inflict misery, lie our asses off, screw our customers, and make a whole shitload of money. Yeah... you talk to Oracle, it's like, 'no, we don't fucking make dreams happen -- we make money!' ...You need to think of Larry Ellison the way you think of a lawnmower. You don't anthropomorphize your lawnmower, the lawnmower just mows the lawn, you stick your hand in there and it'll chop it off, the end. You don't think 'oh, the lawnmower hates me' -- lawnmower doesn't give a shit about you, lawnmower can't hate you. Don't anthropomorphize the lawnmower. Don't fall into that trap about Oracle."
[0] https://www.youtube.com/watch?v=Zpnncakrelk#t=15m16s
LetsEncrypt did us a huge favor by forcing automation vs having the guy who knows how to update the SSL certs every 4.9 years and left 6 months ago. I'd like to see the RHEL stability model go away too and force people to complete their automation and solve the problems of being able to rebuilding on demand - and actually doing it.
(I know, most HN folk are well disciplined but there are a lot of corporate cultures that are not.)
SSLMate is about as old as LE, they both started around the same time.
Whenever there's a new distribution release it invariably breaks a bunch of things with the automation and you spend more time massaging your playbook so it works again than it would have taken to do it by hand
But even still, things like snaps, the way Debian handles system Python, and various little changes that have an outsize effect on automated deployment do cause a good amount of churn with automation.
However, paying an extra engineer to sort through all the changes possibly will.
On the other hand, there's some interesting trends like monokernels and minimal OS images that leverage services running off-machine instead of expecting so many local services removing some of the complexity/volatility (DNS, SMTP, federated login)
Both these things should not be an issue for anyone, just one or the other.
The rolling-release life is that things break constantly, during each week’s upgrade, but only a little bit at a time (and hopefully in staging). I don’t know if this is better for system administration, necessarily, but if you’re used to a stable-release dynamic of heavy discrete breakage and piles of backported patches, then you might be imagining the same scale of breakage every upgrade, which is isn’t the experience at all. So don’t discard the rolling-release option because of this preconception.
If you only update every few years, each update becomes a full project distracting from and conflicting with other projects.
That entirely depends on your operations model. There's a difference between, say, a nuclear power plant and a colo web hosting shop. With the latter, sure, no problem risking "minor" weekly breakage. With the former, I'd much rather have scheduled, heavily tested and carefully monitored maintenance windows.
And HN tends to underestimate the number of places like the former exist. Backbones of global finance and telecom, industrial facilities of all kinds, etc.
And yes, upgrading that is a full blown project.
It is very different from a "living" software environment with ongoing development processes.
Changes to graphics drivers, can, do, and have impacted how things like MRI results get rendered by software. It's going to have at least some networking with the rest of the hospital and difficult to completely airgap, but at the same time you cannot update it willy nilly with the latest and greatest uncertified drivers.
That's precisely where "enterprise" distros are sometimes necessary.
The vendor (Siemens or whoever) should make updating the OS part of their development cycle and certify accordingly. If drivers are relevant part of the product there, they have to take enough ownership on the drivers to ensure they do the right thing.
Another example would be systems setup for CAD. Certain levels of CAD require certified video drivers in order to unsure tolerances are met. Error introduced at this level destroys product and possibly people.
I could be convinced the former were the answer in many cases, but I’ve never seen the argument for that move beyond a bare proclamation like you’ve just made. It may also be that the argument for this is so mired in the particulars of a situation that it essentially can’t be articulated, but for me that’s mutually exclusive with it applying to a broad, vaguely defined classes of deployments like you’ve just done. (A complex argument is bound to produce an intricately shaped class.)
My (admittedly theoretical) fear in my initial comment was that the LTS people read about kernel updates every month, think of the breakage their LTS encounters on every kernel update, and nope out. Yet without the humongous pile of backported patches the LTS requires kernel updates are the most benign thing in the world if you can afford the reboots—a decade of them with literally not a single issue can and does happen.
Do you work in an industry that requires a specific service level to be maintained? How many hours per day? What does downtime cost? What does redundancy cost?
We have situations that cost millions of dollars per hour when downtime occurs. Now how do you protect those systems?
A definition of Hubris is not believing something exists because you haven't seen it before.
Our tasks as Systems Administrators is not to make a career out of updating systems for maintenance patches. Our task is to design systems to minimize updates and then redesign systems to eliminate wasted user interaction and downtime.
There are too many places where the current guard is going to have to die off before they even _start_ automation. So we're looking at 20-30 years.
LTS is the actual sane solution for these places, despite how utterly insane it is.
Folks who know such things better than I do, have suggested that it would've been far better at like a 64-week rollover (or just chop it to 52 and leave part of the code space unused), that way everyone would have to have a plan for it. Nobody could claim they don't expect their hardware to be in use 64 weeks in the future therefore they can ignore rollover.
So how could this be accomplished?
A Nix-OS style approach coupled with an immutable OS core?
It is much harder to offer stability guarantees than to just publish updates in a rolling release fashion. And yet big organizations pay big money for that stability or the support to poke an enterprise software provider to get stuff working for their needs.
The issue is that a rolling-release approach does not have stability guarantees and forces everything to be upgrading all the time.
This does not work very well if you have specialized hardware or scientific equipment. If the drivers for your lab equipment work with a given release of an enterprise linux, you can't just jump on the next release until you have working drivers ready.
The same is true if you are working with some enterprise software which is only certified to work with a given release of an enterprise linux. Would you really want to run business critical software on a version of the operating system which is not (yet) supported by the vendor?
Also shout-out to Arch ... I've been using it since ... forever and never really had an issue in update.
Rolling updates mean that I have to carefully choose what to install in order to keep the maintenance costs down. This has the side effect of reducing the attack surface.
I also manually review updates, which means that I keep up with the news in OS land.
I have to reboot once in a while because of updates, which means that I test resilience of my infrastructure.
By comparison, RedHat stack at work feels creepled and ancient.
You need a RedHat account (read: subscription) for pretty much everything, even the most basic documentation or downloads and yet my bugs in their bugzilla linger for months with none even trying to reproduce, let alone fix.
If you dont have comprehensive integration tests it's less risky just to not upgrade unless you really need to.
Exactly. Most of us do not have sqlite-level (https://www.sqlite.org/testing.html) testing so we just do the best we can with the resources we have.
This tends to make most "enterprise" shops risk-averse and the motto becomes "if it's not broke, don't fix it".
How does Qubes OS work with drivers for specialized hardware such as scientific lab equipment?
Qubes OS doesn't really solve much in regards to stability and work required to update in a professional setting though, I'd say.
Using a GPU passthrough indeed decreases the security, but it is still much more secure than anything else. More details: https://groups.google.com/group/qubes-devel/browse_frm/threa...
> Qubes OS doesn't really solve much in regards to stability and work required to update in a professional setting
Couldn't disagree more. All software runs in VMs. The Admin VM never goes to the Internet or runs anything. Therefore it's less necessary to update and reboot it: https://www.qubes-os.org/doc/supported-releases/#note-on-dom...
All VMs are easy to backup/restore in a few clicks; cloning for testing and upgrading are amazingly smooth. All this with a great GUI.
Regardless, even with the ease of virtual machines, Qubes OS doesn't really solve problems involved in professional management, nor should it be considered for that given the overhead of the system. It's a neat system, and for general purpose use it's pretty cool, but for stability and work required to update a system, it really doesn't.
Sure, it makes general use-cases pretty easy to update, but those aren't really much of a problem if you've set up a PXE server and have a base image together with default configuration. The issues occur when you're updating servers, which is what I was talking about, because the end-user isn't that important in this context.
Whether you have it running in a hypervisor or bare metal, it all comes back to properly configured backups. Qubes OS doesn't solve this, nor is it meant to. It increases security at the cost of convenience and complexity. Nor does it solve stability in a professional environment, because while it does give you an isolated OS per application or stack of applications, you've now increased your maintenance surface. Servers need to be updated, within a reasonable time frame of ones being provided, and general computing often requires more of that too.
And at the end of the day, while I do like Qubes OS and do like virtual machines, they're not the be all, end all, in regards to security. Exploits exist, and as with all things, the more common they become, the more will be made.
I do still hope for a system like Qubes OS in the future, just not Qubes OS.
In this model, what happens when the next Python2->Python3 breaking change comes along?
Also, Enterprise Linus is one of the reasons (definitely not the only) that the migration took such a long time. Too many enterprise shops that stuck with Python 2 because it's the lazy thing to do. The tech debt grows every year you don't move with the ecosystem.
How far down does "own your application environment" extend? How about libc? What is the role of the underlying OS?
It depends on the needs of your application.
>> How about libc?
If you need to make sure the underlying libc has what you need, you must either bring your own libc or have sufficient feature test macros and adapters to account for possible differences.
>> What is the role of the underlying OS?
It depends on what the application requires. What operating system features, if any, do you require? Do you have any timing or scheduling requirements that are sensitive for your application? Do you need real-time responsiveness?
How does the operating system handle failure scenarios? What guarantees, if any, does it make when hardware fails? Is it okay for your application to crash if a portion of the computer's memory or disk borks?
Ideally none, with scratch containers for applications and the bare minimum running under your orchestrator which becomes your main interface.
Main issue is drivers, of course. It's hard to beat Linux. It contains open source drivers and server vendors usually target Linux and Windows with their driver efforts.
As for 10% security - it’s more than 10%. Take my same example, the Switch. No bugs have been found to launch unapproved software in the last 4 years. There’s always the Secure Boot bug by NVIDIA in earlier consoles, but not even a WebKit bug will get you homebrew on a Switch. Kind of a big deal…
Another example of this would be Microsoft’s experiments with what would happen if an OS was built with all apps running in managed code - no compiled apps. Performance cost? They got it down to just 7% (though, admittedly, Midori never shipped, but it did host Bing in a few countries for a few years.)
The five year reign of RHEL7 was too long and did result in the very issues you bring up, but the ~3 year duration of RHEL 5,6 & 8 was short enough to avoid problems due to attrition in enterprise settings (unlike startups which have higher turnover, and not counting bus factors of one - no release cycle can't solve that).
And like others have pointed out, automation doesn't help as much when moving between releases. We have everything configuration controlled with kickstart and ansible and/or docker, and it is great for reproducibility within a release cycle, but it doesn't save much time or knowledge between releases. And Ubuntu is even worse in that regard despite having a shorter release cycle.
Admittedly, Yahoo was an extreme case. It never solved the really building problem - the culture from the early days was to compile, ship and forget. Once a RHEL-6 package was pushed to our dist/yinst system (packages), it would never be rebuilt unless it was 1) necessary, or 2) It was time to try and figure out how to build it on RHEL-7.
A lot of effort was spent in the later years to try and address this (by burning the old tech stack to the ground), but the culture was pervasive for the longest time. If 10-year-RHEL didn't exist we would have been forced to address the building processes.
If it's hard or error prone, then do it frequently until you get the process nailed down.
Major life lesson -- practice makes perfect.
The average person could practice violin for 500 years and never be invited to play Carnegie Hall.
I assume you mean major releases. Less common than minor releases but, especially for air-gapped equipment, it's not that unusual.
Industry specific - but finance world, we still had straggling RHEL5 machines up until a year or two ago and still have a bunch of RHEL6 machines and have basically NO RHEL9. The vast majority of the machines are all sitting on RHEL7/8.
The ability to stay abreast of major version updates is a wonderful attribute of many environments, but definitely far from all.
I probably have some details wrong in the story above. I worked with those people, but never on the mainframe. I think the point stands though, if you don't do something often it can't be done.
Also Pascal and BASIC compilers with several extensions, e.g. VMS Pascal and VMS BASIC.
I always call them mainframes, regardless if the name is micro or whatever is the pedantic nomenclature.
The string handling functions in chapter 20 require the programmer to pass around separate pointer and length arguments, which is not what I would call proper strings.
Only half joking.
A mainframe has hardware different enough to require a different approach to the OS.
Of course a modern variant of System 390 happily runs tons of Linux VMs.
If a medical device or train works but needs support for years and years, should someone be constantly updating Linux? What about the software that runs on Linux and is tested there?
Considering just the modern cloud environment really limits where enterprise Linux runs and is useful. And, where there are calls for really long support contracts.
Yes. If you have embedded software in the field and it is running on hardware that has not reached its EOL, then you absolutely should be fixing bugs and vulnerabilities, doubly so when that hardware is attached to any kind of network, and triply so when the software talks to some kind of cloud services.
For most products, the customer should have the power decide when that hardware reaches EOL. In other words, it should be illegal (and severely punished) to disable or downgrade devices remotely, whether by abdicating the responsibility to maintain their software or by shutting down network services that those devices require to operate fully.
At the very least, that would prevent the proliferation of pervasive networking features that have no business communicating with anyone except their owners.
Talk to Qualcomm and NXP and MediaTek and Broadcom and STMicroelectronics and....
Seriously, most of this is completely out of the product engineers' control. We can't update the hardware because our vendors (ALL of our vendors) control the kernel selection, almost never upstream, and never keep the board support packages up to date. Never.
IME the only exceptions to that rule are RaspberryPi (kinda sorta), AMD, and Intel. I only recently managed to finally extricate all my projects from Linux 2.6 which I considered a minor miracle. In 2023.
Ok, I'll be the unpopular person here.
Some bugs, even security ones: are ok.
I know that a very significant amount of software is mature these days, but sometimes upgrading causes different bugs, which are either harder to diagnose or even potentially more deadly.
I work in gamedev though, releases are tradeoffs with which bugs we accept.
It is very frequently the case that fixing one bug will incur several other bugs, it's just a case of understanding if they're worse or not.
For example: I don't care that my TV will have a 1/100 crash when launching the settings. I will just launch settings again-
Coincidentally: having everything constantly updating and internet connected is counter-intuitive. I used to spend entire evenings waiting for my PS4 to install new software on itself, but the PS2 (which contained many bugs!) worked much more often.
When you only have a couple of hours free time, spending it updating firmware means that I just don't update or use it.
I mostly agree, but the severe punishment for not keeping software or services updated and running should be to release as much of the software as you own and a full hardware spec, well before you turn off the lights. People have to be allowed to get out of a business.
If you bought a product and the vendor said it will be supported for X years with updates — that’s all you get — X years. If you want to keep using a product for X+1 years, that’s fine, but that’s on you. I agree with your sentiment that hardware shouldn’t be disabled, but I don’t expect security updates past EOL. You can’t expect a business to keep supporting a product longer than they originally said they would (without compensation).
But these are enterprise products… support contracts last a long time. More than enough time to migrate (or past the time when you should be migrating). That’s part of why the long term support exists in the first place.
Where is your other embedded equipment???
I love that movie.
Usually you cannot do the same with a flight computer or a medical device controller.
And HN is full of developers.
On a related note, the new EU Cyber Resilience Act will make upgrades a necessity even in the most constrained environments like pacemakers.
But it can be really hard to change the culture at a place that has a long history of "ain't broke, don't fix it" engineering.
The places that don't do constant upgrades also don't usually have teams looking after that. If they time it right they can do with less people.
Of course it's less reliable not having as much active knowledge but I do think it can be cheaper if nothing goes wrong.
Upgrades however are a different story. The major version changes require a ton of testing and manual massaging. This is why enterprises like to have that infrequently. For the systems that are easier you can still choose to follow the releases quickly.
Because security patches are being back ported is usually not a real issue.
The main issue is that certificates should really be automated by every web server by default. At least for those with public IP addresses. There are servers like Caddy which implemented it, but it should be basic feature that just works without any additional configuration.
Those problems can also be mitigated with mandatory environment rebuilds which is trivial for a lot of setups with infrastructure as code.
At the extreme end, you have Kubernetes/CNCF where 6 months go by and you're many versions behind with a huge changelog of breaking changes you have to fix first. Stable APIs and stable ABIs are very useful here (which enterprise Linux provides).
An example of argument that belongs to a weird class of arguments you at the same time want to agree and disagree.
This is known as the out-of-the-loop performance problem. [1]
[1] https://en.wikipedia.org/wiki/Out-of-the-loop_performance_pr...
Not in my experience. There's still a guy who goes around and updates (manually) all the LetsEncrypt certificates every year.
Sounds like my org :)
LetsEncrypt certificates don't last for one year, they only last for 90 days, no exceptions. You may be thinking about something different.
Although it's possible that newer versions are more secure, I couldn't find evidence of it yet. (Links are welcome).
What that doesn’t consider is that the vast majority of programmers simply don’t care to keep up with security concerns. And that’s why two of the most dangerous, easy to exploit, easy to stop attacks out there (SQL injection, XSS) are still among the most prevalent.
As long as features are growing, the claims that new is more secure is difficult to believe.
With an unknown vulnerability, I don't have any information about it, it could be affecting me right now.
More or less setting up a rigorous de facto compliance regime and making the arms race for enterprise distros be about who can establish the best regime and best gauntlet automation.
> In April 2019, it was announced that feature development for Scientific Linux would be discontinued
Many times, even Debian has avoided being hit by a vulnerability, just due to a good wack of issues being in fresh code changes.
And many times, even when the vulnerability has been a long existing one: RHEL is pretty focused on defense-in-depth, through compiler flag choices, SELinux setups, etc.
And there are bigger differences between the versions of Windows than Linux too.
[1]: https://lkml.org/lkml/2012/12/23/75 [2]: https://news.ycombinator.com/item?id=36805308
That is different kind of stability. No rolling release distro offers API (nevermind ABI) stability the way rhel etc do, which is the big selling point. You can install updates on enterprise distros without needing to worry too much about your application breaking because someone decided to have some fun with the API.
Is this 10 year assumption true?
Are people running 10 year old versions of operating systems today in non-safety or non-high security aspects?
10 years ago today is when the following were roughly released:
- Linux 3.8.
- RHEL 6.4 with Linux 2.6.32-358
- RHEL 7 was Beta with Linux 3.10.0
Are these still alive out there?
Should these systems have been upgraded sooner? One year has usually allowed plenty of time for testing and rollout of the next major release.
RHEL 6 is still under extended support until next year.
https://techcommunity.microsoft.com/t5/windows-it-pro-blog/t...
Though for newer releases they partially reduced it to 5 years.
This is not a Linux problem - it's a culture problem (and probably goes well beyond the US).
[0] - https://infrastructurereportcard.org/cat-item/bridges-infras...
The initiative begets the spend. The spend and roadmap stretches 10-12 years, chassis fans and PSUs have MTBF that conveniently coincide with Moore’s law and the dark arts of finance, where ownership is good but only to that boundary.
Note how “Ksplice was the first project for live patching the Linux kernel; however, ksplice was sold to Oracle and eventually changed to a closed-source tool.”
https://www.redhat.com/en/topics/linux/what-is-linux-kernel-...
I only recently registered the site and set up the blog because I had nowhere else to post it and did not want to use a hosted platform like Medium.
I was watching the recent Red Hat debacle from the sidelines and had some thoughts to put together about the general Enterprise Linux model.
Can you see how that appeal for trust hasn't been built on firm ground?
Anyway, I'm not sure what else you'd have me do here. We all have to start somewhere. I put my thoughts out there under something I directly control and I'd rather people focus on the central thesis of the thoughts being proposed over resorting to personal attacks.
I'll continue posting small essays on relevant topics under the domain.
As to the merits of your thesis, I'll outline my understanding of it & my thoughts in response:
1) You're proposing OEL as an alternative for RHEL (and other such offerings)
2) People might object to it based on Oracle's reputation (well deserved)
3) People should put that repulsion aside though because...
4) Oracle really is letting people use it for free so users don't have to enter into a $ relationship with this predatory company
However:
People choose RHEL (or any paid enterprise distro) primarily because of the support contracts that guarantee service. This undermines your appeal in #3 & #4 above because it requires the exact thing you're saying users can avoid to overcome their repulsion.
Also, if enough people start using this distro then I have no doubt that Oracle will at some point switch things up and try to monetize that userbase under threat of crippling legal bills.
I say all of this as someone who has actually been deposed by lawyers in a lawsuit with Oracle, where events leading up to the lawsuit were very much a bait-and-switch. I won't bog down this comment with the details but if you're interested in the exact nature of that bait & switch let me know and I'll comment in reply-- it may change your mind (or not) about trusting Oracle not to do something like that with OEL
Sometimes it’s interesting to read or hear from someone with a high trust score.
Other times it’s interesting to read or hear a line of reasoning without - or even negative - trust score.
Only going by trust arguably is one of the fundamental problems with tribalism and echo chambers.
It's implicit within the statement that they don't have a conflict of interest. Trust is precisely what conflicts of interest are about, where their presence often (maybe not always) introduces the need to be more skeptical of the person's motives, i.e., less trust.
Your original comment said 1) you didn't see an appeal for trust and 2) went on to make broader statements that trust isn't necessarily required when considering an opinion (on it's merits, independently of high/low trust, was my interpretation of what you wrote)
#2 Does not modify #1 in a way that would prohibit fair treatment of #1 when considering it alone. #1 is a straightforward question about the content of the article.
I didn't choose to address #2 because I agree, trust is not a requirement when considering someone's reasoning. In agreeing with you, there wasn't much for me to add.
In friendly dialog, it's quite common to make agreements equally clear to disagreements.
In addition, ghosting your conversation partner's agreed to points is indistinguishable from ignoring points you simply don't have a good retort to. If you were profusely nodding in agreement while reading parts of my comment, that was unfortunately invisible to me.
In addition, my original post had the phrase "particular appeal for trust", followed by the very relevant sentence "Not mentioning anything about non association would have seemed weirder." You also ghosted that in your response.
So you simply ignored my entire larger implication that it's not worth talking about the trustworthiness of the author of the OP, since there's no tangible evidence one way or the other.
That's why I gave up and deleted my prior post.
That seemed to be what we were having, up until this reply from you where you seem to accuse me of conversing in bad faith or rudeness. My apologies if that was not your intent and I am reading too much into this accusation of ghosting. I even took the time, after seeing your reply before deletion and trying to address its contents, to still reply elsewhere to let you know, in essence, I found the deleted comment to be valid and deserved a reply of its own. Far from ghosting you, I un-ghosted your ghosting of me! :) Okay, now I'm having a bit of fun with the "ghosting" idea because I really do view this as a friendly dialogue as well, and I hope you take it in the spirit intended. So let me explain myself further, both in my original original comment and in reply to what you've written here.
1) My original comment: I merely chimed in to reply to the author of the linked article, who felt personally attacked by the previous comment by someone else, to give a perspective of why someone might be skeptical of their motivations. As you say there is no tangible evidence either way but the circumstances of their post-- new domain, anonymous, first-post... those do provide grounds for an inference that the blogger might not be authentic. I am being very careful to say might, because it is that benefit of the doubt that the harsh comment directly accusing sock puppetry failed to give to the author, an accusation I think was over stated and inflammatory. I attempted to temper that harshness with a less accusatory view, and thereby begin engaging the author-- if they chose-- in a more level headed conversation. They kindly took up that implicit invitation and I tried to provide a more thoughtful reply regarding the contents of their blog post than the original accuser did.
2) You replied to this by addressing only part of my comment-- that about the appeal for trust-- rather than address my underlying claim about why a person might be skeptical of the motives of the author. You ghosted me on that point! (again I jest) I don't see anything wrong with you choosing to only address the portion you disagreed with-- if I was mistaken in reading an "appeal" into the author's post then why not discuss that mistake before moving on? As you say, it's a dialogue, there's no need to assume I considered it ended and wouldn't engage in more depth on that or other points if you were willing & continued to reply.
3) Now I'll explain why I didn't consider the "...would have seemed weirder" context necessary to explicitly address when replying to you: We were then discussing whether or not an appeal had been made, and that is a question that is wholly separate from the question of it being weird to not make such an appeal. The appeal for trust can be there (or not) regardless of whether or not its absence would would be strange. Its existence is independent from the issue of it being expected or normal to include it. "...weirder" may be relevant in the broader conversation regarding the pragmatics involved (I use that in the linguistic sense of meaning) and logical inferences that might reasonably be made or not made by the author claiming a lack of conflicts of interest.
Side note: I hope you don't see the above dissection as adversarial. I am dissecting my own responses to explain them further, and dissecting yours in an effort to put what you wrote into my own words, in my own understanding, to make it clear what my understanding was, in case I did not accurately interpret your intended meaning. I enjoy questions of pragmatics & meaning in discourse, intended message vs. received message. In that vein, we've opened up 3 topics much different than the author's enterprise linux argument. I think we can leave that to the side as we pursue broader questions of meaning and trust. I'll be a little more comprehensive in this reply because we arrived at a point where you believed that my...
I honestly didn’t think, that my first point had ignored anything in your first post about trust, but I see now that I misinterpreted your intent in that post. And so I happily concede that I inadvertently “ghosted” that intent (I’m glad you appreciated the weird sense of enjoyment I had in appropriating that term into an uncommon context - and that you ran with it in your response!).
Your point about mentioning the less charitable route in my thoughts is valid, yet in my defence, I typed up that post articulating why I had deleted the other post. So I shared what were originally intended to remain private thoughts, not to attack, but to explain the post deletion. In my mind, deleting that post had been the more charitable thing to do. But when you took it upon yourself to resurrect it in some way, I thought it would be more charitable to explain the deletion very honestly - including my up to then private thoughts - rather than to stay silent and making the ghosting permanent.
Sharing those previously private thoughts was intended as an act of charity, much in the sense that spouses or friends explain to each other why they were angry, rather than keeping it to themselves. And since this forum doesn’t have private chatting, our exchange is now ingestible by all of those present and future ML/AI bots for better and worse :-)
Human (and pretty much any) communication is indeed a long and deep topic that much smarter minds than mine have not “solved”. Suffice it to say, that it’s a miracle that it ever works at all.
And in this specific case - somewhat counterintuitively - our disagreement also unearthed underlying common ground in carrying the conversation into philosophical meta-topics far removed from the original RH/IBM and Oracle.
On the old Slashdot we probably would have been buried in off-topic ratings. Here on HN (for unrelated reasons) we’re buried under a long flagged and dead post. Douglas Adams would have been able to spin that fact into an entire chapter, if not into a full book!
Thanks again for your investment of time and emotional energy. I appreciate that very much and also look forward to future exchanges! Warmest regards from this anonymous coward!
Yes, and we spin up RHEL in them for some of our application servers.
I’ll be impressed when they open source Oracle DB and let Microsoft fork it and sell support for it. It’s pretty easy to commit to “open source” a copy of RHEL. Especially when linux isn’t core to their business but a bait and switch to get their customers on Oracle’s platform.
For me was when I was trying to incorporate Redhat, Intel and Mirantis as a secure platform for on-prem-cloud as a service offering - and we were setting up the environment with RH, Intel and Mirantis (I was the Mirantis tech PM) - and it was a nightmare of IBM-esque beurocracy.
it reminded me of the nightmares of installing equipment in an IBM/Intel DC in the 90s...
Too fn many non-technical stakes in an undefined landscape and RH attempting to "feel enterprise" with Intel - and it was a disaster. (two FN months to get IP allocations????)
Yeah - I signed off on redhat way before this - but this is what made me hate RHEL.
They got too smarmy, just as LinuxCare did...
But to answer the Q -- They attempted to get 'too enterprisey with it' and emulate those who they were previously trying to take down.
(want some linuxcare stories)
Sure, if you already had Oracle as a hardware vendor then the cat is out of the bag, barn doors open w/ horse gone. After that, using their distro isn't going to increase your attack surface much, the killer is already inside the house.
And the music maker side of me thinks your post has the beginnings of a promising blues lyric. :-)
I can't for two reasons:
1) The only way I avoid the repulsive side of Oracle in this is if I don't care about having a reliable guaranteed enterprise support contract. No one is going to have more expertise in how this distro is built & functions than Oracle, and that's the no-go zone.
2) It's not just about the current status of the project but also the future and what happens if this author is successful and lots of people switch over to Oracle Linux. Nothing stops them from switching things up at short notice (unless I pay them for a service guarantee!) and pulling the rug out from underneath the whole thing. And the more that people adopt it as their distro of choice, the more likely Oracle is to look around for some way to monetize it at gun-point (well, crippling legal bills at least)
It’s more important to them type be able to tell auditors that they are following a mitigation process.
In my case that means all I’m doing is pulling Amazon Linux images and running automatic security updates.
Using Amazon Linux is the path of least resistance because it comes with AWS tooling and it’s already optimized for AWS.
Everything else is basically “who cares?”
You are always limited to their opinionated decisions on what your deployment should look like* (or risk losing support), but at the same time, the support you actually get is next to none.
If I can't fix it myself, it ain't gonna be fixed.
At this point, I don't know what we are paying for anyways.
*As an example, more recent versions of RHEL only allow the use of NetworkManager for permanent network configuration. In a production hypervisor system, NM is completely unsuitable in my opinion. It's full of footguns and that will bite us at some point
You get to shift your blame on someone else. It's a commercial "covering your behind as a service", that you just blame $vendor when things go bad. It's the game big corps play, sometimes called "compliance".