Ceph is not becoming Spectrum Ceph. Along with ODF, it will remain 100 percent open source. IBM said it will assume Premier Sponsorship of the Ceph Foundation, whose members collaborate on the Ceph open source project.
That is the key take-away for me that I hope remains true and does not go the route of Sequent. Should that be the case I also hope the Ceph Foundation keep Ceph alive and independent from IBM perhaps by forking it in the same manor that was done with MySQL (MariaDB and others) when Oracle acquired it.
Red Hat Ceph Storage and ODF continue. The same team will continue to develop newer releases, now with dual IBM and RHT corporate access.
Open Source commitment, upstream first, and foundation continue as before.
I am a member of the team, and while we are certainly adjusting to the news, we are also pretty excited. Red Hat has been generous to Ceph, but it could not take Ceph beyond platform storage for OpenStack and Kubernetes — we want to reach for the full “Linux of Storage” original vision.
Makes me curious what the internal politics is like. IBM does seem to have mostly let RedHat continue working as it did before, not meddling on a broad scale.
Now we have some RedHat employees getting the news they are being transferred to an IBM area. I wonder how that's viewed from the RedHat side.
> Now we have some RedHat employees getting the news they are being transferred to an IBM area. I wonder how that's viewed from the RedHat side.
Shocking absolutely no one. Every acquisition comes with the promise of "X will let Y operate exactly like it always has been, and we'll never steer their direction" and it has never been true.
In this case, I had asked some RH folks about it when the acquisition happened. There was a lot of bravado that IBM would leave it be. This is small in relative terms, but appears to be the first crack. Thus, I'm curious if it's a hot topic there.
This is for sure the biggest crack in the divide. Red Hatters have had multiple occasions for pause, however, beginning with IBM obtaining the list of Red Hat customers and trying to upsell OpenShift/IBM Software customers to Cloud Paks (IBM's embedded software solutions on OpenShift).
Well the presumption is that the culture at RedHat is preferable to the one at IBM. So, no, those two moves didn't cause me to wonder about whether those teams had concern about the move.
Former Red Hatter here. It has also gone the other way, where Red Hat obtained IBM teams that were better suited to live on our side and those products were more successful via integration with other Red Hat products.
It would likely frighten me to know how much JBoss and Websphere is still used.
It seems 15 years ago I was asking ?why aren't we running this in vanilla Tomcat?
I know, I know, it's like "Beetlejuice", you say "Integration Integration Integration" and five salesmen appear to sell you something that provides a bad UI in front of configuration files you'll eventually hand-edit anyway.
Plenty of it in big companies but you won't see a new release. The existing 8.5.5 and 9 will be supported through 2030 but customers are being encouraged to migrate their existing WAS apps to Liberty on OpenShift. There are several migration tools that help analyze the code and runtime for the best way to dissect legacy Java.
At Red Hat very talented people leave money on the table to align their work with their passion and idealism. The main risk with “you work for IBM now” is that this bubble is burst and that the smartest people decide to just go somewhere and get paid.
"We did in fact have high hopes for words like “Nubes”, amazingly the latin word for cloud, also a play on words with “newb” reflecting our introduction of a new architecture for reliably retaining data.
So then it’s time to lock in your domain name. GoDaddy had nubes.com on sale for $800, of course wayyy beyond the budget of this self-funded entrepreneur (and BTW still available. Fortunately GoDaddy’s algorithm sensed our cheapness, suggested alternative “NooBaa.com” for $3.99 per year, and the mouse went 'click!
But....NooBaa didn’t mean anything. Nobody spelled it right. Everybody asked us what it means. We invented creative facts about African tribes. Then we observed that it’s attention-getting. It’s intriguing."
For some reason, technical founders (like me) really love Latin and Greek names for companies. It is such a trap: nobody knows what your company does just from the etymology, and literally all of the URLs for common Latin words are already registered.
I'm certain many are very grateful they didn't lock down "Nubes" since it would definitely have ended up being conflated with a very different market in searches.
It's almost the real life equivalent of Tobias Funke's "A Nu Start".
Oh wow, an odd convergence of two of my IBM careers... Dennis Kennelly was in charge of the part of Tivoli I worked for for a while about 15 years ago and then about 7 years ago I consulted for a year with the Storwize storage folks...
Well, it'll happen, unfortunately. Everything that Big Blue buys will eventually be assimilated. Unfortunately the next phase is usually a few years of tightening budgets, then beyond that 'buying another thing to replace it'.
> Unfortunately the next phase is usually a few years of tightening budgets...
This happens because post-Blue Washing, execs want to quickly drive cash towards shareholders and bonuses towards themselves, and the easiest way to do that is to slash budgets. This translates into losing the best and brightest, and...
> ...then beyond that 'buying another thing to replace it'.
This doesn't happen at 1:1 product granularity as much as many think it does. It is more a big swap of strategic focus, which happens to include large, splashy purchases of other assets that gets lots of attention while existing steady revenue generators get metaphorically tossed aside if not budget-wise, then management focus-wise.
christ the IBM acquisition is turning out to be the WORST timeline.
- Redhat kneecaps centos because some mid manager was worried he might not get this years bonus
- Redhat kneecaps Ansible by making ansible-core and forcing everyone to re-install community plugins they were using from EPEL, or buy the shiny Redhat version of ansible.
- Redhat decides to turn Ceph into yet another garbage storage product with the same exorbitant support fees and contract payments that were the original motivation for an open source storage solution in the first place.
All this is being done under the guise that "oh its not proprietary, its enterprise grade now!" like the very same devsecops engineers and SRE's who made this shit work in the first place arent wholly capable of running it in production without forking over most of their $round of investment to the same company that turned softlayer into a swirling vortex of DDoS gangs spam relays and double extortion rings.
to date it feels like all IBM wants to do with Redhat is turn it into IBM so if thtas all these juggernaut dinosaurs can do im quietly waiting to see Microsoft acquire Ubuntu and quietly turn it into a teams plugin the same way they acquired github and turned it into an outage response simulator for developers.
I was at IBM (research) at the turn of the century after the Lotus acquisition. IBM was all in on Notes and I was developing notes applications for a department. (I actually came to appreciate Notes, and understand that Notes was being sold for mail, when mail is just another notes database, which is why it isn't great...)
But it was clear later when I met from people from Lotus there was tension between the Lotus division which IBM had bought. The cultures didn't mesh well and they resented everything being IBMified. Though from the IBM side it seemed they really were all in on Notes. Really pushing it hard. Lotus's "smart suite" of desktop application not as much.
Another friends company got bought by IBM a couple years ago. It seems they didn't really want it as they let half the people go (core people, not just overhead).
Its kinda sad really. It seems its better after an acquisition to let the company keep doing what it was doing, with minimal changes (excepting removing the corporate overhead stuff).
Notes was always pretty horrible, slow and buggy. It was right handy to make little databases in it, which wormed their way into business critical processes and became dependencies long after their developer left the company. I think this is what kept Notes going, because there was no migration path for all this legacy stuff. We spent years moving away from notes, rebuilding each little 'handy Joe's database' that was caught somewhere in a critical business process manually in modern technologies. And this was after we already discarded the 90% that weren't important.
I'm pretty sure that if not for this we would have moved away in the early 2000s. Instead we waited until the situation became so untenable that we had no choice to invest the millions and years to make it happen. I remember Notes constantly crashing when someone would send me a 'funny' GIF on Sametime, and of course it would take all my draft emails with it because the autosave never seemed to do what it was supposed to.
The only thing I did like about it was collapsible sections in emails. And the fact that you could write really advanced 'agents' for it. I had one that would remove the "return read receipt" flag and insert a button in the email so I could send it whenever I did wish to send it instead.
Anyway, when IBM took over Lotus we were still full-on using it and I was really hoping they'd clean up the product and make it functional. Of course none of that happened and they just let it continue on its path to extinction. I have no idea why they would buy something way after its prime and then just let it wither away.
Oh yeah and the notes mail databases. When there was a new version we'd get tons of questions like "why does my mail still look like the old app but notes itself is new??". Well, click on the upgrade your database. Ooops now it got corrupted completely well good luck bye :D
Seriously why did the email UI look and behaviour have to be embedded inside each user's mail database??? This makes totally no sense and breaks pretty much every software design concept. As well as create tons of vulnerabilities for malware to embed itself. Data and code don't belong in the same place. Though luckily Notes got deprecated before that really became a thing.
We didn't have any of the stability problems though is was kind of "quirky" (I also used it at another big corp which was only mail, which is terrible). I see you are not a fan. I had to use it and by the end kind of liked it. I didn't have to migrate from notes to something else though. (I migrated a non Y2K database to Notes). I suspect "Mail" was the way to get an installed base of Notes applications across a business.
>The only thing I did like about it was collapsible sections in emails. And the fact that you could write really advanced 'agents' for it. I had one that would remove the "return read receipt" flag and insert a button in the email so I could send it whenever I did wish to send it instead.
Thats pretty good.
I see notes as a basically a NoSQL database attached to a primitive gui and a very bad scripting language (lotus script). You could make a little application and deploy it to a team pretty easily. IBM had a bunch of little Notes databases/applications that could be used by the dept to track project and other things. I haven't seen much that replaces its sort of simplicity and ease of use. I guess the web, but it seems it takes more to stand up a custom app these days.
I have no idea how you would export the unstructured data that lives in those databases..
I'm surprised you didn't have the stability problems. So so so many times have I been looking at a standard Windows "OK" popup box with so much java traceback in it that it literally filled the entire screen. Some periods this happened literally every 2 days or so. What helped stability extremely was installing the separate Sametime client and disabling the sametime functionality in Notes itself. However our admins frowned on that for some reason (never found out why). So I always had to get it from dubious sources.
> I see notes as a basically a NoSQL database attached to a primitive gui and a very bad scripting language (lotus script). You could make a little application and deploy it to a team pretty easily.
Too easily, yes. It was too handy. I wonder if this was intentional because these little databases had a way of becoming indispensable over time. Leading to Notes being set in stone as a business requirement.
> IBM had a bunch of little Notes databases/applications that could be used by the dept to track project and other things. I haven't seen much that replaces its sort of simplicity and ease of use. I guess the web, but it seems it takes more to stand up a custom app these days.
Yep.. MS Access roughly had the same and it was actually structured as a decent relational database, not some unstructured mess. But file based, a bit like sqlite with a forms and scripting tool built in. So it was also easy to migrate to SQL Server once an app became so big that it needed a real server. MS even had (has?) a wizard for this. It was really piss-easy to make a little tool in it and its scripting language was easier and known by more people (being VBA which most Excel experts know). It was even quite possible to make decently performing fairly complex apps in it. Not sure how it stands now as I haven't used it in years, but from what I've seen with some quick looks throughout the years, Access has not really changed at all since 2005 or so.
However, Access did not reach this ubiquity and the reason is that MS left it out of the cheapest office editions. Which meant most standard users never had it on their PC and they would have to request it and provide a business reason for the cost.
Notes on the other hand was always there and available, ready for people to tinker with to their heart's desire and create a whole shadow IT world :) Very smart move from Lotus, I have to say.
> I have no idea how you would export the unstructured data that lives in those databases..
Oh don't start... It was terrible. None of it was documented because said "handy Joe"s were usually not experienced developers and had almost always left the company years ago.
Also, most of these had been unmaintained for many years and thus not really fitting business requirements anymore. So lots of ugly workarounds were used, and the business side tried to use the project to inject new business requirements. Which in some cases made sense but in other ways would derail the project completely.
In the end most of them were converted to web using a low-code/zero-code platform.
I remember when Notes came out and they said "It's Groupware!" and I said "what does it do?", and they just said it does Groupware stuff and I said what does it do? Then more marketing that it was groupware and I said WTF does it do???
Regarding Ansible, I still default to installing with `pip3 install ansible`, which installs the "Ansible Community Distribution" (which is still called "Ansible" even though it seems "ansible-core" is also called "Ansible", but there's also "Ansible by Red Hat" which is the 'Ansible Automation Platform' which includes Tower and other related products...).
The ACD (or what I still call "Ansible") includes everything that was traditionally part of Ansible, and then some, just bundled up into one big install instead of requiring you to install ansible-core then pick and choose collections to install on top.
This covers about 99.9% of all use cases (for Ansible for automation work) still, and for anything else, you can add your own roles or install an additional community collection or two.
After much struggle, I reached the same conclusion. And it gets around the problem of downloading collections only to find that they don't support the version of python on the system you're trying to use it on. But since those collections get stored in python site-packages instead of anything in ANSIBLE_COLLECTIONS_PATH, it's still kind of confusing. And I still haven't checked what happens if a teammate accidentally installs the latest collection version locally in their home from Galaxy as well (a stupid default imho).
I understand why collections got split out due to development needs, but the switch over to them has been pretty crappy from a user perspective.
This isn't really reality-based. There are/were a bunch of problems, but it's not these.
> - Redhat kneecaps centos because some mid manager was worried he might not get this years bonus
Red Hat paid (pays, probably) bonuses quarterly, and they were (are, probably) based on stock performance. They are not and never were based on cost saving measures.
The reality is that, probably, CentOS should have never become part of the "Red Hat family". The Chinese firewall concept didn't really work internally, the vast majority of RH engineers ran Fedora anyway, and it was basically just giving CentOS better build infrastructure plus giving customers a way to buy their way out of nonsense situations. The Red Hat support agreement always said "if you're not running RHEL, we'll close your case", which frequently meant customers who ran CentOS in dev/qa/preprod and RHEL in production who were forced to install another RHEL system (which RH made no money on anyway, since they never charged license costs, only subscription fees for updates) just to reproduce an issue so the sosreport didn't say "CentOS Linux ...".
In general, the same engineers who worked bugs (or were escalated to cases) on RHEL were often part of the community of upstream maintainers anyway, and RH's policy was "upstream first, and if it needs an emergency downstream bandaid, take it out once upstream accepts a patch and backport it properly", but it was easier for customers to get attention on problems if they could back it with dollars.
Rocky, Alma, and OEL (however you feel about that) exist anyway and basically picked up where they left off. Stream probably always should have been a thing. It was (probably is) an enormous amount of work to branch, re-sync, and generally re-sync again any given Fedora snapshot to a new RHEL release, and getting it into shape was (probably is) a nightmare, because the difference between the code which is currently running on any given RHEL release and what upstream is is huge. sysvinit -> upstart/sysv -> systemd, for example.
- Redhat kneecaps Ansible by making ansible-core and forcing everyone to re-install community plugins they were using from EPEL, or buy the shiny Redhat version of ansible.
The Ansible leadership was generally terrible about open source philosophy regardless. This is nothing new.
- Redhat decides to turn Ceph into yet another garbage storage product with the same exorbitant support fees and contract payments that were the original motivation for an open source storage solution in the first place.
Open source Ceph still exists. I don't see the difference here. It's just moving it from one independent company to another independent company, at least nominally.
The systemic problem with IBM acquiring Red Hat was in a lack of good answers for many of the questions employees had (policies on open source, WFH, etc), the fact that the purchase was not communicated even to management until news articles broke, a six month cliff for "business critical" employees, and following that, the fact that IBM fundamentally believed that they could do what everyone always thought Red Hat was doing and drive the market, so they were constantly pouring money and engineers into projects that had no chance to succeed in the market rather than adopting and fostering upcoming solutions, and *that* drove engineers out of the company very rapidly.
> Redhat decides to turn Ceph into yet another garbage storage product
Is Ceph still a viable alternative in hardware and operational cost and reliability to HDFS to store hundreds of PBs of data, if not EBs? I kept hearing people recommend Ceph over HDFS, but was unable to find examples of large deployment of Ceph for HDFS-like workload.
> all IBM wants to do with Redhat is turn it into IBM
Reading the histories of both companies, it seems to me that Redhat was always IBM. The acquisition was a marriage of separated twins more than anything else, the dominating culture in both organizations is largely sales and consulting driven, with public strategy not quite identical but strongly rhyming. The technology aspects (while perhaps once important) were far from the prime focus of either organization long prior to the merger. I think of IBM's acquisition more as a desire to access marketing expertise and established sales channels than anything to do with technologies, most of RH's open source technology is after all mostly just an advanced form of disguised marketing
I don't think that's right; Red Hat, culturally, does much more community engagement and open source development. I know several RH developers from working on Ceph-adjacent things and RH has been much more open to contributions and collaborations in academia and industry, plus they funded a significant chunk of research around storage in quite a few labs around the area. IBM never had that kind of culture imo, I would take RH anyday.
From what I know Red Hat devs are also very open in other areas of OS research and development, from the new io_uring, eBPF, etc. IBM is trying to stamp it out, but it's still there.
Maybe GP is referring to IBM having an equity stake in Redhat since the late '90s? Or, maybe because they both generate the majority of their revenue via support contracts with the same sorts of customers buying said contracts?
We manage this POS for some clients, and it's the worst distribution you can have. Stuff you can "just install" under any other popular distro is often silosed off into special packages you need to buy, despise it being OSS and "just available" on EPEL.
I'm working with academics that have code/support from Red Hat, and rumours are that funding is not gonna get renewed. Red Hat has been a very valuable collaborator, very open to the academia side, and funded a lot of cool stuff around the ecosystem, looks like that's all going away now :(
I have a feeling IBM doesn't even know what they want to be themselves, let alone their acquisitions.
What they do now is dabble in a bit of consultancy, a bit of datacenter ops, a bit of software and some servers here and there. And of course all the support on their legacy mainframe stuff. But when they have something that really stands out like the Thinkpad line they divest it.
It's almost as if they don't want to be noticed or something. I don't know.
I never thought their RedHat acquisition was a good thing for RedHat, though I never thought RedHat was a very good company to begin with. When Microsoft acquires Ubuntu (and I'm surprised this hasn't happened yet) they have more good stuff to mess up. Which I'm sure they will. And it seems like Canonical in anticipation of the acquisition has already started going the wrong way themselves.
I've been through an IBM acquisition a few years ago, peak Openstack time. They don't know what they're doing or what they want to achieve. They're trying to buy something that would keep em relevant but the end results are usually just sad. Early version of IBM cloud was unstable to a point I couldn't even use it for personal experiments. Last I heard it's still not very close to being reliable, while their middle management is always obsessing about Gartner magic quadrant or whatever that is. It's a dinosaur firmly on its way to extinction along with everything they buy along the way.
They already stopped building packages for Ubuntu for Ceph...
>to date it feels like all IBM wants to do with Redhat is turn it into IBM so if thtas all these juggernaut dinosaurs can do im quietly waiting to see Microsoft acquire Ubuntu and quietly turn it into a teams plugin the same way they acquired github and turned it into an outage response simulator for developers.
RedHat was eating into their business so they bought it.
> For instance, it is possible to deploy Ceph Quincy on Ubuntu 20.04 LTS via the UCA or on Ubuntu 22.04 LTS via the regular Ubuntu package archive. See Ceph and the UCA for details.
Red hat was a THIRTY FOUR BILLION DOLLAR acquisition. For a company that dresses up free software for enterprises.
No way Red Hat is staying like some gilded ivory tower of open source purity. They are a brand name for selling services and licenses, so if Red Hat still looks after ... wait that was only 3 years ago? Well anyway if Red Hat is still recognizable after that acquisition in some non-IBM way then IBM is showing enormous restraint.
The previous 10 years at IBM were "buy company, attach Websphere to the front of their product names, sell, abandon product". Still better than Computer Associates software, but not much better.
IBM has left us alone. I don’t get paid by IBM, my benefits are Red Hat, My CEO is Red Hat’s CEO and funny enough when we work with IBM we lead the calls because IBM is letting us do what we do best.
I could sense storage was changing. We had product delays, we kept pushing things out to future releases to expand ODF to our managed services. Evrything was always next quarter. Now it makes sense.
If you're going to buy storage from redhat/ibm you might as well skip all the pain and just use GPFS (which is currently branded as spectrum scale)
Ceph is fine, but its a toy.
You can slam hooks into everything, and the raid replacement (which is probably only allow on IBM tin, the bastards.) is brilliant. (rebuilding stuff doesn't take 24 hours)
Want to have a global filsystem, no problem, want to mange wrote clashes? no problem, everything is based on rules. Want to have a storage array in the US, but a copy in the EU, no problems. just set your rules and you're away.
Ceph started out as an object storage system with vaguely universal front end slammed on top. It is a good object store, but it's not a file system, and this shows. It has Forward Error Correction scheme that dealt with underlying hardware failure. so your blocks that you store are reasonably safe. But thats the point, its a blockstore/object store.
Thats the easy part. allowing concurrent access to the same block, managing write clashes, decent cacheing are hard. Even harder when you are doing it with posix semantics.
The main issue with Ceph is that the filesystem is an addon and required a huge amount of resources to make it work properly. (search out the ceph plan for gitlab, before they were bought out by a company that insisted on bringing in sysadmins to sort infra out)
the big and largest elephant is that cephfs is still not really "stable" by that I mean supported fully by a large support company. This means that if something goes wrong, you need to figure it out yourself.
> Ceph started out as an object storage system with vaguely universal front end slammed on top.
The opposite. Ceph started out with the underlying load balancing algorithm (May 2004), then with the plan for the filesystem metadata management (Nov 2004). The block storage was created to support the filesystem, virtual block devices were an afterthought to enable private clouds, object storage was an afterthought in a world where S3 got popular.
Of course, stabilizing the most complex part of the stack, depending on almost everything else already being stable (everything except RBD and the S3-compatible gateway, which both are relatively slim layers on top of RADOS), takes the longest. Meanwhile, business users were very keen on getting the benefits of RBD for "private clouds" and the object storage, so the filesystem was on the back burner.
I've had the misfortune of using ceph both supported and unsupported. IMO, it's a nightmare to use, it constantly has problems, there's tons of bugs in every release. We're currently several releases behind because there's some kind of major problem with each of the newer versions.
It's sad, because there's a lot of genius level people that work on Ceph. But the project as a whole is not high enough quality to be reliable.
I imagine most people that work on ceph (eg, the code, not the rest of the org) will want to move to greener pastures soon. Anyone that wants to work for IBM pretty much already did.
> It's 10x cheaper to build your own infrastructure than use AWS on-demand instances
It’s always cheaper on the bill of materials. Who is going to build and maintain the complex infrastructure. It would be interesting to see the TCO over 5 years.
In our company it certainly is not cheaper. But what they do is ridiculous. They just moved from physical datacenters to AWS instances. Every "server" instance is running 24/7 independent of load, and you have to fill in an excel of 10 tabs to get permission to fire up an instance. In fact, the only thing they changed is the physical rack location tab to an AWS zone one :P
So now we are "in the cloud" but we have almost none of the benefits said cloud offers. And we're using it in such an inefficient way that we're wasting money by the millions. And we kept all the internal overhead we already had when we managed the datacenters itself (that's what that excel is for).
Once we actually start dynamically spinning up instances based on load and shutting them down when not needed, then yeah I could see it being cheaper than having a bunch of metal in a rack somewhere. But I guess we won't get there in the next 10 years or so. We'll probably be all-in on cloud by the time everyone has already gone on to edge computing or whatever the next big thing is, lol.
79 comments
[ 3.1 ms ] story [ 151 ms ] threadThat is the key take-away for me that I hope remains true and does not go the route of Sequent. Should that be the case I also hope the Ceph Foundation keep Ceph alive and independent from IBM perhaps by forking it in the same manor that was done with MySQL (MariaDB and others) when Oracle acquired it.
Open Source commitment, upstream first, and foundation continue as before.
I am a member of the team, and while we are certainly adjusting to the news, we are also pretty excited. Red Hat has been generous to Ceph, but it could not take Ceph beyond platform storage for OpenStack and Kubernetes — we want to reach for the full “Linux of Storage” original vision.
Now we have some RedHat employees getting the news they are being transferred to an IBM area. I wonder how that's viewed from the RedHat side.
Shocking absolutely no one. Every acquisition comes with the promise of "X will let Y operate exactly like it always has been, and we'll never steer their direction" and it has never been true.
You align software to where it works best. IBM is better suited to develop and sell Software Defined Storage than Red Hat.
I saw some people furious because of personal prejudices against IBM but many seem to be looking at this opportunistically.
It wasn't clear which products those would be and when, though.
It seems 15 years ago I was asking ?why aren't we running this in vanilla Tomcat?
I know, I know, it's like "Beetlejuice", you say "Integration Integration Integration" and five salesmen appear to sell you something that provides a bad UI in front of configuration files you'll eventually hand-edit anyway.
Most of the big banks
Ah, here we go... https://blog.noobaa.com/blog/the-noobaa-brand-brilliant-foun...
"We did in fact have high hopes for words like “Nubes”, amazingly the latin word for cloud, also a play on words with “newb” reflecting our introduction of a new architecture for reliably retaining data.
So then it’s time to lock in your domain name. GoDaddy had nubes.com on sale for $800, of course wayyy beyond the budget of this self-funded entrepreneur (and BTW still available. Fortunately GoDaddy’s algorithm sensed our cheapness, suggested alternative “NooBaa.com” for $3.99 per year, and the mouse went 'click!
But....NooBaa didn’t mean anything. Nobody spelled it right. Everybody asked us what it means. We invented creative facts about African tribes. Then we observed that it’s attention-getting. It’s intriguing."
It's baaaaaad.
Technical founders are really a special bunch.
It's almost the real life equivalent of Tobias Funke's "A Nu Start".
Well, it'll happen, unfortunately. Everything that Big Blue buys will eventually be assimilated. Unfortunately the next phase is usually a few years of tightening budgets, then beyond that 'buying another thing to replace it'.
> Unfortunately the next phase is usually a few years of tightening budgets...
This happens because post-Blue Washing, execs want to quickly drive cash towards shareholders and bonuses towards themselves, and the easiest way to do that is to slash budgets. This translates into losing the best and brightest, and...
> ...then beyond that 'buying another thing to replace it'.
This doesn't happen at 1:1 product granularity as much as many think it does. It is more a big swap of strategic focus, which happens to include large, splashy purchases of other assets that gets lots of attention while existing steady revenue generators get metaphorically tossed aside if not budget-wise, then management focus-wise.
- Redhat kneecaps centos because some mid manager was worried he might not get this years bonus
- Redhat kneecaps Ansible by making ansible-core and forcing everyone to re-install community plugins they were using from EPEL, or buy the shiny Redhat version of ansible.
- Redhat decides to turn Ceph into yet another garbage storage product with the same exorbitant support fees and contract payments that were the original motivation for an open source storage solution in the first place.
All this is being done under the guise that "oh its not proprietary, its enterprise grade now!" like the very same devsecops engineers and SRE's who made this shit work in the first place arent wholly capable of running it in production without forking over most of their $round of investment to the same company that turned softlayer into a swirling vortex of DDoS gangs spam relays and double extortion rings.
to date it feels like all IBM wants to do with Redhat is turn it into IBM so if thtas all these juggernaut dinosaurs can do im quietly waiting to see Microsoft acquire Ubuntu and quietly turn it into a teams plugin the same way they acquired github and turned it into an outage response simulator for developers.
But it was clear later when I met from people from Lotus there was tension between the Lotus division which IBM had bought. The cultures didn't mesh well and they resented everything being IBMified. Though from the IBM side it seemed they really were all in on Notes. Really pushing it hard. Lotus's "smart suite" of desktop application not as much.
Another friends company got bought by IBM a couple years ago. It seems they didn't really want it as they let half the people go (core people, not just overhead).
Its kinda sad really. It seems its better after an acquisition to let the company keep doing what it was doing, with minimal changes (excepting removing the corporate overhead stuff).
I'm pretty sure that if not for this we would have moved away in the early 2000s. Instead we waited until the situation became so untenable that we had no choice to invest the millions and years to make it happen. I remember Notes constantly crashing when someone would send me a 'funny' GIF on Sametime, and of course it would take all my draft emails with it because the autosave never seemed to do what it was supposed to.
The only thing I did like about it was collapsible sections in emails. And the fact that you could write really advanced 'agents' for it. I had one that would remove the "return read receipt" flag and insert a button in the email so I could send it whenever I did wish to send it instead.
Anyway, when IBM took over Lotus we were still full-on using it and I was really hoping they'd clean up the product and make it functional. Of course none of that happened and they just let it continue on its path to extinction. I have no idea why they would buy something way after its prime and then just let it wither away.
Oh yeah and the notes mail databases. When there was a new version we'd get tons of questions like "why does my mail still look like the old app but notes itself is new??". Well, click on the upgrade your database. Ooops now it got corrupted completely well good luck bye :D
Seriously why did the email UI look and behaviour have to be embedded inside each user's mail database??? This makes totally no sense and breaks pretty much every software design concept. As well as create tons of vulnerabilities for malware to embed itself. Data and code don't belong in the same place. Though luckily Notes got deprecated before that really became a thing.
>The only thing I did like about it was collapsible sections in emails. And the fact that you could write really advanced 'agents' for it. I had one that would remove the "return read receipt" flag and insert a button in the email so I could send it whenever I did wish to send it instead.
Thats pretty good.
I see notes as a basically a NoSQL database attached to a primitive gui and a very bad scripting language (lotus script). You could make a little application and deploy it to a team pretty easily. IBM had a bunch of little Notes databases/applications that could be used by the dept to track project and other things. I haven't seen much that replaces its sort of simplicity and ease of use. I guess the web, but it seems it takes more to stand up a custom app these days.
I have no idea how you would export the unstructured data that lives in those databases..
True. This is 15 years of frustration talking.
I'm surprised you didn't have the stability problems. So so so many times have I been looking at a standard Windows "OK" popup box with so much java traceback in it that it literally filled the entire screen. Some periods this happened literally every 2 days or so. What helped stability extremely was installing the separate Sametime client and disabling the sametime functionality in Notes itself. However our admins frowned on that for some reason (never found out why). So I always had to get it from dubious sources.
> I see notes as a basically a NoSQL database attached to a primitive gui and a very bad scripting language (lotus script). You could make a little application and deploy it to a team pretty easily.
Too easily, yes. It was too handy. I wonder if this was intentional because these little databases had a way of becoming indispensable over time. Leading to Notes being set in stone as a business requirement.
> IBM had a bunch of little Notes databases/applications that could be used by the dept to track project and other things. I haven't seen much that replaces its sort of simplicity and ease of use. I guess the web, but it seems it takes more to stand up a custom app these days.
Yep.. MS Access roughly had the same and it was actually structured as a decent relational database, not some unstructured mess. But file based, a bit like sqlite with a forms and scripting tool built in. So it was also easy to migrate to SQL Server once an app became so big that it needed a real server. MS even had (has?) a wizard for this. It was really piss-easy to make a little tool in it and its scripting language was easier and known by more people (being VBA which most Excel experts know). It was even quite possible to make decently performing fairly complex apps in it. Not sure how it stands now as I haven't used it in years, but from what I've seen with some quick looks throughout the years, Access has not really changed at all since 2005 or so.
However, Access did not reach this ubiquity and the reason is that MS left it out of the cheapest office editions. Which meant most standard users never had it on their PC and they would have to request it and provide a business reason for the cost.
Notes on the other hand was always there and available, ready for people to tinker with to their heart's desire and create a whole shadow IT world :) Very smart move from Lotus, I have to say.
> I have no idea how you would export the unstructured data that lives in those databases..
Oh don't start... It was terrible. None of it was documented because said "handy Joe"s were usually not experienced developers and had almost always left the company years ago.
Also, most of these had been unmaintained for many years and thus not really fitting business requirements anymore. So lots of ugly workarounds were used, and the business side tried to use the project to inject new business requirements. Which in some cases made sense but in other ways would derail the project completely.
In the end most of them were converted to web using a low-code/zero-code platform.
The ACD (or what I still call "Ansible") includes everything that was traditionally part of Ansible, and then some, just bundled up into one big install instead of requiring you to install ansible-core then pick and choose collections to install on top.
This covers about 99.9% of all use cases (for Ansible for automation work) still, and for anything else, you can add your own roles or install an additional community collection or two.
I understand why collections got split out due to development needs, but the switch over to them has been pretty crappy from a user perspective.
https://access.redhat.com/articles/6968468
> - Redhat kneecaps centos because some mid manager was worried he might not get this years bonus
Red Hat paid (pays, probably) bonuses quarterly, and they were (are, probably) based on stock performance. They are not and never were based on cost saving measures.
The reality is that, probably, CentOS should have never become part of the "Red Hat family". The Chinese firewall concept didn't really work internally, the vast majority of RH engineers ran Fedora anyway, and it was basically just giving CentOS better build infrastructure plus giving customers a way to buy their way out of nonsense situations. The Red Hat support agreement always said "if you're not running RHEL, we'll close your case", which frequently meant customers who ran CentOS in dev/qa/preprod and RHEL in production who were forced to install another RHEL system (which RH made no money on anyway, since they never charged license costs, only subscription fees for updates) just to reproduce an issue so the sosreport didn't say "CentOS Linux ...".
In general, the same engineers who worked bugs (or were escalated to cases) on RHEL were often part of the community of upstream maintainers anyway, and RH's policy was "upstream first, and if it needs an emergency downstream bandaid, take it out once upstream accepts a patch and backport it properly", but it was easier for customers to get attention on problems if they could back it with dollars.
Rocky, Alma, and OEL (however you feel about that) exist anyway and basically picked up where they left off. Stream probably always should have been a thing. It was (probably is) an enormous amount of work to branch, re-sync, and generally re-sync again any given Fedora snapshot to a new RHEL release, and getting it into shape was (probably is) a nightmare, because the difference between the code which is currently running on any given RHEL release and what upstream is is huge. sysvinit -> upstart/sysv -> systemd, for example.
- Redhat kneecaps Ansible by making ansible-core and forcing everyone to re-install community plugins they were using from EPEL, or buy the shiny Redhat version of ansible.
The Ansible leadership was generally terrible about open source philosophy regardless. This is nothing new.
- Redhat decides to turn Ceph into yet another garbage storage product with the same exorbitant support fees and contract payments that were the original motivation for an open source storage solution in the first place.
Open source Ceph still exists. I don't see the difference here. It's just moving it from one independent company to another independent company, at least nominally.
The systemic problem with IBM acquiring Red Hat was in a lack of good answers for many of the questions employees had (policies on open source, WFH, etc), the fact that the purchase was not communicated even to management until news articles broke, a six month cliff for "business critical" employees, and following that, the fact that IBM fundamentally believed that they could do what everyone always thought Red Hat was doing and drive the market, so they were constantly pouring money and engineers into projects that had no chance to succeed in the market rather than adopting and fostering upcoming solutions, and *that* drove engineers out of the company very rapidly.
Is Ceph still a viable alternative in hardware and operational cost and reliability to HDFS to store hundreds of PBs of data, if not EBs? I kept hearing people recommend Ceph over HDFS, but was unable to find examples of large deployment of Ceph for HDFS-like workload.
Reading the histories of both companies, it seems to me that Redhat was always IBM. The acquisition was a marriage of separated twins more than anything else, the dominating culture in both organizations is largely sales and consulting driven, with public strategy not quite identical but strongly rhyming. The technology aspects (while perhaps once important) were far from the prime focus of either organization long prior to the merger. I think of IBM's acquisition more as a desire to access marketing expertise and established sales channels than anything to do with technologies, most of RH's open source technology is after all mostly just an advanced form of disguised marketing
From what I know Red Hat devs are also very open in other areas of OS research and development, from the new io_uring, eBPF, etc. IBM is trying to stamp it out, but it's still there.
We manage this POS for some clients, and it's the worst distribution you can have. Stuff you can "just install" under any other popular distro is often silosed off into special packages you need to buy, despise it being OSS and "just available" on EPEL.
What they do now is dabble in a bit of consultancy, a bit of datacenter ops, a bit of software and some servers here and there. And of course all the support on their legacy mainframe stuff. But when they have something that really stands out like the Thinkpad line they divest it.
It's almost as if they don't want to be noticed or something. I don't know.
I never thought their RedHat acquisition was a good thing for RedHat, though I never thought RedHat was a very good company to begin with. When Microsoft acquires Ubuntu (and I'm surprised this hasn't happened yet) they have more good stuff to mess up. Which I'm sure they will. And it seems like Canonical in anticipation of the acquisition has already started going the wrong way themselves.
>to date it feels like all IBM wants to do with Redhat is turn it into IBM so if thtas all these juggernaut dinosaurs can do im quietly waiting to see Microsoft acquire Ubuntu and quietly turn it into a teams plugin the same way they acquired github and turned it into an outage response simulator for developers.
RedHat was eating into their business so they bought it.
> https://ubuntu.com/ceph/docs/supported-ceph-versions
> For instance, it is possible to deploy Ceph Quincy on Ubuntu 20.04 LTS via the UCA or on Ubuntu 22.04 LTS via the regular Ubuntu package archive. See Ceph and the UCA for details.
Instead of being a fork, it’s upstream.
No way Red Hat is staying like some gilded ivory tower of open source purity. They are a brand name for selling services and licenses, so if Red Hat still looks after ... wait that was only 3 years ago? Well anyway if Red Hat is still recognizable after that acquisition in some non-IBM way then IBM is showing enormous restraint.
The previous 10 years at IBM were "buy company, attach Websphere to the front of their product names, sell, abandon product". Still better than Computer Associates software, but not much better.
IBM has left us alone. I don’t get paid by IBM, my benefits are Red Hat, My CEO is Red Hat’s CEO and funny enough when we work with IBM we lead the calls because IBM is letting us do what we do best.
I could sense storage was changing. We had product delays, we kept pushing things out to future releases to expand ODF to our managed services. Evrything was always next quarter. Now it makes sense.
I wouldn’t be surprised if Jboss is next.
Ceph is fine, but its a toy.
You can slam hooks into everything, and the raid replacement (which is probably only allow on IBM tin, the bastards.) is brilliant. (rebuilding stuff doesn't take 24 hours)
Want to have a global filsystem, no problem, want to mange wrote clashes? no problem, everything is based on rules. Want to have a storage array in the US, but a copy in the EU, no problems. just set your rules and you're away.
and its all posix and really rather scalable.
Because?
Thats the easy part. allowing concurrent access to the same block, managing write clashes, decent cacheing are hard. Even harder when you are doing it with posix semantics.
The main issue with Ceph is that the filesystem is an addon and required a huge amount of resources to make it work properly. (search out the ceph plan for gitlab, before they were bought out by a company that insisted on bringing in sysadmins to sort infra out)
the big and largest elephant is that cephfs is still not really "stable" by that I mean supported fully by a large support company. This means that if something goes wrong, you need to figure it out yourself.
The opposite. Ceph started out with the underlying load balancing algorithm (May 2004), then with the plan for the filesystem metadata management (Nov 2004). The block storage was created to support the filesystem, virtual block devices were an afterthought to enable private clouds, object storage was an afterthought in a world where S3 got popular.
See https://www.ssrc.ucsc.edu/person/sage.html for the chronological order.
Of course, stabilizing the most complex part of the stack, depending on almost everything else already being stable (everything except RBD and the S3-compatible gateway, which both are relatively slim layers on top of RADOS), takes the longest. Meanwhile, business users were very keen on getting the benefits of RBD for "private clouds" and the object storage, so the filesystem was on the back burner.
[Disclaimer: I worked on Ceph pre-RedHat]
It's sad, because there's a lot of genius level people that work on Ceph. But the project as a whole is not high enough quality to be reliable.
I imagine most people that work on ceph (eg, the code, not the rest of the org) will want to move to greener pastures soon. Anyone that wants to work for IBM pretty much already did.
It’s always cheaper on the bill of materials. Who is going to build and maintain the complex infrastructure. It would be interesting to see the TCO over 5 years.
In our company it certainly is not cheaper. But what they do is ridiculous. They just moved from physical datacenters to AWS instances. Every "server" instance is running 24/7 independent of load, and you have to fill in an excel of 10 tabs to get permission to fire up an instance. In fact, the only thing they changed is the physical rack location tab to an AWS zone one :P
So now we are "in the cloud" but we have almost none of the benefits said cloud offers. And we're using it in such an inefficient way that we're wasting money by the millions. And we kept all the internal overhead we already had when we managed the datacenters itself (that's what that excel is for).
Once we actually start dynamically spinning up instances based on load and shutting them down when not needed, then yeah I could see it being cheaper than having a bunch of metal in a rack somewhere. But I guess we won't get there in the next 10 years or so. We'll probably be all-in on cloud by the time everyone has already gone on to edge computing or whatever the next big thing is, lol.
https://www.redhat.com/en/blog/red-hat-storage-strategy-upda...