>2) The mainframe's reliability-centric culture vs. today's flaky software design culture
Very underrated comment! Ran into the same thing with VOIP vs. traditional PBX systems. The IP Telephony guys were sloppy with uptime and reliability, and that's being generous :p
A university I know has an old Nortel PBX. The telecom room is absolutely massive. Filled with cabinets, bix panels and stuff. All of that is at least 35 years old and is still in operation to this day, but a transition to VOIP is planned.
I have to respect the reliability aspect of those old systems they were built to last a generation.
It kinda depends on the application. My home phone line gets 95% telemarketer / spam calls. I moved it to VOIP about 5 years back, saving about $35 a month. Even a standard POTS line, with all the taxes and crap, was over $40/month around here. If it's "down" a few % of the time I don't care. I'll just fall back to my cell phone.
New development on the mainframe started tapering off in the mid 1990's. So here's my take on IBM's list:
- 45 of the top 50 banks
- 8 of the top 10 insurers
- 8 of the top 10 telcos
- 7 of the top 10 retailers
- 4 of the top 5 airlines
All had significant operations prior to the mid 1990's and so have extensive mainframe investments that are very difficult and costly to migrate from. I know. I work for one of the top electrical utilities in the United States and at the heart of our customer operations lies an IBM mainframe. We're in the midst of launching a multi-year, billion dollar program to replace that mainframe. We've been talking about doing this for years and are just now pulling the trigger on it. There's still time to tell whether we actually go through with it or not, or get $100 million into the project and realize we're looking at a $1.5 billion - $2 billion effort and abandon the project.
I would not call this state of affairs thriving. Is the mainframe alive? Yes. Is it thriving? No. Most CIOs are looking for a means for retiring those assets.
I’m too young to have used one, but from what I’ve read, it’s the programming model of a single machine with the scale and resilience of a distributed system.
A mainframe is effectively a mail-order on-premise cloud, from the hardware up, developed 3 to 4 decades before things like OpenStack or Kubernetes. Mainframes supported hardware-accelerated virtualization way back in the early 1970s. You could have thousands or tens of thousands of VMs running on a single machine (granted, a machine the size of a room).
One of my older greybeard coworkers tells stories of how sometimes an IBM employee would show up at 6 AM to replace a water pump (or some other component) that had phoned home the night before to report that it was beginning to fail. The hardware is super reliable and resilient but also heavily instrumented to detect & resolve failures before they caused problems. And they would send their techs out to pre-empt those problems before the customer ever noticed anything.
Not that it would have been too much a problem if it had failed - because there was so much hardware redundancy built into those machines.
One of his other stories, is a tech demo that IBM did for his company, where they put a mainframe in the back of a trailer and would drive it around to customer sites. The demo consisted of starting a huge computation on the machine, and pulling up a screen that showed the available hardware resources, then a tech would reach into the machine and start pulling out CPUs and RAM, and you'd watch live as it detected that the hardware had disappeared and shuffled the jobs around to other hardware resources without crashing. And then the tech would start swapping hardware back in, and it would add the resources back into the pool and start doing work with them once again.
One of the reasons for that instrumentation is that they used to be operated in a cloud-like environment where the machine is sliced into smaller ones and time on them is billed much like cloud computing is billed today.
> One of the reasons for that instrumentation is that they used to be operated in a cloud-like environment where the machine is sliced into smaller ones and time on them is billed much like cloud computing is billed today.
IIRC, that's still true. I think IBM mainframes are typically leased, and the customer is billed on usage.
There's some complexity to this, where I think the metering only applies to programs in the "traditional" mainframe environment, while things running on the mainframe Linux environment are exempt from metered billing.
They still pay for licensed capacity when running things like z/OS. The machine is shipped "full" but the hardware has to be activated by paying a license.
One of the interesting features I heard when they launched z15 was that, if one needed to restart a partition, it could tap unlicensed CPUs to start faster and then go back to its licensed capacity.
I don't know the details of billing these days; it's been a while since I followed the space. But when Linux came out on Z, it ran on something called IFLs. These were not AFAIK physically different from the processors running more traditional IBM workloads. They were simply priced at a lower level. I don't remember if there was capacity on demand for Linux workloads or not.
I worked at a place that had a little mainframe (essentially a large server) from IBM to port some software to. Most of the people didn't know anything about mainframes.
On a hot day they had an air condition problem in the data center and had to turn off a lot of machines. They tried to turn off the mainframe using the big button, but it just kept running. Then they decided to pull the power plug. System still kept running. It turned out it had an built-in UPS. After some more attempts they finally managed to turn it off.
> Basically, rather than handling redundancy in software, they have a ton of hardware-level redundancy to get the high uptime #s.
But at what cost? Instead of getting suppliers to out-bid one another you are now stuck in a single supplier situation. [0] Building the redundancy in software allows you to run on commodity hardware that can be purchased from anyone.
> it’s the programming model of a single machine with the scale and resilience of a distributed system.
Single-system-image clustering delivers on that promise with no dependence on proprietary platforms. Linux is now slowly getting the underlying features needed to enable this (e.g. process migration and checkpointing, namespacing of all system resources), largely as a side-effect of earlier developments such as cgroups. You can already experiment with Linux-CRIU to see what the basic tech is like, it's not fully mainlined yet but it's slowly getting there.
For most of their history, IBM mainframes were used for two tasks: batch workloads and interactive users.
Batch workloads were things like billing: you're a utility and you need to do the accounting and billing for the million customers you have. You're MCI and you've spent a year advertising Friends and Family discounts and now need to do custom billing every month that figures out all those intersections of friends & family and sends custom billing. Not rocket science, just a lot of boring data crunching you have to do over and over again.
Interactive was everything you do today on personal systems: stores had terminal displays that hooked to mainframes for inventory control so salespeople could put together an order or cash registers would transmit sales near real time to a central mainframe and pump into/out of your ERP system.
Depending on your company's view of the mainframe world, you might have an MVS system doing all of the batch processing and a VM/TSO system providing interactive. And the MVS system could run under the VM system as a guest O/S, or you could partition the single mainframe to have MVS running on some CECs and VM running on others (e.g. you might have more interactive users during the daytime, then tune down the systems so that the MVS systems had more capacity overnight for batch operations).
Mainframes systems could be clustered together in a single data center and you could move workloads back and forth across systems while the workload was in progress. Eventually (circa 1994) you could link mainframe complexes together across DCs to provide redundancy (if a failure occurred in the mainframe equivalent of an AWS AZ the workload could, theoretically, transfer to another AZ with minimal interruption or reconfiguration).
On the interactive side…you could have thousands of systems connected to a single mainframe via various forms of I/O systems. All of your retail stores, all of the printers at those stores, frequently all of the security systems, would be set up using APPC or SNA to connect to your regional mainframe over one or more leased POTS lines.
The 3270 protocol that drove those green screens was screen oriented, if you changed two fields on the screen and hit ENTER, only the changed data from the fields would be sent, so having a bunch of terminals share a whopping 33kbps line wasn't outrageous.
From a programming and systems management perspective, the systems were designed to run as close to maximum utilization as possible. You just spent 10-15-20 million setting up this complex, you want to run it at 90-95% utilization 24 hours a day. So there's a whole set of workload management tools that developed by IBM, CA, and others to tweak applications and the operating environment to run it as hot as possible. There's all sorts of accounting built into the operating system(s) and language environments, going back to when you set literal budgets for how much a given department or program could spend.
So, this is the environment that ran from effectively the 1950s through the 1980s into the 1990s. The S/360 line launched in the early 1960s but offered some backwards compatibility to some IBM systems of the 1950s.
COBOL gets tagged as the most popular mainframe language but PL/1 (or PL/I) was frequently used (IBM had an internal variant called PL/X which was mostly used for systems software). C was used a lot as well inside IBM but usually for smaller products running inside VM/CMS.
I tried writing a web server for MVS in my spare time circa 1995–1996 and even though I was working with people inside MVS development we had a hard time getting it to work with what turned out to be a pretty crappy TCP/IP stack. The batch oriented nature of MVS wasn't the problem, IIRC the TCP/IP stack just could not handle the set up/tear down of the typical HTTP/0.9 and 1.0 connections. When a web server was eventually developed for MVS it was under the posix flavored &q...
Given the cost of mainframe operations vs. cost of cloud, no, I can't imagine anyone adopting a mainframe. The mainframe is extremely expensive compared to say, AWS - especially if you're running AWS' auto-managed services. Let me put it this way - we're looking at a multi-year, $1 billion project and we're able to justify it. Keep in mind too that we're a regulated utility and the regulators agree with our analysis. Trust me, whatever your AWS bill is it's a pittance compared to what your IBM bill would be.
No. The hardware is leased - you can't buy an IBM mainframe. All the software is priced per MIP per CPU core for your installation - regardless of how much you actually use the software. For all intents and purposes that software is leased as well. There's also retroactive charges - if you bump of the capacity of your mainframe mid-cycle on your license then you have to retroactively pay for the upgrade. We had a situation where that happened with one of our vendors and somehow it got missed for a few years and we ended up owing millions of dollars.
So imagine an environment where you can't buy your hardware, you can't buy your software, and when you've made an upgrade to your hardware your software gets even more expensive. That's a mainframe.
Our mainframe, which is mostly used for customer billing (electrical utility billing is slightly complex due to all the different tariffs, localities and taxes), and customer relationship & service management, is costing us $15 million - $20 million per year. On the one hand you'd say holy smokes that's expensive! But on the other you could say that's only $3-$4 per customer per year. Thing is you could do the same amount of processing in AWS for under $250,000 per month or $3 million per year, making the mainframe at least 5-7 times more expensive than AWS.
It's the business risk of not being able to replace resources for a critical system and the fact that projects are exceedingly expensive and very slow because of dealing with an old spaghettilith (I just made that up!) architecture. Frankly, we need solutions to be delivered faster so we can "move at the speed of business" and this platform just isn't cutting it.
Is that worth $1 billion? That's above my pay grade and I'm glad I don't have to make that decision. For the record, our CFO doesn't think so - at least not until he's closer to retirement so if he gets canned for it he'll be alright! He's in his mid 50's now so we may have to wait another five years!
Out of curiosity, given the nine figure migration costs you're throwing around, why bother? How is it hurting you day to day (to the tune of a billion dollars) enough to migrate. Obvs all else being equal there's no way you'd start on mainframe today, but now that you have it why jump off?
The reason given is it's getting difficult to find COBOL programmers and the platform isn't as agile as other platforms. I have to admit that IBM is working to make the platform more agile and you can do more than run COBOL - you can run Java and Linux on Z. I've been in IBM training sessions where I've been running Linux on Z and then running Docker on top of that and unless you had told me otherwise the experience was normal: you're running bash, running Docker and everything runs as you'd expect.
BUT - the problem is the core of our system is written in COBOL. There's still CICS, JCL, Copybooks and DB/2 and those who can work with those technologies are now within 10 years of retiring and the new generation doesn't know them or have much of a desire to learn them. Heck, I'm part of the first generation of PC developers from the mid-1980's and I've never done any mainframe development!
The thinking then is we'd better jump off before all these people retire and you end up having to pay $300+/hr for resources capable of working with these legacy technologies. Pay now and buy yourself a new technology set and new capabilities or pay later and just keep the system running.
> the new generation doesn't know them or have much of a desire to learn them
On one hand, I'd say that this is true. But on the other, I'm not so sure.
15 years ago, I interviewed for a job that turned out to be for building software on OpenVMS running on Alpha.
I had honestly never heard of either thing.
The interviewer, who would be my manager, said that isn't a problem, it's just a computer that works a little different than others. Do you want to learn it? I said ok.
Over the years, the hiring process changed, and there is no way that someone like me would even be sitting at that interview where they could ask if I wanted to learn.
So companies can go ahead and spend all that money on legacy consultants and all that money on migrating their systems.
Companies that want to hire new people haven't change. If you want a standard web app you can demand someone with experience in the framework of your choice and get it. If you want someone to write in your custom language - you know they have to learn it so you hire people who can learn.
And not even a super custom language. Get out of the bay area and you'll see tons of job postings that ideally want Go or Scala experience, but will gladly teach you if you have the right fundamentals. That's the only way to get their hiring top of funnel large enough to fill seats.
I really think big iron's customer's problems are that they don't want to pay for quality engineers, but for some reason will pay those higher rates or more for some overpriced consultants. Literally just accepting that the outsourcing dream of 2000 was not a great move and using that money on internal engineering would go soooo far. You see even .gov getting the hint with 18F and the US Digital Service building and supporting non contractor government engineers.
How hard is it, really, to learn? Big Tech is constantly hiring people to work in technologies they don't know yet. We were all in on Go before anybody off the street knew it. Facebook has its own language. I'm sure businesses would prefer instant productivity, but is the training expense really bigger than the migration expense?
Yeah, but... can you hire for it? Plenty of forward-looking developers are willing to learn Go, or whatever new hotness the SV scene is cooking up. I think pitching mainframe development to that same crowd is a much harder proposition.
nah, I think both ideas are wrong. that crowd would never want to work for a bank or insurance company in the first place unless desperate. due to their org structure they can never get any real value out of any 1 person, especially an engineer. so that salary will never exist in the first place.
This is absolutely untrue. All of my colleagues and friends put up with corporate SV crap the same way they'd put up with bank / insurance company crap only for the pay.
If a bank were paying SV salaries to do COBOL programming on a mainframe, I'd jump for it, honestly.
Might not be the same crowd, but some of us wouldn't mind to learn COBOL and get our hands on a mainframe (while at the same time wondering whether the world really needed Go).
The question isn’t how hard it is to learn but who is willing to embrace an old technology which for now it doesn’t even make financial sense. When $300/hour becomes the norm you’ll have plenty of resources. For now COBOL jobs aren’t very attractive
> BUT - the problem is the core of our system is written in COBOL.
This is the key issue. You can migrate it off the mainframe or you can migrate it to sexier tech on the mainframe.
> and the new generation doesn't know them or have much of a desire to learn them.
One of the barriers is that these companies are not willing to pay the same money we (I think we are from the same generation - I was paid to write Apple II software) can get from working with anything that's not COBOL.
I'd love to spend some time with mainframes - they are the fastest Linux machines you can get - but don't want it enough to combine COBOL and JCL with a significant pay cut.
I've always wondered what exactly banks DO see as a competitive advantage. If I want to take my money somewhere, what's the single-slide pitch for why should I choose Chase or Bank of America versus a regional/local competitor?
It's not pricing. As a consumer, it always seemed like the largest banks tended to be relatively non-aggressive on their rates. Isn't "we have economies of scale and pass the savings to you" the classic competitive play?
It's barely, if at all, public facing apps/websites. Okay, you've all got a branded app, you've all started offering Zelle, and you all let me take a photo of the (average of) five paper cheques I recieve in a year to deposit them.
It feels like for a while it was the vague promise of "we've got the largest network so you don't need to worry about finding a machine." A concern largely mooted by smaller banks refunding out-of-network fees and credit union networks.
A depositor is not the customer banks are chasing. It's the businesses who want loans and financial services like M&A and IPOs.
Their competitive advantage is not in their software at all. It's in their relationships. The software helps them scale to large numbers of depositors. That's all.
Yeah, RHEL and SLES have supported mainframe for a long time. Tons of mainframe users are running "modern" software stacks on top of the hardware. You can even do it in parallel with running the legacy stuff (and they do).
>One of the barriers is that these companies are not willing to pay the same money we (I think we are from the same generation - I was paid to write Apple II software) can get from working with anything that's not COBOL.
As a younger person (graduated 2012) who entered the mainframe space for my first job then left after 2y, I can very much vouch that compensation for software devs working on zOS systems writing COBOL or HLASM is peanuts. You'd be looking at 1.5-2x'ing your income even if you took a job at a series A startup writing CRUD apps in Rails/etc.
There's a good argument for unionizing here. There are few enough COBOL devs that they could demand what they are worth and support it with industrial action.
Most RandomBigCo just sees any kind of software development as a cost center, and eventually outsources it, to keep their focus on selling their actual products, while some else writes CRUD applications for stock management.
A lot of mission critical systems are Java on the mainframe.
Not just Cobol.
Why don't people switch, then?
The mainframe gives you a predictible runtime.
Long running jobs don't get rescheduled or hogged by other loads.
I know about a firm that uses a mainframe to control a factory line.
The problem with replacing stuff that works (and the mainframe does) is then your job is on the line if the transition fails, and there are a lot of business moving parts inside.
Re: The reason given is it's getting difficult to find COBOL programmers
Cobol programming is a skill that is trivially taught. Any student with a Cmpt. Science degree from a credible institution, given a curriculum, can learn any programming language in a month with a bit of focus/appropriate curriculum. Can pick up the necessary frameworks in a reasonable period of time. Then it's just a matter of teaching them the line of business, and the practices and procedures.
I would claim that, given twelve months, you could teach a cohort of recent Cmpt. Science graduates all the skills they needed to be successful in any Insurance/Banking/Finance programing environment.
And these are developers that you can then pay a very reasonable salary - say, $150K/year - which is $75/hour, not $300/hr, at the cost of 1 year of salary as they come on board, come up to speed (say at an out-of-college salary of $100K/year.) - You need 1,000 of those developers - that's an up-front cost of $100mm, at which point you have your developers.
Compared to a multi-billion dollar transition cost off of a mainframe, I know which one I would chose if I were the PHB.
> Cobol programming is a skill that is trivially taught.
And this used to be very common. My first job out of school in the early 1990s was with a "big consulting firm" they had a boot camp where they taught COBOL to batches of new hires. You did that for a month, and then were deployed to a client site writing COBOL about 50 hours a week.
I wonder if it's less that COBOL is hard and more that there's a lot of design philosophies in legacy codebases that more recently trained developers are going to find frustrating/error prone.
I've done some work in the financial space. It's interesting to compare, for example, expressing a credit card transaction in an ISO 8583 message versus a modern JSON/REST API format. The same data might be there, but one of them is going to be much harder to debug without a lot of specialized tooling.
I worked for a large retail company that had mainframes, our own DCs, and a large cloud presence. There was and still is an ongoing project to retire the mainframes. From what I heard, the mainframes don't always solve the problems we have today, such as realtime information about all the stores and inventory. They are also harder and harder to get people that know and understand them to fix issue or work on integrating them into newer systems. Finally, there was a serious power outage at a DC after I left and the backup generators did not kick in properly. And the mainframes and other older hardware was not booting properly. That is a major risk that you don't have with the cloud, or even more modern application design with your own hardware. As long as you have the data, you can just provision new hardware and go. It would be expensive and time consuming, but this is a Fortune 50 company and they can pay the cost if it happens.
They are also harder and harder to get people that know and understand them
Bootcamps saturating the market with webdevs armed with skills that barely last a year, when they could be training people with skills that would see them employed for a whole career.
There is a yearly Master The Mainframe competition for students (but anyone can participate, get an account and do the exercises and go through the materials). It's fun. z/OS is completely alien for someone who grew up in a Windows/Unix ecosystem.
Also IBM is launching a new program for students called "IBM Z ambassadors" trying to reach students and evangelize about mainframes... I know it because Im currently an IBM Z ambassador.
https://zambassador.com/
The pay for people who know and understand mainframes is insufficient to incentivize people to become knowledgeable and familiar with mainframes.
It has nothing to do with bootcamps. If the mainframe knowledge was in sufficient demand, then they would be selling mainframe training too since people would be wanting to buy it.
> It has nothing to do with bootcamps. If the mainframe knowledge was in sufficient demand, then they would be selling mainframe training too since people would be wanting to buy it.
A large part of that is probably due to American management practices, where the preference is to skimp on maintenance until they find themselves in a crisis. They don't want to pay for what they need, they want to pay for whatever is cheap.
"harder and harder to get people that known and understand them"
This is not surprising. IBM could fix this overnight if it wanted to. How? By providing no-cost educational licenses for its mainframe software for use on Hercules. Why don't they do this? I believe that there are 2 reasons why: (1) they're terrified of people running their production mainframe workloads on a non-mainframe, and (2) I believe that some of IBM's existing mainframe customers keep pressure on IBM to prevent them from opening up the 'mainframe secrets to the world'. The second point is the belief in security through obscurity. I don't have any evidence to back up this belief, it's just my theory.
The other thing that IBM could do is provide low-cost (or even no-cost) options to real mainframe environment over the internet. I know that they have some of this, but it's too restrictive. I'm talking about opening it up like AWS did with free for 1 year. But that's expensive? It wasn't free for AWS to do it, and it wouldn't be free for IBM to do it. However, it would eliminate the biggest barrier to entry for new talent.
Getting back to my theory of security through obscurity, I believe that this is also in play with AIX and iSeries (AS/400). A number of years back, I bought a used RS/6000 workstation on eBay for the purpose of learning and I contacted IBM to get a licensed copy of AIX 4.3. It was as if I were speaking in tongues trying to get through to them.
IBM: "Sir, what's your IBM customer number?"
Me: "Umm, I don't have one. I'm just an individual who bought a used RS/6000 on eBay."
IBM: "We can't sell you an AIX license without a customer number."
Me: "Ok, well give me one."
IBM: "What's the name of the company?"
Me: "There is no company. It's just me as an individual who wants to learn about the technology."
IBM: "We have to have a company name."
Me: "Put my name as the company name."
...
...
...
Eventually, get assigned an IBM customer number and did finally get AIX CD-ROMs from IBM.
I can only imagine that it would be 10x worse to do something similar with their iSeries (AS/400). When you see this over and over, I'm left with the conclusion that IBM doesn't want individuals getting their hands on their hardware or their software.
IBM does have the zPDT for running an emulated mainframe on Intel hardware for development, and education purposes. However it is not free. IIRC it is about $1K/year. Not a bad cost for a business, but too expensive for someone to go learn on their own.
You are absolutely right about how hard it is to do business with IBM. I worked for IBM for nearly 7 years, running a small datacenter. It was such a pain in the ass every time I needed to order something. IBM sales and support process is very high touch, and is designed for the large enterprise that has a team of IBM sales people supporting them. If you are that customer it is not a big deal because you have someone to navigate the process for you. If you are a smaller customer, they want you to go through a VAR. Which is still a pain in the ass.
Windows NT and the AS/400 are about the same age. This pain in the ass sales process is a big part of why Windows NT won. The IBM sales people didn't want to sell it because you didn't need to buy services with it and it was too hard to buy on your own. Where as if you wanted Windows NT, you just bought a copy of Windows NT and installed it on an easily available PC.
We won't even talk about the time I tried to use IBM Smart Cloud Enterprise.
Thus exact thing happened to me also. Thet were unwilling to sell me a screen for my RS6000 system because I was not a company.nuo until that poiint I looked uo at IBM. Ever since that point I"ve never looked back.
It's also not very clear what target environment to migrate to with costs in that ballpark and expectations for longevity comparable to mainframes. Java? Cloud (lol)? Web (lol)? Just blinks of an eye compared to what these are supposed to replace. And last decade's post-standards rush hasn't exactly helped either so POSIX etc might not help much.
You're not wrong. Oracle is making a big play in the utility space and so they have a lot of solutions available that can be hosted on-prem or in their "cloud" - which really is somebody else's data center. Their solutions are all JEE based and run on WebLogic.
What gives me pause for concern is the JEE/WebLogic platform is now as old as COBOL was in the 1980's when we were calling it "dead" and we're still looking at 5-7 years of development and migration time to get to the new system. What will we think of JEE/WebLogic in 2030? My biggest concern is they're going to spend all this money to get onto another soon-to-be-legacy platform that they're going to realize on day 1 they need to start planning on getting away from. This is why mainframes have been so "sticky."
'Vendor Lock in' was also a problem. When it came time to renew leases/licenses, IBM could say whatever. There was no recourse but to pay. Funny/sad that we believed Oracle when they said they were the solution to 'Vendor Lock in'. The billion+ dollar grift was thick.
To add on to this, this results in an odd job market where demand for COBOL programmers is pretty low most of the time since there isn't a ton of work, but when a company does need to hire COBOL programmers its much more difficult to find them and hire them. Which leads to these common stories about COBOL programmers being in demand, when the reality is that most have a hard time finding new COBOL jobs, especially without needing to move.
Off-topic but do you work for Eversource? An ex-coworker of mine found a bug that he immediately recognized as a COBOL bug while working with Eversource CT's EDI
>We're in the midst of launching a multi-year, billion dollar program to replace that mainframe. We've been talking about doing this for years and are just now pulling the trigger on it. There's still time to tell whether we actually go through with it or not, or get $100 million into the project and realize we're looking at a $1.5 billion - $2 billion effort and abandon the project.
So, you're looking at a high risk, multi-year, multi-billion dollar project to remove your own mainframe dependency. And the business isn't fully committed to it.
> Most CIOs are looking for a means for retiring those assets.
And finding extremely expensive, high risk, long duration software projects are the usual alternative, per your own experience. Why waste prestige and budget on that when you can keep all the critical LOB software running exactly as it is (or faster) forever? Mainframes are the OG backward compatibility and virtualization platforms.
Given that, why do you think the mainframe business isn't thriving, especially given the billions in revenue that the companies you list in your post undoubtedly bring in for IBM? It's not sexy, and they're probably not winning new customers. But they're making a shit ton of money on a long lived, successful product and continuing to fund research and development on the platform.
Yes, the mainframe business is 'thriving' in that environment. Much like how SARS-CoV-2 has been 'thriving' in a non-immunized population who don't like wearing masks...
> Why waste prestige and budget on that when you can keep all the critical LOB software running exactly as it is (or faster) forever?
Hum... Probably (and that was the case at my workplace) because business as usual has higher risks and ongoing costs than the migration. It's not like mainframes are cheap to own, or guaranteed to stay in existence in the future.
> why do you think the mainframe business isn't thriving
"Thriving" on my lexical implies on growing or at least sustainable. A market where new players don't want to buy at any cost and old players are all trying to get out is not this.
Your definition thriving isn't too far from mine, I think we just differ as to the expectations we have about the platform in the future.
IBM offers a compelling option:
- yes, your software will continue to run, because that's what the mainframe is for
- yes, it will scale with whatever transaction volume you need in the future, because that's what the mainframe is for
- yes, you can get 99.999999999999999% uptime, because you have reliability built into the platform in astounding depth
- yes, we'll happily help develop new functionality and support your in house staff in managing this system; the mainframe can actually detect faults and arrange a service call.
It's expensive, but it's a safe bet in at very least the medium term. The alternative is quite risky in most cases, and perhaps less expensive over the medium term if it works out.
Nothing is sustainable forever, but that sounds like a pretty hopeful story for IBM over the medium term. Long term, who knows, but eventually we're all dead anyway.
> - yes, your software will continue to run, because that's what the mainframe is for
Maybe if you are using zos without any niche configuration. IBM has discontinued many mainframe platforms in the past, and there's no guarantee that next year they won't approach you with a 10x markup on your renting costs, that will become 50x in a couple of years and will keep going up until you decide to stop paying.
> yes, it will scale with whatever transaction volume you need in the future, because that's what the mainframe is for
Only if you gave-up on data dependency from the beginning. Transactions here means independent data processing, with possible rollbacks, not the relational concept people got used to on databases. If one needs relational data, it won't scale. NoSQL can scale with non-relational data too, it's not an exclusivity of mainframes.
> yes, you can get 99.999999999999999% uptime, because you have reliability built into the platform in astounding depth
That one is ridiculous. Uptime is limited by business code bugs, network and power outages, integration problems, and well, all that causes 90% of the downtime on a single-server shop. All the mainframe promises is that the hardware won't be a problem.
> yes, we'll happily help develop new functionality and support your in house staff in managing this system; the mainframe can actually detect faults and arrange a service call.
Yes, mainframes come with support contracts. You can get support for other platforms too, with the advantage that you won't need to hire IBM.
Anyway, none of that changes the fact that nobody wants to start using a mainframe nowadays, and that the people using them are either looking on how to replace them or avoiding increasing their usage. IBM will certainly be able to milk that market over the medium term, even more if they keep increasing their prices. They revenue may even go up for a while. But that's not thriving.
You are completely missing the point. I'm not arguing that any other solution can't offer the same kinds of things that the mainframe can.
Imagine you are a CIO. You have mainframe apps. Everything I said is true of your existing solution. If you keep it as is, it works just fine for the foreseeable future. Every other way forward requires a total rewrite of systems that have been stable for years and are typically mission critical financial and transaction processing systems.
That's what I mean by IBM having a compelling offering.
Problem is you can't keep it long term. As CIO you are also aware that the code has some ugly parts that are hard to maintain. You are also well aware of the cost to update the code (tax laws, or new features). You are very well aware of how hard it is to hire anyone competent who is willing to work in the language that it uses - mainframe languages are seen as a dead end. There is a lot less risk long term in leaving the mainframe behind, the only question is can you there without destroying the company.
There are advantages to the mainframe, but there are disadvantages too. As a CIO your job is to consider both.
Do mainframes only run on COBOL? What are these languages? Or is it that some program written in 1972 is still running everything at United Airlines today?
My understanding is at least Java is well supported on mainframes. It's not the latest greatest language, but it sure beats ALGOL.
I had some colleagues who had worked on mainframe operating systems tailored for the airlines (PARS), and it's built upon TPF as I understand it. I don't know which language(s) it's implemented in, but I do recall them talking about assembler having a major role.
They run most anything from common to rare. The problem isn't the language, it is the years of cruft that never was fully documented. The people who made the early design decision are probably dead. Same problem any code you write today will have it is still in use in 50 years. Well maybe not quite as bad, if you paid attention you know some modern best practices that didn't exist then.
I wonder if one of the bigger arguments is that there's no way you can really migrate in a way that doesn't actually cost you more money in the short term.
Even if you say "March 14, 2021 is the grand final feature freeze for the Mainframe, and day one of the Cloud Initiative", it's not going to happen. You're not going to have a MVP for several years, and a full replacement for five million undocumented lines of COBOL for even longer. Production fires will have to be fought. Regulators will demand you change something, and won't be keen to hear "we're supporting it in the Cloud Initiative, which customers will see in 2025." So you're gonna be writing quite a few more cheques to IBM and your team of COBOL jockeys.
After 5 or 6 years, as you've unloaded most of the customers to your painstakingly built replacement, you can finally begin to scale down the mainframe staff and perhaps return/downgrade some of the rented hardware.
Given that, it may be less a put-your-own-head-in-the-guillotine move just to spend those years hiring in a few new graduates with the promise of "We'll apprentice you to handle our unique and obscure codebase, and as long as you show up 40 hours a week and don't take a dump on your manager's desk, you're going to have a reliable paycheque for the next 20 years."
> IBM has discontinued many mainframe platforms in the past, and there's no guarantee that next year they won't approach you with a 10x markup on your renting costs, that will become 50x in a couple of years and will keep going up until you decide to stop paying.
Which ones has it discontinued? I thought once the 360 platform got going, it was basically forever (unbroken chain of compatibility). What am I missing?
If they're not gaining enough new customers to offset the losses of those that eventually leave the platform or go out of business, and if we take it as a given that over long periods of time all businesses will eventually disappear or reshuffle their technology stack, then the mainframe market is shrinking. Possibly very slowly but it's still attrition.
That might make it good business for a long time but it would be a stretch to call it thriving, and it provides gentle but salient cause for businesses to consider migration.
No software is bug free including software written for the mainframe in JCL, REXX, Cobol, Mark IV, etc. Someone has to decide to either keep fixing the bugs as they crop up or migrate off of the platform. Turns out writing a whole new platform is very expensive.
> We're in the midst of launching a multi-year, billion dollar program to replace that mainframe. We've been talking about doing this for years and are just now pulling the trigger on it. There's still time to tell whether we actually go through with it or not, or get $100 million into the project and realize we're looking at a $1.5 billion - $2 billion effort and abandon the project.
I work at a company that basically built its current business on mainframes in the 70s/80s/90s. The migration effort to get off them has been going on for more than 20 years and still isn't done yet.
> Sounds about like Interstate 35, by the time you're done, it is time to do it again.
Yep. They're already spinning up a big cloud migration project that will eventually replace many of the re-engineered on-premise systems, and they're still not yet off the mainframe (they're close, but that last 20% is always the hardest).
Anecdata. I work for a top 10 bank. The mainframe isn't going away here. We're investing in new development, new tools, and just (last 2 years or so) had a hiring spree to support those efforts. I am also very familiar with strategy at several other peer banks (they're doing the same thing), a number of the telcos (same) and 2 of the major airlines (who will look at you like you're insane if you suggest they get rid of their mainframe).
I've been hearing "mainframes are dead" since the 1980s. The market for them has dramatically changed, but by the same token there's just more people in IT who have no clue what the mainframes do from a business perspective, so "obviously" it's a dead end.
No, mine is anecdata too, just more of it. And neither wins because there's actual data out there about the size and velocity of the mainframe market. I just don't much care to look it up every time someone makes an unfounded declaration that mainframes are dead. And, since we're keeping score, zero content snark is bottom of the heap.
I think the difference is that the mainframe isn’t going anywhere and the large companies themselves have been thriving and will likely continue to do so. My employer has a number of them and they work great although it’s challenging to find the developers. If it was my decision to migrate I’d be hesitant because it seems as though infrastructure is priced low enough to convince people to switch but then will likely creep up over time in either prices or lock-in.
I don't dispute that; every niche has a growth phase and a decline phase. See: Minicomputers or RISC workstations. However, given a nearly 60 year run so far, and an expectation that there's a couple more decades of life it it, we should be so lucky if whatever niche we work in 'fails to thrive' like that.
The replacement for a RISC workstation was an Intel one serving the same role for the same kind of customer. The mainframe doesn't have that sort of path.
Datacenter-scale compute on non-proprietary platforms (e.g. leveraging OCP standards and best practices, which have been developed w/ significant input from FAANG and 'cloud' vendors) can easily tackle any mainframe workload. The implied path is obvious.
Yes, and you can see the mainframe origins in the modern cloud. It was funny to see the predecessor to Kubernetes (Borg) had a language that was even a wink and a nod to JCL called BCL that then evolved into Kubernetes's resource config files.
That being said, it takes a massive commitment and pretty much a rewrite from scratch to achieve a jump from mainframe to modern DC. You need to embrace individual node failure and that's not something you can just hack in.
This is actually a good comment - look at a datacenter and squint, and tell me - is it not justa a new mainframe?
AWS, Azure and GCP are all proprietary, and all are producing larger and larger servers and VMs. They are basically rentable, virtualised mainframes at this point
Wasn’t Nathan Myhrvold spouting this continuously at Microsoft in the 80’s that RISC was going to take over imminently? While that’s more likely presently than it ever was then, it still isn’t clear that it is even close to happening is it?
The only reason it failed is because there was little incentive to port software away from x86.
Most of us didn't consider writing cross-platform software back then and the few who did were writing software for Unix, not Windows, and that because there wasn't a single dominant OS (Unix was a fuzzy target) and binary architecture you could write for, so you had to think about portability.
Oddly enough, there was a lot of cross-platform work on games in the 80's because the home computer market was very fragmented.
To someone going into IT or programming at a young age (so, not me), would you recommend them to get into COBOL and mainframes? Will it be a career option for 10+ years?
I understand and agree with your point but hey, I got a lot of benefit from learning Latin, arguably more than I got for learning Lisp (which was is what I used for the first 10 years of my career. It was a fun and quite lucrative period).
For graduates of my (very small) high school, Classics was the number one choice of college major and though I'm not in close touch with all of them I never heard any of them claim that they had made a mistake.
Cobol is sort of like Lisp* in that regard: the options are quite limited, but as the supply is too you'll probably do better than fine financially.
Lisp is interesting because it always has a first mover advantage in any field, i.e. I can code up something that does 80% of what I want in 10% of the time of any other language. Cobol's only use is that it's entrenched.
In a nutshell, Cobol fails to inspire throngs of youngsters to write "toy Cobol" implementations.
It is also relatively rare for new languages cite Cobol as an inspiration for any of their features.
Cobol has a feature whereby dissimilar structures can be assigned. The structure members which have the same names are identified between the structures and the data is moved accordingly. Or something like that. A programming language which has such a thing could claim to have had that inspired by Cobol.
If I ever add that to TXR Lisp, I will be sure to drop a Cobol reference into the documentation.
I might have learned about this via Knuth; I think there is a mention of it in some passage in TAOCP.
> Cobol has a feature whereby dissimilar structures can be assigned. The structure members which have the same names are identified between the structures and the data is moved accordingly. Or something like that.
This horrifies me on several levels, though I suppose it could be considered a sort "cast to a superclass".
It seems to make sense in data-oriented programming. Say you want to create an invoice from a purchase_order or something. They have the same the order number, items, and figures. Just assign one to the other (or use it as a template for construction) to get the common values in, and fill in the rest.
> I might have learned about this via Knuth; I think there is a mention of it in some passage in TAOCP.
Indeed, see Volume 1, section 2.4 (Multilinked Structures). Page 426 has the example of the statement “MOVE CORRESPONDING DATE OF SALES TO DATE OF PURCHASES”, which, if I understand correctly, means something like “for each field foo in Purchases.Date, copy Purchases.Date.foo into Sales.Date.foo”.
The discussion is in the context of what kinds of data structures would be needed to compile such statements, which I guess makes sense considering that the original publisher offer for TAOCP was to write about everything a programmer would need to know to write a (60s-era) compiler.
(Coincidentally: I've started attending an online meetup that someone is running for reading TAOCP—we usually read about four pages in the two hours—and this is the very topic that we're going to read tomorrow.)
Unless your job is to teach Latin :). Quite a few schools around the world are teaching Latin or even Ancient Greek.
Are Latin and Ancient Greek dead? Ostensibly yes. But not useless. Besides teaching, there is a tremendous amount of books and texts written in Latin and Ancient Greek. To study them, one would invariably need to learn the language.
I believe this comment betrays a mean ignorance of philology -- a discipline to which I'd expect most hackers would be sympathetic, at least insofar as the art of n-way merges may apply, and a vain commitment to the search for (optimal) objective truth.
Texts from the ancient world are handed down to us not from on high, or a single authoritative source (please consider the problem of duplication prior to computers or mimeographs), so considerable effort and erudition is entailed in comparing sources, filling in lacunae (here's a true warrant for what most consider a sesquipedelian word), all of which is predicated on perusing (in vain) an intuition for "pure et clare latine" and a hard-nosed insistence on _attested sources_ on matters of usage and expression.
I treasure, uncountably, the continuity afforded by training new generations in "antique" languages and dialects. It addresses a much deeper and more valuable problem than shitting out the next side-hustling platform.
Loquendi ars latine vivendius quam lingua ista, ut mihi videtur. Non doctus sum (et plurima menda facio faciamque), et clare non dives causae haec lingua, at vita mea bene ornatur a hac lingua.
The following is hearsay based on a few people I know in that field, so take it with a grain of salt.
These days, most companies seem increasingly resigned/accepting of the fact that they're not going to find young, dynamic COBOL programmers in the wild. On top of that, old-school COBOL has a knack for creating idiosyncratic software architectures. Learning how some particular monstrosity was taped together is more important than the language itself, which is brain-dead simple. And there are few such COBOL codebases out there in the wild to read.
The result is a move towards hiring and training good programmers in-house, quite counter to the general trend in the industry. Which makes for good job security but also makes it hard to enter directly.
System Z is not just COBOL. C/C++, etc and Linux has run over there for years.
If I could go back 30 years, I’d have made more money, slept better and would have more hair on Z. Not that I can say for a new generation but now that everyone is a “full stack” developer and the stack changes every few years, I’d still prefer longer term niches if I was 22 again.
Anecdote too. I work for a large bank too, and >95% of our business has now moved off the mainframe. Took about 8 years to migrate.
I’m not savvy on the costs, but I think the main benefit of ditching the mainframe is you can start using the IT tools, expertise, workflows, languages, frameworks, best practices etc. that 99% of the world is using.
With the mainframe, you’re in sort of a shielded of nook of the software world, where everything is fast and stable but also juuust a little different.
Key theme is slimming down the mainframe to bare basics, then take out either batch, or DB2, and then see to pull the trigger on the rest, 6y horizon...
Why don't people migrate the systems to vms, then you could run them on vastly cheaper hardware? You have all this data across the complex storage system, but all of that could be virtualized if they wanted. The vms I worked on in the 1980s had so many different os variants, cics etc but if I was working on this approach I'd try to run things in vms, eventually teeze out the pieces I could into new services. But the parts that are too hard could just keep running in vms. You have to virtualize your data sources and provide apis to read those old disk drives or virtualized tape drives etc.
But if you are spending a billion dollars, how much code do you have? Also, on the lack of programmers, just pay them more and you'll get a lot more applicants.
What does the mainframe do that makes it take years and cost $1B to replace? I'm guessing that it didn't take that much time and money to set up the mainframe in the first place. So why does it cost that much to build an equivalent system today?
In most cases, it's running millions of lines of legacy code that interacts with all sorts of crufty, decades-old proprietary software components. (iow, it's not COBOL itself that blocks a migration, it's the fact that real-world COBOL code tends to rely on this terrible proprietary stuff. I mean, it's nothing that couldn't be dealt with cleanly if a gigadollar was literally on the line, but I suspect that such figures are somewhat exaggerated.)
Later in the article it describes many companies stance on mainframe, hybrid cloud and cloud.
Hybrid cloud just means you have all the security problems of hosting your own infra, AND all the security problems of the cloud. Not to mention other non security problems of both systems.
When these CIO / CTOs make this "middle of the road" decision, they are dooming their tech budget to spend more on infrastructure and processes than actual product.
It's one thing to have a "we are both until we finish migrating to one" but to try and maintain both forever is just punting the problem to the next person.
It's usually FOMA. Our CIO loves to brag about being in the cloud, but we have less than a handful of ec2 instances running, and barely use more than RDS (oh and Snowflake). But it lets him deflect any criticism, despite the added cost and management overhead. Plus, he'd have a hard time explaining more than 3 of the services AWS offers.
I certainly question the orthodox view that software should be fault-tolerant and horizontally-scalable. If programmer time is more valuable than machine time, then why can't my team get a single server with a few TiB of RAM and a few hundred cores and forget all the gnarly problems of distributed computing? Why do we accept that our hardware is unrelaible and then spend so much effort making our software tolerate failure?
I do not know whether or not mainframes are the right solution, though. I don't really know anything about them. I don't know whether or not the architecture is the correct approach.
>IBM has grown usage as measured by MIPS, a method of calculating the raw speed used, by 350% over the past ten years.
That's an awful metric. MIPS isn't even a good way of measuring aggregate performance and aggregate performance is a terrible way of measuring adoption.
> If programmer time is more valuable than machine time
This is probably not true in industries such as banking, manufacturing, agriculture, etc. I worked as a field engineer in a food processing plant once, during the peak season for the crop being processed. The plant manager told me that downtime could cost the plant up to a quarter million USD per day.
> Why do we accept that our hardware is unrelaible and then spend so much effort making our software tolerate failure?
From what I understand, the whole point of mainframe design is to tackle the issue that hardware is unreliable. Mainframes feature many hardware redundancies, moreso than a commodity rack server.
> I do not know whether or not mainframes are the right solution, though. I don't really know anything about them. I don't know whether or not the architecture is the correct approach.
Mainframes are designed for throughput, not performance. They are good at things like transaction processing, not machine learning.
Right now we're seeing a proliferation of states, counties, local health authorities, hospitals and drug stores all setting up their own individual Covid19 vaccination booking applications. Just about every single one is falling flat on its face when confronted with demand. No mainframes involved.
What, from a black box perspective (i.e. performance, reliability, cost/compute, maintenance-cost/compute) distinguishes a modern mainframe from a modern onsite private compute cluster?
Let's say both are running the same OS and application using VMs, and neither is doit specialized types of compute (i.e GPU stuff), just processing gobs of transactions.
Obviously there are big internal architectural differences, but ultimately these must manifest in some quantitative and qualitative differences and tradeoffs from a black box perspective.
From the buying manager POV, one is cheap commodity and the other is expensive single vendor equipment.
From the programmer POV, one is a proprietary environment with uniformly addressable memory (not UMA, just addresses) and the other is a distributed blank canvas where you must write the IPC yourself (or use a library, of course). Both have a huge IO capacity, but mainframes can integrate data much faster (and usually, process it much slower).
From the ops POV, one is a single machine (a heavy one), and the other is an entire datacenter.
It's a single server with massive amounts of cache (like 1Gbyte or more of the L3 cache) and optimized everything.
When you login via ssh to your server, each keypress creates an interrupt which is handled by the CPU. If you are using emacs , even on the terminal it might be multiple interrupts due to screen refresh etc.
Mainframe: generate the form or the screen data. Send it. 1 interrupt. Your terminal or terminal program handles everything until you send 1 or more lines (not characters) back. A dedicated processor handles communication with the terminal. It then bundles up whatever the changes are and sends it to the CPU. 1 interrupt.
Disk, network etc. is the same, with dedicated processors on the network card or disk controller that go out and send/retrieve the data without CPU intervention. Database table retrieval: the table is sent to the CPU, already sorted in the requested order. Etc etc.
Since most of the mainframe projects goes to big tech consulting cos, as a developer with many years of mainframe experience who these companies don't want to hire or have moved to unrelated work, what can that developer do provide consulting or other business opportunity around mainframes ? Asking for a friend.
Root causes of the perceived mainframe problem as I see it are:
1. It’s difficult to obtain skilled personnel, that’s it.
Why, because you can only truly learn mainframe technologies on a mainframe. Hercules and all the things that IBM have done themselves such as MTM, VS Code extensions are not enough. The barrier to learn mainframe technologies is too high.
Curious what VS Code extensions have been made for mainframe stuff? Sounds like fun to play with.
I agree with your main point though. Occasionally I’ll see someone recommended people to learn COBOL to get a job as if the language is the most relevant part of that industry.
I know its not trendy, but, if you want to not bother writing software to handle redundancy, you can pay IBM to do it in hardware.
Seriously. if you pay IBM for a couple of z series machines, you can link them together in step, and literally blow one up, and you'll only loose a certain number of CPU cycles.
sure they are expensive, but its cheaper than buying a team of ten engineers for x years to make K8s do the same thing.
Seeing how banks and other industries are throwing money at the wall to migrate away from the mainframe, and seeing how money speaks louder than words, I’m doubting that last statement.
Also, you don’t buy mainframes, you rent mainframe CPU cycles. Or actually, you will pay for the rolling window 4 hour peak of CPU utilization.
IBM will gladly ship out the fridge sized box that holds those CPU’s to your data center for free, as soon as you turn it on their cash starts coming in.
The very first sentence contains a glaring error when it lists the "seven dwarves" (the other mainframe companies) as being "Burroughs, Unisys...". Unisys was formed in the 1980s from the two remaining "dwarves", Burroughs and Sperry (which itself had previously merged with Remington Rand).
By that time there were a few other mainframe class manufacturers, most notably Tandem and Amdahl.
Notably missing (and most relevant to the 2020s tech scene) was DEC, who by the late 60s was making smaller mainframes as well as minicomputers. What makes them different was their decision to tae a different, and what Clay Christainsen would call disruptive path. And while highly influential they were themselves nuked by a new class of disruptive companies.
IBM has reinvented itself several times over in its almost 150 year history. Not sure the current one is successful, but I suppose it's still too soon to tell.
So we know that these large companies use mainframes, in part, because they've always used mainframes. But it also seems like there are compelling reasons now in terms of large amounts of memory & compute & I/O & speciality co-processors. So how come it's not a sexy thing now? Having a ton of compute + memory + I/O in AWS/GCP/Azure can be really, really expensive... I'd love to know a cost comparison.
Is it even possible for an ordinary person to rent a VM on one of these machines and do some compute? If I have a startup that needs to crunch a bunch of models or would have an easier time running some system centrally versus having to have the skills & time to create a purely distributed system, why aren't mainframes a good/better/easier option? How come AWS doesn't have any? Does IBM have quick/easy on-ramps to hosted mainframes via Softlayer (I can't find them if they do)?
Pretty sure commodity servers have now eclipsed any compute, IO, etc, advantage mainframes ever had. With perhaps a few exceptions for some really niche things...like VISA using TPF (which is an unusual OS on a mainframe) because it's basically a huge, high transaction rate, distributed NoSql database.
The only real remaining advantage is probably the overall reliability, redundancy, engineering, etc. Like being able to swap out a CPU while the system is running.
No. A mainframe is not a supercomputer. You can get more cores, more dram, more iops for less money if you don't need high reliability in a single machine (because your software handles nodes failing) and if you don't need compatibility with binaries from the 60s.
The chip is impressive, because they don't skimp on costs, because they sell them for a million a pop, but consider how many ryzen on xeon cores you can get for that price, with all the cool DRAM and IO tech you want.
Max cores is 190, so no on CPU. Memory and IO, perhaps, but it's unusual to put more into a single partition than you would into a typical large X86-64 server. And it's a lot more pricey.
A true "cost comparison" would have to include existing on-prem platforms, and account for new players like Oxide Computer Systems who are working to bring newer, cloud-native technologies to the on-prem datacenter. The mainframe may be competitive with "cloud" solutions, but that's not where the most obvious comparison is.
Hah no. When my company was looking to justify buying a new mainframe, they tried to leverage our server team by saying "You can run Linux in LPARs just like you do on VMware etc." So we met with the sales guys. They asked the size of our average Oracle VM (RAM mostly), and when they heard 256GB, they smiled. Then the mainframe manager asked how much it would cost to provision that much RAM for 10-12 LPARs. I wish I had taken a photo of his expression. Needless to say the meeting ended shortly after, and we didn't pursue spending almost 100x the amount we were spending on our ESX hosts...
While you can run Linux in their own LPARs, I think the much more common scenario is to run them as guests under z/VM. Why is this? Because a single LPAR typically represents a sizable chunk of the overall resources (and cost). On the other hand, you can create an LPAR for z/VM (say with 2-4 IFLs) and have those hardware resources spread out across many Linux z/VM guests.
If you have a lot of hardware in the same facility, configured in a way such that you can hotswap and load balance and do all of the other things you might want, you only call it a "mainframe" if it's from IBM, who is I believe the only player left in that business. Otherwise, you call it a "server farm", and pretend that it's an entirely different beast.
A mainframe (in recent decades) is essentially a proprietary, high cost, tricked out server farm in a box (which is often big enough to be a small building, btw), which might make sense in a few cases but mostly is not as good as a much larger, non-proprietary mainframe, called a "server farm", which is supported by several competing vendors.
> Otherwise, you call it a "server farm", and pretend that it's an entirely different beast.
Well, server farms don't usually expose a single system image. You can't set the OS to "migrate" workloads seamlessly across hosts the way it can migrate them across CPU cores. Linux is now getting some of the underlying features that are needed to enable this use case, but integrating them into a coherent whole will take some time.
What's the difference between the system image in a huge zOS machine (which, from what I understand supports all kinds of NUMA and hot-pluggable everything, including CPUs, memory, tapes, and disks because they're designed to never, ever need to be shut down) and the cluster "image" provided by a high-availability k8s cluster (which also supports all kinds of hot-pluggable everything)?
In a way, the kubelet is the kernel of a k8s cluster. You wouldn't run masters on every node in a cluster, because consensus gets harder quadratically. Maybe workload migration doesn't have to live in the (linux) kernel.
Kubernetes is just middleware, it's not a full distributed operating system like, e.g. Amoeba or Inferno. It was never designed from the ground up to expose a meaningful view of the whole cluster to application developers or to those managing the system. As a result of this history, it also features a significant amount of incidental complexity.
CAP theorem. CA_ is mainframe. Crazy that Azure or AWS haven't put Z15 "frames" in a region. Large network latency kills a lot of otherwise easy integrations.
Why cannot the mainframes be replaced with virtual machines and the target binaries replaced with the identical output from a language with more than one compilation target? It seems from the outset this is entirely possible and practical given how much more compute is available from modern architectures. Why is this not now an entirely feasible and a pragmatic approach ?
> Why cannot the mainframes be replaced with virtual machines and the target binaries replaced with the identical output from a language with more than one compilation target? It seems from the outset this is entirely possible and practical given how much more compute is available from modern architectures.
Because mainframes are more than just a quantity of computation. Their hardware and software has prioritized reliability and resiliency far more than the PC architecture, which pretty much everything else is based on nowadays (and which started out and remains mainly about low-cost).
In some ways the approach you describe would be like replacing a ZFS RAID array with an a single large disk with a FAT filesystem. It's possible, but you probably don't want to.
Full disclosure: I'm not a mainframe developer, but I've read quite a bit about them.
These options exist to varying degrees. There can be many obstacles to migrations. Very few of these systems can be untangled to migrate a bit at a time though.
Mainframe replacement is a non goal I think for reasons others brought up. Mainframe languages can be implemented in a language runtime like .net or jvm. I believe this has already been done with cobol and allows you to interop with new code in popular languages
Everytime I see mainframes/COBOL mentioned, there's always anecdotes on how constantly short-supply COBOl programmers are[0] and that a surefire way to get a job is to learn about COBOL and mainframes. I'm always curious - are there any people here who have deliberately gone into a career with mainframes for salary/job security reasons? How has it worked out?
I did, my first job out of college was mainframe cobol at a bank in 2002. It was a huge mistake. The mainframe apps were a cost center the company sought to minimize. Salaries were low, raises were zero to 3%, no bonuses. It was a dead end job. I recall sitting in a group meeting when the VP announced the offshoring initiative. The plan was to replace attrition with offshore developers, max 10%. A few quarters later same VP told the same group the same plan, only this time the target had risen to 20%. Ding ding!
That’s when I got out. When it came to switching jobs I had zero experience in current tech and had to take an entry level position and a pay cut. A few years later that same company had massive layoffs and all development outsourced by 2008 or so.
So yeah whenever you hear about opportunities in mainframe cobol don’t believe it. I’ve got 2,000 former cobol developers from that company alone who will attest that its farce. What puzzles me is who’s behind this idea and what’s their goal in promoting it?
I'm surprised to read these recent anecdotes (going back to that New Jersey unemployment COVID programmer shortage) from various people that COBOL jobs are in high demand yet shockingly underpaid. That would seem to imply markets don't actually work the way we've been told.
No, this is the way a market normally works. You have a supply curve, and a demand curve. As you go lower in price, the demand curve rises, and the supply curve declines. So you have low price, high demand, and low supply. This is completely normal when the price is too low.
The difference in this case is that the press is showing sympathy to those who won't pay the price and then complain about the missing supply. That doesn't happen in most markets.
Why is the press listening? Because the people complaining are people that the press normally listens to - banks and big, big businesses. It seems that many such businesses would rather complain, publicly, than pay a market-clearing rate. Well, I don't have a lot of sympathy, to be honest...
Banks and big businesses? Whenever I read articles about COBOL programmers needed, it’s always in regards to government positions. Which better explains why the pay is low, especially compared to big tech.
> The mainframe apps were a cost center the company sought to minimize.
a bit off topic, but this is a huge thing that I at least didn't pick up on across jobs for many years - for accounting reasons, many firms will always see what your team is doing as an 'expense' to be 'minimized', no matter what you do, and in many cases, there is nothing you, or your manager, or even your managers manager can do to change this, since it is often decided at the highest levels of the firm based on broad-stroke accounting policies that have a direct impact to how the firms entire books (and therefore revenue/profit/etc) are calculated.
IT/tech related? goes in a cost center.
What's a cost center? an optional expense which can and should be minimized, with manager compensation often tied to how effectively costs are controlled.
If anyone working in tech is not aware of the various accounting models used in business, it is well worth your time to get familiar with the basics - assignment of 'cost center' to technology groups in firms that use cost/revenue center models has a huge impact on how teams are likely run, decisions are made, and potential for career advancement for employees in those groups.
this is not the only costing model (job costing, activity based costing, etc) , but one in my view that is often hugely detrimental to technology teams in non-tech companies unfortunate enough to find themselves pre-categorised as a 'cost center'.
We've had some examples of cost-center-based thinking leading to disasters, SolarWinds and Boeing being two recent ones I can think of. I'm sure there are others. I wonder if b-school/management types put serious thought into managing complexity they don't understand. It's a serious question; nobody can understand everything and yet you have to make decisions.
> I wonder if b-school/management types put serious thought into managing complexity they don't understand.
Think this is highly dependent on the school/student; In my experience (data point size=1), MBA teaching hasn't really caught up to the fact that basically everything is technology these days, so it shouldn't be treated as something 'separate'
to the 'real' business, which it often still is - i'd wager that the resulting compartmentalization in thinking around how to manage tech is the main driver behind allowing poor systems to capture IT value from the financial perspective...
New Parts? Depending on your meaning it might be worth pointing out that the raw engineering IBM put into these things is pretty mega, i.e. (not sure about z15) but power10 is a 7nm monster of a chip.
The rest of the market is for the Japanese business -- Hitachi sells a very competitive offering with an in-house processor, but the machines are made by IBM. Fujitsu might be the last one selling an entire system developed in-house, and they're pushing their cloud offering fairly hard.
In the age of modern CD/CI on container/vm/serverless, there might be innovative ideas and even startup opportunities based on something like SimH (https://github.com/simh/simh) to build a Jurassic Park for these dinosaurs.
Since they are all Turing machines perhaps we can also use AI to generate virtual COBOL programmers from disk images.
I also work in a bank and I am a Db2 z/os database administrator with 13yr experience. We did a little exercise a month back involving all mainframe smes(sysprogs/storage/Db2/cics/mq). Numbers are like, mainframe costs only 8%(including people and licenses), carbon foot print is 3% in data centre and 85% of all Tier 1 applications run on mainframe. Offhost side of people who saw stats were amazed.
Two years back Db2 has become strategical database previously it was oracle in the decision tree. Security and up time has been solid.
Just seeing the trend at workplace on work related to mainframe, I would say it's picking up. Basically we know the work(projects), so we are focussed on onboarding process for newer applications as we are more and more working with people who have no idea about mainframe or who were once on the bandwagon saying mainframe is dying once.
Interns are trained internally after college. I would say that's the best training.... At first, they hate it. It's all black and green text, but slowly they pick up.
199 comments
[ 3.3 ms ] story [ 222 ms ] thread1) the Lindy Effect: https://en.wikipedia.org/wiki/Lindy_effect
2) The mainframe's reliability-centric culture vs. today's flaky software design culture
Very underrated comment! Ran into the same thing with VOIP vs. traditional PBX systems. The IP Telephony guys were sloppy with uptime and reliability, and that's being generous :p
I have to respect the reliability aspect of those old systems they were built to last a generation.
- 45 of the top 50 banks
- 8 of the top 10 insurers
- 8 of the top 10 telcos
- 7 of the top 10 retailers
- 4 of the top 5 airlines
All had significant operations prior to the mid 1990's and so have extensive mainframe investments that are very difficult and costly to migrate from. I know. I work for one of the top electrical utilities in the United States and at the heart of our customer operations lies an IBM mainframe. We're in the midst of launching a multi-year, billion dollar program to replace that mainframe. We've been talking about doing this for years and are just now pulling the trigger on it. There's still time to tell whether we actually go through with it or not, or get $100 million into the project and realize we're looking at a $1.5 billion - $2 billion effort and abandon the project.
I would not call this state of affairs thriving. Is the mainframe alive? Yes. Is it thriving? No. Most CIOs are looking for a means for retiring those assets.
One of my older greybeard coworkers tells stories of how sometimes an IBM employee would show up at 6 AM to replace a water pump (or some other component) that had phoned home the night before to report that it was beginning to fail. The hardware is super reliable and resilient but also heavily instrumented to detect & resolve failures before they caused problems. And they would send their techs out to pre-empt those problems before the customer ever noticed anything.
Not that it would have been too much a problem if it had failed - because there was so much hardware redundancy built into those machines.
One of his other stories, is a tech demo that IBM did for his company, where they put a mainframe in the back of a trailer and would drive it around to customer sites. The demo consisted of starting a huge computation on the machine, and pulling up a screen that showed the available hardware resources, then a tech would reach into the machine and start pulling out CPUs and RAM, and you'd watch live as it detected that the hardware had disappeared and shuffled the jobs around to other hardware resources without crashing. And then the tech would start swapping hardware back in, and it would add the resources back into the pool and start doing work with them once again.
All handled by the hardware and the OS.
IIRC, that's still true. I think IBM mainframes are typically leased, and the customer is billed on usage.
There's some complexity to this, where I think the metering only applies to programs in the "traditional" mainframe environment, while things running on the mainframe Linux environment are exempt from metered billing.
One of the interesting features I heard when they launched z15 was that, if one needed to restart a partition, it could tap unlicensed CPUs to start faster and then go back to its licensed capacity.
On a hot day they had an air condition problem in the data center and had to turn off a lot of machines. They tried to turn off the mainframe using the big button, but it just kept running. Then they decided to pull the power plug. System still kept running. It turned out it had an built-in UPS. After some more attempts they finally managed to turn it off.
But at what cost? Instead of getting suppliers to out-bid one another you are now stuck in a single supplier situation. [0] Building the redundancy in software allows you to run on commodity hardware that can be purchased from anyone.
[0] https://www.gwern.net/Complement
Single-system-image clustering delivers on that promise with no dependence on proprietary platforms. Linux is now slowly getting the underlying features needed to enable this (e.g. process migration and checkpointing, namespacing of all system resources), largely as a side-effect of earlier developments such as cgroups. You can already experiment with Linux-CRIU to see what the basic tech is like, it's not fully mainlined yet but it's slowly getting there.
Batch workloads were things like billing: you're a utility and you need to do the accounting and billing for the million customers you have. You're MCI and you've spent a year advertising Friends and Family discounts and now need to do custom billing every month that figures out all those intersections of friends & family and sends custom billing. Not rocket science, just a lot of boring data crunching you have to do over and over again.
Interactive was everything you do today on personal systems: stores had terminal displays that hooked to mainframes for inventory control so salespeople could put together an order or cash registers would transmit sales near real time to a central mainframe and pump into/out of your ERP system.
Depending on your company's view of the mainframe world, you might have an MVS system doing all of the batch processing and a VM/TSO system providing interactive. And the MVS system could run under the VM system as a guest O/S, or you could partition the single mainframe to have MVS running on some CECs and VM running on others (e.g. you might have more interactive users during the daytime, then tune down the systems so that the MVS systems had more capacity overnight for batch operations).
Mainframes systems could be clustered together in a single data center and you could move workloads back and forth across systems while the workload was in progress. Eventually (circa 1994) you could link mainframe complexes together across DCs to provide redundancy (if a failure occurred in the mainframe equivalent of an AWS AZ the workload could, theoretically, transfer to another AZ with minimal interruption or reconfiguration).
On the interactive side…you could have thousands of systems connected to a single mainframe via various forms of I/O systems. All of your retail stores, all of the printers at those stores, frequently all of the security systems, would be set up using APPC or SNA to connect to your regional mainframe over one or more leased POTS lines.
The 3270 protocol that drove those green screens was screen oriented, if you changed two fields on the screen and hit ENTER, only the changed data from the fields would be sent, so having a bunch of terminals share a whopping 33kbps line wasn't outrageous.
From a programming and systems management perspective, the systems were designed to run as close to maximum utilization as possible. You just spent 10-15-20 million setting up this complex, you want to run it at 90-95% utilization 24 hours a day. So there's a whole set of workload management tools that developed by IBM, CA, and others to tweak applications and the operating environment to run it as hot as possible. There's all sorts of accounting built into the operating system(s) and language environments, going back to when you set literal budgets for how much a given department or program could spend.
So, this is the environment that ran from effectively the 1950s through the 1980s into the 1990s. The S/360 line launched in the early 1960s but offered some backwards compatibility to some IBM systems of the 1950s.
COBOL gets tagged as the most popular mainframe language but PL/1 (or PL/I) was frequently used (IBM had an internal variant called PL/X which was mostly used for systems software). C was used a lot as well inside IBM but usually for smaller products running inside VM/CMS.
I tried writing a web server for MVS in my spare time circa 1995–1996 and even though I was working with people inside MVS development we had a hard time getting it to work with what turned out to be a pretty crappy TCP/IP stack. The batch oriented nature of MVS wasn't the problem, IIRC the TCP/IP stack just could not handle the set up/tear down of the typical HTTP/0.9 and 1.0 connections. When a web server was eventually developed for MVS it was under the posix flavored &q...
So imagine an environment where you can't buy your hardware, you can't buy your software, and when you've made an upgrade to your hardware your software gets even more expensive. That's a mainframe.
Our mainframe, which is mostly used for customer billing (electrical utility billing is slightly complex due to all the different tariffs, localities and taxes), and customer relationship & service management, is costing us $15 million - $20 million per year. On the one hand you'd say holy smokes that's expensive! But on the other you could say that's only $3-$4 per customer per year. Thing is you could do the same amount of processing in AWS for under $250,000 per month or $3 million per year, making the mainframe at least 5-7 times more expensive than AWS.
So how does saving $17 million per year justify spending $1 billion?
Is that worth $1 billion? That's above my pay grade and I'm glad I don't have to make that decision. For the record, our CFO doesn't think so - at least not until he's closer to retirement so if he gets canned for it he'll be alright! He's in his mid 50's now so we may have to wait another five years!
The reason given is it's getting difficult to find COBOL programmers and the platform isn't as agile as other platforms. I have to admit that IBM is working to make the platform more agile and you can do more than run COBOL - you can run Java and Linux on Z. I've been in IBM training sessions where I've been running Linux on Z and then running Docker on top of that and unless you had told me otherwise the experience was normal: you're running bash, running Docker and everything runs as you'd expect.
BUT - the problem is the core of our system is written in COBOL. There's still CICS, JCL, Copybooks and DB/2 and those who can work with those technologies are now within 10 years of retiring and the new generation doesn't know them or have much of a desire to learn them. Heck, I'm part of the first generation of PC developers from the mid-1980's and I've never done any mainframe development!
The thinking then is we'd better jump off before all these people retire and you end up having to pay $300+/hr for resources capable of working with these legacy technologies. Pay now and buy yourself a new technology set and new capabilities or pay later and just keep the system running.
On one hand, I'd say that this is true. But on the other, I'm not so sure.
15 years ago, I interviewed for a job that turned out to be for building software on OpenVMS running on Alpha.
I had honestly never heard of either thing.
The interviewer, who would be my manager, said that isn't a problem, it's just a computer that works a little different than others. Do you want to learn it? I said ok.
Over the years, the hiring process changed, and there is no way that someone like me would even be sitting at that interview where they could ask if I wanted to learn.
So companies can go ahead and spend all that money on legacy consultants and all that money on migrating their systems.
I really think big iron's customer's problems are that they don't want to pay for quality engineers, but for some reason will pay those higher rates or more for some overpriced consultants. Literally just accepting that the outsourcing dream of 2000 was not a great move and using that money on internal engineering would go soooo far. You see even .gov getting the hint with 18F and the US Digital Service building and supporting non contractor government engineers.
If a bank were paying SV salaries to do COBOL programming on a mainframe, I'd jump for it, honestly.
This is the key issue. You can migrate it off the mainframe or you can migrate it to sexier tech on the mainframe.
> and the new generation doesn't know them or have much of a desire to learn them.
One of the barriers is that these companies are not willing to pay the same money we (I think we are from the same generation - I was paid to write Apple II software) can get from working with anything that's not COBOL.
I'd love to spend some time with mainframes - they are the fastest Linux machines you can get - but don't want it enough to combine COBOL and JCL with a significant pay cut.
If banks routinely started paying Google salaries for good COBOL developers, you'd see those shortages disappear in a big hurry.
It's not pricing. As a consumer, it always seemed like the largest banks tended to be relatively non-aggressive on their rates. Isn't "we have economies of scale and pass the savings to you" the classic competitive play?
It's barely, if at all, public facing apps/websites. Okay, you've all got a branded app, you've all started offering Zelle, and you all let me take a photo of the (average of) five paper cheques I recieve in a year to deposit them.
It feels like for a while it was the vague promise of "we've got the largest network so you don't need to worry about finding a machine." A concern largely mooted by smaller banks refunding out-of-network fees and credit union networks.
Their competitive advantage is not in their software at all. It's in their relationships. The software helps them scale to large numbers of depositors. That's all.
As a younger person (graduated 2012) who entered the mainframe space for my first job then left after 2y, I can very much vouch that compensation for software devs working on zOS systems writing COBOL or HLASM is peanuts. You'd be looking at 1.5-2x'ing your income even if you took a job at a series A startup writing CRUD apps in Rails/etc.
No top 50 CS program teaches these tech. No unicorn uses them. Not a single FAANG or FANNG+ if you include other large players.
Finally, no engineering-centric company uses these tech. That's the difference between learning Go vs COBOL.
There are only a couple of FAANG+, with crazy hiring practices, and most businesses aren't engineering centric, the world is bigger than that.
It sets the trends and build what's next. RandomBigCo just follows.
Why don't people switch, then?
The mainframe gives you a predictible runtime. Long running jobs don't get rescheduled or hogged by other loads.
I know about a firm that uses a mainframe to control a factory line.
The problem with replacing stuff that works (and the mainframe does) is then your job is on the line if the transition fails, and there are a lot of business moving parts inside.
Cobol programming is a skill that is trivially taught. Any student with a Cmpt. Science degree from a credible institution, given a curriculum, can learn any programming language in a month with a bit of focus/appropriate curriculum. Can pick up the necessary frameworks in a reasonable period of time. Then it's just a matter of teaching them the line of business, and the practices and procedures.
I would claim that, given twelve months, you could teach a cohort of recent Cmpt. Science graduates all the skills they needed to be successful in any Insurance/Banking/Finance programing environment.
And these are developers that you can then pay a very reasonable salary - say, $150K/year - which is $75/hour, not $300/hr, at the cost of 1 year of salary as they come on board, come up to speed (say at an out-of-college salary of $100K/year.) - You need 1,000 of those developers - that's an up-front cost of $100mm, at which point you have your developers.
Compared to a multi-billion dollar transition cost off of a mainframe, I know which one I would chose if I were the PHB.
And this used to be very common. My first job out of school in the early 1990s was with a "big consulting firm" they had a boot camp where they taught COBOL to batches of new hires. You did that for a month, and then were deployed to a client site writing COBOL about 50 hours a week.
I've done some work in the financial space. It's interesting to compare, for example, expressing a credit card transaction in an ISO 8583 message versus a modern JSON/REST API format. The same data might be there, but one of them is going to be much harder to debug without a lot of specialized tooling.
Bootcamps saturating the market with webdevs armed with skills that barely last a year, when they could be training people with skills that would see them employed for a whole career.
It has nothing to do with bootcamps. If the mainframe knowledge was in sufficient demand, then they would be selling mainframe training too since people would be wanting to buy it.
A large part of that is probably due to American management practices, where the preference is to skimp on maintenance until they find themselves in a crisis. They don't want to pay for what they need, they want to pay for whatever is cheap.
Maybe, but it may also be true that people simply aren't aware that mainframes exist or these jobs exist.
This is not surprising. IBM could fix this overnight if it wanted to. How? By providing no-cost educational licenses for its mainframe software for use on Hercules. Why don't they do this? I believe that there are 2 reasons why: (1) they're terrified of people running their production mainframe workloads on a non-mainframe, and (2) I believe that some of IBM's existing mainframe customers keep pressure on IBM to prevent them from opening up the 'mainframe secrets to the world'. The second point is the belief in security through obscurity. I don't have any evidence to back up this belief, it's just my theory.
The other thing that IBM could do is provide low-cost (or even no-cost) options to real mainframe environment over the internet. I know that they have some of this, but it's too restrictive. I'm talking about opening it up like AWS did with free for 1 year. But that's expensive? It wasn't free for AWS to do it, and it wouldn't be free for IBM to do it. However, it would eliminate the biggest barrier to entry for new talent.
Getting back to my theory of security through obscurity, I believe that this is also in play with AIX and iSeries (AS/400). A number of years back, I bought a used RS/6000 workstation on eBay for the purpose of learning and I contacted IBM to get a licensed copy of AIX 4.3. It was as if I were speaking in tongues trying to get through to them.
IBM: "Sir, what's your IBM customer number?" Me: "Umm, I don't have one. I'm just an individual who bought a used RS/6000 on eBay." IBM: "We can't sell you an AIX license without a customer number." Me: "Ok, well give me one." IBM: "What's the name of the company?" Me: "There is no company. It's just me as an individual who wants to learn about the technology." IBM: "We have to have a company name." Me: "Put my name as the company name." ... ... ... Eventually, get assigned an IBM customer number and did finally get AIX CD-ROMs from IBM.
I can only imagine that it would be 10x worse to do something similar with their iSeries (AS/400). When you see this over and over, I'm left with the conclusion that IBM doesn't want individuals getting their hands on their hardware or their software.
You are absolutely right about how hard it is to do business with IBM. I worked for IBM for nearly 7 years, running a small datacenter. It was such a pain in the ass every time I needed to order something. IBM sales and support process is very high touch, and is designed for the large enterprise that has a team of IBM sales people supporting them. If you are that customer it is not a big deal because you have someone to navigate the process for you. If you are a smaller customer, they want you to go through a VAR. Which is still a pain in the ass.
Windows NT and the AS/400 are about the same age. This pain in the ass sales process is a big part of why Windows NT won. The IBM sales people didn't want to sell it because you didn't need to buy services with it and it was too hard to buy on your own. Where as if you wanted Windows NT, you just bought a copy of Windows NT and installed it on an easily available PC.
We won't even talk about the time I tried to use IBM Smart Cloud Enterprise.
What gives me pause for concern is the JEE/WebLogic platform is now as old as COBOL was in the 1980's when we were calling it "dead" and we're still looking at 5-7 years of development and migration time to get to the new system. What will we think of JEE/WebLogic in 2030? My biggest concern is they're going to spend all this money to get onto another soon-to-be-legacy platform that they're going to realize on day 1 they need to start planning on getting away from. This is why mainframes have been so "sticky."
Off-topic but do you work for Eversource? An ex-coworker of mine found a bug that he immediately recognized as a COBOL bug while working with Eversource CT's EDI
So, you're looking at a high risk, multi-year, multi-billion dollar project to remove your own mainframe dependency. And the business isn't fully committed to it.
> Most CIOs are looking for a means for retiring those assets.
And finding extremely expensive, high risk, long duration software projects are the usual alternative, per your own experience. Why waste prestige and budget on that when you can keep all the critical LOB software running exactly as it is (or faster) forever? Mainframes are the OG backward compatibility and virtualization platforms.
Given that, why do you think the mainframe business isn't thriving, especially given the billions in revenue that the companies you list in your post undoubtedly bring in for IBM? It's not sexy, and they're probably not winning new customers. But they're making a shit ton of money on a long lived, successful product and continuing to fund research and development on the platform.
Hum... Probably (and that was the case at my workplace) because business as usual has higher risks and ongoing costs than the migration. It's not like mainframes are cheap to own, or guaranteed to stay in existence in the future.
> why do you think the mainframe business isn't thriving
"Thriving" on my lexical implies on growing or at least sustainable. A market where new players don't want to buy at any cost and old players are all trying to get out is not this.
IBM offers a compelling option:
- yes, your software will continue to run, because that's what the mainframe is for
- yes, it will scale with whatever transaction volume you need in the future, because that's what the mainframe is for
- yes, you can get 99.999999999999999% uptime, because you have reliability built into the platform in astounding depth
- yes, we'll happily help develop new functionality and support your in house staff in managing this system; the mainframe can actually detect faults and arrange a service call.
It's expensive, but it's a safe bet in at very least the medium term. The alternative is quite risky in most cases, and perhaps less expensive over the medium term if it works out.
Nothing is sustainable forever, but that sounds like a pretty hopeful story for IBM over the medium term. Long term, who knows, but eventually we're all dead anyway.
Maybe if you are using zos without any niche configuration. IBM has discontinued many mainframe platforms in the past, and there's no guarantee that next year they won't approach you with a 10x markup on your renting costs, that will become 50x in a couple of years and will keep going up until you decide to stop paying.
> yes, it will scale with whatever transaction volume you need in the future, because that's what the mainframe is for
Only if you gave-up on data dependency from the beginning. Transactions here means independent data processing, with possible rollbacks, not the relational concept people got used to on databases. If one needs relational data, it won't scale. NoSQL can scale with non-relational data too, it's not an exclusivity of mainframes.
> yes, you can get 99.999999999999999% uptime, because you have reliability built into the platform in astounding depth
That one is ridiculous. Uptime is limited by business code bugs, network and power outages, integration problems, and well, all that causes 90% of the downtime on a single-server shop. All the mainframe promises is that the hardware won't be a problem.
> yes, we'll happily help develop new functionality and support your in house staff in managing this system; the mainframe can actually detect faults and arrange a service call.
Yes, mainframes come with support contracts. You can get support for other platforms too, with the advantage that you won't need to hire IBM.
Anyway, none of that changes the fact that nobody wants to start using a mainframe nowadays, and that the people using them are either looking on how to replace them or avoiding increasing their usage. IBM will certainly be able to milk that market over the medium term, even more if they keep increasing their prices. They revenue may even go up for a while. But that's not thriving.
Imagine you are a CIO. You have mainframe apps. Everything I said is true of your existing solution. If you keep it as is, it works just fine for the foreseeable future. Every other way forward requires a total rewrite of systems that have been stable for years and are typically mission critical financial and transaction processing systems.
That's what I mean by IBM having a compelling offering.
There are advantages to the mainframe, but there are disadvantages too. As a CIO your job is to consider both.
My understanding is at least Java is well supported on mainframes. It's not the latest greatest language, but it sure beats ALGOL.
Then Algol, COBOL, RPG, NEWP, PL/I.
Even if you say "March 14, 2021 is the grand final feature freeze for the Mainframe, and day one of the Cloud Initiative", it's not going to happen. You're not going to have a MVP for several years, and a full replacement for five million undocumented lines of COBOL for even longer. Production fires will have to be fought. Regulators will demand you change something, and won't be keen to hear "we're supporting it in the Cloud Initiative, which customers will see in 2025." So you're gonna be writing quite a few more cheques to IBM and your team of COBOL jockeys.
After 5 or 6 years, as you've unloaded most of the customers to your painstakingly built replacement, you can finally begin to scale down the mainframe staff and perhaps return/downgrade some of the rented hardware.
Given that, it may be less a put-your-own-head-in-the-guillotine move just to spend those years hiring in a few new graduates with the promise of "We'll apprentice you to handle our unique and obscure codebase, and as long as you show up 40 hours a week and don't take a dump on your manager's desk, you're going to have a reliable paycheque for the next 20 years."
Which ones has it discontinued? I thought once the 360 platform got going, it was basically forever (unbroken chain of compatibility). What am I missing?
That might make it good business for a long time but it would be a stretch to call it thriving, and it provides gentle but salient cause for businesses to consider migration.
I work at a company that basically built its current business on mainframes in the 70s/80s/90s. The migration effort to get off them has been going on for more than 20 years and still isn't done yet.
Yep. They're already spinning up a big cloud migration project that will eventually replace many of the re-engineered on-premise systems, and they're still not yet off the mainframe (they're close, but that last 20% is always the hardest).
I've been hearing "mainframes are dead" since the 1980s. The market for them has dramatically changed, but by the same token there's just more people in IT who have no clue what the mainframes do from a business perspective, so "obviously" it's a dead end.
> And, since we're keeping score, zero content snark is bottom of the heap.
That's just hurtful, man.
Datacenter-scale compute on non-proprietary platforms (e.g. leveraging OCP standards and best practices, which have been developed w/ significant input from FAANG and 'cloud' vendors) can easily tackle any mainframe workload. The implied path is obvious.
That being said, it takes a massive commitment and pretty much a rewrite from scratch to achieve a jump from mainframe to modern DC. You need to embrace individual node failure and that's not something you can just hack in.
AWS, Azure and GCP are all proprietary, and all are producing larger and larger servers and VMs. They are basically rentable, virtualised mainframes at this point
Most of us didn't consider writing cross-platform software back then and the few who did were writing software for Unix, not Windows, and that because there wasn't a single dominant OS (Unix was a fuzzy target) and binary architecture you could write for, so you had to think about portability.
Oddly enough, there was a lot of cross-platform work on games in the 80's because the home computer market was very fragmented.
For graduates of my (very small) high school, Classics was the number one choice of college major and though I'm not in close touch with all of them I never heard any of them claim that they had made a mistake.
Cobol is sort of like Lisp* in that regard: the options are quite limited, but as the supply is too you'll probably do better than fine financially.
* wow is it painful to write that.
Which is why I used the vatican comparison.
It is also relatively rare for new languages cite Cobol as an inspiration for any of their features.
Cobol has a feature whereby dissimilar structures can be assigned. The structure members which have the same names are identified between the structures and the data is moved accordingly. Or something like that. A programming language which has such a thing could claim to have had that inspired by Cobol.
If I ever add that to TXR Lisp, I will be sure to drop a Cobol reference into the documentation.
I might have learned about this via Knuth; I think there is a mention of it in some passage in TAOCP.
This horrifies me on several levels, though I suppose it could be considered a sort "cast to a superclass".
Indeed, see Volume 1, section 2.4 (Multilinked Structures). Page 426 has the example of the statement “MOVE CORRESPONDING DATE OF SALES TO DATE OF PURCHASES”, which, if I understand correctly, means something like “for each field foo in Purchases.Date, copy Purchases.Date.foo into Sales.Date.foo”.
The discussion is in the context of what kinds of data structures would be needed to compile such statements, which I guess makes sense considering that the original publisher offer for TAOCP was to write about everything a programmer would need to know to write a (60s-era) compiler.
(Coincidentally: I've started attending an online meetup that someone is running for reading TAOCP—we usually read about four pages in the two hours—and this is the very topic that we're going to read tomorrow.)
Are Latin and Ancient Greek dead? Ostensibly yes. But not useless. Besides teaching, there is a tremendous amount of books and texts written in Latin and Ancient Greek. To study them, one would invariably need to learn the language.
Texts from the ancient world are handed down to us not from on high, or a single authoritative source (please consider the problem of duplication prior to computers or mimeographs), so considerable effort and erudition is entailed in comparing sources, filling in lacunae (here's a true warrant for what most consider a sesquipedelian word), all of which is predicated on perusing (in vain) an intuition for "pure et clare latine" and a hard-nosed insistence on _attested sources_ on matters of usage and expression.
I treasure, uncountably, the continuity afforded by training new generations in "antique" languages and dialects. It addresses a much deeper and more valuable problem than shitting out the next side-hustling platform.
These days, most companies seem increasingly resigned/accepting of the fact that they're not going to find young, dynamic COBOL programmers in the wild. On top of that, old-school COBOL has a knack for creating idiosyncratic software architectures. Learning how some particular monstrosity was taped together is more important than the language itself, which is brain-dead simple. And there are few such COBOL codebases out there in the wild to read.
The result is a move towards hiring and training good programmers in-house, quite counter to the general trend in the industry. Which makes for good job security but also makes it hard to enter directly.
If I could go back 30 years, I’d have made more money, slept better and would have more hair on Z. Not that I can say for a new generation but now that everyone is a “full stack” developer and the stack changes every few years, I’d still prefer longer term niches if I was 22 again.
Compared to webdev? I'm very skeptical. The following scenario seems more likely:
https://news.ycombinator.com/item?id=26431085
I’m not savvy on the costs, but I think the main benefit of ditching the mainframe is you can start using the IT tools, expertise, workflows, languages, frameworks, best practices etc. that 99% of the world is using.
With the mainframe, you’re in sort of a shielded of nook of the software world, where everything is fast and stable but also juuust a little different.
Key theme is slimming down the mainframe to bare basics, then take out either batch, or DB2, and then see to pull the trigger on the rest, 6y horizon...
But if you are spending a billion dollars, how much code do you have? Also, on the lack of programmers, just pay them more and you'll get a lot more applicants.
May I ask Why? If they are working perfectly well with a decent roadmap and upgrade path and quality assurance from IBM?
Hybrid cloud just means you have all the security problems of hosting your own infra, AND all the security problems of the cloud. Not to mention other non security problems of both systems.
When these CIO / CTOs make this "middle of the road" decision, they are dooming their tech budget to spend more on infrastructure and processes than actual product.
It's one thing to have a "we are both until we finish migrating to one" but to try and maintain both forever is just punting the problem to the next person.
I do not know whether or not mainframes are the right solution, though. I don't really know anything about them. I don't know whether or not the architecture is the correct approach.
>IBM has grown usage as measured by MIPS, a method of calculating the raw speed used, by 350% over the past ten years.
That's an awful metric. MIPS isn't even a good way of measuring aggregate performance and aggregate performance is a terrible way of measuring adoption.
This is probably not true in industries such as banking, manufacturing, agriculture, etc. I worked as a field engineer in a food processing plant once, during the peak season for the crop being processed. The plant manager told me that downtime could cost the plant up to a quarter million USD per day.
> Why do we accept that our hardware is unrelaible and then spend so much effort making our software tolerate failure?
From what I understand, the whole point of mainframe design is to tackle the issue that hardware is unreliable. Mainframes feature many hardware redundancies, moreso than a commodity rack server.
> I do not know whether or not mainframes are the right solution, though. I don't really know anything about them. I don't know whether or not the architecture is the correct approach.
Mainframes are designed for throughput, not performance. They are good at things like transaction processing, not machine learning.
Let's say both are running the same OS and application using VMs, and neither is doit specialized types of compute (i.e GPU stuff), just processing gobs of transactions.
Obviously there are big internal architectural differences, but ultimately these must manifest in some quantitative and qualitative differences and tradeoffs from a black box perspective.
Please discuss.
From the buying manager POV, one is cheap commodity and the other is expensive single vendor equipment.
From the programmer POV, one is a proprietary environment with uniformly addressable memory (not UMA, just addresses) and the other is a distributed blank canvas where you must write the IPC yourself (or use a library, of course). Both have a huge IO capacity, but mainframes can integrate data much faster (and usually, process it much slower).
From the ops POV, one is a single machine (a heavy one), and the other is an entire datacenter.
It's a single server with massive amounts of cache (like 1Gbyte or more of the L3 cache) and optimized everything.
When you login via ssh to your server, each keypress creates an interrupt which is handled by the CPU. If you are using emacs , even on the terminal it might be multiple interrupts due to screen refresh etc.
Mainframe: generate the form or the screen data. Send it. 1 interrupt. Your terminal or terminal program handles everything until you send 1 or more lines (not characters) back. A dedicated processor handles communication with the terminal. It then bundles up whatever the changes are and sends it to the CPU. 1 interrupt.
Disk, network etc. is the same, with dedicated processors on the network card or disk controller that go out and send/retrieve the data without CPU intervention. Database table retrieval: the table is sent to the CPU, already sorted in the requested order. Etc etc.
1. It’s difficult to obtain skilled personnel, that’s it.
Why, because you can only truly learn mainframe technologies on a mainframe. Hercules and all the things that IBM have done themselves such as MTM, VS Code extensions are not enough. The barrier to learn mainframe technologies is too high.
I agree with your main point though. Occasionally I’ll see someone recommended people to learn COBOL to get a job as if the language is the most relevant part of that industry.
Seriously. if you pay IBM for a couple of z series machines, you can link them together in step, and literally blow one up, and you'll only loose a certain number of CPU cycles.
sure they are expensive, but its cheaper than buying a team of ten engineers for x years to make K8s do the same thing.
Also, you don’t buy mainframes, you rent mainframe CPU cycles. Or actually, you will pay for the rolling window 4 hour peak of CPU utilization.
IBM will gladly ship out the fridge sized box that holds those CPU’s to your data center for free, as soon as you turn it on their cash starts coming in.
By that time there were a few other mainframe class manufacturers, most notably Tandem and Amdahl.
Notably missing (and most relevant to the 2020s tech scene) was DEC, who by the late 60s was making smaller mainframes as well as minicomputers. What makes them different was their decision to tae a different, and what Clay Christainsen would call disruptive path. And while highly influential they were themselves nuked by a new class of disruptive companies.
IBM has reinvented itself several times over in its almost 150 year history. Not sure the current one is successful, but I suppose it's still too soon to tell.
Is it even possible for an ordinary person to rent a VM on one of these machines and do some compute? If I have a startup that needs to crunch a bunch of models or would have an easier time running some system centrally versus having to have the skills & time to create a purely distributed system, why aren't mainframes a good/better/easier option? How come AWS doesn't have any? Does IBM have quick/easy on-ramps to hosted mainframes via Softlayer (I can't find them if they do)?
The only real remaining advantage is probably the overall reliability, redundancy, engineering, etc. Like being able to swap out a CPU while the system is running.
The chip is impressive, because they don't skimp on costs, because they sell them for a million a pop, but consider how many ryzen on xeon cores you can get for that price, with all the cool DRAM and IO tech you want.
https://en.wikichip.org/wiki/ibm/microarchitectures/z15
compare with:
https://en.wikichip.org/wiki/intel/xeon_platinum
https://en.wikichip.org/wiki/intel/microarchitectures/cascad...
A mainframe (in recent decades) is essentially a proprietary, high cost, tricked out server farm in a box (which is often big enough to be a small building, btw), which might make sense in a few cases but mostly is not as good as a much larger, non-proprietary mainframe, called a "server farm", which is supported by several competing vendors.
Well, server farms don't usually expose a single system image. You can't set the OS to "migrate" workloads seamlessly across hosts the way it can migrate them across CPU cores. Linux is now getting some of the underlying features that are needed to enable this use case, but integrating them into a coherent whole will take some time.
In a way, the kubelet is the kernel of a k8s cluster. You wouldn't run masters on every node in a cluster, because consensus gets harder quadratically. Maybe workload migration doesn't have to live in the (linux) kernel.
Now that is crazy cool.
Because mainframes are more than just a quantity of computation. Their hardware and software has prioritized reliability and resiliency far more than the PC architecture, which pretty much everything else is based on nowadays (and which started out and remains mainly about low-cost).
In some ways the approach you describe would be like replacing a ZFS RAID array with an a single large disk with a FAT filesystem. It's possible, but you probably don't want to.
Full disclosure: I'm not a mainframe developer, but I've read quite a bit about them.
[0] https://www.cnbc.com/2020/04/06/new-jersey-seeks-cobol-progr...
That’s when I got out. When it came to switching jobs I had zero experience in current tech and had to take an entry level position and a pay cut. A few years later that same company had massive layoffs and all development outsourced by 2008 or so.
So yeah whenever you hear about opportunities in mainframe cobol don’t believe it. I’ve got 2,000 former cobol developers from that company alone who will attest that its farce. What puzzles me is who’s behind this idea and what’s their goal in promoting it?
I'm surprised to read these recent anecdotes (going back to that New Jersey unemployment COVID programmer shortage) from various people that COBOL jobs are in high demand yet shockingly underpaid. That would seem to imply markets don't actually work the way we've been told.
The difference in this case is that the press is showing sympathy to those who won't pay the price and then complain about the missing supply. That doesn't happen in most markets.
Why is the press listening? Because the people complaining are people that the press normally listens to - banks and big, big businesses. It seems that many such businesses would rather complain, publicly, than pay a market-clearing rate. Well, I don't have a lot of sympathy, to be honest...
The other way of reading the situation is: There's nobody interested in working in COBOL at the suggested rates.
a bit off topic, but this is a huge thing that I at least didn't pick up on across jobs for many years - for accounting reasons, many firms will always see what your team is doing as an 'expense' to be 'minimized', no matter what you do, and in many cases, there is nothing you, or your manager, or even your managers manager can do to change this, since it is often decided at the highest levels of the firm based on broad-stroke accounting policies that have a direct impact to how the firms entire books (and therefore revenue/profit/etc) are calculated.
IT/tech related? goes in a cost center.
What's a cost center? an optional expense which can and should be minimized, with manager compensation often tied to how effectively costs are controlled.
If anyone working in tech is not aware of the various accounting models used in business, it is well worth your time to get familiar with the basics - assignment of 'cost center' to technology groups in firms that use cost/revenue center models has a huge impact on how teams are likely run, decisions are made, and potential for career advancement for employees in those groups.
https://en.wikipedia.org/wiki/Cost_centre_(business)
https://en.wikipedia.org/wiki/Profit_center
this is not the only costing model (job costing, activity based costing, etc) , but one in my view that is often hugely detrimental to technology teams in non-tech companies unfortunate enough to find themselves pre-categorised as a 'cost center'.
Think this is highly dependent on the school/student; In my experience (data point size=1), MBA teaching hasn't really caught up to the fact that basically everything is technology these days, so it shouldn't be treated as something 'separate' to the 'real' business, which it often still is - i'd wager that the resulting compartmentalization in thinking around how to manage tech is the main driver behind allowing poor systems to capture IT value from the financial perspective...
Shame IBM won't sell 'em to us...
https://popcon.debian.org/
There are more Debian GNU/Hurd systems than Debian s390x systems :)
https://www.theregister.com/2017/05/24/hitachi_exits_mainfra...
https://en.wikipedia.org/wiki/Amdahl_Corporation#Amdahl_cust...
Since they are all Turing machines perhaps we can also use AI to generate virtual COBOL programmers from disk images.
Anyone interested? ;-)
Two years back Db2 has become strategical database previously it was oracle in the decision tree. Security and up time has been solid.
Just seeing the trend at workplace on work related to mainframe, I would say it's picking up. Basically we know the work(projects), so we are focussed on onboarding process for newer applications as we are more and more working with people who have no idea about mainframe or who were once on the bandwagon saying mainframe is dying once.
Interns are trained internally after college. I would say that's the best training.... At first, they hate it. It's all black and green text, but slowly they pick up.