Ask HN: Why is there so little info on the web about IBM mainframe programming?
I recently heard on a podcast that IBM is selling more mainframes today than at any other time in its history. This piqued my curiosity. I don't know the first thing about mainframes so I was interested to learn how one goes about coding for a mainframe. But a Google search on this topic brings up very little. Most of the results are on IBM's own website, and even that contains only high-level details. The usual places like Stack Overflow and YouTube have a dearth of info on IBM mainframe programming. So, my question is: why?
The most likely explanation would be that there are hardly any mainframe developers. But IBM's website proclaims that "ninety percent of Fortune 500 companies rely on the IBM Mainframe" and "eighty percent of the worlds corporate data resides or originates on the mainframe." If that's true then surely there must be a reasonable number of mainframe developers out there? And if there are, then why is there so little info for them on the web? Is the knowledge proprietary and secret? Or is it just somewhere other than the usual places?
I would be really interested to hear from anyone with knowledge about this.
137 comments
[ 4.6 ms ] story [ 246 ms ] threadhttps://www.ibm.com/blogs/ibm-training/free-course-announcin...
I am sure there are others out there too.
* The documentation for mainframe stuff is incredibly comprehensive compared to what most people are used to, so there's less need for something like Sewer Overflow.
* The code people are working on is likely to be proprietary, so no GitHub.
* Mainframes are often used in security-conscious environments where "openness" is not a core value and might even be considered negative.
* People working in that area might like being part of a relatively closed community (lower supply = better pay) so they don't advertise.
And then there's just critical mass. Since there aren't already a lot of mainframe programmers Out There, and mainframe programmers' work has few technical "touch points" with the rest of the world (e.g. different languages, libraries), it almost makes more sense to flip the question around.
Why would there be more info on the web about IBM mainframe programming?
So even if you do see a mainframe programmer in the wild asking a question, you probably won't recognize it.
2. developerworks site on IBM had some info on mainframes and mainframe programming.
3. It is not "cool".
I've no knowledge of IBM mainframes - but I assume they're sold as part of a large consultancy/training package.
I would also dig through Project Guttenburg https://www.gutenberg.org/ for IBM Mainframe Programming books if you're interested in the history.
They weren't the types to go looking on StackOverflow or Youtube for answers.
They would also help train new joiners, and IBM offers good training too.
The kinds of systems running on mainframes don't lend themselves to copy/paste from StackOverflow type programming. It's a lot of credit-card and banking transactions where you want experienced people writing and helping on it.
Secondly, even if IBM is selling more than before, that's still an incredibly tiny fraction of the computing scene. Companies don't tend to have more than one (or possibly 2, with one for DR mainframes). So, that 80-90% of the Fortune 500 is really just 1000 mainframes total. Generously, if every Fortune 500 had a pair in each geography (roughly EU, Americas, Asia for big corps), that ends up being 6, so 3000 globally, in production. There are also some spread around academia, government labs, etc. But the numbers are dwarfed by standard x86 machines. eg. at one bank we had "the mainframe" which was actually one hot and one DR, and then 40k windows servers and 40k linux servers. Similarly, there were thousands of engineers in technology, but the mainframe team with single digits.
Given the wealth of information out there, even if there was "a lot" of information for some value of lots, it's dwarfed by the incredible amount of standard tech info.
There are F500s that have 1k mainframes each, today, in 2022.
This helps you build up an immune system that is skeptical. If most wrong answers were removed/flagged/hidden then the remaining wrong but unflagged would be even more a problem.
And unlike stackoverflow, there’s not really any good reason for anyone to think a top-level comment is somehow authoritative, especially because the best comments on HN tend to be replies
Good for debate and free speech in general, but bad for your grasp of the facts in the case every time there don't happen to be any corrective replies.
Hard to weighs pros vs cons.
(ObSillyPun: Naah, just put them on the scales! But remember that a lighter pro will often beat a heavier con anyway!)
There might be less mainframes in total, though. The bank where my parents worked had 4 at different sites, but now runs on a single one, AFAIK.
Right. The main limitation is the core software platform. This is an extraordinarily complex system that requires expensive regulatory sign-off/audit/etc.
There are some vendors in the space who are trying to do a new bank + new core. Even if someone had an awesome x86 clustered core system with 100% regulatory approval, any existing mid-tier bank converting to it would be an ~8 figure ordeal.
As others have posted, your estimates are WILDLY off.
First, fortune 500 companies dont own "two" mainframes, but a LOT more.
Next, many non-F500 companies own a lot of them as well.
Years ago I worked for a local cable company in Canada.
Not a "F500" company, owned at least two mainframes for the "head office" and each different 'subdivision' owned their own mainframe.
> Really just 1000 mainframes total.
I'd be willing to bet that any major city has more within its borders.
And they're minicomputers, not mainframes :)
Multiply his estimate to 1000 per F500 company ... that's only half a million
Put a pair in every major geography ... that's only 5,000,000
And, for kicks and giggles, multiply by another 10 for funzies
That 50,000,000
Yes, it's "a lot" ... but it's nothing compared to commodity hardware
According to a quick web search, Google alone had in excess of 1,000,000 x86 servers about a decade ago
That's one (large) company
I've worked with myriad companies who're running 10s or 100s of 1000s of x86 servers (let alone x86 desktops/workstations/laptops) - with nary a mainframe in sight (or maybe a couple ... which is on the order of "none" next to 100s of 1000s or millions of x86 boxes)
Per Microsoft [0], there are 1.4 billion Windows 10 and 11 devices (yes, I know that's not a server count, but it speaks to the orders of magnitude difference between x86 devices and mainframes ...even with my generous estimates posted above)
-----------
[0] https://news.microsoft.com/bythenumbers/en/windowsdevices
Mainframes on the other hand or more likely than not to have software that only runs on a handful of machines at most.
So number of machines is not a good proxy for amount of software written.
On the other hand I’ve never touched a mainframe so it’s all speculation…
I find that claim remarkably improbable - the overwhelming majority of all code written is for in-house business tasks
It is a good proxy for the relative need of developers, however - with billions of x86 devices out there vs max a couple-few million mainframes...the scale of need is just wildly different
https://github.com/FuzzyMainframes/Awesome-Mainframes
There is lots of content there.
The equivalent from IBM is called "IBM Z Development and Test Environment" and the license for that costs $5,540 per user per year. There was (maybe still is?) a "learners edition" that you have to contact them to "see if you qualify" and it is a far more reasonable $120/year (there are no links on IBM Marketplace to even try to order it). This includes several compilers and 3 databases: CICS, DB2 & IMS.
IBM has a very different model for supporting developers than Microsoft does with MSDN.
Links:
https://en.wikipedia.org/wiki/Hercules_(emulator)
http://www.hercules-390.org/
https://www.ibm.com/products/z-development-test-environment/...
https://ibm.github.io/zdt-learners-edition-about/
https://old.reddit.com/r/mainframe/
Ballmer being excited: https://www.youtube.com/watch?v=wNy2zBG79zU
After all, IBM wants you to pay IBM to do this stuff for you.
It's not like Bob from IT is going to play around with mainframe coding for a year or so and convince the boss to buy one.
So imagine you're a company selling low volumes (by modern standards) of exquisite computing hardware to highly demanding customers, in a world where... (1) there is no Internet, (2) there are no personal computers, (3) most people don't know how to use a computer, much less program one.
You'd probably come up with a model that looked a lot like "We'll sell you the machine + train you how to program and use it!" as a package.
And then after the world changed, but your offering was very polished, you probably just wouldn't overhaul it. Because it's been working fine for 50 years.
not currently, IBM pretty much actively prevents this with their current pricing.
it could happen, and would, if people were allowed to learn this on their own.
Mainframes are very good at some things. Far better than commodity hardware is today.
AFAIR, they offer the possibility for a one year free access to a node (Linux/IFL?). I do not remember if it is part of IBM Z Xplore and I didn't find it again with a quick search, but if you contact them they for sure will tell you more.
Edit: I just found https://www.openmainframeproject.org/
"Hello World on z/OS" (2018) https://medium.com/the-technical-archaeologist/hello-world-o...
Hn discussion:
https://news.ycombinator.com/item?id=17642846
I seem to recall IBM launched z/os in the cloud as partly a production service, and partly an education/hobby resource a while back - but I'm not sure if that's what's now:
https://www.ibm.com/cloud/pricing
I think it came with access to db2 as well?
Ed: I was probably thinking of IBM LinuxOne:
https://developer.ibm.com/articles/get-started-with-ibm-linu...
Interesting comparison, because when I was teaching myself to code in the early 00s, that's how I perceived Microsoft.
MSDN was a paid subscription, and the tools you needed to write MS software were also paid.
Hence me ending up forming a career on open source software, and a generation of programmers who still turn their nose up at Microsoft languages.
The parent comment noted that there was a generation who shunned Microsoft because of their corporate misbehavior. More recent generations of programmers have forgotten why that was, and Microsoft has, once again, successfully established a dominant position in the development ecosystem.
I fully expect that position to be leveraged for profit, and that would see a new generation, 25 years later, who will never trust Microsoft again.
Most things are cyclical.
"Learner's Edition is currently being updated and will return soon"
Let's hope "soon" is actually soon :)
[1] https://www.ibm.com/products/z-development-test-environment/...
Sometime later, a connection on LinkedIn pointed out the following resources:
https://www.openmainframeproject.org/
https://github.com/openmainframeproject
https://github.com/openmainframeproject/cobol-programming-co...
Also, mainframes are really good at high throughput transactional jobs. That's why you see them in banks, transportation, insurance, etc. Big Tech™ doesn't see it as "cool" and are too focused on the Next Big Thing™, so there's not a lot of attention there. Sometimes, boring just gets the job done.
During my vocational training, I did a bit of work with mainframes and found a forum back then I liked, but my work (and the forum) was mostly about sysadmin-stuff, and the little coding I did was in Perl.
There is a sizable library of RedBooks on IBM's web site somewhere, but again, the stuff I was interested in was about system administration.
A lot of the culture in the mainframe world developed a long time ago, and in relative isolation from the rest of the IT world, so I suspect by the time Stack Overflow became popular, their communities had already found other ways and places to mingle and pass knowledge on and around.
With hardware, software, and support contract being so expensive, and corporate culture being what it is, I also suspect the threshold to call up the the vendor for support instead of Stack Overflow is much lower than in most other areas.
(Most of this is speculation on my part, I only spend a few months in this fascinating corner of IT. If I'm wrong, I'd love to hear to from someone with actual experience.)
Indeed, and not just Java. zLinux runs a great many workloads on M/Fs these days, which looks a lot like a normal Linux env.
Then ofc K8S/docker/etc can run on mainframes, presenting another layer of abstraction.
Modern mainfame programming is just a question of getting to know a slightly different set of tools.
wtf why even plow $$$ into hardware at that point?
A corporation might well calculate that, for its typical workloads, mainframes make a better K8s platform in the long run than server farms.
I don't have benchmarks, but googling "mainframe energy efficiency" brings up a load of results, eg:
https://www.bmc.com/blogs/mainframe-sustainability-green-it/
https://www.serverwatch.com/guides/for-energy-savings-consid...
https://www.suse.com/c/mainframe-versus-server-farm-comparis...
has a lot off z howto books
a lot of howto books
StackOverflow in itself is the answer to the question "who you gonna call" -- because you might be alone, or you might be working on a small team where no one has better expertise, and usually you are working with open source and so no one is obligated to pick up the phone at the software "manufacturer". This scenario does not play in a mainframe environment: you are going to be part of a larger team, full of literal graybeards and when push comes to shove then you do have a contact number: your employer is paying absolutely ridiculous amounts of money to the software company and they will have a contact person who will pick up the phone.
Not only that but that redundancy is hidden from everything. Already decades ago the Hungarian electricity distributor had a "RAID" storage unit where one bunch of disks were at the capitol, the other bunch connected with fiber at another city -- and you talked to this thing as if it were an ordinary disk. Short of an invading army nothing could cause data loss on that thing -- simple aerial bombing wouldn't because the unit in the capitol was tens of meters underground... that's not concerns you have on your average web app.
We have the technology and the capability to do things like run a VM and have it move between hardware when that hardware fails or needs replacement.
but that would cost money.
This is one of my favorite features of Google Cloud:
https://cloud.google.com/compute/docs/instances/live-migrati...
Unfortunately, Google continues to be the third-string provider in the US, so I don't often get the opportunity to work with Google Cloud.
I mean, maybe not in an old-school 1970s batch-SIMD mainframe environment with limited storage memory.
But in a modern mainframe environment, where
1. every tenant being processed is hitting their own virtualized environment locked into their own legacy versions of everything (think: separately-SCMed "Enterprise" deployments of SaaS software — just all living together on one machine cluster rather than on-prem per customer); and
2. you've got the spare storage capacity to do everything via async-queued CQRS/ES + writing secondary representations and cubes to a local data lake, rather than ever updating anything in place...
...then why not do "agile" blue-green deploys of your ACH reconciliation logic, starting with the smallest tenants first?
If you ever produce a bug in such an environment, you can always just: stop the job; emit tombstone records for all the CQRS Commands it produced so they'll be ignored on "outbox processing" rather than pushed to consumers; fix the bug; up the version number for OLAP report generation, invalidating + requiring recomputation for any reports produced from the data in the reporting period; and then start the job up again. No external system will ever have to know, because your own system should be set up such that no other external system is synchronously dependent on its outputs. (This is half the reason things like ACH are designed with the async "flex" they have.)
And, in fact, assuming you've got the spare CPU power (which you do), you can even run entire reporting / integration cycles twice — once under the old logic, once under the new logic — and examine the resulting queued commands + reports to see that they all come out the same, before allowing anything to go through. You can do this not just in production, but experimentally, against a corpus of all old input data for each tenant (which you've almost certainly kept around.) And, as such, you don't even have to stick with those virtualized legacy environments that mainframe architectures like z/OS were designed to enable — you can actively migrate clients away from them (even automatically, with CI/CD-triggered graduated rollouts et al) as soon as you can prove that their workloads (both present and historical!) won't be affected by the change. Each tenant can be re-pinned over time to the newest release that can be proven to be equivalent to the original release provided to them on all known data.
One of the key insights about mainframe programming, is that the software architectures of the kinds of systems deployed on mainframes, are designed to be resilient against individual bugs; or, one might say, against individual programmers who don't know what they're doing. Systems engineering in e.g. finance, is to bad individual-component / individual-release logic, as NASA's Mars-rover systems engineering is to bad hardware: designed under the assumption of the potential for failure, and designed with built-in remediation strategies for that failure.
So that cultural value of keep everything close to the vest and only learn through official channels, still persists in IBM mainframe culture today. Especially since there is little to no "hacker culture" surrounding mainframes since it's not like you can download z/OS and run it on your PC (things like Hercules notwithstanding).
At least before 1980, during the IBM System/360 and IBM System/370 generations, abundant documentation about any aspect of any IBM product was available, much more detailed than any modern documentation.
Moreover, for many of the programs provided by IBM with the hardware, the source code was available, even if they were not open-source in the modern sense, because the users were not encouraged to modify and redistribute them, even if such activities happened sometimes, e.g. within the IBM SHARE user group.
When the IBM PC was introduced in 1981, this policy of providing exhaustive documentation was still active, which is why the IBM PCs were delivered with a ton of manuals, including descriptions of the hardware interfaces and the complete listing of the source code for the BIOS firmware.
There is Code in a multitude of languages even when it is not Linux, from C++ to Java, but nowadays also nodejs and whatever you want to run in that architcture.
Edit:
Basically, you should look for "System Z" which is the actual name of the machine, and "redbooks" the name of the documentation manuals.
Example: https://www.redbooks.ibm.com/domains/zsystems
I think this is the equivalent - https://ibmzxplore.influitive.com/users/sign_in
I've had this bookmarked for years, never gotten around to starting it, though. Web development continues to entertain, challenge, and sustain me.
And on the application side you are typically dealing with databases and Runtimes(Java mostly) that also exist on Linux and have an large Linux centric community where any legacy programmers and admins kind of drown in the masses.
There is a whole bunch of companies out there with large COBOL and ALGOL code-bases but but as the problem here is less about generic mainframe skills and more about knowing how to work large COBOL or ALGOL projects.
Check out:
https://www.ibm.com/z/trials
https://www.redbooks.ibm.com/search?query=%22system%20progra...
https://developer.ibm.com/?q=z%2Fos
Also search ibm.com for terms like "PL/I", "COBOL", "CICS", "RPG" and so on and you'll likely turn up more of the mainframe stuff. You can, of course, program mainframes in other languages, especially Java. But from an IBM perspective, references to the former batch of terms will generally turn up stuff about their mainframes, or their minicomputer series (mostly what's called iSeries these days, as far as I know. Formerly the AS/400, S/38, etc).
There are also various print books from 3rd party sources that can be found on Amazon. Ex:
https://www.amazon.com/IBM-COBOL-Complete-Guide-2020/dp/0655...
https://www.amazon.com/Quick-Start-Training-Application-Deve...
etc.
The reasons for this are, so far as I can tell, largely what other commenters have already posted. It's just a different culture and way of doing things, and the mainframe world just works a bit differently. And I think a lot of that is because that world is so insular now with IBM being more or less the only mainframe vendor left. I suppose Unisys or Hitachi or Fujitsu or somebody may still make a mainframe here and there, but the market is almost completely dominated by IBM. So the "mainframe way of doing things" is more or less "the IBM way of doing things" and in particular it's a very old skool version of "The IBM way" at that.