My first job out of college was working at a large financial services company in one of their departments that, among other things, handles corporate actions on behalf of its customers. Much of that corporate actions processing is done as part of a nightly batch cycle that runs on an IBM mainframe. The system itself was first deployed and continuously maintained for longer than I've been alive.
Long before I joined the company, that department tried to rewrite the system to be "modern", porting everything over to Java and having it run across a couple of servers. This was in the late 00's if I remember correctly. Apparently this rewritten system couldn't handle the volume of data that the mainframe can easily process during its batch cycle. However, were they to rewrite the system now (having the tools available to better facilitate real-time processing along with flexible resource scalability) I bet that a new system would be able to keep up with the demand and be just as expensive, if not a little more cost effective to run.
Does anyone else have any experience updating and rewriting old mainframe batch processes to newer systems and architectures?
I work in a bank and we have a team of people writing microservices on top of mainframe "programs". I'm not sure how it works in the mainframe because I work mostly on the front-end, but apparently what we are used to in modern REST apis are just called programs in the mainframe. Anyway, I digress, what we are experiencing is quite the opposite: operations written in the microservice run much faster and we are kind of bottlenecked by the mainframe.
I also recently came to learn that the mainframe run jobs in batches, so I'm not sure how does that affect performance, if it even does.
This is what I see most shops doing, including ours. I think the cost of getting off a mainframe is too high, so this will likely continue to be be the trend for the foreseeable future.
Also, mainframes can run both batch jobs and "online" (CICS) jobs. Our shop is about half and half with batch and online programs. The main thing that batches can be annoying with is that they run on schedules and may have to wait for other jobs to complete before they can kick off, causing delays. We just upgraded from a z13 to z15, so our mainframe is no longer a concern but we are also much smaller than a bank.
IBM's profit margins are so high they sell you a machine that's fully configured because they know you'll pay for the upgrade that's already delivered but deactivated in a year or so.
If moving off the mainframe becomes cheaper, they have a lot of flexibility
I think this hints at the best practice, establishing APIs around the edges of the legacy system, then sectioning off chunks of legacy functionality, building internal APIs inside the legacy code around those chunks and then replacing those chunks with modern systems and making those APIs external (external as in legacy -> modern instead of legacy -> legacy). Some legacy core pieces might never be replaced but those can be contained so the bottleneck is limited. (a good example is genuinely batch operations likely nightly email alert processes, while it might be nice to modernize those)
My limited experience with this is this strategy is that it works, although it is not error proof and can be messy. You also end up discovering that assumptions you made about the legacy software are not correct and then you need to back pedal and potentially throw away previous work (ironically your own assumptions about how to modernize the legacy code become their own legacy that you need to work around as the project ages).
It's good if you have some of the original developers still around and probably worthwhile to give them consulting $$ in order to bring them out of retirement.
Perhaps the best thing about this strategy is if you in the end conclude it isn't worthwhile, you will still have modernized some sections of the codebase. Of course, because of this you need to make sure that the individual section by section modernization efforts are themselves a net gain, instead of just hoping that the final end of the modernization process will be a benefit, because you might never get to that end point.
Also, you might be able to enhance the product early on by being able to iterate and improve the sections that have been modernized, instead of waiting till everything is done to flip a switch and get new features.
The downside is this is a slow and gradual process. In the meantime your developers are filling their heads with details of a unique system that is hopefully going to go away never to be seen again.
If you have specs and documentation and tests and all of that good stuff, it might be better to just start from scratch, and build things up according to requirements.
I can second this - our mainframes average response time is 2-5 seconds based on the call, with 99 percentile responses in the minutes per transaction. This is after a fortune was spent moving us to a "faster" platform.
Also, at our bank, very few applications use the REST API. Since most applications are developed by contractors, most vendors don't want to use our custom API, and instead opt to connect to the mainframe directly, because then they can reuse the code at other banks after charging us to develop it.
Isn't this confusing latency with throughput? Mainframes are designed for throughput, raw TXNs per second, and a lot of the architecture is designed around batching. Which is great for throughput, but the antithesis of latency.
Mainframes (Z/OS) will mainly run batches (most often COBOL programs controlled by JCL jobs) but there is also CICS (https://en.wikipedia.org/wiki/CICS) which runs COBOL (and probably others) "on demand".
My experience with connecting UNIX to Z/OS was not with a real API (HTTPS) but UNIX > TIBCO EMS > something called TIBCO Substation > CICS. It was horrible, mainly because of Substation which needs to translate on the fly between two completely different platforms.
Last time I worked on mainframes in 2012 the price to have a legacy banking app running with around 300 transactions/sec on IMS or DB2 was just insane.
IMO any "modern" architecture can handle the load. After all Google, Apple, Facebook, Microsoft, Amazon, etc. are not running on mainframes (just taking mastodon examples).
The main difference between ARM, x86 or Power vs Z is that you can keep your CPU usage between 95-100% in production and not break everything.
Yep. We migrated from IBM Z13 running COBOL + DB2 to HP machines hosting RHEL VM's running TIBCO, JAVA and Cassandra. Similar load as you mention.
Performance is really not an issue, especially when you factor in the price (we could easily double our servers and still be cheaper than a pretty baseline Z13).
Main pro of the mainframe is: you can basically not touch your code for decades and it'll still run fine, every technology on it and the hardware itself is rock solid.
Main pro of leaving the mainframe is: you don't need to worry as much about expensive/hard technical debt (i.e. EBCDIC text encoding, special hardware for everything), pricing/billing is much simpler and cheaper, way easier to switch technologies.
Not personally, but most every migration attempt I have heard about has gone similarly.
We run a mainframe shop and for us the cost is fairly reasonable, and the high bandwidth/reliability (even with the cpu pinned to 100%) is the most critical aspect to us. Ignoring the risk and cost of migration, I am not even sure that an alternative would be cheaper... but it is quite difficult to determine the cost and my boss obviously factors in the cost/risk of migration anytime this comes up.
I'd be curios to hear of any successful total migration attempts. Most shops I know of keep their mainframe and build around it with modern tools and we do as well.
That's been the experience I've had as well. One company I worked for around 2015 had 5 or 6 mainframes (which they kept upgraded to the latest hardware on an IBM lease program every few years), and it was well understood that the number of distributed computing servers required to keep up with the volume and resiliency of the mainframe would be too great to even consider.
Another company I interviewed with did just what you said, they built REST APIs and other "modern" interfaces to the mainframes so that 99% of app developers in the company don't have to learn anything about the mainframe.
Around 2000, I spent some time developing snazzy new Java e-commerce stuff, at a large company that was mostly based on midrange AS/400's running RPG code. At the time, they announced a huge project to retire RPG and re-write everything in Java.
I've stayed in touch with some friends there all this time, and apparently they still haven't finished this project as of 2020. Disaster after disaster.
The first executive couldn't execute, so he moved on and the project was slow-rolled for awhile. Until enough time had passed for a subsequent executive to claim it as HIS "fresh new idea".
That executive came it very aggressively, actually pushing out all of the RPG employees and contractors well before the company was really ready. So that executive was pushed out, and his replacement had to hire IBM Global Services to come in and take over everything. IBM hired most of those employees and contractors who were let go. So the net result was the same people doing the same job, with the company paying 2-3x more than they had been paying.
It was around 2015 when they finally reached the tipping point, to where the company was "a Java shop with some legacy big iron" rather than "an RPG shop with some Java e-commerce pieces". But there's still a lot of AS/400 on the inventory side of things, and it will probably be another couple of decades before that finally goes away.
A lot of people on HN and Reddit, who are either students or startup employees, just have NO IDEA how things work for large businesses in the other 49 states. Stuff lasts forever, it's just a completely different world.
What people fail to understand is that the rules/logic of a particular business that has been around for awhile can be absolutely enormous, complex, and poorly documented outside of the thousands of lines of code in production systems .A lot of this is due to historical choices that nobody remembers why a decision was made, but another problem is that many employees hate documenting what they know as it is either boring or they think it makes them less valuable, and management usually doesn't push it because it isn't a big quantifiable win on their resume and most aren't going to stick around long enough to see the fallout from the loss of institutional knowledge over time and turnover. Rewriting can be very challenging even with the domain knowledge, onboarding people to be competent in the business can take close to a decade... etc. They're kind of like programming languages where there is no spec other than the interpreter.
In 2015 I took a contract job with a large team at Verizon because I am an old man and know both Java and COBOL. The idea was to replace COBOL code with Java. The Java ran on IBM iron that was similar to the existing COBOL system. The COBOL code consistently ran at least 3-4 times faster than the Java code, no matter what we did. I think that the bottleneck was access to the huge Verizon customer database, but even the straight Java ran slower than the equivalent COBOL.
They moved the project to another data center and I lost my job (my last job before I retired). I don't think that they ever got things going. I'm not sure that they ever will.
COBOL is ridiculously efficient when it comes to doing bulk IO when coupled with the right hardware that is optimized for COBOL. Attempting to use some other language to do the same thing will have the same effect as inefficient memory access patterns have on a GPU.
There seems to be a moral to every Old language replacement story.
I am Not sure why languages are suddenly considered obsolete, I think it is that all the programmers suddenly run over to the next big thing leaving massive gaps in staff that mean recruitment becomes a problem. If I was in the market today I would be running the other way and embracing what currently works.
"Maintained a COBOL system that processes twenty zillion transactions per hour" just doesn't sparkle like "replaced 40-year-old monolith with a galaxy of JavaScript microsoervices that get winded at 100 requests per hour."
> Does anyone else have any experience updating and rewriting old mainframe batch processes to newer systems and architectures?
A fair bit and key is that the business process and understanding of all that is more key to getting a good transition over anything else.
The other key is small chunks, identify aspects that can be shifted across bit by bit - reports that are using daily data can be a datawharehoused data affair with your reporting system tapping that - that opens up tools for the the various departments to do ad-hoc reports by themselves without suddenly spiking the key-business loads.
But so many avenues and aspects there is no simple list to go thru, as each one depends upon the set-up and more so - business logic.
So I'd take a COBOL programmer who knows the bushiness over a top-end java programmer who does not know the business for a migration project as be easier to teach the COBOL programmer java than it would to teach the java programmer the business logic as would take sooo long and to learn all the nuances and what needs to happen when things go wrong and what is priority and the dependencies. Alas many management and project types will see a glossy top-end java programmer and pick them and ignore their in-house talent all too often.
However - batch stuff is maybe the easiest area to work upon - less users in the equation and with that - less variables to go wrong ;).
Mainframes are also not just large legacy number crunchers - the whole aspect of resilience and fault tolerance goes much deeper than ECC memory, RAID and dual NICs and PSU's. Much deeper and that is why they have up times above and beyond your X86 server class. This with database and messaging systems equally tried, tested and robust. Which gives many factors to work upon.
Though as always - business logic and process along with legal requirements and other constraints for data-flow are key and certainly flowcharting that all out and the dependencies would be key to have anyhow and essential for any migration.
I worked at * bank [Australias oldest bank!] within the last 3 years. The entire bank CRM (~12 million active customers) is a Z/OS CICS system which was first installed in about 1972-73.
The same goes for all wire transfers, credit card transactions, ATM infrastructure and so on.
The banks solution was to simply write about 40 different Javascript and Visual Basic interface apps for staff to use, which are poorly maintained failures, running inside on-premises VPS. If you wanted to access the apps for one of the banks subsidiary banks... you had to open a VPS via remote desktop INSIDE the remote desktop instance of the VPS you were already working on.
Where it got really stupid though is they wrote a windows program which emulates the 'green screen' environment. My coworkers in their 50s were using the same IBM commands to update client account details that they learned during training in the 1970s as branch staff.
>Does anyone else have any experience updating and rewriting old mainframe batch processes to newer systems and architectures?
What we did is... use AutoHotKey to record keystroke macros which are then executed into CICS via a windows emulator. It sounds stupid, but we managed to automate about 50,000 FTE (full time equivalent) of man-hours from our India based back office teams. The MBAs loved that. Of course, this was achieved by using an auto hot key script on an IBM terminal emulator running CICS inside a windows vps running inside a second windows vps, running inside a desktop computer. All written in a mixture of VBS/javascript/AHK
IBM missed the 21st century turn, and buying Red Hat will probably not save everything [1].
Mainframes are just too expensive, even with open-source software running from the Canonical or Red Hat ecosystem. You just can't keep billing CPU consumption and sell memory sticks like it's made of diamonds. Running heavy "modern" workloads will give you a crazy stupid high R4HA MSU bills ... the clash between the relatively cheap commodity hardware philosophy of the last 10 years and the IBM way of billing high-end things is too big.
I would say mainframes are still really good at handling an ultra large bus with sky-high bandwidth but that's pretty much it. The power platform is a lot cheaper and also good at that.
IBM should stop trying to copy AWS/Azure/GCP with their cloud offerings and instead offered a "mainframe" based cloud, they might get more traction. Customers could submit their COBOL/CICS jobs and just pay for the cpu milliseconds or I/O blocks used etc. getting the cloud scale and elasticity for a mainframe without having to actually "own" a mainframe. I guess that's where AWS is going with some of their ML & analytic offerings (batch oriented, but not mainframe based).
I have no idea about the current level of activity in this sector (mainframe proceesing as a service), but would not be at all surprised to find it greatly diminished from it's heyday when dinosaurs roamed the raised floor.
I don't think it takes a huge leap to draw a connection between these businesses and the cloud providers of today.
IBM cloud for learners itself not cheap that's the problem. But the thing is they cannot do it like that each software takes a huge load on the lpar/Plex and run as a subsystem. So cost of running a small stack itself will be huge. Now they got zVM don't know how effective that is, but currently it helps Devops process.
Sure IBM System Z is a marvel to work with, and it tickles the fascination of an engineer to be able to hot swap CPU banks and memory banks from a live system without downtime! I have had my mind blown away doing exactly that with a test Z10 in 2008.
However, the big System Z is surrounded by a lot of marketing and sales people who only know how to hype the shit out of the world. You talk to them for 2 minutes, and they bring up Watson just like in the linked article. For no logical reason. It is a big percentage of your job to cut the bull shit if you are dealing with IBM, particularly the System Z crowd. Unfortunately.
As expensive it is. It might be a crown jewel of enterprise computing even in the modern era if the ecosystem around it evolves to modern practices and cut out the bullshitters.
I have run a few thousand Linux VMs under Z/VM (hypervisor) on the Z10 hardware, using specialty Z processors made for running Linux. It was lovely albeit being inaccessible to the masses.
I completely agree with you. I have been completely fascinated and astounded by the capabilities of the System Z (and also System P in my case), yet utterly flabbergasted and disgusted by the incompetence of IBM in business dealings.
For example, the hypervisor on P series has been around since the introduction of Power5 and it allows you to create micro-LPARS with a small fraction of the CPU resources. Even back in the Power5 days, there was something called entitled capacity that lets you specify the minimum amount of CPU resources your LPAR would receive even if the entire server is being hammered simultaneously. It's an engineering marvel. Back then it was called APV. I believe it's called PowerVM these days. I'm sure that most people in our industry have no idea that this capability exists, let alone that it's been around since Power5. Sad, really.
This isn't really accurate WRT what is happening, the IFL (https://www.ibm.com/products/integrated-facility-for-linux) is little more than a normal fully unlocked PE that is only licensed for running linux. Its just a cheaper way to get an unlocked PE.
openSUSE Tumbleweed and SUSE Linux Enterprise Server also run on IBM Z mainframes. I don't see how "Red Hat" has revived mainframes, they are not doing anything SUSE, openSUSE, Debian and many others are already doing.
“IBM mainframe sales grew some 69% during the second quarter of this year, achieving the highest year-over-year percentage increase of any other business unit. Some industry observers attribute the unexpected performance to the fact the z15, introduced a year ago, is still in its anticipated upcycle. Typically, mainframe sales level off and dip after 12 to 18 months until the release of a new system. But that might not be the case this time around.”
They don’t give any more context to know how unusual that 69% is. Maybe there is a big jump every year in the second quarter, and this one was just a bit bigger. I don’t know.
One reason they cite is the increase in online shopping that prompted customers to pay for activation of the extra hardware that is included, disabled until you pay, in each mainframe:
“Some industries are in slumps, but online sales are up and that means credit card and banking systems are more active than normal. They liked the idea of being able to turn on ‘dark’ processors remotely.”
IO; reliability. That and only that. Until recently you could have added 'large amounts of memory' to that too, but you can now get TB+ machines for very little money elsewhere and most problems an ordinary business will run into will fit in that footprint.
Good point, yes, it can be a way to get out of party A pointing to party B and party B pointing to party A. That said, I would hate to have IBM be the highest level contractor for anything that really mattered to me.
> That said, I would hate to have IBM be the highest level contractor for anything that really mattered to me.
My experience is with highly-skilled staff who are very comfortable with using the IBM troubleshooting facilities of the product, you very quickly (a few days on normal support cases, 2-6 hours on Sev 1 cases, about an hour on Sev 1 where you're paying for enhanced support) get to real developers with real source code access.
The problems come when you aren't using skilled staff. There has been a huge upturn in support cases I've see in clients' IBM Support case histories that are really "professional services/training-through-support" incidents, where it is painfully obvious the requesting parties don't know the most basic aspects of the products they're opening support cases upon. Both customers and IBM are shooting themselves in the foot here: customers for leaning on IBM with unskilled staff, IBM for accepting it. No good solutions without spending more money (on both sides), unfortunately.
Everyone who snarks about enterprise software: the road to enterprise scale dominance is paved with a maze of twisty little requirements, all alike (on the surface, then you peel it back and the cockroaches of edge cases come swarming out). There are more things in IT, Horatio, / Than are dreamt of in your philosophy.
> There are more things in IT, Horatio, / Than are dreamt of in your philosophy.
That was a fine comment right up to the snark. I've been in IT for 35 years now, a portion of that on mainframes.
Yes, customers are often also to blame. But IBM is no longer equivalent to competent. They have plenty of incompetents on staff and do not hesitate to let those fend for themselves where customer data is on the line. A recent case involved fairly massive data loss for a customer, and I'm still surprised that never made the news.
IBM is still quality hardware, and very good documentation on that hardware but the professional services organization is a mixed bag.
Not meant as a snark, a constant reminder I keep telling myself every day, that there is way more in IT than I'll ever have time to learn. All I have to do if I ever think I'm beginning to get my arms around it all is look to someone like Fabrice Bellard or innumerable others. We only win as a team, not many such Free Electrons in the world to pass around, I'm certainly not such a talent, I just heave against the entropy with the gang and share like everyone else.
For IBM professional services, you'll have no quibble from me. Very mixed bag from what customers tell me, and customers nearly have to be domain experts to make smart purchase decisions on contractors. A non-trivial chunk of big name professional services placements (not just IBM) are third parties, most of those third parties get run through mandatory sub-sub-contracting relationships and there is even a thriving industry feeding off of sub-sub-sub-contracting relationships.
IBM Support however, IME while it has declined markedly in L1 and marginally in L2 (you need to get to the L2 leads quickly if you know what you're doing and want to save time), is still quite good in L3, and if you pay for the enhanced support, exceptionally good (but it is so expensive that few people outside the IBM ecosystem have even heard it is an option).
IBM's documentation has always been very thorough, but I've seen a decline in accuracy and editing (to organize the typically voluminous standards) that corresponds with their gutting of the US technical writing teams who when I worked with Redbooks upon were typically very technically proficient in their own right. I try to help out these days by insisting upon a documentation APAR when I run across documentation defects in my accounts, but I'm likely in the minority of users. Ever since they took away the ability to comment on individual pages in Knowledge Center, they've considerably increased the friction for users to help increase the quality of documentation.
Possibly security. When the IBM 370-class mainframe was born, it was often in a data center with competitors running on the same machine. That machine was a "universe" and its many software components (various DBs, communications, file structures, language modules, etc.) EACH have its own security. No one can do ANYTHING without explicit tediously-granted authorizations - there is no such thing as a global "root" access. If anyone has EVER hacked the traditional core systems (where the data is), it's a well-kept secret - versus the not uncommon news of "modern" systems being hacked.
If you REALLY want to know some tech details, the freely-downloadable Redbooks can tell you more than most humans would ever want to know. Two of many are available below (and you can browse the library for others too):
I think a lot of people at HN just hate IBM for no real reason. An economy doesn't have to be composed entirely of scrappy startups, bleeding-edge tech (and IBM does do a lot of great systems research). Super odd to me that in such a 'rational' community there's a ton of brand affiliation - something like nationalism reduced to the tech industry.
Ultimately, you actually often need something like an IBM to be around to acquire your small company so you can run away with the money.
There's probably a lot of people who similarly hate on the mainframe. Except in the case of the mainframe (the technology, as opposed to the vendor), the reason is most likely because they view it as a competitor to the stacks that provide their paychecks.
That's a silly way to see it. A lot of their pains is that they are trying to build reliable things out of unreliable components when they could, instead, just enjoy the reliability five decades of R&D can get you.
> I think a lot of people at HN just hate IBM for no real reason
I disagree. From a marketing perspective, they are a company that is known to over promise and under deliver. Their marketing is vague pie in the sky without much to show for in the market. Watch this IBM ad and tell me what is it that this company actually does:
They are a company that fails to innovate, and fails to be relevant, yet they put a ridiculous amount of effort to try to convince people otherwise.
From a business strategy perspective, IBM has spun off or sold off most of their great product lines and business units. Some of which were failing under IBM leadership, and became profitable once divested. Every company they acquire, they seem to dissolve after a few years.
From an internal culture perspective, they are known for laying off their talented experienced employees who have been with the company for over 20 years in favor of younger, cheaper employees. In some departments, they have resorted to simply not promoting people and let people to leave on their own accord.
That being said, most of my experience in working with IBM has been while they were under CEO Ginni Rometty who led from 2012 to 2020. This was one of the biggest bull runs in history, yet the stock went from about $205 per share down to about $120 per share. They have a new CEO now, so things might change.
disclaimer: Red Hat employee, which technically means IBM employee
They do innovate, it's just that the places where they have been putting all their flashy marketing over the past couple of years have not been the places where they've been innovating.
They have not one but two custom CPU architectures designed entirely in-house (POWER and Z) which they've kept competitive over the years (in terms of hardware if not price). They're one of a very few entities (corporate or academic) at the very leading edge of quantum computing and one of the few builders of high-end supercomputers (#2 and #3 most powerful computers in the world at the moment are POWER systems)
The problem is that while they are legitimately extremely good at many things, those things make up a small fraction of the "surface area" of IBM and it's not what they have have historically been strategically emphasizing for the past couple decades. The average engineer's experience with IBM is more likely to involve their software products and consulting services and/or sales team than it is any of those things I mentioned.
Every time one of these articles comes along we hear about fast I/O and how other hardware can't handle the volume. Why not? What makes mainframe hardware so special that it can't be duplicated in other architectures?
Great questions. I've often wondered the same thing. I strongly suspect that the biggest impediment might be the threat of litigation for patent infringement. IBM has fought fiercely in the past to protect their crown jewels.
The chips are huge and numerous & you would have to boostrap a market to compete, that's hard. Technically it can be "duplicated", but in the same sense that Apple can make a state of the art processor better than Intel: you have to assemble a big first class team, a shitload of money, and persist during 10 years. If you want to compete on a similar field as mainframes you also need extreme reliability, and that's even more expensive to design. If you relax the requirements and make some quite special purpose hardware arguably it should be "easy" enough to make some fast I/O that can handle volume; but then your market will be small, niche most of the time.
Mainframe stack is very simple COBOL-Db2, online will be CICS-COBOL-Db2, batch is JCL, COBOL with sorts dfsort or syncsort, SAS, much modern will be COBOL/JAVA with Db2 Queue replication. Scripting language mostly REXX.
Last 3 years, lots of things are happening, more offloads to zIIP than General CP. In terms of applications more and more Db2 Queue replication for the time monitoring. CDC target being Kafka, IDAA for analytics and after acquisition of RHEL, now I see admins, sysprogs installing Spark, Anaconda, Jupyter in mainframe.
Usually in mainframe for detecting frauds, it used to be rule based now some companies are planning for machine learning in mainframe and be close to source and detect frauds.
Only thing mainframe cannot support is Deep learning as that requires GPU, no GPU here.
Anyones companies already started ML and spark in mainframe and what's your experience till now.
> Only thing mainframe cannot support is Deep learning as that requires GPU, no GPU here.
You can have racks full of PCIe slots if you want. I don't think there's anything that prevents you from adding a dozen GPUs on every IO drawer if that's what you want. And, unlike PC-based servers, there's a ton of silicon dedicated to keeping all IO channels fed, so, you'd probably have very low CPU usage.
Could someone who has had experience with mainframes explain to me why, in 2020, if I were developing a brand new application with no backwards compatibility baggage, what scenarios in which I would want to choose a mainframe rather than a collection of off the shelf servers? Let's assume I have the financial capacity to run my own data center.
I'm legitimately curious. I already know how stupid reliable they are in terms of correctness guarantees and staying up when hardware failure and replacement occurs, but what truly sets them apart, considering their cost?
At least for we 20-something engineers, I'd wager that nearly all of us have never had an opportunity to be exposed to mainframes.
The reliability is a huge thing. Some of the ancient software these things run are the backbones of major systems: Power grids, water systems, air traffic control systems, flight booking systems, weapons controls, or banking systems.
Sometimes, these systems are unable to be run in what you'd think about as a traditional distributed systems model. You don't just run that one COBOL program 100 times and kill one process everytime it jams up.
The software inside of some of these systems was written in a time when it could be optimized to the absolute bit. At that level of optimization and skill in development, you're not really going to do better or get more throughput with a rewrite. Instead, you just keep wrapping that thing in outer layers and message queues.
So, if you need a computer that can absolutely 99.99999% never stop working, including redundant everything (Power, ram, cpu, network) and you're running something that's super old but super efficient and has worked basically perfectly for 30 years, you just buy another one to replace the original, and add some containers into the mix for the queues, or storage adapters or whatever, rather than spend a billion dollars re-engineering the whole thing from scratch as microservices that may or may not work.
That, or it's a giant fucking database server. That's usually the case: one big floppy database that runs everything in the company and it's SAP system.
IBM can sell you a LinuxONE machine that's mainframe that runs Linux under the z/VM hypervisor. It's probably the biggest Linux server money can buy. And can be carved out into thousands of really nice average servers. You won't have to worry about network components or latency or unreliable servers or anything like that - you can treat it as a data center in a couple racks.
Another interesting feature is that they are designed to be monitored, because a traditional business model was to lease a mainframe and rent out capacity to smaller companies that'd pay for things like CPU seconds or bytes read/written.
> In the environments where COBOL thrives, multi-threading is usually left to application containers like CICS or IMS which are good at it, rather, than programmers who tend to be bad at it.
My first RHEL laptop was a T40 thinkpad with a 31-bit (not a typo) z-series emulator complete with dongle. I think they spent 20k on the crazy thing. Such a weird market to sell into.
The demise of mainframes is entirely IBM's own doing. Mainframes are capable of incredible throughput, compute power, and reliability. They allow you to design applications with extremely tight SLA's without the complex and bloated distributed architectures you see spammed everywhere nowadays.
However... then you have to:
1) Deal with IBM's ridiculous sales staff. They're rarely technical and hype their random software offerings like there's no tomorrow
2) Straight up purchase these mainframes and put them in a datacenter somewhere. No easy way to just run one of these things on eg; AWS even if it's dedicated just to you.
3) IBM licensing is beyond stupid. The typical workloads people run today wouldn't be cost effective purely due to licensing.
4) The tooling is ancient and highly proprietary. People want to be able to write modern C++/Go/Rust/Java and if you want to use specialized libraries to interact with the hardware that's fine. But the current ecosystem is BAD.
5) How do you even learn to use Z/OS? If nobody can just play with it even in a free emulator then why would anyone even consider it?
If IBM bothered to address 1-5 maybe we'd actually see more people considering mainframes at larger companies. But they're digging their own grave.
I think 5 is probably the biggest — in terms of emulation, you can run TK4 in Hercules, but it’s literal decades behind the modern mainframe. How am I supposed to learn if the concepts are hidden behind a massive monetary barrier? Reading manuals can only get you so far
75 comments
[ 420 ms ] story [ 4585 ms ] threadLong before I joined the company, that department tried to rewrite the system to be "modern", porting everything over to Java and having it run across a couple of servers. This was in the late 00's if I remember correctly. Apparently this rewritten system couldn't handle the volume of data that the mainframe can easily process during its batch cycle. However, were they to rewrite the system now (having the tools available to better facilitate real-time processing along with flexible resource scalability) I bet that a new system would be able to keep up with the demand and be just as expensive, if not a little more cost effective to run.
Does anyone else have any experience updating and rewriting old mainframe batch processes to newer systems and architectures?
I also recently came to learn that the mainframe run jobs in batches, so I'm not sure how does that affect performance, if it even does.
Also, mainframes can run both batch jobs and "online" (CICS) jobs. Our shop is about half and half with batch and online programs. The main thing that batches can be annoying with is that they run on schedules and may have to wait for other jobs to complete before they can kick off, causing delays. We just upgraded from a z13 to z15, so our mainframe is no longer a concern but we are also much smaller than a bank.
If moving off the mainframe becomes cheaper, they have a lot of flexibility
My limited experience with this is this strategy is that it works, although it is not error proof and can be messy. You also end up discovering that assumptions you made about the legacy software are not correct and then you need to back pedal and potentially throw away previous work (ironically your own assumptions about how to modernize the legacy code become their own legacy that you need to work around as the project ages).
It's good if you have some of the original developers still around and probably worthwhile to give them consulting $$ in order to bring them out of retirement.
Perhaps the best thing about this strategy is if you in the end conclude it isn't worthwhile, you will still have modernized some sections of the codebase. Of course, because of this you need to make sure that the individual section by section modernization efforts are themselves a net gain, instead of just hoping that the final end of the modernization process will be a benefit, because you might never get to that end point.
Also, you might be able to enhance the product early on by being able to iterate and improve the sections that have been modernized, instead of waiting till everything is done to flip a switch and get new features.
The downside is this is a slow and gradual process. In the meantime your developers are filling their heads with details of a unique system that is hopefully going to go away never to be seen again.
If you have specs and documentation and tests and all of that good stuff, it might be better to just start from scratch, and build things up according to requirements.
Also, at our bank, very few applications use the REST API. Since most applications are developed by contractors, most vendors don't want to use our custom API, and instead opt to connect to the mainframe directly, because then they can reuse the code at other banks after charging us to develop it.
My experience with connecting UNIX to Z/OS was not with a real API (HTTPS) but UNIX > TIBCO EMS > something called TIBCO Substation > CICS. It was horrible, mainly because of Substation which needs to translate on the fly between two completely different platforms.
IMO any "modern" architecture can handle the load. After all Google, Apple, Facebook, Microsoft, Amazon, etc. are not running on mainframes (just taking mastodon examples).
The main difference between ARM, x86 or Power vs Z is that you can keep your CPU usage between 95-100% in production and not break everything.
Performance is really not an issue, especially when you factor in the price (we could easily double our servers and still be cheaper than a pretty baseline Z13).
Main pro of the mainframe is: you can basically not touch your code for decades and it'll still run fine, every technology on it and the hardware itself is rock solid.
Main pro of leaving the mainframe is: you don't need to worry as much about expensive/hard technical debt (i.e. EBCDIC text encoding, special hardware for everything), pricing/billing is much simpler and cheaper, way easier to switch technologies.
We run a mainframe shop and for us the cost is fairly reasonable, and the high bandwidth/reliability (even with the cpu pinned to 100%) is the most critical aspect to us. Ignoring the risk and cost of migration, I am not even sure that an alternative would be cheaper... but it is quite difficult to determine the cost and my boss obviously factors in the cost/risk of migration anytime this comes up.
I'd be curios to hear of any successful total migration attempts. Most shops I know of keep their mainframe and build around it with modern tools and we do as well.
Another company I interviewed with did just what you said, they built REST APIs and other "modern" interfaces to the mainframes so that 99% of app developers in the company don't have to learn anything about the mainframe.
I've stayed in touch with some friends there all this time, and apparently they still haven't finished this project as of 2020. Disaster after disaster.
The first executive couldn't execute, so he moved on and the project was slow-rolled for awhile. Until enough time had passed for a subsequent executive to claim it as HIS "fresh new idea".
That executive came it very aggressively, actually pushing out all of the RPG employees and contractors well before the company was really ready. So that executive was pushed out, and his replacement had to hire IBM Global Services to come in and take over everything. IBM hired most of those employees and contractors who were let go. So the net result was the same people doing the same job, with the company paying 2-3x more than they had been paying.
It was around 2015 when they finally reached the tipping point, to where the company was "a Java shop with some legacy big iron" rather than "an RPG shop with some Java e-commerce pieces". But there's still a lot of AS/400 on the inventory side of things, and it will probably be another couple of decades before that finally goes away.
A lot of people on HN and Reddit, who are either students or startup employees, just have NO IDEA how things work for large businesses in the other 49 states. Stuff lasts forever, it's just a completely different world.
I am Not sure why languages are suddenly considered obsolete, I think it is that all the programmers suddenly run over to the next big thing leaving massive gaps in staff that mean recruitment becomes a problem. If I was in the market today I would be running the other way and embracing what currently works.
"Maintained a COBOL system that processes twenty zillion transactions per hour" just doesn't sparkle like "replaced 40-year-old monolith with a galaxy of JavaScript microsoervices that get winded at 100 requests per hour."
A fair bit and key is that the business process and understanding of all that is more key to getting a good transition over anything else.
The other key is small chunks, identify aspects that can be shifted across bit by bit - reports that are using daily data can be a datawharehoused data affair with your reporting system tapping that - that opens up tools for the the various departments to do ad-hoc reports by themselves without suddenly spiking the key-business loads.
But so many avenues and aspects there is no simple list to go thru, as each one depends upon the set-up and more so - business logic.
So I'd take a COBOL programmer who knows the bushiness over a top-end java programmer who does not know the business for a migration project as be easier to teach the COBOL programmer java than it would to teach the java programmer the business logic as would take sooo long and to learn all the nuances and what needs to happen when things go wrong and what is priority and the dependencies. Alas many management and project types will see a glossy top-end java programmer and pick them and ignore their in-house talent all too often.
However - batch stuff is maybe the easiest area to work upon - less users in the equation and with that - less variables to go wrong ;).
Mainframes are also not just large legacy number crunchers - the whole aspect of resilience and fault tolerance goes much deeper than ECC memory, RAID and dual NICs and PSU's. Much deeper and that is why they have up times above and beyond your X86 server class. This with database and messaging systems equally tried, tested and robust. Which gives many factors to work upon.
Though as always - business logic and process along with legal requirements and other constraints for data-flow are key and certainly flowcharting that all out and the dependencies would be key to have anyhow and essential for any migration.
The same goes for all wire transfers, credit card transactions, ATM infrastructure and so on.
The banks solution was to simply write about 40 different Javascript and Visual Basic interface apps for staff to use, which are poorly maintained failures, running inside on-premises VPS. If you wanted to access the apps for one of the banks subsidiary banks... you had to open a VPS via remote desktop INSIDE the remote desktop instance of the VPS you were already working on.
Where it got really stupid though is they wrote a windows program which emulates the 'green screen' environment. My coworkers in their 50s were using the same IBM commands to update client account details that they learned during training in the 1970s as branch staff.
>Does anyone else have any experience updating and rewriting old mainframe batch processes to newer systems and architectures?
What we did is... use AutoHotKey to record keystroke macros which are then executed into CICS via a windows emulator. It sounds stupid, but we managed to automate about 50,000 FTE (full time equivalent) of man-hours from our India based back office teams. The MBAs loved that. Of course, this was achieved by using an auto hot key script on an IBM terminal emulator running CICS inside a windows vps running inside a second windows vps, running inside a desktop computer. All written in a mixture of VBS/javascript/AHK
Mainframes are just too expensive, even with open-source software running from the Canonical or Red Hat ecosystem. You just can't keep billing CPU consumption and sell memory sticks like it's made of diamonds. Running heavy "modern" workloads will give you a crazy stupid high R4HA MSU bills ... the clash between the relatively cheap commodity hardware philosophy of the last 10 years and the IBM way of billing high-end things is too big.
I would say mainframes are still really good at handling an ultra large bus with sky-high bandwidth but that's pretty much it. The power platform is a lot cheaper and also good at that.
[1] Earlier this year: IBM’s Lost Decade https://news.ycombinator.com/item?id=22224782
https://en.wikipedia.org/wiki/Service_bureau#Histories
I have no idea about the current level of activity in this sector (mainframe proceesing as a service), but would not be at all surprised to find it greatly diminished from it's heyday when dinosaurs roamed the raised floor.
I don't think it takes a huge leap to draw a connection between these businesses and the cloud providers of today.
Generally Billing is in peak MIPS.
However, the big System Z is surrounded by a lot of marketing and sales people who only know how to hype the shit out of the world. You talk to them for 2 minutes, and they bring up Watson just like in the linked article. For no logical reason. It is a big percentage of your job to cut the bull shit if you are dealing with IBM, particularly the System Z crowd. Unfortunately.
As expensive it is. It might be a crown jewel of enterprise computing even in the modern era if the ecosystem around it evolves to modern practices and cut out the bullshitters.
I have run a few thousand Linux VMs under Z/VM (hypervisor) on the Z10 hardware, using specialty Z processors made for running Linux. It was lovely albeit being inaccessible to the masses.
For example, the hypervisor on P series has been around since the introduction of Power5 and it allows you to create micro-LPARS with a small fraction of the CPU resources. Even back in the Power5 days, there was something called entitled capacity that lets you specify the minimum amount of CPU resources your LPAR would receive even if the entire server is being hammered simultaneously. It's an engineering marvel. Back then it was called APV. I believe it's called PowerVM these days. I'm sure that most people in our industry have no idea that this capability exists, let alone that it's been around since Power5. Sad, really.
This isn't really accurate WRT what is happening, the IFL (https://www.ibm.com/products/integrated-facility-for-linux) is little more than a normal fully unlocked PE that is only licensed for running linux. Its just a cheaper way to get an unlocked PE.
https://www.openmainframeproject.org/
Also Debian runs on IBM Z mainframes for folks who prefer that to RedHat:
https://www.debian.org/ports/s390/
They don’t give any more context to know how unusual that 69% is. Maybe there is a big jump every year in the second quarter, and this one was just a bit bigger. I don’t know.
One reason they cite is the increase in online shopping that prompted customers to pay for activation of the extra hardware that is included, disabled until you pay, in each mainframe: “Some industries are in slumps, but online sales are up and that means credit card and banking systems are more active than normal. They liked the idea of being able to turn on ‘dark’ processors remotely.”
Is there any justifiable reason for a greenfield project in 2020 to choose mainframes as a platform?
Their reputation is no longer deserved imnsho.
My experience is with highly-skilled staff who are very comfortable with using the IBM troubleshooting facilities of the product, you very quickly (a few days on normal support cases, 2-6 hours on Sev 1 cases, about an hour on Sev 1 where you're paying for enhanced support) get to real developers with real source code access.
The problems come when you aren't using skilled staff. There has been a huge upturn in support cases I've see in clients' IBM Support case histories that are really "professional services/training-through-support" incidents, where it is painfully obvious the requesting parties don't know the most basic aspects of the products they're opening support cases upon. Both customers and IBM are shooting themselves in the foot here: customers for leaning on IBM with unskilled staff, IBM for accepting it. No good solutions without spending more money (on both sides), unfortunately.
Everyone who snarks about enterprise software: the road to enterprise scale dominance is paved with a maze of twisty little requirements, all alike (on the surface, then you peel it back and the cockroaches of edge cases come swarming out). There are more things in IT, Horatio, / Than are dreamt of in your philosophy.
That was a fine comment right up to the snark. I've been in IT for 35 years now, a portion of that on mainframes.
Yes, customers are often also to blame. But IBM is no longer equivalent to competent. They have plenty of incompetents on staff and do not hesitate to let those fend for themselves where customer data is on the line. A recent case involved fairly massive data loss for a customer, and I'm still surprised that never made the news.
IBM is still quality hardware, and very good documentation on that hardware but the professional services organization is a mixed bag.
For IBM professional services, you'll have no quibble from me. Very mixed bag from what customers tell me, and customers nearly have to be domain experts to make smart purchase decisions on contractors. A non-trivial chunk of big name professional services placements (not just IBM) are third parties, most of those third parties get run through mandatory sub-sub-contracting relationships and there is even a thriving industry feeding off of sub-sub-sub-contracting relationships.
IBM Support however, IME while it has declined markedly in L1 and marginally in L2 (you need to get to the L2 leads quickly if you know what you're doing and want to save time), is still quite good in L3, and if you pay for the enhanced support, exceptionally good (but it is so expensive that few people outside the IBM ecosystem have even heard it is an option).
IBM's documentation has always been very thorough, but I've seen a decline in accuracy and editing (to organize the typically voluminous standards) that corresponds with their gutting of the US technical writing teams who when I worked with Redbooks upon were typically very technically proficient in their own right. I try to help out these days by insisting upon a documentation APAR when I run across documentation defects in my accounts, but I'm likely in the minority of users. Ever since they took away the ability to comment on individual pages in Knowledge Center, they've considerably increased the friction for users to help increase the quality of documentation.
YMMV of course, just sharing my perspective.
IBM z15 Technical Introduction (3.3m PDF) http://www.redbooks.ibm.com/Redbooks.nsf/RedbookAbstracts/sg...
IBM z15 Technical Guide (22.1m PDF) http://www.redbooks.ibm.com/Redbooks.nsf/RedbookAbstracts/sg...
(Please forgive my URL handling, newbie here!)
Ultimately, you actually often need something like an IBM to be around to acquire your small company so you can run away with the money.
I disagree. From a marketing perspective, they are a company that is known to over promise and under deliver. Their marketing is vague pie in the sky without much to show for in the market. Watch this IBM ad and tell me what is it that this company actually does:
https://www.youtube.com/watch?v=gsEwmumEsKU
They are a company that fails to innovate, and fails to be relevant, yet they put a ridiculous amount of effort to try to convince people otherwise.
From a business strategy perspective, IBM has spun off or sold off most of their great product lines and business units. Some of which were failing under IBM leadership, and became profitable once divested. Every company they acquire, they seem to dissolve after a few years.
From an internal culture perspective, they are known for laying off their talented experienced employees who have been with the company for over 20 years in favor of younger, cheaper employees. In some departments, they have resorted to simply not promoting people and let people to leave on their own accord.
That being said, most of my experience in working with IBM has been while they were under CEO Ginni Rometty who led from 2012 to 2020. This was one of the biggest bull runs in history, yet the stock went from about $205 per share down to about $120 per share. They have a new CEO now, so things might change.
They do innovate, it's just that the places where they have been putting all their flashy marketing over the past couple of years have not been the places where they've been innovating.
They have not one but two custom CPU architectures designed entirely in-house (POWER and Z) which they've kept competitive over the years (in terms of hardware if not price). They're one of a very few entities (corporate or academic) at the very leading edge of quantum computing and one of the few builders of high-end supercomputers (#2 and #3 most powerful computers in the world at the moment are POWER systems)
The problem is that while they are legitimately extremely good at many things, those things make up a small fraction of the "surface area" of IBM and it's not what they have have historically been strategically emphasizing for the past couple decades. The average engineer's experience with IBM is more likely to involve their software products and consulting services and/or sales team than it is any of those things I mentioned.
It also says a lot how relational database models outlive business logic api-s.
http://www.redbooks.ibm.com/redpieces/pdfs/sg248850.pdf
Last 3 years, lots of things are happening, more offloads to zIIP than General CP. In terms of applications more and more Db2 Queue replication for the time monitoring. CDC target being Kafka, IDAA for analytics and after acquisition of RHEL, now I see admins, sysprogs installing Spark, Anaconda, Jupyter in mainframe.
Usually in mainframe for detecting frauds, it used to be rule based now some companies are planning for machine learning in mainframe and be close to source and detect frauds.
Only thing mainframe cannot support is Deep learning as that requires GPU, no GPU here.
Anyones companies already started ML and spark in mainframe and what's your experience till now.
You can have racks full of PCIe slots if you want. I don't think there's anything that prevents you from adding a dozen GPUs on every IO drawer if that's what you want. And, unlike PC-based servers, there's a ton of silicon dedicated to keeping all IO channels fed, so, you'd probably have very low CPU usage.
I'm legitimately curious. I already know how stupid reliable they are in terms of correctness guarantees and staying up when hardware failure and replacement occurs, but what truly sets them apart, considering their cost?
At least for we 20-something engineers, I'd wager that nearly all of us have never had an opportunity to be exposed to mainframes.
Sometimes, these systems are unable to be run in what you'd think about as a traditional distributed systems model. You don't just run that one COBOL program 100 times and kill one process everytime it jams up.
The software inside of some of these systems was written in a time when it could be optimized to the absolute bit. At that level of optimization and skill in development, you're not really going to do better or get more throughput with a rewrite. Instead, you just keep wrapping that thing in outer layers and message queues.
So, if you need a computer that can absolutely 99.99999% never stop working, including redundant everything (Power, ram, cpu, network) and you're running something that's super old but super efficient and has worked basically perfectly for 30 years, you just buy another one to replace the original, and add some containers into the mix for the queues, or storage adapters or whatever, rather than spend a billion dollars re-engineering the whole thing from scratch as microservices that may or may not work.
That, or it's a giant fucking database server. That's usually the case: one big floppy database that runs everything in the company and it's SAP system.
Another interesting feature is that they are designed to be monitored, because a traditional business model was to lease a mainframe and rent out capacity to smaller companies that'd pay for things like CPU seconds or bytes read/written.
http://dtsc.dfw.ibm.com/
They might also know others who do it too. I must admit I've never heard of such a thing from non-IBM sources.
[https://medium.com/selency-tech-product/your-balance-is-0-30... - all those problems are addressed.
Also, per [this thread](https://softwareengineering.stackexchange.com/questions/1129...):
> In the environments where COBOL thrives, multi-threading is usually left to application containers like CICS or IMS which are good at it, rather, than programmers who tend to be bad at it.
However... then you have to: 1) Deal with IBM's ridiculous sales staff. They're rarely technical and hype their random software offerings like there's no tomorrow 2) Straight up purchase these mainframes and put them in a datacenter somewhere. No easy way to just run one of these things on eg; AWS even if it's dedicated just to you. 3) IBM licensing is beyond stupid. The typical workloads people run today wouldn't be cost effective purely due to licensing. 4) The tooling is ancient and highly proprietary. People want to be able to write modern C++/Go/Rust/Java and if you want to use specialized libraries to interact with the hardware that's fine. But the current ecosystem is BAD. 5) How do you even learn to use Z/OS? If nobody can just play with it even in a free emulator then why would anyone even consider it?
If IBM bothered to address 1-5 maybe we'd actually see more people considering mainframes at larger companies. But they're digging their own grave.
All points listed are very valid BTW.
I wonder if the math works out to buy or lease one and rent out cloud compute instances.