Ask HN: Why hasn't the cloud killed the mainframe?
I was surprised to see how few mainframes Big Tech companies that handle a huge amount of transactions (Netflix, meta, google,...) use relative to legacy industries (banking, retail, insurance,...)
This makes me suspect that the real reason why mainframes continue to exist are due to industry inertia, vendor lock-in or even legacy code rather than any performance/cost reasons.
87 comments
[ 5.0 ms ] story [ 137 ms ] threadIt shows that they did want to switch but it was harder than they thought.
I would rather be without Netflix and Google, than banking and food ... but each to their own..
While some is inertia (mostly doing to rewriting truly large applications are hard and expensive), there is also the the point that most of those industries cannot easily handle "eventually consistent" data..
Not all transactions are created equally, the hardest usually have a set of requirements called ACID.
ACID in the classic RDBMS is not a random choice, but driven by real requirements of their users (the database user, i.e. applications in the business sense - and not the users as people). The ACID properties are REALLY hard to do in scale in a distributed system with high throughput. Think of the rate of transactions in the bitcoin system (500k/day with many, many "servers") vs. visa (500M+/day) - the latter is basically driven by two (!) large mainframes (with 50ish km distance) the last I heard of any technical details.
None of the companies you mention need to have strict ACID, as nobody will complain if different users see slightly different truths - hence scaling writes is faily easy.
From quick google: "Legacy industries are those that have been around for a long time. These industries dominate a specific market and have not always had a positive approach to innovative ideas."
Google is itself starting to sound a bit like that!
And I have to strongly agree with OP, I couldn't care less about fate of FAANGs of these days, but I do care about those 'legacy' businesses tremendously.
As for original topic - if it works for 3-4 decades, don't be the stupid guy and change it. Tremendous risk to core business with little to gain.
Relying on the existence of one vendor with highly unportable and unmaintainable code carries its own risk and my post is asking whether it justifies the cost.
I do the same, I have run PosgreSQL for more than 2 decades now and I don't care about portability to any other database, all of which I consider inferior (and yes, I do follow most of their releases).
1: I wonder how many transactions the largest e.g Postgres clusters (or other classic RDBMS) handles per day. 500M+/day doesn't seem that incredibly high?
2: Google Spanner, which I would classify as cloudy, promises ACID guarantees at a global distributed scale. Couldn't that be used?
I've listened to a Swedish developer podcast where they interviewed an old school mainframe developer in the banking sector. He brought up similar points about the scale and correctness of database transactions, and it didn't feel convincing to me.
What does Paypal, Klarna, or even maybe Amazon give up by not using mainframes? Does any company founded in the last 10-15-20 years use mainframes? If not, does that mean that "modern" companies can't compete in these high-demand industries like retail or insurance?
I think it's much more in the inertia-point, the cost of rewriting these enormous applications is simply too large.
The hard parts are updates in a shared system with a single consistent "view" requirement..
But then I mulled over it for a while, and it occurred to me it's likely Sqlite does orders of magnitude more than that planet wide.
Spanner's 1 billion per second is more impressive: https://cloud.google.com/blog/topics/developers-practitioner..., assuming it's returning a consistent view across many tables. But the Sqlite comparison still stands.
Visa claims 24,000 TPS, but in reality runs at a 10'th of that. It would be interesting to see if Spanner could process the same 2,000 transactions per second. Sqlite definitely can't.
Young developers often think that banks, insurance companies etc. should just rewrite these "legacy" systems because it will bring all of these magical benefits with no risk. Whereas older developers who have worked on (a) mission-critical applications, (b) major re-platforming efforts and (c) projects in a highly regulated industry know the score.
Doing just one is hard. Doing all three at the same time is suicidal. The chance of project success is basically in the single digits. And the risk of failure is billions in lost revenue and your future prospects in the company and within the broader industry ruined.
Because (a) major decisions like choosing a mainframe are not made by tech managers and (b) every company is built around vendor lock-in.
Who do you think companies like Atlassian, Oracle, Salesforce etc sell to ?
Simplicity. Because instead of having an enormous team to maintain their multi-million line Kubernetes configuration files that automatically spawns thousands of servers, having an enormous team that tries to fix the CAP problem, you go grab IBM, you tell them "this mainframe goes down, you're dead, here's half a billion in cash" and you run all of your software on a single, massive machine.
Sure, it adds other problems, but to be fair I'd rather deal with a mainframe than yet another microservice that doesn't work because Omega Star doesn't handle ISO timestamps and it blocks Galactus
>Does any company founded in the last 10-15-20 years use mainframes?
The biggest issue with mainframes are:
- High initial costs (although that's been changing)
- Nobody knows how to work on IBM Z mainframes
- The current zeitgeist about having a billion servers spread throughout the world because it's really important to have an edge CDN server for your cat trading card game
These industries didn't care about that because they could absorb these high initial costs, had the knowledge of the people building the mainframes, and were already highly centralized. Decentralization just adds more problems, it doesn't fix anything.
>I think it's much more in the inertia-point, the cost of rewriting these enormous applications is simply too large.
Rewriting just because it's not following the latest trend is garbage. These applications work. They're not going to work better because you're using Spanner now.
Rewriting your system to a mainframe architecture is equally as expensive.
Lets say that you save a hundred megabucks per year by going cloud (I am really not convinced, that you save any money at all, but lets say). That is what 1500 man-years of work? No way you can rewrite even a simple "banking" system in that amount of work, and the second problem is that you need to feature-freeze the old system (and hence your business) while you convert. So maybe 5 to 10 years of lost competitive power on top of that..
Also, remember these are not "begin work; select * from table where id = 123; commit;" transactions. These have maybe 50 to 200 queries (selects and updates) in each transaction (has the paying party the funds, is the receiving party blacklisted (terror organisation for example), does this look like money laundring.... etc... and a very detailed logging requirement by law). All of these MUST usually be in the same "snapshot" (in the RDBMS definition).
It makes no sense to talk about "transaction rates" without an intimate knowledge of what it does, as especially marketing departments have a tendency to use the simplest possible "transactions" to get a large number..
And in the end, it is only "money" the might loose, and any choice you make makes some other choices in the future easier (and some harder). That is called path dependency.
But.
> Rewriting your system to a mainframe architecture is equally as expensive.
There was a new bank mentioned in this thread that actually started using mainframes from scratch, but other than that I've never heard of any "modern" fintech (or really any) company introducing mainframes. Organisations actually rewriting functioning systems TO mainframe must be almost never heard of (in the last 10-20 years at least).
If System Z, Cobol and DB2 are so obviously superior, why are so many successful new competitors in industries where they are the norm in older companies choosing to not use them?
I'm not saying banks should rewrite their stuff in node.js (or deno - even better of course), it makes sense for them to stay.
I just have a hard time believing that mainframe systems are so technically impressive, to the point where some people claim it's almost impossible to build a similar system on non-mainframe technologies.
The machines themselves are impressive (hardware wise) and reliability wise, for example you can swap mainboards one by one in a full frame without ever taking down the machine (think raid on a mainboard level, RAIMB ?).
But the high start-up cost makes most startups going the other road. I am not convinced that the vertically scaling is cheaper than horizontally, if you need the ACID guarantees... but it is hard to say.
The reason why us old dogs say it is hard (not impossible) is due to the single-image and acid requirements. There is no good way to do that distributed (look up the CAP theorem).
So having a massive computer (with double digit terabytes of memory AND cache, and truly massive i/o pipes.. just makes building the need-to-work stuff simpler.
As an example, a few years ago I was (on my own money) on a mainframe conference (not doing mainframe work in my work day).. at that time the machine had more bandwidth to the roughly 200 PCIe adapters that a top-of-the-line intel CPU had between the L1 cache and the computing cores) - and that meant that given enough ssd's you could move more data into the system from disk that you could move into an intel cpu from cache...
Also mainframes can run two mainframes lockstep (as long as they are less than 50km apart), that means if one of them dies during a transaction (which in itself is extremely rare), the other can complete it without the application being any the wiser.. Try that in the cloud :)
Bitcoin is slow because of the many servers not in spite of it. Because of the design of the network, all servers need to receive every transaction and servers need to be able to be pretty small, which limits the transaction rate.
Bigtable: https://cloud.google.com/bigtable/docs/replication-overview
> When using replication, reads and writes to the same cluster are consistent, and between different clusters, reads and writes are eventually consistent. If an instance does not use replication, Bigtable provides strong consistency, because all reads and writes are sent to the same cluster.
DynamoDB: https://docs.aws.amazon.com/amazondynamodb/latest/developerg...
> Both tables and LSIs provide two read consistency options: eventually consistent (default) and strongly consistent reads > Eventually consistent reads are half the cost of strongly consistent reads
In a big enterprise, the mainframes give CIOs leverage for other stuff too - they sell at high margin and IBM will “give away” or subsidize other services by moving money around in the backend.
(IBM should send me money for this tagline).
Seriously though, there are businesses where sending the data to a third party is absolutely impossible. You can spend a lot of money building your own private cloud... or, you can buy a mainframe, which will be cheaper, and if your industry is mainframe-friendly, there will be a lot of support and accumulated experience.
IMO the theory, technology, stack, libraries, ... for distributed computing has evolved a lot over the last decades that if you really had to have on-premise it is easier to create your own on-premise EC2 than rely on IBM to supply you with mainframes.
And if you rely on systems integrators to do that, it will be quite expensive, and you are not sure if this particular integrator will be around in the next 30 years. (Amazon will probably be around, which is why AWS is popular, but sometimes public cloud is not an option at all).
In this context, mainframes are quite competitive. Their costs are predictable (and comparable to, if not less than to private cloud offers from systems integrators), IBM will probably be around for long or at least acquired by somebody who will carry the support contracts (like Sun -> Oracle), and they will provide full technical support, again with predictable costs.
Put another way if IBM says they are closing shop in a month time, or they were stop upgrading a particular product line this will royally screw many of their clients who will start scrambling for alternative solutions (although I would imagine they baked in significant notice periods into their contracts).
But if you use the cloud and have done it right, it is much easier to migrate from a cloud service to another.
You spend a lot of money, and the best case is you end up with the same services as before.
Not that legacy will do better, but keeping your own fate in own hands does feel better. And these are at this point pretty proven systems. They most likely would have already failed.
> Why hasnt cloud killed the mainframe
A mainframe is typically a very powerful machine/cluster. You may be able to get those in the cloud (although I doubt you can get a single machine with >32TB RAM, for example) , but why pay a premium to the cloud provider on top of the base cost?
> Why do newer companies not use mainframes
Some do, but with your examples, its because they have been engineered to scale horizontally instead of vertically - i.e. using lots of less powerful machines instead of a single big machine.
> Are mainframes used because of lock-in
In some cases, sure. But there are many cases where big powerful machines are needed (where you cant have any of the issues that come with distributed computing - i.e. CAP trade-offs)
And cloud is not only option, you could also instead of mainframe, build your own data centres and get most what cloud offers. CAPEX vs. OPEX is a thing, but it is not like these are startups that need to scale. Or the scaling limits are pretty well known.
It is irrelevant because those companies don't care about cost/performance.
They care about risks e.g. operational, security, regulatory etc.
And I would trust a mainframe over the cloud for those criteria any day of the week.
I think that most customers lease their mainframe. IIRC IBM will often ship more hardware than the customer requires, with additional CPU/memory etc. being activated if a licence is purchased.
System Z - as I understand it, this is descendant from IBMs earliest computers like System 360. Typically apps are written in COBOL, PL1, tied together with JCL. “Newer” apps use DB2, but there are older non-relational databases as well.
System P - an IBM alternative server architecture based on the Power chipset. I think these mostly run UNIX, but I’ve never been hands on.
AS400 - technically an IBM minicomputer, but similarly esoteric with a bespoke OS and some nifty functionality for business apps (very SQL centric).
HP Tandem - traditionally used in high availability transaction processing applications.
Others? I think most of the other surviving old stuff is basically UNIX
Typically people are thinking of System Z. I think both the inertia and scalable ACID explanations are relevant. That said, I’m not sure how DB2 on System Z compares to say Oracle RAC.
While there's technically overlap in performance between the lowest spec Z and the higher end P, and these days lots of performance overlap between P and X ranges, P-series is called "midrange" by everyone I've every met.
The Power hardware has been the basis for both the AS-400 and P-Series/AIX line for a long time now, although you can buy p-series with linux too.
As an asside, AIX, with the exception of its volume manager which irks me deeply, is an incredibly usable Unix.. I mean you can really manage the whole thing via the SMIT UI without ever touching the command line. Not my preference, but you can. Some things, like cpu usage reporting are still better on AIX ( core utilisation via hyperthreads for example )
I'm still not a fan as such, mostly I'm annoyed by how good it is, but credit where credit is due.
Of course there are some people, (usually older but not always), who will call any server, or anything in a server room "the mainframe", in the same way the any desktop tower gets called "the cpu" (technically correct via some broader definitions but still..), or they'll call whichever office app they primarily use as "Microsoft", and of course you need to take that into account when communicating with people, the the incorrect usage of mainframe seems pretty rare these days just because mainframes themselves are rare.. and people in businesses who have them generally can distinguish.
They are not at all a trash or a product of corruption or nepotism as many tend to think. That stuff works all the time, and did since ~1960, while we serfs spend our lives fixing bugs resulting from never-ending updates in that hodgepodge of javascript libraries our "efficient", "cheap", "FOSS-based" products are made of.
We all don't use mainframes because marginal or downright dodgy business cases of our products simply won't pay for it, thus we are stuck in this race to the bottom.
Not just because of the expensive hardware, but also because it's all but impossible to get your hands on them as a developer. With Java, JS, Python, .NET, whatever companies have an insanely large pool of people to choose from... quite a few self-taught, some who went to university and studied CS, some who went to more or less decent bootcamps, some who "grew into" programming from other roles. On top of that, last I heard (admittedly a decade ago) the tooling to work on mainframe systems is just as old as the code they're running, so no modern IDEs, debuggers and whatnot.
With mainframes however, companies have to spend their own money, and quite a lot of it, to get trained developers, and on top of that pay a hefty premium for those who submit to fossilized tooling willingly.
Going through the challenges is worth it just to see how the mainframe way of doing things works, even if you never plan to use a mainframe in your career. There's a lot of core concepts and cultural touchpoints that help you build more reliable systems everywhere else.
It feels like non mainframe world decided to not put effort into hardware reliability and tries to fix it at the architecture and software lvl, which is kinda sad.
You're also often dealing with very sensitive information. Zenbleed style attacks are an unacceptable risk, so you'll need separate hardware from the rest of the cloud anyway. Maybe you can find a data center that's reliable and secure enough to put all of that sensitive data, but there's a good chance you'll be wiring up your basement with fiber optics if you're dealing with finance.
The cloud, as in "other people's computers", is horrifically expensive. If you're going to set up mass throughput systems, you'd better start your own data center. This is expensive as well, but it's not impossible.
If you want scalability to reduce power consumption and deal with burst workloads, you'll have to separate out your processing systems from your storage systems. Your average data center probably runs a lot of iSCSI or similar remote disk tech, possibly based on some kind of software wrapper at the cost of latency and performance but with the benefit of quickly swapping drives and expanding capacity.
Then you'll have to architect your software, of course; if you're doing batch work, you'll probably want to distribute programs in batches and run similar programs on similar chips, making optimal use of data locality and CPU cache. Maybe do the whole Hadoop thing depending on your workload.
You'll also want to figure out maintenance. You can hire a team of your own techs dealing with replacements or upgrades, but in many cases hiring external talent for a limited amount of time per month is probably cheaper and saves you the effort of keeping your workers trained.
When a machine fails and the SKU you've selected has gone out of production you'll need to figure out a replacement. This doesn't have to be a problem, but servers are fickle things and you'll probably want to use something that works with the other server vendors' tools, so you're stuck buying hardware from a limited number of suppliers in a limited number of configurations.
For management, you'll want to pretend all the computers are part of the same system. Whether you pick Kubernetes or OpenStack, you want central control to save you the headache of a million dashboards.
When you're done, you've built yourself a mainframe, except you've spent your own money on R&D and need to spend money to upgrade your hardware rather than buying licenses for the rented machines your mainframe vendor already shipped to you, waiting to be licensed. Have you saved money compared to buying overpriced mainframe hardware? Tough to say, mainframe hardware is often better at its job than normal servers. You'll save money on developers, as you no longer need to keep the old COBOL around, but when that happens you've probably paid more than you've saved when the decade long project to rewrite the backend to another language is finally done.
Mainframes are just computers good at batch jobs. They're not better or worse than regular computers, they're just different.
"""
Many factors from cost to regulatory requirements to cryptography and other security requirements play a role in making a decision to run a core banking platform in the cloud. Whilst the cloud is good for managing certain services (which we indeed use), it becomes a challenge to manage a bank’s core banking platform.
Once infrastructure is in a cloud, it is outsourced to that cloud provider. That means that the business is bound by those agreements covering aspects such as scalability, usage, capacity-on-demand and disaster recovery. These costs could grow exponentially as volumes grow. By using our own infrastructure for our core banking platform, we have full control over these factors.
Our chosen mainframe solution provides the ability for us to grow exponentially, while controlling all factors (CPU, Memory, Disk, Network, DR, Remote capability, etc) allowing us to manage the environment efficiently and effectively.
"""
I also found this extremely brief archived article about their setup, running LinuxONE on IBM Z mainframe(s?): https://www.businesslive.co.za/bd/companies/financial-servic...
Finally, Wikipedia links this article with mostly pictures: https://mybroadband.co.za/news/cloud-hosting/283199-bank-zer...
When you have reliability/continuity as a top business requirement and ACID transactions (e.g. billing) must be processed at scale, then mainframes shine.
The argument "Never change a running system." is not a wrong one, but it does not on its own explain the existence of mainframes; there are non-legacy scenarios where using a mainframe is the most reasonably choice. Finally, "cloud" as a term denoting outsourced compute/storage capacity can also apply to mainframes, see e.g. IBM's pricing brief at https://www.ibm.com/downloads/cas/YM94KV6N - some own their mainframe, some rent it - like a prive or public (internal or external) cloud.
No business in their right mind would want to be on out of date, proprietary technology with hard to source skills, but the cost and effort of migration is enormous. There are scare stories of SAP migrations costing $billions and I assume a mainframe migration could be multiples of this.
What would be interesting is if they let actual enlightened techies size up and run these projects, rather than giving to Accenture and the like. Maybe it wouldn’t be insurmountable then?
Google and Amazon don't use "the cloud" in the normal sense of a third-party public offering. They own the infrastructure that is also used by other customers as a cloud so in one sense it is private cloud but closer to on-prem than anything else.
Many other large businesses run on-prem instead of "the cloud" because the cost savings of a op-ex cloud system start to diminish when you already have your own infrastructure/networks/specialist staff as these large businesses have. Again, this is often a mix of traditional on-prem infrastructure and private clouds that offer some sharing of resources to applications that do not need an entire physical server.
Netflix, I think is a mixture of on-prem and public cloud but not sure.
So I'm not sure if you are asking "why mainframes instead of distributed e.g. microservices systems" or "why on-prem instead of using the cloud".
I used the phrase "quiver of mainframes" as the compute tasks are best served by having a cluster of architectures - dedicated RAM in the tens of terrabytes, dedicated fast local storage in the petabytes, large custers of computer nodes optimised for pipelined throughput, others optimised for hyper cubed deep mesh computations, other architures again purely for graphical representations, etc.
See: https://pawsey.org.au/supercomputing/
Lack of impetus
There are plenty of neobanks out there that have built scalable, secure infrastructures using modern development practices in the cloud. But despite growing rapidly, none of them has the scale to challenge the big banks. Similarly in areas like the airline industry, the big airlines are all old and back-end technology is rarely the deciding factor in whether or not one airline is more efficient than another. There simply isn't as strong an impetus to change as you'd expect.
Risk
There is very little incentive for any given executive at one of these firms to take the risk involved in staking their reputation on a big technology migration. Equally, there's nothing quite like a failed transformation project to destroy the careers of those associated with it. If you think mainframe is antediluvian, take a look at the ERP software the same companies are running. Layer upon layer of legacy with custom code built to manage myriad edge-cases that nobody understands anymore. Why take the risk when you can build a new system that integrates with the mainframe using (for example) a modern database that gives you a modern transactional API while micro-batching updates back to the mainframe. The incentives are all to create additional cruft.
Outsourcing
Most if not all of the big banks, airlines etc. have outsourced considerable parts of their operations over the years. In doing so, institutional knowledge was shifted out of the business into those outsourcers. The outsourcers in turn have little incentive to drive transformation of the mainframe given that a move to cloud sees their revenue deriving from the infrastructure management go to near zero. The outsourcers don't even have to act in bad faith for this to be a major problem. McKinsey and the rest thrive on complexity and by advising clients to outsource, they layered organizational and contractual complexity on the technology complexity, making the problem of transformation increasingly irreducible.
After risk, outsourcing is probably the most important factor since it is extremely difficult to create outsourced structures which maintain and develop an organic link between those responsible for business processes, and those responsible for technology. The result is an ever growing pile of sclerotic processes, dysfunctional governance bodies and uni-functional teams (often themselves outsourced to different parties for competitive purposes) that purport to control but which really just create complexity.
Outsourcing has served to worsen the organizational complexity that most mainframe users already suffered from. The result is a situation in which any programme of work to get off mainframe becomes fearsomely complex. I've worked in places which would have regular meetings of large parts of the company to try to coordinate major business process change in a single area. I've seen companies nearly break themselves trying to bring a single outsourced business function back in house. The question is why, when they're so incredibly inefficient and inflexible, they aren't competed away. That's a different question on which I have my own opinions, but this comment is too long already.
Knowledge
The loss of COBOL and other mainframe technology knowledge is real. I remember working at a bank in the EU around 2010 where I sat with a bunch of elderly gentlemen (walking sticks were a theme) who had been contracted back into the bank to develop integration between an ancient mainframe application and something modern the bank was building.
But that stereotype aside (there are surprising numbers of younger mainframe experts in India thanks to outsourcing), the problem is real, particularly when it comes to migration of software from mainframe to cloud using modern development practices. Any migration away from mainframe software requires understanding the whole technology stack and more importantly, how that stack interacts ...