I can’t speak to mainframes directly, but the greatest joy and the greatest learning I’ve experience have always seemed to come from legacy systems.
If you ever get to visit Seattle,
Paul Allen, before his death, set up something called “The Living Computer Museum” [1]. I think it was maybe $4 in admission, but you could literally sit down and program a working PDP 1 (via a teletype machine for both input and output), PDP 11, Xerox Alto, Windows 3.1, and spectacularly, a Cray 1 supercomputer.
>> What do I mean by that? How about a couple of examples: It should not be an acceptable practice to just insert a CD and indiscriminately install software onto a production machine. Mainframe systems have well-documented and enforced change management procedures that need to be followed before any software is installed into a production environment.
>>
>> Nor should it be acceptable to just flip the switch and reboot the server. Mainframe systems have safeguards against such practices. And mainframes rarely, if ever, need to be restarted because the system is hung or because of a software glitch. Or put into words that PC dudes can understand: there is no mainframe “blue screen of death.” Indeed, months, sometimes years, can go by without having to power down and re-IPL the mainframe.
This can also be true for a standard cloud server running several web services or other processes. How are any of the author's points specific to mainframes?
There are lot of interesting technologies that we should pay attention to from mainframes particularly in managing distributed processes, interesting IO and accelerator configurations, etc. But this article seriously misses the mark and makes some bad arguments that show this person hasn't really administered a non-mainframe server in the last decade or so. He's also an "IBM Gold Consultant" we should take this with a huge grain of salt.
> It should not be an acceptable practice to just insert a CD and indiscriminately install software onto a production machine.
This assumes your machine is set up correctly. I've worked on some mainframes where everyone could access customer data and swap out production executable freely. Why? Because no engineer wanted to deal with 6 layers of bureaucracy to debug an issue or update a library. Segmenting the data and isolating specific infrastructure has proven out to be more efficient and more secure.
> Nor should it be acceptable to just flip the switch and reboot the server.
These are more about the fact that there is one mainframe. The reality is for distributed systems that anyone can flip the switch at any time and you better plan for that up front. Its a good thing that I can restart my laptop, or production servers whenever I need for updates, and don't need to do 6 months of business continuity planning for "update day".
> In mainframe shops there is an entire group of people in the operations department responsible for protecting and securing mainframe systems, applications, and data.
As does any non-negligent mid-sized organization. Having a security team is very standard at this point.
> Ever wonder why there are no mainframe viruses?
Because no hacker can afford to have an IBM mainframe running in their garage?
> There is even a term – the accidental DBA – that has been coined in the SQL Server world for developers who become the DBA because nobody else is doing it. Such a situation is unheard of in the mainframe world – indeed, you’d be laughed at if you even suggested it!
Great now I need to hire extra people to specifically administer my very expensive database deployments instead of relying on a cloud provided database and relying on general "DevOps" people to have enough knowledge to troubleshoot some specific problems. The DBA didn't go away for no reason, they've just largely outlived their economic usefulness.
Regarding the "insert a CD and install software" statement, as someone who hasn't worked on this sort of environment I've always wondered - I have multiple test machines running Windows and Linux. I can push software to a canary installation on an identical server to production. Yes, I can do so indiscriminately, but by the time I push canary to production it's had a certain level of testing.
I knew a mainframe engineer and he certainly didn't have multiple spare mainframes laying around to play with. There might have been some form of partitioned install they called "test" but by the time people bureaucracy gets involved noone is "playing with it" and ergo, there's a lot less actual experts involved.
Bonus question: Was there never a firmware bug that would take down an entire mainframe? A certain server line was advertised with "mainframe like redundancy" on Windows, yet that's exactly my experience.
For test systems, you can create LPARs on IBM z mainframes. I don't have much mainframe experience but I know they have access to test systems and development systems
Not sure about bugs, but I suspect mainframes have fairly redundant software with enough hardware and software to handle unknown bugs.
> Was there never a firmware bug that would take down an entire mainframe?
Yes, there was quite an ugly one not that long ago in fact. A local mega-corporation which still has some mainframes had its main (and redundant) systems just roll over and die suddenly, plus their hot backup systems which were offsite. I don't remember all of the details, but IIRC this was a case of "We [the vendor] knew we had a major bug, and we had a fix for it, but we were remiss about getting it out to all of our customers in a timely manner." This type of thing is shockingly rare in the mainframe world, though.
I am skeptical of the claims of superior mainframe security.
1. The IT Security Profession is young, in many ways, it's still in its infancy. I say this because today you can still get a B.S. in Computer Science without being taught secure coding best practices or having that be integrated into the curriculum. So given that the state of the art is young, and yet mainframes are quite old and have a reputation of being slow to change, how could they possibly keep up with best practices?
2. The mainframe world tends to be highly proprietary, even with proprietary protocols. I am skeptical that they adopted appropriate information sharing and training mechanisms, and how can they recruit security talent when access to the system is so closed? How would we even know of attacks if they are kept secret and not reported?
Security through obscurity should never be your only line of defense, but it's a good thing to layer upon the other defenses.
Hercules is a great teaching tool, but it will limit the kind of exploits you can develop (the most interesting being hardware based ones anyway).
On top of that, mainframes are very observable machines. Because a lot of companies rented out extra capacity and billed it by usage (70's AWS), everything is audited, logged, measured, traced and, unless the intrusion went really deep, you'll have detailed information on how it happened.
There is a course I never took ("Hack the mainframe") by a couple nice folks I know from Twitter that teaches a lot about vulnerabilities in default settings and how to secure mainframe based services.
Mainframe security (or mainframe era security) was stronger simply because of access. Introducing media required access to a connected peripheral and console (or TSO, at least) access, both of which were probably in a secure, raised-floor computer center. You couldn't rely on leaving a tape sitting on a bench and expect an operator to just load it, a la Stuxnet.
Having spent my entire career with one foot on the "legacy" side of things (mainframe-like systems) and the other on the Wintel side, I can tell you that I've had to deal with any number of problems on the Wintel side that generally just wouldn't happen - and indeed in many cases couldn't happen - on the legacy side.
As an example of how things are done on the legacy side, I was once dealing with a piece of Power hardware which could be configured as either a Linux system or an iSeries system. There was a USB port in the hardware which was enabled on Linux systems but disabled on iSeries systems, and AFAIK it couldn't be enabled, either. This puzzled me at first, until I found out how USB ports can be used to attack systems and then it made perfect sense - they had chosen to disable the USB port for security reasons.
If an attacker wants to learn to attack IBM mainframes, they don't need a mainframe in their garage. Using the open source Hercules emulator, you can run IBM mainframe operating systems emulated on x86. Legally, you can only run really old versions, but piracy of newer versions is widespread. (IBM will sell you a legitimate way to emulate newer versions on x86 for a few thousand dollars annual subscription - but we can assume an attacker isn't going to be bothered with obeying copyright laws.)
Security through obscurity is a bad practice, but sometimes it really works. Exotic platforms tend not to get much malware, not because they lack security vulnerabilities, but simply because knowledge of how they work (and in many cases, even opportunity to acquire that knowledge) is rare.
> Security through obscurity is a bad practice, but sometimes it really works. Exotic platforms tend not to get much malware, not because they lack security vulnerabilities, but simply because knowledge of how they work (and in many cases, even opportunity to acquire that knowledge) is rare.
More importantly, the return per hour spent to write that malware is lower. There are fewer macs, though knowledge is widespread, so there are more windows viruses because there is a larger pool of victims.
This reads like it was written by someone who has never worked with a properly managed distributed system.
What I can believe is that it's much easier to mismanage a distributed fleet of consumer PCs, as a result of mainframes mostly only having been used in situations with extensive change management and change audit processes.
> significantly less capable than a normal computer
less capable than the underlying mainframe, you mean. Unless you have tons of separate VMs running on your mainframe, you can have tens of terabytes of RAM available to your job and a fridge-sized cabinet of PCIe channels to do its IO (which is done by specialized CPUs).
Mainframe environments are partitioned. Usually, when you want to validate a change, you deploy it to a separate partition of the same machine that is running your current workload.
> Parts of AWS are down right now, as I understand it.
I haven't heard about this, production looks fine. Even if that's the case: the key point being "parts". A meteor can blow up a whole AWS datacenter, and your production systems should still survive. There may be a blip while capacity is provisioned. If you _really_ want to be safe and invest the engineering resources, you can deploy across regions. Or even clouds.
Done it for 10 years from programmer to os support to dba. It taught you some discipline but as one poster said you can learn that using any large system environment. The problem of that Env is its too protected and limited in its variety. Good luck live out of the glass room.
This sounds like: If you can afford a mainframe and people to manage it, you get an experienced team of people who can manage mainframes well. Not that I disagree, but... this is about how money can give you sophisticated environments, not about mainframes. All of those points can be replicated with a distributed system of classic consumer hardware.
There is something to be said about finding work when the client is completely unable to assess your skills. Or just doesn't care because the environment is so restrictive and bulletproof you can't damage it no matter how incompetent you are.
Not only that, almost every job posting I see for mainframe development wants a decade of experience working with the systems, and that's not something you can reasonably expect to self teach. I imagine most of the people who have significant experience with mainframes are retiring nowadays
>It should not be an acceptable practice to just insert a CD and indiscriminately install software onto a production machine. Mainframe systems have well-documented and enforced change management procedures that need to be followed before any software is installed into a production environment.
This applies to pretty much any production environment. Why would you have to work on a mainframe to understand this.
As a side note I just realized I’ve never physically touched any sever I’ve worked on professionally.
I just realized I’ve never physically touched any sever I’ve worked on professionally.
In my career, nothing felt quite as high-tech in its moment as hands-on mainframe operation, oddly enough.
- Loading a custom removable boot drive.
- actually dialing in a hex address of a boot device on a console
- knowing and using command sequences to bring up and down systems with thousands of simultaneous users
- using quiesce to halt a mainframe, leaving all processes intact but frozen, so major hardware components could be worked on... then all processes resumed in place
- bailing somebody out of a screwup that would otherwise require hours and thousands of dollars to redo
- working on technology than spans 20+ years in the same shift
Best of all, I was mostly on my feet all night rather than sitting at a desk.
As for subsequent realms (UNIX, PC, VAX, etc), I started before servers were banished to racks and always had hands-on access back then. It was a big improvement when we could do DOS builds using cross-compilers on Sun and then serve the binaries over PC-NFS (welcome to 1993).
Right after college I worked with one of those stereotypical sys admins that were very smart, into IETF newsletters, loved Sun, etc etc.
He had 2 Solaris servers under his desk and I loved to listen to everything he said about them and Sun. Too bad his talent was wasted when his job was outsourced to cheaper locales, I mean RIF-ed excuse me.
I remember feeling like I had really "arrived" that day I got my very own Sparcstation on my desk. But it was still not as much fun as operating an IBM 3081 with two banks of tape drives by myself throughout the night.
By "mainframe environment" the author of the article is presumably referring to IBM's zOS. Unlike minicomputers, DOS, etc, it's been continuously developed and improved over time. Other than using weird terminology for everything that comes from not being Unix-derived, nowadays it's a modern server OS.
This isn't solely a mainframe thing. The pty/tty designation for terminals on Linux is a reference to when the computer had no display, but output to a typewriter (to give one such example).
Pretty much all of the related documentation is publicly available, AFAIK, and freely accessible on the internet from what I can tell. This availabilty is actually a legal requirement from times past, IIRC.
My cousin recently went to a weeks-long class and got zOS certified. He's been a sysadmin at a university for years, and they don't use zOS there at all. But I think he did it just to learn the system and have some career insurance. I was fascinated that people are doing this.
Part of this is laughable. "Security" of mainframes in particular. I have a friend who does security work on mainframes, and, like, no.
Also, I will someday write a blog post about how to get into the field, starting with "at first, grow up on a farm". The point being that there are hundreds of ways to get into this field and have serious fun without following some Official way.
Part of the appeal of mainframes, and of consultancies that focus on them is the seriously low salaries. Oh, also hostility to new ideas and lack of development tools.
Still, some of the other points are more correct than at first sight - the author's failure is to compare mainframes with single servers when, in reality, they behave more as tightly coupled clusters of specialized machines. When you switch to this mindset, a lot of the problems we face managing our clusters have mainframe counterparts since the mid 80's.
I would note that we do have something analogous to the mainframe in the regular PC-server world: the virtualization-host control-plane server cluster. Taken as a unit, a vSphere controller requires just as much hand-holding and just as many techno-priests as a z/OS deployment.
Really, it's not the technology that's unique with a mainframe, that gives it the properties it has; rather, it's the scale, the multitenancy, and the SLA of the use-cases that the technology is being put to. In other situations where those same things are true (such as the hypervisor control-planes of big public clouds, or tier-1 network switches, or telecom equipment) you see the same kind of top-down operational planning form around them.
I think that's the proper analog for a mainframe in the Linux space - a cluster. A mainframe is, ultimately, a very tightly coupled cluster of specialized nodes, some coupled so tightly they can share memory, some less coupled that deal with IO.
And, I think, this is where we can learn the most from mainfrfames - they have been where we are for a long time now and solved some of our problems in vastly different ways.
One thing I really miss from mainframes is their stability.
Case a) Lots of years ago IBM changed CPU architecture on their System i. Every program we made was transparently recompiled to the new instruction set on first run, and just worked after that.
Case b) Legacy technologies are taken very seriously. Decades old programs made in obsolete versions of COBOL can still be compiled without issues on the modern systems. No unexpected side effects, no new bugs, it just works. It provides a comfortable path to gradual upgrade.
Case c) The tech stack builds upon itself, which means you can devote years to learn the stack and still enjoy the fruits of your effort for decades. You just have to keep up with the changes, not relearn everything every five years.
Case d) The manuals and reference are exhaustive and absolutely superb, you can become an expert just by reading them. How many questions about mainframe development are created daily in Stack Overflow?
Their biggest drawback is their cost. Mainframe as a Service exist, but it needs to be affordable enough to run cheap experiments, otherwise it won't be attractive to startups.
The biggest drawback is that it's inflexible, slow to work on and pretty much the definition of proprietary. Everything is a secret, everything has as much red tape as a lawyers office and nobody wants to share.
Slow to work on? Oh no, you didn't. I'd even contest it's more productive, as the developments tools come preinstalled and they integrate seamlessly with the platform.
It isn't as flexible as a PC, but you shouldn't expect it to. You learn the design, conventions and idiosyncrasy of the platform and work with them, not against them.
Slow as in you can't do as many release cycles on it as say, a distributed cluster. We're hosting about 700 services with about 6 instances each and release each of them multiple times a day. That's about 4000 production releases a day with no downtime, errors, or security issues. Includes rollbacks, roll forwards, AB-tests, versioned releases etc. No mainframes anymore either, and for 10% of the cost.
Maybe this isn't the general standard, or maybe this wouldn't work for multinationals or companies/workforces with different values, but it works here. I suppose it also depends on what your company is doing; we're in fast moving online retail, if you are in a slower operating environment (i.e. only doing batch processing or bulk messaging with ETL of different kinds) where changes aren't giving you an edge over competition if must be a different set of operational requirements too.
Certainly not the i. There is so much documentation that it might be more of an issue of it being overwhelming. there are multiple active communities sharing code.
this is a system which is based on DB2 meaning SQL in native, development tools are varied, from COBOL, RPGLE, JAVA, and more. You can web face nearly any existing process, node.js work is one area we have lots of current work. no down time for reorganizing files, indexes always maintained, and tools open to all on the platform to optimize sql statements. I have developers using all this and more. most use an eclipse based platform (RDi for Cobol/RPGLE) or eclipse itself to work.
admin wise it is a breeze, no need to hunt down a tcp/ip stack, db tools, schedulers, or such. its all built in.
Legacy systems are only limited by the imagination of those using them. the reason the systems persist and grow is because they are not frozen in time and their architecture readily adapts to new technology because they were designed to not be locked down with an inflexible design.
This is actually one of my favorite things about Linux. I learned Unix back around 1989, using IBM AIX. The AT&T books (blue star-field covers) were mostly applicable (except for IBM specific items), and everything I learned then I'm still putting to good use today on modern Linux systems. Printing has improved somewhat, disk storage a bit more, and of course my favorite, the /proc file system. But all my shell, awk, sed, grep, sort, join, vi commands still work as expected. I even had something similar to gnu screen, called VTCS (virtual terminal control system), but I wasn't on the Internet at the time so I never released it. Would love to dig up the source code for that again though.
Right. And with open source software decades old programs can be compiled without issues too using last known good compilers, toolchains, libraries, preserving all the old bugs and vulnerabilities, just like on mainframes, but for free.
So everything he misses is there. It's just that running legacy code is too risky these days unless you are completely isolated from the internet. And modern infrastructure has a lot more essential complexity.
I'd say this kind of source-based backwards compatibility is superior to what's happening in Windows world -- not only it encourages you to keep the source code open, but you also don't have to keep as many hacks in the kernel/system libraries.
It was fairly evident to me by the mid 1990s that Unix likely had staying power over other. options -- VMS, MVS, Mac (pre-OSX), and Windows. And yes, I'd worked on all of those, plus a few others.
> The tech stack builds upon itself, which means you can devote years to learn the stack and still enjoy the fruits of your effort for decades. You just have to keep up with the changes, not relearn everything every five years.
Then you get laid off at 50 and have no hope of finding another job at the same salary.
Anecdata: I'm more used to see senior mainframe people change jobs than be laid off. The pay is not that higher compared to other platforms, so maybe that softens the temptation to save on salaries.
> IT professionals should work in a mainframe environment at some point (2015)
... in order to experience and learn that:
* Max depth of your file system tree is 2
* Password must be changed every month
* Max password length is 8
* Alphanumeric chars only
* No upper/lower case distinction
* There's nothing wrong about using java 1.7 in 2019
* "if <customer == XYZ> then <feature>" is just fine as a code pattern.
* To release uncompiled & untested code is OK.
* To release code with compilation errors is OK, too.
* To use CVS could be a huge step forward. Because the thing you've been using can manage only code files but not configuration files. Albeit "manage" is a huuuge overstatement here.
* Print out the list of changes and stamp it with a rubber stamp is the way to fix it.
* Permanently.
* Permanently-Partially. Half of the team was printing the other half was copy-pasting the list of changes to a file.
* For years. On the PCs. Not on the mainframe.
* In MS Word documents, not as a plain-text. Because reasons.
* Ok, several months after my predecessor left I gained enough confidence to ask for the permission to abandon the printing in favor of the MS Word. I was injured at the time and I thought like the pitiful picture of me limping between the printer and the desk 4 times a day could help me in an argumentation.
* There are mainframe developers who don't even know how e.g. XML looks like. No kidding.
* And they won't get fired neither asked to learn anything new. And their salary has 6 digits.
I interviewed someone who worked in the IT department at one of the largest manufacturers of commercial vehicles, and a mainframe / COBOL shop. He described their development process. It involved printing out your source code, hole-punching the paper, putting it into a three-ring binder, and placing it in the physical inbox on the desk of the person who was assigned to conduct code review.
At least where I worked, those same developers are close kin to satan to work with. Not even working together on anything. Just being in their presence.
(rewritten by suggestion - same content, different vibe)
Many of you came here to talk ish about mainframes, and by doing so you precisely illustrate why new IT people need to learn about mainframes.
It's disingenuous to say that GNU/Linux is getting better when that is hardly the case. It has business-itis. One distro's lessons don't translate well or sometimes at all to another's. It's growing subsystems and methods that don't have well thought out or even decently documented rationale, and the rate of change doesn't appear to be slowing.
By trying to make examples of some of the worse characteristics of mainframes, some of you are showing that you're not speaking to the article at all.
The point is that what we call modern with regards to reliability is simply aggregation and redundancy. It's a collection of solutions to symptoms and they don't address the underlying problems at all. This is nohow like a mainframe. Not even a little bit.
Heck - a modern virtual machine farm can't even cluster as well as VMS can cluster. Why? Because our virtual machine farms have evolved to fix the symptoms that come from the problems of less reliability, and have taken little or nothing from mainframes.
Mainframes have already learned about the problems of reliability, addressed those problems, then moved on, and they did that half a century ago.
You have some good points here, but please don't post in the flamewar style to Hacker News. We're trying for something a bit better than internet default discussion, and flamewars point the other way.
But throwing together a lot of cheap unreliable systems is, cheaper, easier and more flexible. I don’t care if my containers are crashing several times a day. I don’t care if a node in my kubernetes cluster falls over. I’m not working at the level of a single machine.
Yes, that. Modern cluster systems are designed on the premise that hardware and software fails.
Netflix's 'chaos monkey' software is an example of something that goes around randomly breaking things within clusters just to see if something fails to transparently recover without service interruption.
It reads like it was written by someone who just happens to like mainframes. None of the listed aspects are mainframe-exclusive. On the other hand: yes, there are a whole lot of "IT" places that just run broken workflows and cheap out on everything. Those places will never buy a mainframe.
The gist of the whole article (as already stated a few times) is "work in a properly managed place".
Summer of 2000 I started working as an intern at a Fortune 500 w/ a series MF. Ostensibly, I was to be a dedicated (primarily) web developer for the DBA team (around 8 people excluding me). The primary RDBMS was DB2 on the MF, also IMS.
The first year, most of my time was spent doing VB Script in ASP and some occasional DB query tuning. My second year, .Net was new and I convinced my boss to migrate to ASP.net (I'd done a project in school with it, so I had a decent understanding of .Net for the time).
Anyways, a couple of highlights:
Pioneering use of BLOB storage in DB2 at the company (was a huge pain in the ass working with BLOBs from classic ASP).
I crashed the development LPAR (logical portion, sort of a VM in MF parlance) tuning a SQL query (hit a bug in the DB2 query optimized that somehow knocked out the entire OS). My green-screen term disconnect shortly after I tried to run my query. I got a call from the NOC along the lines of "whatever you did, dont do it again. You just took down the entire dev LPAR". Knocked about 10000 people off with a SQL query.
I spent an entire summer trying to get DB2 Connect (required software, at least at the time, to talk to the MF DB from either Windows or *nix), spent 8 hours a day most days of the week on calls with IBM support trying to get it to work on a windows server cluster. Followed their documented instructions to the letter. Never got it to work. After the summer ended, they came back with: "this is an unsupported configuration", despite clear documentation on how to set it up in their printed manuals.
Had a bug against DB2 where the "select" columns could cause different number of records to be returned. It was a difference between using a coalesce vs "case when null". We happened to have a 3rd party DB tuning expert on site teaching a class at the time. When I showed the 2 theoretically queries she was stumped. IBM got a copied of our non-sensitive DB to replicate the results. Never did hear about a resolution on that.
Apparently about a month or so before I started, they had just retired the last punch card reader. The extra unused punched made for great notecards.
Finally, one of my favorites was when I attended a series of meetings at the datacenter. The DC was over 100 miles away from the office, because apparently it got the company a huge cost savings in disaster insurance. The building was basically a bunker in the northern Midwest. 1 window in the entire building, a little more than a foot square. For the security guard. It was also inch thick bullet proof glass. No heaters, but around 2 dozen or so industrial air conditioners, each the size of about 2 residential refrigerators.
So, along with getting rid of the punch card reader, they had recently upgraded to a tape silo, instead of having human runners get a tape and load it, when requested. The new tape silo also had incredibly higher storage density. So, all the old tapes were gone, nearly every was in the silo. Thus, the old tape room adjacent to the server was no longer needed for tapes, and the room was turned into a meeting room. The tape shelves were removed, and conference tables and chairs were added. Nothing else change. Including the automated door shutting and HALON activating in case of a fire. IIRC, they warned us that you had about 15 seconds from the time the fire alarm sounded to evacuate that room before the door forcibly closed and the HALON activated and near certainly killing you.
Halon activation wasn't fatal to people. It was designed to be the least harmful method to people, equipment, and materials alike. (The ozone layer, on the other hand, ...)
I started my career on a IBM 360 but I learned far more useful skills with ed then vi and Unix (along with C, Lex and yacc). Those skills have stayed fresh for 30+ years.
122 comments
[ 3.1 ms ] story [ 208 ms ] threadIf you ever get to visit Seattle, Paul Allen, before his death, set up something called “The Living Computer Museum” [1]. I think it was maybe $4 in admission, but you could literally sit down and program a working PDP 1 (via a teletype machine for both input and output), PDP 11, Xerox Alto, Windows 3.1, and spectacularly, a Cray 1 supercomputer.
[1] https://livingcomputers.org/
>>
>> Nor should it be acceptable to just flip the switch and reboot the server. Mainframe systems have safeguards against such practices. And mainframes rarely, if ever, need to be restarted because the system is hung or because of a software glitch. Or put into words that PC dudes can understand: there is no mainframe “blue screen of death.” Indeed, months, sometimes years, can go by without having to power down and re-IPL the mainframe.
This can also be true for a standard cloud server running several web services or other processes. How are any of the author's points specific to mainframes?
> It should not be an acceptable practice to just insert a CD and indiscriminately install software onto a production machine.
This assumes your machine is set up correctly. I've worked on some mainframes where everyone could access customer data and swap out production executable freely. Why? Because no engineer wanted to deal with 6 layers of bureaucracy to debug an issue or update a library. Segmenting the data and isolating specific infrastructure has proven out to be more efficient and more secure.
> Nor should it be acceptable to just flip the switch and reboot the server.
These are more about the fact that there is one mainframe. The reality is for distributed systems that anyone can flip the switch at any time and you better plan for that up front. Its a good thing that I can restart my laptop, or production servers whenever I need for updates, and don't need to do 6 months of business continuity planning for "update day".
> In mainframe shops there is an entire group of people in the operations department responsible for protecting and securing mainframe systems, applications, and data.
As does any non-negligent mid-sized organization. Having a security team is very standard at this point.
> Ever wonder why there are no mainframe viruses?
Because no hacker can afford to have an IBM mainframe running in their garage?
> There is even a term – the accidental DBA – that has been coined in the SQL Server world for developers who become the DBA because nobody else is doing it. Such a situation is unheard of in the mainframe world – indeed, you’d be laughed at if you even suggested it!
Great now I need to hire extra people to specifically administer my very expensive database deployments instead of relying on a cloud provided database and relying on general "DevOps" people to have enough knowledge to troubleshoot some specific problems. The DBA didn't go away for no reason, they've just largely outlived their economic usefulness.
I knew a mainframe engineer and he certainly didn't have multiple spare mainframes laying around to play with. There might have been some form of partitioned install they called "test" but by the time people bureaucracy gets involved noone is "playing with it" and ergo, there's a lot less actual experts involved.
Bonus question: Was there never a firmware bug that would take down an entire mainframe? A certain server line was advertised with "mainframe like redundancy" on Windows, yet that's exactly my experience.
Not sure about bugs, but I suspect mainframes have fairly redundant software with enough hardware and software to handle unknown bugs.
Yes, there was quite an ugly one not that long ago in fact. A local mega-corporation which still has some mainframes had its main (and redundant) systems just roll over and die suddenly, plus their hot backup systems which were offsite. I don't remember all of the details, but IIRC this was a case of "We [the vendor] knew we had a major bug, and we had a fix for it, but we were remiss about getting it out to all of our customers in a timely manner." This type of thing is shockingly rare in the mainframe world, though.
1. The IT Security Profession is young, in many ways, it's still in its infancy. I say this because today you can still get a B.S. in Computer Science without being taught secure coding best practices or having that be integrated into the curriculum. So given that the state of the art is young, and yet mainframes are quite old and have a reputation of being slow to change, how could they possibly keep up with best practices?
2. The mainframe world tends to be highly proprietary, even with proprietary protocols. I am skeptical that they adopted appropriate information sharing and training mechanisms, and how can they recruit security talent when access to the system is so closed? How would we even know of attacks if they are kept secret and not reported?
Hercules is a great teaching tool, but it will limit the kind of exploits you can develop (the most interesting being hardware based ones anyway).
On top of that, mainframes are very observable machines. Because a lot of companies rented out extra capacity and billed it by usage (70's AWS), everything is audited, logged, measured, traced and, unless the intrusion went really deep, you'll have detailed information on how it happened.
There is a course I never took ("Hack the mainframe") by a couple nice folks I know from Twitter that teaches a lot about vulnerabilities in default settings and how to secure mainframe based services.
As an example of how things are done on the legacy side, I was once dealing with a piece of Power hardware which could be configured as either a Linux system or an iSeries system. There was a USB port in the hardware which was enabled on Linux systems but disabled on iSeries systems, and AFAIK it couldn't be enabled, either. This puzzled me at first, until I found out how USB ports can be used to attack systems and then it made perfect sense - they had chosen to disable the USB port for security reasons.
> Because no hacker can afford to have an IBM mainframe running in their garage?
There have been IBM mainframe malware in the past, the most famous incident being https://en.wikipedia.org/wiki/Christmas_Tree_EXEC
If an attacker wants to learn to attack IBM mainframes, they don't need a mainframe in their garage. Using the open source Hercules emulator, you can run IBM mainframe operating systems emulated on x86. Legally, you can only run really old versions, but piracy of newer versions is widespread. (IBM will sell you a legitimate way to emulate newer versions on x86 for a few thousand dollars annual subscription - but we can assume an attacker isn't going to be bothered with obeying copyright laws.)
Security through obscurity is a bad practice, but sometimes it really works. Exotic platforms tend not to get much malware, not because they lack security vulnerabilities, but simply because knowledge of how they work (and in many cases, even opportunity to acquire that knowledge) is rare.
More importantly, the return per hour spent to write that malware is lower. There are fewer macs, though knowledge is widespread, so there are more windows viruses because there is a larger pool of victims.
None of it works for actual distributed systems.
He once mentioned in the pub at lunch that "oh my first boss was Diskaja".
What I can believe is that it's much easier to mismanage a distributed fleet of consumer PCs, as a result of mainframes mostly only having been used in situations with extensive change management and change audit processes.
less capable than the underlying mainframe, you mean. Unless you have tons of separate VMs running on your mainframe, you can have tens of terabytes of RAM available to your job and a fridge-sized cabinet of PCIe channels to do its IO (which is done by specialized CPUs).
This is a huge simplification.
I haven't heard about this, production looks fine. Even if that's the case: the key point being "parts". A meteor can blow up a whole AWS datacenter, and your production systems should still survive. There may be a blip while capacity is provisioned. If you _really_ want to be safe and invest the engineering resources, you can deploy across regions. Or even clouds.
Try that with a mainframe.
...And a cloud-based budget.
finds job board posting $60-$80 USD
> we need someone to replace multiple existing mainframe systems...
Doh. Well. Somebody tell the author, someone's out for their job!
There is something to be said about finding work when the client is completely unable to assess your skills. Or just doesn't care because the environment is so restrictive and bulletproof you can't damage it no matter how incompetent you are.
This applies to pretty much any production environment. Why would you have to work on a mainframe to understand this.
As a side note I just realized I’ve never physically touched any sever I’ve worked on professionally.
Sadly, the age of the 3278 is gone.
- Loading a custom removable boot drive.
- actually dialing in a hex address of a boot device on a console
- knowing and using command sequences to bring up and down systems with thousands of simultaneous users
- using quiesce to halt a mainframe, leaving all processes intact but frozen, so major hardware components could be worked on... then all processes resumed in place
- bailing somebody out of a screwup that would otherwise require hours and thousands of dollars to redo
- working on technology than spans 20+ years in the same shift
Best of all, I was mostly on my feet all night rather than sitting at a desk.
As for subsequent realms (UNIX, PC, VAX, etc), I started before servers were banished to racks and always had hands-on access back then. It was a big improvement when we could do DOS builds using cross-compilers on Sun and then serve the binaries over PC-NFS (welcome to 1993).
He had 2 Solaris servers under his desk and I loved to listen to everything he said about them and Sun. Too bad his talent was wasted when his job was outsourced to cheaper locales, I mean RIF-ed excuse me.
I worked on mainframes for years and cheered when minis came along.
I worked on minis for years and cheered when PC's came along.
I worked on DOS for years and wept when Windows came along.
I worked on windows for years and sang choruses of joy when Linux came along.
I haven't moved off Linux because it just keeps getting better and better.
Would I recommend anyone go back off Linux to ye olde mainframes?
Not on yer Nelly. Never.
Linux and UNIX used to be installed using stacks of diskettes and piles of patience.
http://www.redbooks.ibm.com/abstracts/sg246366.html?Open
Also, I will someday write a blog post about how to get into the field, starting with "at first, grow up on a farm". The point being that there are hundreds of ways to get into this field and have serious fun without following some Official way.
Part of the appeal of mainframes, and of consultancies that focus on them is the seriously low salaries. Oh, also hostility to new ideas and lack of development tools.
Disagree strongly.
Still, some of the other points are more correct than at first sight - the author's failure is to compare mainframes with single servers when, in reality, they behave more as tightly coupled clusters of specialized machines. When you switch to this mindset, a lot of the problems we face managing our clusters have mainframe counterparts since the mid 80's.
EBCDIC
/s :-)
Really, it's not the technology that's unique with a mainframe, that gives it the properties it has; rather, it's the scale, the multitenancy, and the SLA of the use-cases that the technology is being put to. In other situations where those same things are true (such as the hypervisor control-planes of big public clouds, or tier-1 network switches, or telecom equipment) you see the same kind of top-down operational planning form around them.
And, I think, this is where we can learn the most from mainfrfames - they have been where we are for a long time now and solved some of our problems in vastly different ways.
Case a) Lots of years ago IBM changed CPU architecture on their System i. Every program we made was transparently recompiled to the new instruction set on first run, and just worked after that.
Case b) Legacy technologies are taken very seriously. Decades old programs made in obsolete versions of COBOL can still be compiled without issues on the modern systems. No unexpected side effects, no new bugs, it just works. It provides a comfortable path to gradual upgrade.
Case c) The tech stack builds upon itself, which means you can devote years to learn the stack and still enjoy the fruits of your effort for decades. You just have to keep up with the changes, not relearn everything every five years.
Case d) The manuals and reference are exhaustive and absolutely superb, you can become an expert just by reading them. How many questions about mainframe development are created daily in Stack Overflow?
Their biggest drawback is their cost. Mainframe as a Service exist, but it needs to be affordable enough to run cheap experiments, otherwise it won't be attractive to startups.
It isn't as flexible as a PC, but you shouldn't expect it to. You learn the design, conventions and idiosyncrasy of the platform and work with them, not against them.
Maybe this isn't the general standard, or maybe this wouldn't work for multinationals or companies/workforces with different values, but it works here. I suppose it also depends on what your company is doing; we're in fast moving online retail, if you are in a slower operating environment (i.e. only doing batch processing or bulk messaging with ETL of different kinds) where changes aren't giving you an edge over competition if must be a different set of operational requirements too.
Certainly not the i. There is so much documentation that it might be more of an issue of it being overwhelming. there are multiple active communities sharing code.
this is a system which is based on DB2 meaning SQL in native, development tools are varied, from COBOL, RPGLE, JAVA, and more. You can web face nearly any existing process, node.js work is one area we have lots of current work. no down time for reorganizing files, indexes always maintained, and tools open to all on the platform to optimize sql statements. I have developers using all this and more. most use an eclipse based platform (RDi for Cobol/RPGLE) or eclipse itself to work.
admin wise it is a breeze, no need to hunt down a tcp/ip stack, db tools, schedulers, or such. its all built in.
Legacy systems are only limited by the imagination of those using them. the reason the systems persist and grow is because they are not frozen in time and their architecture readily adapts to new technology because they were designed to not be locked down with an inflexible design.
So everything he misses is there. It's just that running legacy code is too risky these days unless you are completely isolated from the internet. And modern infrastructure has a lot more essential complexity.
>It is amazing how well things work – the modern assembler, linker, and debugger handles the code generated by GCC 1.27 without any problems.
https://kristerw.blogspot.com/2019/01/building-gcc-127.html
And the write-up he based his experiments upon:
https://miyuki.github.io/2017/10/04/gcc-archaeology-1.html
I'd say this kind of source-based backwards compatibility is superior to what's happening in Windows world -- not only it encourages you to keep the source code open, but you also don't have to keep as many hacks in the kernel/system libraries.
Learning `vi` in 1979 then using it for 40 years seems like a good return on investment.
Both would also be examples of survivorship bias.
Then you get laid off at 50 and have no hope of finding another job at the same salary.
... in order to experience and learn that:
* Max depth of your file system tree is 2
* Password must be changed every month
* Max password length is 8
* Alphanumeric chars only
* No upper/lower case distinction
* There's nothing wrong about using java 1.7 in 2019
* "if <customer == XYZ> then <feature>" is just fine as a code pattern.
* To release uncompiled & untested code is OK.
* To release code with compilation errors is OK, too.
* To use CVS could be a huge step forward. Because the thing you've been using can manage only code files but not configuration files. Albeit "manage" is a huuuge overstatement here.
* Print out the list of changes and stamp it with a rubber stamp is the way to fix it.
* Permanently.
* Permanently-Partially. Half of the team was printing the other half was copy-pasting the list of changes to a file.
* For years. On the PCs. Not on the mainframe.
* In MS Word documents, not as a plain-text. Because reasons.
* Ok, several months after my predecessor left I gained enough confidence to ask for the permission to abandon the printing in favor of the MS Word. I was injured at the time and I thought like the pitiful picture of me limping between the printer and the desk 4 times a day could help me in an argumentation.
* There are mainframe developers who don't even know how e.g. XML looks like. No kidding.
* And they won't get fired neither asked to learn anything new. And their salary has 6 digits.
* etc.
Many of you came here to talk ish about mainframes, and by doing so you precisely illustrate why new IT people need to learn about mainframes.
It's disingenuous to say that GNU/Linux is getting better when that is hardly the case. It has business-itis. One distro's lessons don't translate well or sometimes at all to another's. It's growing subsystems and methods that don't have well thought out or even decently documented rationale, and the rate of change doesn't appear to be slowing.
By trying to make examples of some of the worse characteristics of mainframes, some of you are showing that you're not speaking to the article at all.
The point is that what we call modern with regards to reliability is simply aggregation and redundancy. It's a collection of solutions to symptoms and they don't address the underlying problems at all. This is nohow like a mainframe. Not even a little bit.
Heck - a modern virtual machine farm can't even cluster as well as VMS can cluster. Why? Because our virtual machine farms have evolved to fix the symptoms that come from the problems of less reliability, and have taken little or nothing from mainframes.
Mainframes have already learned about the problems of reliability, addressed those problems, then moved on, and they did that half a century ago.
https://news.ycombinator.com/newsguidelines.html
Netflix's 'chaos monkey' software is an example of something that goes around randomly breaking things within clusters just to see if something fails to transparently recover without service interruption.
The gist of the whole article (as already stated a few times) is "work in a properly managed place".
Summer of 2000 I started working as an intern at a Fortune 500 w/ a series MF. Ostensibly, I was to be a dedicated (primarily) web developer for the DBA team (around 8 people excluding me). The primary RDBMS was DB2 on the MF, also IMS.
The first year, most of my time was spent doing VB Script in ASP and some occasional DB query tuning. My second year, .Net was new and I convinced my boss to migrate to ASP.net (I'd done a project in school with it, so I had a decent understanding of .Net for the time).
Anyways, a couple of highlights:
Pioneering use of BLOB storage in DB2 at the company (was a huge pain in the ass working with BLOBs from classic ASP).
I crashed the development LPAR (logical portion, sort of a VM in MF parlance) tuning a SQL query (hit a bug in the DB2 query optimized that somehow knocked out the entire OS). My green-screen term disconnect shortly after I tried to run my query. I got a call from the NOC along the lines of "whatever you did, dont do it again. You just took down the entire dev LPAR". Knocked about 10000 people off with a SQL query.
I spent an entire summer trying to get DB2 Connect (required software, at least at the time, to talk to the MF DB from either Windows or *nix), spent 8 hours a day most days of the week on calls with IBM support trying to get it to work on a windows server cluster. Followed their documented instructions to the letter. Never got it to work. After the summer ended, they came back with: "this is an unsupported configuration", despite clear documentation on how to set it up in their printed manuals.
Had a bug against DB2 where the "select" columns could cause different number of records to be returned. It was a difference between using a coalesce vs "case when null". We happened to have a 3rd party DB tuning expert on site teaching a class at the time. When I showed the 2 theoretically queries she was stumped. IBM got a copied of our non-sensitive DB to replicate the results. Never did hear about a resolution on that.
Apparently about a month or so before I started, they had just retired the last punch card reader. The extra unused punched made for great notecards.
Finally, one of my favorites was when I attended a series of meetings at the datacenter. The DC was over 100 miles away from the office, because apparently it got the company a huge cost savings in disaster insurance. The building was basically a bunker in the northern Midwest. 1 window in the entire building, a little more than a foot square. For the security guard. It was also inch thick bullet proof glass. No heaters, but around 2 dozen or so industrial air conditioners, each the size of about 2 residential refrigerators.
So, along with getting rid of the punch card reader, they had recently upgraded to a tape silo, instead of having human runners get a tape and load it, when requested. The new tape silo also had incredibly higher storage density. So, all the old tapes were gone, nearly every was in the silo. Thus, the old tape room adjacent to the server was no longer needed for tapes, and the room was turned into a meeting room. The tape shelves were removed, and conference tables and chairs were added. Nothing else change. Including the automated door shutting and HALON activating in case of a fire. IIRC, they warned us that you had about 15 seconds from the time the fire alarm sounded to evacuate that room before the door forcibly closed and the HALON activated and near certainly killing you.
I always went for the chair closest to the door.
https://www-01.ibm.com/events/wwe/ast/mtm/audit.nsf/enrollal...?
As a counterpoint, I think the author should go work in a salt mine at some point. You know, so he can gain a deeper appreciation for table salt.