HN Explorer

Ask HN: Questions for two 40 year IBM Mainframe repair and programmers

114 points by elsherbini ↗ HN
Hi HN, I’m visiting my grandfather for the holidays. He worked for IBM starting in 1966 as a repairman in Alabama servicing everything from IBM 407/403 “grey iron” to 1401s, 360s, 700s and beyond. He got his training in the Marines, and become one of the first crop of IBM service workers that didn’t need an engineering degree. My grandfather would be on call and often have to drive hours in the middle of the night to service a computer that had issues when factories or banks depended on them to print checks or keep the business running. Biggest service customers in the area were a Goodyear plant, the Life Alabama insurance company, the local community college, a steel plant, and many banks.

His friend is coming to visit who was a programmer for these computers at a Goodyear plant that had 3000+ employees starting in the mid 1960s until around 2000. He started in the mail room and volunteered to become a programmer, back when IBM had training programs for companies to learn to use and program their their equipment. After the year 2000 he moved to a Life Insurance company as a programmer for another 10 years. During his career he programmed on punch cards, in assembly language, and later FORTRAN and COBOL.

There are all kinds of debugging in the field stories, such as when a magnetic drum memory at an airfield kept having issues, and finally they figured out when the radar from the tower pointed just right at the drum it’d flip some of the memory, or when fumes from the Goodyear plant were found to be eating through solder joints, making the mainframe at that plant the 2nd worst for IBM to maintain in North America (the worst was a tire plant in Canada, those fumes!).

What questions do you have for two ~80 year old computer professionals in small town Alabama who started work back when you could walk into the computers, and ended their career in the 2000s?

They are coming in about 4 hours from this post, and I’ll record the whole thing and do my best to answer your questions in replies on this thread!

42 comments

[ 3.4 ms ] story [ 98.7 ms ] thread
norwayjose
I worked as a field engineer on Honeywell mainframes in the late 1970s and later moved into programming so I'm always keen on hearing about the good old days when computers looked impressive. I'd be interested in hearing about their most challenging debugging problems. I'd also like to hear about how your grandfather serviced core memories back in the day. Honeywell core memories had spare bits built in which made is easy to bypass a failing sense amp.
pixelmonkey
My Dad worked at Honeywell on mainframes in the 1970s, too. In NY. In case the last name rings a bell: https://amontalenti.com/contact
norwayjose
I worked in the northern VA district. On weekends we also had to cover DC and parts of MD which was a wide area to cover. Once I was at a site in VA and got a call at the Naval Academy which was over 100 miles away. I wasn't able to make the obligatory 2 hour response time for that call.
elsherbini
I do not remember any transistors being inside the core house but sometimes the welds inside the house would break loose and if you could find the broken weld you could resolder it. IBM core units did not have spare positions. Sometime the entire core house would have to be replaced. - Grandfather
norwayjose
Thanks. That's interesting. I've only seen the outside of IBM mainframes in the few computer rooms which had both types of machines.
elsherbini
I think my most challenging diagnostic problem was a channel hanging the system. I used a 360 I/O channel test box that connected to the channel and when a error or hang condition would occur it had recorded the last several instructions and the channel responses to those instruction. IBM sent a factory engineer to the site and it was determined that a test I/O instruction to a 3271 was generating a device end status when it should not generate any new status. The fix was a EC to the wiring of the 3271. My most unusual repair was not in the maintenance manual. One morning about 2 am I had a bad 12v power supply in a disk controller unit. I asked the customer to bring me a 12volt car battery. I rewired the 2841 to the battery sitting in the floor beside it. The 2841 powered up and ran for about 15 minutes before it started getting disk read errors. I asked the customer to bring me a battery charger which I connected to the battery. The computer ran two days with the battery supplying the 12v until I could get a power supply. Sometimes experience and good luck are the troubleshooters best tools. In 1988 I was in China helping the Chinese with their computers and received a call from The Bank Of China as their system was down and they had closed the bank because of it. My assistant Chen Wei and I walked into the computer room and I looked at the console. The console showed a MPX channel error. I looked around the computer room and saw several sections of the raised floor were pulled up which indicated the Chinese service company had been moving boxes around. Due to past experience I knew the buss and tag cables could be switched and would cause a MPX channel error. The first box I checked had the cables switched and Chen Wei switched them correctly. Everyone in attendance thought I was a true computer expert but it was experience and good luck.
markus_zhang
I'd very much like the two gentlemen to sit down and write a memoir about their mainframe days. The older technologies are particularly fascinating. It doesn't have to be a book, but if it is then they can crowd fund.
mixmastamyk
Also might simply let them talk and record it. Someone could later make a transcript.
mdaniel
https://storycorps.org/about/ is the formalized version of that, as well as archiving the stories in the Library of Congress. They air every Friday morning on my local NPR station, and the stories they air are very, very diverse so I'm of the impression they'll take anything they can get in the vein of preserving first-hand accounts of "the way it was"
[deleted]
(comment deleted)
matt_s
Might be too late but would love to hear some stories of things they had to do because of space/speed constraints (storage, RAM, registers, CPU, etc.)

Edit to add: we are very, very spoiled in todays computing age as far as constraints goes

throwaway81523
Look at the exercises at the beginning of Knuth vol 3.
CoastalCoder
I don't have specific questions, but I'd love to hear whatever stories they're most interested in sharing.
[deleted]
(comment deleted)
zed716
I'm curious in regards to repairs (and perhaps this is just stupidity on my part): Were spare parts for repairs kept on-site, did they have to be ordered/shipped, or did your grandfather have a, er, service truck with parts, etc?
brk
My father (and I for a bit) worked in IBM field service. The answer is all of the above. You’d carry some common things with you, but most parts needed were picked up at local parts depots that IBM stocked.

For larger clients (like the K Mart stores) we had private pilots on call that could be sent to another city to get a ~$500 part. Especially during holiday shopping season. Some clients had onsite spares but that wasn’t very common. They relied on IBM service to just handle everything. For some customers we had a 2 hour on-site SLA.

norwayjose
For Honeywell mainframes, the maintenance contract required an onsite office for field engineers. In those offices were stored spare parts (mostly for peripheral devices), larger tools such as oscilloscopes, and the repair documentation which took up a lot of space. The CPUs for Honeywell mainframes contained about 80 large circuit boards (about 12" square as I recall) which were too expensive to have a full set of spares on each site. For those, we had several district offices with more extensive parts departments. Field engineers had to carry a small toolkit with them which in addition to traditional tools contained soldering irons and wire-wrap tools.
elsherbini
I was in a resident territory 60 miles from the home office and we stocked a good supply of parts but far from any part you would ever need. My branch office stocked a higher level of parts and would send them to me via taxi if needed. The regional parts center in Atlanta would air freight over night to the Birmingham airport and I would meet the airplane. I had a part removed from a production line machine and sent to me for a repair. Part support was very good. In the 1990s when I was working for TSS we did get vans so we could have some parts with us.
bjornlouser
Did either ever come close to leaving the field? If so, what made them want to quit tech?
elsherbini
I never considered leaving my IBM Field Engineering job. The job was rewarding and gave me a great sense of purpose; however it was very stressful and had a lot of high expectations. With the 360 line of computers the field was flooded with engineering changes (ECs) to be installed and IBM eventually hired part time people just to install ECs. The ECs eventually fixed the many problems with the 360s and the job demands settled down to more normal.
cschep
Curious if they think they would be interested in a career of programming starting today?
1920musicman
Would love to know whether security was a concern at all on those systems (eg industrial espionage, internal program meddling for the purpose of tricking auditors, etc). And if yes, then what are some interesting stories about investigating and resolving security issues.

Thanks for doing this!

elsherbini
Hacking was not a problem in the early days of computing as there was no internet and computers talked to each other over dedicated telephone lines. - Grandfather
norwayjose
In the late 1970s, we used dial up modems (the kind with the klunky acoustic coupler) to dial into Honeywell's response center to close out repair calls. I also discovered they had games on the response center computer and was able to entertain myself on quiet night shifts by playing Adventure and Star Trek on a TI Silent 700 terminal.

Some of the mainframes we servicced were also connected to ARPANET which provided networked access to university and various government agencies. I know there was an ARPANET worm around 1980. Not sure if there was hacking prior to that.

julianeon
What parts of the computing landscape today remind them the most of what it was like to be a mainframe programmer?
scrapcode
Were there any parallels between the systems they worked on and helped bring into this world and the modern era with "supercomputers" and AI, as far as people being worried about the technology replacing their jobs?

Did they have much of a network or reference to lean on when novel problems would come up, or was it more of a "figure out a way to make it work" way of troubleshooting?

elsherbini
I do not remember people being concerned about computers replacing them because in the 60s and 70s computers were creating a lot more jobs than they were replacing and automation had not taken over at that time. IBM had a very good support network all the way to the plant if necessary.
KyleSanderson
Stance on the golf club memberships winding down? Alcohol at socials and blue Jean Fridays would be interesting.
me_here_alone
As a recently retired IBMer, I would love to know what the think of IBM today vs when they worked there.
elsherbini
I do not know about IBM today but during my time working for the company they were the greatest company to work for if business was good but in bad times for the company their number one priority was the bottom line.
shrubble
Did he ever work on the System/3 or System/32 devices? (I am looking for a System/32 but there are very few around!) Also what was his favorite machine and why?
elsherbini
I did not work on the computers you referred to and my favorite computer was the 360 model 40 (2040) because it was very reliable and relatively easy to work on because by then I understood the microcode much better. grandfather
fuzzer371
What was it like "volunteering to become a programmer" back in those days? Did you know what you were getting into, or did it just sound better than working in the mail room? Was "Programmer" a well respected job at the time? Were you seen as a sort of magician working on the mainframe? Or the equivalent of a "grease monkey" changing oil in a car?
helf
Oh wow. The Goodyear Plant must be the one in Gadsden AL which is my home town.

I would love to hear what they think of the advances in technology from when they started till now; on the software side: which machine series they liked dealing with the /least/ and wich the most and why. What operating systems they liked the most.

Any fun anecdotes about user issues, idiotic design quirks they dealt with etc.

Also, if they have any old ephemeral like notebooks, manuals, etc? That sort of thing would be a treasure to scan in and send to archive.org and bitsavers!

helf
Oh! I would like to know what sort of reliability and headaches they had out of different types of memory! Drum, core, film memory etc.
elsherbini
Every form of memory storage I was exposed to worked well except for the delay line storage unit in the controller for 2260 CRTs. IT WAS VERY TEMPERATURE SENSITIVE and required a lot of maintenance. THANKFULLY it was not around very long.
skissane
Do they still have any documentation (service manuals, schematics, etc)? Might they be willing to donate any of them to Bitsavers for scanning?
shermantanktop
The current set of developers are often tempted to start again - to start a new software codebase which will surely fix all mistakes of the old. Was this a temptation in the days of big iron?
elsherbini
I was a hardware tech and we installed updates or patches to the computer micro code many times without replacing the entire code. I think most programmers preferred to fix the problems rather than start over. grandfather
graymatters
Given what they’ve seen almost 60 years ago and today - can they extrapolate what will the computing field look like in another 60 years? What tools/techniques/methods from the past do they miss the most? Which of these do they think could/should be adapted to current realities and used to greater effect? What are their top 3 do’s and don’ts from those times?
elsherbini
I have no idea where computing will be in 60 years. The tool I miss most is the oscilloscope. It was great fun and satisfaction to set up the scope and computer and be able to see the display of the electronic signals while you looked for a missing or erroneous signal. You had outsmarted the computer. The top do in my days was if you did not have a good handle on a problem in two hours call for help. grandfather
dpflan
When will there be replies? I see OP’s latest comment is from 4 months ago…