I don't think this is a case of knowing Cobol. I think it's a case of company's hiring the experts who were let go.
Person A has X years experience with company Y. Person is let go. Mission critical infrastructure starts to fail. Person A is now a contractor with X years experience on that exact system.
Sure they can hire a new guy for cheaper, however that appears to be what led up to A being let go.
They suddenly need experience and A is willing at a much higher rate.
It's cheap to portray COBOL programmers as cowboys, but COBOL and the ecosystem around it is actually one of the most organized and fit-for-purpose programming languages around even if it isn't modern by any means. Time will tell if banks and insurances will fare better with J2EE-based systems (meaning classic Java-based enterprise stacks such as EJBs and portlets running under shudder Websfear, which count as new-fangled compared to COBOL in these circles). I could imagine classic J2EE maintenance is being neglected as career path for millenials, just as COBOL is/was for the generation before. J2EE exposes much more escape hatches into system programming and rope to hang yourself, and at the same time will be around for a long time to come, since Java has been the go-to language for almost all big-time eCommerce and FinTech projects at banks/insurances since 2000.
Edit: a-ok, "cowboys" is part of name of that guy's consulting business
I don't know for COBOL, but for JEE, we've been able to migrate out of weblogic (to jboss) and then from oracle to postgresql (on code which is 10 years old). So even if JEE might be hard to maintain, it allows some flexibility in the base tools you choose. (my view of COBOL is that once you're into a vendor, it's pretty hard to leave to another one because COBOL is close to the metal, I may be totally wrong)
but in the long run, the problem is also the business logic. What disappears is the knowledge of the hacks that were applied at the business logic level. And this is much harder than to recover from the code. So I guess one has to make the difference between business logic maintenance and the actual COBOL maintenance.
But having rewrote some huge COBOL app in Java, I can say that Java is better structured, at a framework complexity cost.
Better structured, easier to debug, and the tooling exists to incrementally refactor in a relatively safe way. There are also migration paths to better worlds. IntelliJ can auto-rewrite Java into Kotlin, preserving comments and formatting along the way, and even detecting common Java idioms that have better Kotlin equivalents and rewriting them too. So, not only can you migrate to better app containers, database engines, upgrade your garbage collector and compilers etc, but also even switch to a much more modern language nearly pain free.
I've been using Intellij's Java to Kotlin conversion a bit lately, and the code it spits out is fairly decent Kotlin, and with a pass through on nullability and switching out Java types with native Kotlin types afterwards. It gets just as good as it would get if I hand translated my self, but it takes a lot less time.
I suspect that part of the reason that this works well, is that Java and Kotlin isn't that dissimilar. Doing automated conversion between Java and Scheme for example, would be a lot harder to get right.
Hard-coded business logic: yuck. IIRC there's proprietary software (SDKs / integrations) specifically for specifying, running, testing and enforcing business rules efficiently so that applications just present an API or respond to events rather than having to know everything about the business, and so not having to change N apps when a business rule changes / track down weird bugs when a business rule changes because another system didn't get a change also.
Some people prefer to "hardcode" business logic in Java instead of hardcoding it in XML or whatever business logic specification language the proprietary tools use. At least you can hire Java programmers fairly cheaply.
I would pick hardcoding for most projects. You often end up creating a new DSL for your business logic, but because most of us aren’t language designers our DSLs end up being worse than what ever language the project is written in. Almost every time you will fail to correctly anticipate the future needs and your DSL won’t be sufficient anyway.
That quite deepnds on the business. Some of our data scenario are super complex (thousands of datapoints, where one influence the others depending on how they happen in time, depending on manual intervention, things that happen here and there). Because of that, the usual tools to write your business logic fall short on the testing part. They're good at writing/maintaining simple rules, versioning them, but when a time dimension is crucial, then well, it's not that good anymore.
Yes, they're definitely from Texas: not the usual "cowboy coders" == "coders whom take too much initiative / don't follow a plan / directions."
J2EE usually means GC pauses (hell), XML (hell again) and dependency injection (hell also). Thankfully, hells tend to pay well if someone is willing to schlep the muck around quickly and reliably.
JVM (IBM's, Oracle's) and CLR runtimes are battle-tested and much more widely deployed to run civilization than most non-enterprise developers realize.
Also, I should probably look that guy up as I'm in North Texas at the moment.
I just had to go through an open source project that uses guice, in order to try to trace a particular code path, since the user group was less than helpful.
Trying to figure what was running was an absolute nightmare of nested magic spread out through a score of files. Meaning as soon as I had a clue, I'd lose it as I had to retrace back through all those files.
I'm sure it's clear as crystal to the people who wrote it, but DI can be obtuse as hell if you have to figure what's actually running on your own.
hand rolling DI, yep brilliant, generally quite easy to work your way through because it's easy to see where classes are being used.
Hand rolling can get complicated when your applications get BIG and you want to do fancy things like having modular code (whether you ever use the module somewhere else or not) or lazy loading or multiple scopes.
Then you use a framework and you find your dependencies take dependencies, your wireup code is auto-magical, you inject dependencies into private properties, you want multiple implementations of particular interfaces but want to ensure only the right one gets used in each circumstance.
Then multiple people start getting their fingers into the pie and the DI/IoC explodes into config files throughout the application, and the only way to understand what's being wired up without a map, is to "find usages", pin files, and cry
(I'm not bitter or anything ;) DI/IoC can work wonders for handling complexity, it can also increase your complexity if you aren't very careful)
That last sentence is the important part: "if you aren't very careful". I work in multiple large java/spring codebases and the rule we use is "Keep all configuration related to the same base type/interface in the same file". That way if you want to find out which logger is injected you only have to check 1 file.
Another easy way to find out what is injected at runtime is to attach a debugger and set a breakpoint.
:( with the particular project I've been looking through, trying to debug via running the entry point was a non-starter. The part of the system I needed to trace would only be able to run if I had 4 other services running (all started seperately from the same main file) and a full dataset loaded.
Once I finally discovered a hint about which files governed the behaviour I was investigating I was finally able to isolate the tests to begin the debug cycle.
Where @GetMeALogger does some magic at compile time to pipe in some kind of logger class based on various config options. Then to unit test it, you use different config than production.
Mainly because once you get more than a few arguments, it gets tedious to start newing them all up and all of their dependencies and on down the stack. So some people built frameworks to do it for you but each of these frameworks requires their own configuration/documentation and suddenly instead of 50 lines of news, you have annotations in twenty different files that effect the way your code runs without actually being code. Each step to get from A to Z seems perfectly logical but the end result is a difficult place to start.
It's only a problem if you let it become a problem. Just like with code style, naming and formatting, the DI configuration should be enforced during code review.
Spaghetti code would not pass code review, so why would you allow spaghetti configuration.
In the code bases I work in I know if I need to know what Logger is injected I need to look at the LoggerConfiguration class to see which Logger implementations are used in production, staging and dev.
And even if I don't know the name of the LoggerConfiguration file, I have an IDE that can get me from injected property to configuration in one click.
But, your experienced in your teams code. Now, let's say you and your team get fired except maybe some person on another continent they hired to keep it up. OK, 20 years from now. Your reincarnated and you just got hired to dig through the code and you've hard of java, but nobody uses that relic anymore.. Good luck.
Declaring "I need an ILogger" is the easy part. The complicated part is the configuration related to which ILogger implementation will be injected at runtime.
This can depend on several things like configuration properties, environment variables, libraries loaded, feature flags enabled, etc
edit: also, you don't always want to manually pass the argument to every function. At application startup you typically want to build a full graph of objects with all the dependencies "wired up". The big complicated frameworks (Spring for example) read in a bunch of configuration, resolve all dependencies, and then construct the graph for you.
They were fired by banks which are/were stupid enough to cut corners on people who know things about their infrastructure before migrating to more modern systems.
FTA:
>One COBOL programmer, now in his 60s, said his bank laid him off in mid-2012 as it turned to younger, less expensive employees trained in new languages.
>In 2014, the programmer [...] was brought in as a contractor to the same bank to fix issues management had not anticipated.
Also FTA:
>Accenture’s Starrs said they go through a “black book” of programmer contacts, especially those laid off during or after the 2008 financial crisis.
>The job ultimately took five years and cost more than 1 billion Australian dollars ($749.9 million).
So, in short: a bank fired some old people to pay a billion dollars to a contractor who would hire the same (!) people to do the job.
Self made ... by both the banks and the programmers.
As the article notes, these developers don't seem to have done a great job of documentation. Perhaps deliberately, given the comment about personal vindication. "You can't survive without me" is not a great look for either employee or employer.
Perhaps they are cowboy developers in more ways than one.
Exactly. I'm always surprised when people think that it's easy to find time for documentation and that you only skipped that step because you are lazy and/or unprofessional.
Honestly, I know that's the conventional wisdom, but I've never once heard an engineer go to their manager and say, "I'd like to spend some time documenting this important process you asked me to implement" and being told "no, we don't have time to document! just keep it all in your head!"
I'm sure this literally never happens.
What I do see quite frequently is developers who don't like writing documentation finding ways to not do it or justify doing a poor job.
If you want to blame someone, you can always say that he didn't document his stuff enough.
The problem is, that there is no standard, what's "enough", so it is easy to move the goal posts. Combine that with fast paced environment and putting out fires constantly, and you have a recipe for disaster.
>If you want to blame someone, you can always say that he didn't document his stuff enough.
This, he could have documented it perfectly well in a time when everyone knew COBOL.
But that documentation would all be useless to me today with my knowledge of Java and Kotlin, but understandable to someone else from is generation that works with COBOL.
Tbf, it only really makes sense for the individual programmer to properly document code if there’s a decent chance they’ll forget (eg some obscure part of the codebase), or if you expect a reasonable chance that someone will have to read the code without you. Otherwise, its useful, but not necessary: you just go to the guru (perhaps yourself); his living body is the documentation.
If you’re thinking that only you’ll ever read it, or you’ll be working on this codebase for the next 30 years, it doesn’t really matter. Peoper Documentation is a cultural activity required for teams, especially those with shifting workers, as is common now. And should really be enforced by the managers, not the programmers thenselves.
Its significantly less necessary when you can expect to spend most of your career on a single codebase, a workstyle these cobol programmers presumably came from.
I try to design and name interfaces clearly and comment code for myself. If it's been a few weeks since I've looked at something it helps me snap back.
Also, I would never approach a codebase as if I will be working on it for 30 years. Job security comes from marketability, not obscurity.
Was that true when these systems were first made? Many of them are decades old, and the culture of large institutions is usually resistant to change, and that comes from management on down. If the system didn't start with those values in mind, what would have prompted them to change?
It's not that I don't expect the codebase to age or design it with long-term thinking, I just don't expect to be working on it for that long. Someone will replace me, and I'd like to leave them with well-designed, readable code that they will be able to understand.
I'm talking about that as well. The modern era of jumping ship every 2-3 years on average in software is not how its always been. Jobs used to be very long term
It's not always about jumping ship either. In many companies, you will move between areas/needs or get promoted to other work after a couple years. Also, companies will outsource maintenance to lower-cost teams. It seems there is better project lifecycle management, less waterfall development, and fewer cushy open-ended government contracts.
But, you're right. The expectation that you'll be able to do the same thing indefinitely for you career is gone.
This is a terrible way to maintain a system. You never know when the guru is going to die or quit; you have to have a way of getting other people to maintain the system. This is why humans invented written language: to transmit information to each other and to subsequent generations.
However, writing useful documentation isn't trivial, and is best done with a technical writer, and takes time away from development. If management isn't willing to invest the time and resources necessary to produce useful documentation, then they're going to have problems in the future as the gurus leave.
You, obviously, have no idea how hard it is to write decent documentation for any system. Especially when it has been in existence for many years.
Good documentation is hard and anyone who thinks it is not does not have a clue as to what is needed for the purposes of maintaining any piece of software.
It is a very rare event to see documentation of a quality that explains all the interrelationships, reasons for the choice, etc, etc, etc.
I have in the past been required to provide all the documentation for projects I have worked on, functional and technical specifications, user guides and cheat sheets, detailed design documents and all of these take considerable time to write well. I can't say I am expert in this area because I am not. But I have done a lot better than most of the projects that I have had to interface to over the decades.
Documentation is a low priority for many management teams, they do not see the use in the kind of details that are needed. They do not give it sufficient support to do the job well.
I have seen and worked with people who have the skill to write the kind of documentation that is actually needed and they are a rare breed, well worth any money you pay them.
When companies complain about "cowboys", they often do not see that it is they who are the "cowboys".
You, obviously, have no idea how hard it is to write decent documentation for any system. Especially when it has been in existence for many years.
I've been programming for decades and my current project is frequently praised by other developers who work with it (it's a framework) for the quality of its documentation.
So, in fact, I do have such an idea. It's hard work. We do it anyway because we're professionals. The fact that people frequently don't do it doesn't change this.
One technique I use is to keep documentation in the same repository as the code, and force people to write docs via the code review process. That is, a proposed change doesn't pass review until it also updates the documentation files. People want to get their code merged, so they know they'll have to take docs seriously or else it'll get picked up in review.
Now maybe back in the mainframe days there was no culture of code review, but these are banks we're talking about. They excel at process. So it seems kind of unlikely they'd not want things documented, if management had understood what needed to be documented in the first place. But ultimately it was only the employees who knew that.
Your current project is praised for its documentation, that's actually good. However, if you are taken out by a bus today in some permanent fashion, how easily or quickly can someone come up to speed in supporting your project and not have to try an puzzle out why you did things the way you did? What assumptions are there that you have not documented? What relationship to other pieces of code or system functionality is there that you have not covered? What changes cannot be done because of the "brittleness" of interrelationships between different sections of the code? What are the expected business rules that are applicable to the software? What constraints do they put on others who want to use the software? What are the meanings of all the magic numbers that are littered throughout your code? Keep in mind all those other questions that they will have along the way.
It doesn't matter how "professional" you might be, you will not have covered all your bases. All you can do is spend lots of time in building up this documentation and if it is not being paid for in some way or time constraints get in the way, then none of it will get done properly.
It is good to see that you have some sort of process in place to manage the problem, but code reviews are limited in what they will capture.
I am documenting a piece of my own code at the moment for the purposes of submitting it to a specific project. It provides an additional functionality relating to something that the main system doesn't (at this point) handle. It is intended to be a simple workaround until the main system can supply this functionality at the VM and compiler level.
I have just looked at one method that simply converts a hexadecimal number to a potentially multi-byte sequence. The detailed reasoning for the code (the assumptions, choices of numeric values used, etc) amount to 10 times the number of lines that is actual code and I still don't think it is complete. I think that it will take at least two, three or maybe even four more revisions in the documentation to get it to the place where someone else could make changes without causing problems elsewhere. The code itself is really simple. But, there is an additional purpose to this documentation, in that, it will provide a baseline against which the final system changes can be compared.
I do this because, as a somewhat retired "professional" programmer, I want to see some serious documentation for this system and can afford to put in the time to do so.
So back to the main discussions, unless full and complete documentation is a requirement put in place and strongly enforced, what documentation there is will not be sufficient for the future. It is a rare professional who will instigate this and make sure it happens. From my experience, such professionals that do this, have not come from a programming background but have come from other industries (usually engineering based) where such documentation is considered normal or is mandatory.
I have seen few management teams that have given more than lip service to creation of the required documentation and the time frames they put on projects are such that there is little or no time for the production of quality documentation that is needed. Nor are they interested in paying for such.
Banks have a focus of making money. They are not interested in spending it. The only process orientation they have is increasing the bottom line in whatever way they can. If it means they cut back on programmer and computing costs, they will do this. Ultimately, it is the bank management that holds responsibility to understand that the documentation is required. If they did not understand this, they should not have been in the position of management. It is irrelevant if only the employees (those grunts at the coal-face) knew this.
For enough money, I'd say many. COBOL isn't sexy, but it's not difficult either. And it's demonstrated a lot more staying power than most other languages.
Darn right! I'd cheerfully learn Cobol for a job with the right parameters. I once took an RPG course in order to apply for one (though sadly did not get it).
The only way to have really quality documentation is to have a lot of people reading it and providing feedback, over time.
It also doesn't hurt when a quality technical writer actually writes or edits the documentation in consultation with the people who designed or implemented the system.
The bank and "IT Cowboy's" technology teams have probably spent years on "Ayne Rand's Train." Projects headed for failure. Everyone is concerned with avoiding/pinning blame instead of building software that works. Deep knowledge of contracts instead of deep knowledge of code.
These are common pathologies. I agree though. This is incompetence at technology from strategy to the details.
It's important to understand how much technology gets built this way. Usually these projects are not outright, public failures.
The problem is that these decisions are rational - you get personally rewarded for getting yourself promoted or putting neat projects on your resume (which will increase your salary in your next job), but not for helping the business long-term. Avoiding blame is far more personally valuable, in terms of reward for time, than building good things.
If you take "I swear by my life and my love of it that I will never live for the sake of another man, nor ask another man to live for mine" seriously, this is the end result. Why should any IC or low-level manager at Taggart Transcontinental live for the sake of the Taggarts? But how can the railroad succeed without ICs and low-level managers?
If you want to fix this, don't ask people to live altruistically; instead, fix their incentives. Giving people cool projects such that they feel personally fulfilled when they succeed whether or not the business recognizes it is one option, but not particularly reliable or scalable.
Im not buying her interpretation of why the train scene happens either, but I do think the scene was kind of a brilliant metaphor for this sort of thing.
Whenever you have a systemically pathological situation like big, enterprise software projects often get into... it's probably an incentive problem.
And, this shit has been going on for 20 years, my uncle (now in his 80's) made bank Cobol contracting and subcontracting EXACTLY the same way in the run up to Y2K: hiring developers "early retired".
> a bank fired some old people to pay a billion dollars to a contractor who would hire the same (!) people to do the job
"a bank" is technically incorrect. The decision was a perfectly rational prisoner's dilemma defection from one executive (cut costs by firing the "expensive employees" and hiring "cheaper and newer"), who went on to be promoted on that basis, and it was the next generation of executives who had to deal with the mess and "hired IBM" so they could absolve responsibility of a situation with no visible upside if successful and tons of visible downside if not.
The ironic thing about your reply is that it is crafted in exactly the type of management speak that executives used to convince stakeholders in both occasions.
"absolve relevant parties of their responsibilities" - what relevant parties? What responsibilities? It sounds great and responsible, giving a good impression of dismissal of the point made, but the wording is so general as to be impossible to act upon - just as with "a bank". I'm sure standardised checklists were ticked, committees reviewed the decisions and came to the conclusion that it was best, the board was satisfied with the new direction taken by the CTO's organisation, and the shareholders happy with the cost reduction at the time as described by the board. To paraphrase Thatcher, what colour trousers do the relevant parties wear?
You see this defection pattern everywhere. For example, the never ending stream of useless new features in most Google and some Microsoft products which coincidentally get slower and buggier every year, because you can get promoted for the visible addition of a feature as a PM, but it is much harder to prove that getting the Gmail app load time down by 70% has stopped the feeding of your best power users to the competition (Calendar is too easy a target for criticism).
The classic defector on the tech side is the "ninja rockstar developer" whose prolific output of code and visually attractive but functionally superficial features is matched only by the amount of frustration generated after they are promoted somewhere else or changes jobs if it doesn't happen fast enough, and the rest of the team has to deal with the technical debt and new, unrealistic user expectations.
Security is another good candidate for shortcuts, since there is no visible upside to good security (maybe you were just not attacked) and tons of downside for taking responsibility since you make yourself a visible scapegoat for blame when a crisis does happen.
And note that none of these examples are in "move fast and break things" i.e. "do illegal things and hope they become legal ex-post" territory! I personally take great pains to understand these dynamics so that I can avoid them in my own company, which is a matter of survival.
Except that's not what happened in the specific example given - it was a strategic decision to spend a billion dollars and multiple years to replace an old system that processed transactions in daily batch jobs with a modern system that can do transactions effectively instantly - which is exactly what the parent post was arguing should occur
> cut costs by firing the "expensive employees" and hiring "cheaper and newer"
You should do everything else to cut costs before you start getting rid of your people. Those people understand your business and you're going to need those people when things turn around (and they usually do). Any time you let people go, you're throwing away domain knowledge. You've PAID for that. It is an asset. You'd no more throw away all the computers in the building, why would you throw away your employees!?
For the large companies, it was probably due to pensions. When you take into account ever increasing lifespans and the number of people nearing eligibility for pensions at that time, it was a no brainer for finance and management. A lot of people didn't get fired. Instead they were transferred to one of the big consulting firms like Accenture and they ended up working for the same pay but much less benefits as contractor to their former employer. Some of those affected were smart enough to sue their former employers but I would guess that most did not.
No one who's in a position to decide about COBOL's replacement has any strong incentives to do so. You don't want to be the one exec who made that risky move and crushed half their bank's tech foundation in the process.
Banks were among the first to computerize their systems, and switching systems cost a lot of money, regardless of industry. Hence they have a lot of old systems.
It's not unreplaceable, but it costs money to replace, and it's a risk to replace it; the old system is tried and tested, whereas the replacement might not work quite the same way.
So at any specific time, the safer and cheaper option is to stick with the old system, despite the growing risks and costs in the long run.
Replace COBOL with Visual Basic, Java, Javascript and you have your answer. It's what was used when those finance applications were written and became part of the technical debt of the institutions that wrote them. Your shiny new language/framework today may one day be fondly remember by future generations as 'why are we still using this crap?'
Imagine that you need to replace a live system, downtimes are not allowed, probably provide a new UI while offering 100% of existing features.
There is no documentation available, or even if there is one, you won't find anyone in the building that will put their hand on the fire for how much of it actually maps to the running application.
So a project of reverse engineering existing application into a requirements document or updating the existing documentation usually takes several months.
Now just for a very basic project cost, consider the monthly salary of everyone involved, multipled by the amount of months.
And we are just speaking of phase I, then there is the actual re-write, followed by verification that 100% of the old features are still present on the new system.
Naturally it is cheaper to keep existing systems running.
This applies to any old platform, not only COBOL systems.
Except for the detail that COBOL programmers are outrageously hard to find, thusly expensive, and are only continuing to get harder to find and more expensive.
Because financial sector is run be people who consider engineers a liability[1]. Instead of evolving with time in 80s and 90s as the world moved to modern systems, they clung to their increasingly legacy systems. After all, why would an MBA toting executive from a big bank bother with something as menial as technology? At financial sector firms, Engineers are not meant to be seen or heard. Try proposing a new idea — or a move to a newer system — you’ll be shot down faster than you can say technology — by the same people who have led us from one financial crisis to another in their infinite wisdom.
Most big banks have a bunch of security people whose sole job is to make people’s life hell by spreading as much FUD as possible[2] — that’s what keeps their jobs secure, after all. And they are the only “engineers” who are heard. They make sure that anything new is not adopted till the rest of the world has already started abandoning it.
The primary focus in the technology departments of big banks is to maintain the status quo. New development is rare. New ideas are crushed before they can see the light of the day. As a result, good engineers leave as son as they can and the only ones left behind are the ones who are content living a life of mediocrity — maintaining the status quo.
[1] I know someone who was told something along those lines when they joined a big financial institution as an engineering manager. And not all financial companies work that way. Those who embraced technology reaped the benefits and continue to do so.
[2] I am not suggesting that security is not important. But security people at these institutions take it to a whole different level — to the point where things that should not take more thane a few days end up taking months — if they allow it, that is. Funny how all their objections vanish when someone from upper management asks for the same thing.
One of my jobs was for a subsidiary of a European plane maker. Some of their critical systems were written in Fortran 77 and was decades old in some cases. I have no clue about F77 or aircraft engineering, but my team was told repeatedly that these software ran flawlessly and nobody was ever to touch it (we made some fancy JAVA UI and called F77 executables in the background). I guess a time will come to sunset, but just not yet.
As above, may be these COBOL systems have been battle tested for a long time and they are good. Other reasons could be, "if it ain't broke then why fix it?" or that there aren't enough people to act as a bridge between the old and new systems so it is hard to do such projects (as other comments have pointed out)
I wonder whether it was the same VP who got promotion for decision to cut corners a couple of years back, and another promotion for decision to fix it by contractors.
I'm guessing it was an Accenture project based on the outtakes.
That's what Accenture does: armies of cheap people for huge contracts. Flip a coin on the outcome. Not that IBM or any of the other top-level consultancies/services behemoths are any good, but Andersen Consulting/Accenture basically invented that approach.
While most of your points are valid, you've misunderstood the last quote. CBA didn't fire people and then need to pay a billion dollars to fix something. They specifcally did migrate to a modern system - they completely replaced their entire legacy core banking system with a brand new one.
It isn't just banks either. I'm a greybeard sysadmin type who has seen the insides of hundreds of companies, and for some reason in the late 2000s it seemed like everyone from law firms to insurance companies decided to fire their IT teams and then pay 3x the money for 1/16 the service from contractor/msp-types... and they wonder why they wallow in tech debt... I'd they even knew what it is, they did fire all the people telling them about it after all!
MSP and Cloud are separate concepts. The typical MSP has a bunch of technicians who drive around to client sites and tend to the servers in their closets.
I guess because good programmers are not good in business.
If a bank or any big corp is in trouble with legacy software and they approach you to fix it, charge them 500/h or more. That is still cheaper than the time they need to find someone else.
No, but Cobol + bank experience is different. There's the whole business terminology and overview you need to have to actually understand and fix the code.
I use this news as a ignition for another point: in the past for business reasons we have evolved concept of "platforms" witch means independent software layers build like a trailer to carry on other software on top.
Before we have another concept, those from LispM, the "system" as a single entity of well_integrated stuff.
We have many example of those two way of thinking today: on "platform" side we have snap, flatpack, appimage, lx[cd]/docker, ... on system side we have Emacs, NixOS, GuixSD, ...
Well, for years the "platform" model seems to be the most reasonable, today seems ancient MIT&c hackers ware right, "system" approach is better. Simple to manage, for good software, do not hide bad practice, force collaboration etc.
On datacenters today and not from today we do substantially the same, in the past datacenters was a big collection of independent computers, now their are substantially all "a single computer" (even before The datacenter as a Computer by Google), made of many well_integrated components.
My poor English may not help, but I hope I have been clear up there, if so, what you think?
Using "cowboys" as a pejorative term ignores the fact that cowboys are fine in their natural environment: The wild west, where self-sufficiency, resilience, adaptability and creativity are more important than engineering processes, documentation, and bureaucracy.
A cowboy is a loose cannon in the big city, but a "proper software engineer" is useless in the wild west. These banks have just realised that there are a few ranches left.
It's talking about a company called Cobol Cowboys, itself inspired by Space Cowboys, about a bunch of old guys called back into service to help with an emergency.
Charmingly enough, their motto is "not our first rodeo."
I started in COBOL back in the 90s. When I was working in that shop, one of my older co-workers told me about a guy he knew. That guy left his service/repair gig at IBM to start his own shop, he knew how to fix punch-card machines.
For the decade I worked there, my buddy told me that his pal made a very good living repairing those machines, even though they were long out of fashion and 'there was no market'.
I worked in an IBM shop in the early 1990s, at the time they had a big chart on the wall titled "Punch Card Elimination Project." It was one of those big thermometer charts like United Way drives use to document how close they are to their contribution goal. As I recall it was at about 70%.
Fashion is such a driver for IT study. Just try to find a good book on modern AIX -- they aren't really out there. But AIX is still very much in use at the enterprise level.
Computer trends and actual use don't always match up. Which can be a boon for those who realize they can specialize in unfashionable technology.
This is so stupid. The asshole companies had every opportunity to fix and modernize their crap they just chose to ride the investments beyond any reasonable timeframe. Let them go out of business.
One little side comment: within one of the photos shown as part of this article, they show a photo of Grace Hopper, but no mention nor credit of COBOL's roots to her, nor her contributions to COBOL. So, the past was only made up of cowboys, but not cowgirls, eh? Tsk tsk.
> was brought in as a contractor to the same bank to fix issues management had not anticipated
And continue to ignore. This is not a new issue, but the same story repeating over and over. Management purely via cost control is a terrible approach.
> The risk is “not so much that an individual may have retired,” Andrew Starrs, group technology officer at consulting firm Accenture PLC, said. “He may have expired, so there is no option to get him or her to come back.”
Is "expired" a standard English synonym for pining for the fjords?
> But COBOL veterans say it takes more than just knowing the language itself. COBOL-based systems vary widely and original programmers rarely wrote handbooks, making trouble-shooting difficult for others.
There never seems to be time for documentation, specifications, and clean code
This article really hit home for me because this is exactly what my current career consists of. I am coverting old AS400s running RPG to QNX servers running C. And like the Cobol Cowboy's say the financial institutions will pay what ever you demand. I owe all this work to a college professor that encouraged me to thoroughly learn RPG for this very reason. Thank you Professor Her.
129 comments
[ 3.8 ms ] story [ 146 ms ] threadEdit: a-ok, "cowboys" is part of name of that guy's consulting business
but in the long run, the problem is also the business logic. What disappears is the knowledge of the hacks that were applied at the business logic level. And this is much harder than to recover from the code. So I guess one has to make the difference between business logic maintenance and the actual COBOL maintenance.
But having rewrote some huge COBOL app in Java, I can say that Java is better structured, at a framework complexity cost.
I suspect that part of the reason that this works well, is that Java and Kotlin isn't that dissimilar. Doing automated conversion between Java and Scheme for example, would be a lot harder to get right.
J2EE usually means GC pauses (hell), XML (hell again) and dependency injection (hell also). Thankfully, hells tend to pay well if someone is willing to schlep the muck around quickly and reliably.
JVM (IBM's, Oracle's) and CLR runtimes are battle-tested and much more widely deployed to run civilization than most non-enterprise developers realize.
Also, I should probably look that guy up as I'm in North Texas at the moment.
I just had to go through an open source project that uses guice, in order to try to trace a particular code path, since the user group was less than helpful.
Trying to figure what was running was an absolute nightmare of nested magic spread out through a score of files. Meaning as soon as I had a clue, I'd lose it as I had to retrace back through all those files.
I'm sure it's clear as crystal to the people who wrote it, but DI can be obtuse as hell if you have to figure what's actually running on your own.
e.g. instead of
you'd write Now it's much easier to unit test `foo` with a mock logger object.Is there more to it than that? Why are there big, complicated frameworks for it?
hand rolling DI, yep brilliant, generally quite easy to work your way through because it's easy to see where classes are being used.
Hand rolling can get complicated when your applications get BIG and you want to do fancy things like having modular code (whether you ever use the module somewhere else or not) or lazy loading or multiple scopes.
Then you use a framework and you find your dependencies take dependencies, your wireup code is auto-magical, you inject dependencies into private properties, you want multiple implementations of particular interfaces but want to ensure only the right one gets used in each circumstance.
Then multiple people start getting their fingers into the pie and the DI/IoC explodes into config files throughout the application, and the only way to understand what's being wired up without a map, is to "find usages", pin files, and cry
(I'm not bitter or anything ;) DI/IoC can work wonders for handling complexity, it can also increase your complexity if you aren't very careful)
Another easy way to find out what is injected at runtime is to attach a debugger and set a breakpoint.
Once I finally discovered a hint about which files governed the behaviour I was investigating I was finally able to isolate the tests to begin the debug cycle.
Sometimes it's just complicated.
Spaghetti code would not pass code review, so why would you allow spaghetti configuration.
In the code bases I work in I know if I need to know what Logger is injected I need to look at the LoggerConfiguration class to see which Logger implementations are used in production, staging and dev.
And even if I don't know the name of the LoggerConfiguration file, I have an IDE that can get me from injected property to configuration in one click.
That would be a fun text adventure....
This can depend on several things like configuration properties, environment variables, libraries loaded, feature flags enabled, etc
edit: also, you don't always want to manually pass the argument to every function. At application startup you typically want to build a full graph of objects with all the dependencies "wired up". The big complicated frameworks (Spring for example) read in a bunch of configuration, resolve all dependencies, and then construct the graph for you.
This - in general - is the problem with programming.
They were fired by banks which are/were stupid enough to cut corners on people who know things about their infrastructure before migrating to more modern systems.
FTA:
>One COBOL programmer, now in his 60s, said his bank laid him off in mid-2012 as it turned to younger, less expensive employees trained in new languages.
>In 2014, the programmer [...] was brought in as a contractor to the same bank to fix issues management had not anticipated.
Also FTA:
>Accenture’s Starrs said they go through a “black book” of programmer contacts, especially those laid off during or after the 2008 financial crisis.
>The job ultimately took five years and cost more than 1 billion Australian dollars ($749.9 million).
So, in short: a bank fired some old people to pay a billion dollars to a contractor who would hire the same (!) people to do the job.
The 'problem' is entirely self-made.
As the article notes, these developers don't seem to have done a great job of documentation. Perhaps deliberately, given the comment about personal vindication. "You can't survive without me" is not a great look for either employee or employer.
Perhaps they are cowboy developers in more ways than one.
Wait, no, I've literally NEVER heard of management asking developers to delay the next release to document their work.
Oh, but they can't increase the estimate. The customer signed off on that estimate, so we're committed to deliver by that deadline.
I'm sure this literally never happens.
What I do see quite frequently is developers who don't like writing documentation finding ways to not do it or justify doing a poor job.
Fortunately I had already resigned at that point.
The problem is, that there is no standard, what's "enough", so it is easy to move the goal posts. Combine that with fast paced environment and putting out fires constantly, and you have a recipe for disaster.
This, he could have documented it perfectly well in a time when everyone knew COBOL.
But that documentation would all be useless to me today with my knowledge of Java and Kotlin, but understandable to someone else from is generation that works with COBOL.
If you’re thinking that only you’ll ever read it, or you’ll be working on this codebase for the next 30 years, it doesn’t really matter. Peoper Documentation is a cultural activity required for teams, especially those with shifting workers, as is common now. And should really be enforced by the managers, not the programmers thenselves.
Its significantly less necessary when you can expect to spend most of your career on a single codebase, a workstyle these cobol programmers presumably came from.
However, writing useful documentation isn't trivial, and is best done with a technical writer, and takes time away from development. If management isn't willing to invest the time and resources necessary to produce useful documentation, then they're going to have problems in the future as the gurus leave.
Good documentation is hard and anyone who thinks it is not does not have a clue as to what is needed for the purposes of maintaining any piece of software.
It is a very rare event to see documentation of a quality that explains all the interrelationships, reasons for the choice, etc, etc, etc.
I have in the past been required to provide all the documentation for projects I have worked on, functional and technical specifications, user guides and cheat sheets, detailed design documents and all of these take considerable time to write well. I can't say I am expert in this area because I am not. But I have done a lot better than most of the projects that I have had to interface to over the decades.
Documentation is a low priority for many management teams, they do not see the use in the kind of details that are needed. They do not give it sufficient support to do the job well.
I have seen and worked with people who have the skill to write the kind of documentation that is actually needed and they are a rare breed, well worth any money you pay them.
When companies complain about "cowboys", they often do not see that it is they who are the "cowboys".
I've been programming for decades and my current project is frequently praised by other developers who work with it (it's a framework) for the quality of its documentation.
So, in fact, I do have such an idea. It's hard work. We do it anyway because we're professionals. The fact that people frequently don't do it doesn't change this.
One technique I use is to keep documentation in the same repository as the code, and force people to write docs via the code review process. That is, a proposed change doesn't pass review until it also updates the documentation files. People want to get their code merged, so they know they'll have to take docs seriously or else it'll get picked up in review.
Now maybe back in the mainframe days there was no culture of code review, but these are banks we're talking about. They excel at process. So it seems kind of unlikely they'd not want things documented, if management had understood what needed to be documented in the first place. But ultimately it was only the employees who knew that.
It doesn't matter how "professional" you might be, you will not have covered all your bases. All you can do is spend lots of time in building up this documentation and if it is not being paid for in some way or time constraints get in the way, then none of it will get done properly.
It is good to see that you have some sort of process in place to manage the problem, but code reviews are limited in what they will capture.
I am documenting a piece of my own code at the moment for the purposes of submitting it to a specific project. It provides an additional functionality relating to something that the main system doesn't (at this point) handle. It is intended to be a simple workaround until the main system can supply this functionality at the VM and compiler level.
I have just looked at one method that simply converts a hexadecimal number to a potentially multi-byte sequence. The detailed reasoning for the code (the assumptions, choices of numeric values used, etc) amount to 10 times the number of lines that is actual code and I still don't think it is complete. I think that it will take at least two, three or maybe even four more revisions in the documentation to get it to the place where someone else could make changes without causing problems elsewhere. The code itself is really simple. But, there is an additional purpose to this documentation, in that, it will provide a baseline against which the final system changes can be compared.
I do this because, as a somewhat retired "professional" programmer, I want to see some serious documentation for this system and can afford to put in the time to do so.
So back to the main discussions, unless full and complete documentation is a requirement put in place and strongly enforced, what documentation there is will not be sufficient for the future. It is a rare professional who will instigate this and make sure it happens. From my experience, such professionals that do this, have not come from a programming background but have come from other industries (usually engineering based) where such documentation is considered normal or is mandatory.
I have seen few management teams that have given more than lip service to creation of the required documentation and the time frames they put on projects are such that there is little or no time for the production of quality documentation that is needed. Nor are they interested in paying for such.
Banks have a focus of making money. They are not interested in spending it. The only process orientation they have is increasing the bottom line in whatever way they can. If it means they cut back on programmer and computing costs, they will do this. Ultimately, it is the bank management that holds responsibility to understand that the documentation is required. If they did not understand this, they should not have been in the position of management. It is irrelevant if only the employees (those grunts at the coal-face) knew this.
It also doesn't hurt when a quality technical writer actually writes or edits the documentation in consultation with the people who designed or implemented the system.
These are common pathologies. I agree though. This is incompetence at technology from strategy to the details.
It's important to understand how much technology gets built this way. Usually these projects are not outright, public failures.
If you take "I swear by my life and my love of it that I will never live for the sake of another man, nor ask another man to live for mine" seriously, this is the end result. Why should any IC or low-level manager at Taggart Transcontinental live for the sake of the Taggarts? But how can the railroad succeed without ICs and low-level managers?
If you want to fix this, don't ask people to live altruistically; instead, fix their incentives. Giving people cool projects such that they feel personally fulfilled when they succeed whether or not the business recognizes it is one option, but not particularly reliable or scalable.
Whenever you have a systemically pathological situation like big, enterprise software projects often get into... it's probably an incentive problem.
Tldr agreed
"a bank" is technically incorrect. The decision was a perfectly rational prisoner's dilemma defection from one executive (cut costs by firing the "expensive employees" and hiring "cheaper and newer"), who went on to be promoted on that basis, and it was the next generation of executives who had to deal with the mess and "hired IBM" so they could absolve responsibility of a situation with no visible upside if successful and tons of visible downside if not.
"absolve relevant parties of their responsibilities" - what relevant parties? What responsibilities? It sounds great and responsible, giving a good impression of dismissal of the point made, but the wording is so general as to be impossible to act upon - just as with "a bank". I'm sure standardised checklists were ticked, committees reviewed the decisions and came to the conclusion that it was best, the board was satisfied with the new direction taken by the CTO's organisation, and the shareholders happy with the cost reduction at the time as described by the board. To paraphrase Thatcher, what colour trousers do the relevant parties wear?
You see this defection pattern everywhere. For example, the never ending stream of useless new features in most Google and some Microsoft products which coincidentally get slower and buggier every year, because you can get promoted for the visible addition of a feature as a PM, but it is much harder to prove that getting the Gmail app load time down by 70% has stopped the feeding of your best power users to the competition (Calendar is too easy a target for criticism).
The classic defector on the tech side is the "ninja rockstar developer" whose prolific output of code and visually attractive but functionally superficial features is matched only by the amount of frustration generated after they are promoted somewhere else or changes jobs if it doesn't happen fast enough, and the rest of the team has to deal with the technical debt and new, unrealistic user expectations.
Security is another good candidate for shortcuts, since there is no visible upside to good security (maybe you were just not attacked) and tons of downside for taking responsibility since you make yourself a visible scapegoat for blame when a crisis does happen.
And note that none of these examples are in "move fast and break things" i.e. "do illegal things and hope they become legal ex-post" territory! I personally take great pains to understand these dynamics so that I can avoid them in my own company, which is a matter of survival.
Spending money on upgrades is not the problem. Firing people who know about your systems is.
You should do everything else to cut costs before you start getting rid of your people. Those people understand your business and you're going to need those people when things turn around (and they usually do). Any time you let people go, you're throwing away domain knowledge. You've PAID for that. It is an asset. You'd no more throw away all the computers in the building, why would you throw away your employees!?
"Technical debt is your job." -- is what I once heard some PHB say. Not from a finance person but the attitude, I think, is the same.
In all honesty: why is COBOL so prevalent in the finance sector? It can't seriously be SO GOOD as to be completely un-replacable?
It is not un-replacable in itself, I'd imagine it is just a huge undertaking and many higher ups are afraid to do it and risk something going wrong.
You can't just throw it away.
So at any specific time, the safer and cheaper option is to stick with the old system, despite the growing risks and costs in the long run.
To use hip modern buzzwords, it’s a DSL. In addition, it runs on hardware specifically designed around its idioms.
Imagine that you need to replace a live system, downtimes are not allowed, probably provide a new UI while offering 100% of existing features.
There is no documentation available, or even if there is one, you won't find anyone in the building that will put their hand on the fire for how much of it actually maps to the running application.
So a project of reverse engineering existing application into a requirements document or updating the existing documentation usually takes several months.
Now just for a very basic project cost, consider the monthly salary of everyone involved, multipled by the amount of months.
And we are just speaking of phase I, then there is the actual re-write, followed by verification that 100% of the old features are still present on the new system.
Naturally it is cheaper to keep existing systems running.
This applies to any old platform, not only COBOL systems.
Which by itself shows how much money such re-write would eventually cost.
Most big banks have a bunch of security people whose sole job is to make people’s life hell by spreading as much FUD as possible[2] — that’s what keeps their jobs secure, after all. And they are the only “engineers” who are heard. They make sure that anything new is not adopted till the rest of the world has already started abandoning it.
The primary focus in the technology departments of big banks is to maintain the status quo. New development is rare. New ideas are crushed before they can see the light of the day. As a result, good engineers leave as son as they can and the only ones left behind are the ones who are content living a life of mediocrity — maintaining the status quo.
[1] I know someone who was told something along those lines when they joined a big financial institution as an engineering manager. And not all financial companies work that way. Those who embraced technology reaped the benefits and continue to do so.
[2] I am not suggesting that security is not important. But security people at these institutions take it to a whole different level — to the point where things that should not take more thane a few days end up taking months — if they allow it, that is. Funny how all their objections vanish when someone from upper management asks for the same thing.
As above, may be these COBOL systems have been battle tested for a long time and they are good. Other reasons could be, "if it ain't broke then why fix it?" or that there aren't enough people to act as a bridge between the old and new systems so it is hard to do such projects (as other comments have pointed out)
I wonder whether it was the same VP who got promotion for decision to cut corners a couple of years back, and another promotion for decision to fix it by contractors.
They spent a billion dollars, over 5 years, with a staff of 1,000 strong to replace old software.
I would be surprised if even 20 of the 1,000 were touching COBOL.
That's what Accenture does: armies of cheap people for huge contracts. Flip a coin on the outcome. Not that IBM or any of the other top-level consultancies/services behemoths are any good, but Andersen Consulting/Accenture basically invented that approach.
Operational expenses looks better on the balance sheet (ie. renting versus owning).
MSP and Cloud are separate concepts. The typical MSP has a bunch of technicians who drive around to client sites and tend to the servers in their closets.
If a bank or any big corp is in trouble with legacy software and they approach you to fix it, charge them 500/h or more. That is still cheaper than the time they need to find someone else.
[1] https://www.kalzumeus.com/2015/05/01/talking-about-money/#co...
https://www.bbc.co.uk/programmes/b0bmbhzb
Before we have another concept, those from LispM, the "system" as a single entity of well_integrated stuff.
We have many example of those two way of thinking today: on "platform" side we have snap, flatpack, appimage, lx[cd]/docker, ... on system side we have Emacs, NixOS, GuixSD, ...
Well, for years the "platform" model seems to be the most reasonable, today seems ancient MIT&c hackers ware right, "system" approach is better. Simple to manage, for good software, do not hide bad practice, force collaboration etc.
On datacenters today and not from today we do substantially the same, in the past datacenters was a big collection of independent computers, now their are substantially all "a single computer" (even before The datacenter as a Computer by Google), made of many well_integrated components.
My poor English may not help, but I hope I have been clear up there, if so, what you think?
A cowboy is a loose cannon in the big city, but a "proper software engineer" is useless in the wild west. These banks have just realised that there are a few ranches left.
It's not being used as a perjorative term.
It's talking about a company called Cobol Cowboys, itself inspired by Space Cowboys, about a bunch of old guys called back into service to help with an emergency.
Charmingly enough, their motto is "not our first rodeo."
After the title the first use refers to the name of a company founded by the person mentioned in the first sentence of the article.
For the decade I worked there, my buddy told me that his pal made a very good living repairing those machines, even though they were long out of fashion and 'there was no market'.
Seems the same is true for software, too.
Computer trends and actual use don't always match up. Which can be a boon for those who realize they can specialize in unfashionable technology.
And continue to ignore. This is not a new issue, but the same story repeating over and over. Management purely via cost control is a terrible approach.
Is "expired" a standard English synonym for pining for the fjords?
There never seems to be time for documentation, specifications, and clean code