This is both awful, somehow unsurprising, and COBOL is a piss poor scapegoat.
I've talked on here in the past about how crazy it is that some of my customers had huge financial systems that were written in COBOL, quietly still plugging away.
The most important of which is a system which processes billions of dollars of transactions a day, and has done so since the early 70's (when they purchased the what was then off-the-shelf software). You can imagine what a maintenance nightmare it must be (both because it's old, very important, and written in a programming language which predates the cotton gin).
Assuming that they can't just rewrite the system (which might be a reality), they can certainly sit down a bunch of developers and have them learn COBOL. COBOL isn't even a particularly difficult language, we were writing pretty decently-sized programs after a few months in high school (and if the system is that old, it's probably not even using any fancy language features).
If only the military had some means by which they could take people without past exposure to some skillset and train them to be competent at that skillset. Maybe part of that training could involve having to climb over a wall, or some sort of rope course.
It might not even be an issue of training COBOL programmers. I'm aware of more than one large-scale legacy system that people "somehow" lost the source code for.
Which means fixing bugs means disassembly, hot patching, and keeping your fingers crossed. And if you're lucky and the place is incapable of learning, they'll then lose all documentation for the patch, too. Fixing anything is a nightmare. On the upside, if you can stomach it, there are nice consulting fees to be had.
(I'm not aware if the Pentagon lost source code - my experience is with civilian systems)
> Which means fixing bugs means disassembly, hot patching, and keeping your fingers crossed.
This sounds like something with extremely high overhead, that would benefit from scale. That is to say, instead of trying to reverse-engineer and patch each module every time you need to make a change, just:
1. run the entire project through a decompiler, making sure that each generated code file on its own recompiles to something that works when linked back together with the rest of the existing modules;
2. and then use that decompiled result as the new codebase, and start refactoring.
In a reasonable world, yes. In a world where IT is a cost center, this is what happens:
1) People work around it as much as possible.
2) When the pain is unbearable, you bring in a high-priced consultant, because your engineers say something about "recompile" and "refactor". Which you as VP don't understand, are scared of, and are not willing to spend the money on.
3) The consultant puts in a week or so to hack things together. It's much cheaper than the recompile/refactor thing, and so you as VP are a hero to your bosses.
4) Since the consultant is paid hourly, documentation seems kind of pricey. Let's skip that.
5) Lather. Rinse. Repeat.
I wish this was just Dilbert, but in quite a few companies, that's reality.
I also agree that the language is probably the least of their worries. I can understand reluctance to change a system when it works perfectly, but this obviously doesn't work perfectly. Sounds like a good project for challenge.gov IMO.
I remember ~10 years ago they were trying to get the community to learn COBOL on it's own so they'd have access to cheap programmers. Isn't that how you build a business or successful organization, by exploiting the everyman for cheap labor? It worked for the ____, so I guess not since nobody fell for it.
Problems with pay were one of the primary reasons I didn't reenlist. This was an issue in the 80's. I lost days of personal time talking to yeomen, chiefs and officers just because every other check was late or missing. Turned out for the best, since I would have ended up on the USS Stark since my current ship was going to be decommissioned.
A sad read indeed. Unfortunately this is not new. No country ever loved there soldier as much as their soldiers loved their country.
I remember reading about pay issues in the roman army when I was at the university. It hasn't changed in 2 000 years. Apparently the roman soldiers experience that their families back home didn't get their pay, and wives was given receipts where there was subtracted xx day for "leave" because their husbands was dead at the time.
We can only speculate if there is any motive and reason her, or only incompetence…
Readers may be interested that Britain's joint armed forces pay system, part of JPA (Joint Personnel Admin) is also recognised to be something of a clusterfuck (although not quite as bad now as it was back when it arrived), dragging in its wake myriad unlucky sods who found themselves paid incorrectly and struggled to get anything fixed, including an RAF chap who ended up simply suing the MOD for the money they owed him but found themselves unable to actually give him.
JPA was put together principally by the no-longer existent EDS, whose reputation for being able to handle large complex systems is well known in the industry.
It's worth mentioning that in the industry, smart cookies recognise that payroll is very much a wicked problem, and payroll for something as complicated as the armed forces is massively complex.
Less smart cookies do not recognise this and routinely massively underestimate the work involved and the support required.
It makes me furious that no one bats an eye at spending a BILLION dollars on a single presidential campaign but they can't sort out pay and benefits for our armed forces and veterans.
Surely a team with a background in large systems could take a set of requirements and produce a working system in 12-18 months.
I think that's the key problem: nobody has a complete big-picture idea of what the program is doing. Sure, there are people in each service branch that are familiar with subsets of the overall technology, but I suspect that at the overlap of these subsets there is significant disagreement over interpretation & implementation that has never become a visible problem because nobody runs consistency checks.
Since compiling the mutually contradictory payroll policies into a non-probabilistic master policy would immediately shed light on these issues, it would be a political minefield. Every choice you make would be the wrong choice and nobody is simultaneously high enough on the totem pole to say "live with it" and low enough on the totem pole to actually implement it.
Paychex seems to handle payroll policies for a huge range of businesses without too much trouble. Also I'm not quite sure why the DoD would have several different payroll policies? It seems like everything is determined by the relevant laws, similar to how you can look up the pay frequency and amount of pretty much any pay grade.
Laws often say things like 'significant' which are generally interpreted at the agency level. The DoD has a lot of agency's so these interpretations are unlikely to line up.
The problem is that DoD institutionally is shit at IT now. It didn't used to be this way (DoD was at the forefront in the beginning), but now we just can't do contracting for IT to save our lives.
Everytime someone tries to start a system to fix pay they realize that they will need to change all the personnel systems to match, and figure that they'll go and fix the personnel systems too.
They tried that with DIHMRS, which actually did involve billions, and the project never even got finished it was such a C.F.
Two things are striking:
- The amounts under debate are small
- The DFAS agency's budget is huge (1.36B)
Why not eliminate the agency altogether? If we divide $1.36B by the number of active military members 1,429,995, that results in $951 / active military personnel.
If we assume payment issues happen to 10% of the military, then that means we can just institute a rule that would give every soldier up to $9,510 in free "payment error insurance."
But I guess having a government body of 12,000 people with the task of giving soldiers a hard time makes more sense to some federal bureaucrat.
> If we assume payment issues happen to 10% of the military, then that means we can just institute a rule that would give every soldier up to $9,510 in free "payment error insurance."
Sounds like another scandal just waiting to happen.
this made me think of an editorial I read during the Iraq war (maybe someone else will remember the author's name).
The author was a university professor who was urging his son not to join the military. One of the points he made is that if his son was injured the military would not take care of him. It was tragic ... the son ended up joining anyway and was killed in Iraq. I remember feeling like shit when that happened. Just kept thinking how awful the Dad must have felt.
I can't quite remember the fathers name .. but I'm pretty sure it was Scandinavian. I keep thinking Uwe S.....?
> This agency, with headquarters in Indianapolis, Indiana, has roughly 12,000 employees and, after cuts under the federal sequester, a $1.36 billion budget. It is responsible for accurately paying America's 2.7 million active-duty and Reserve soldiers, sailors, airmen and Marines.
It often fails at that task, a Reuters investigation finds.
31 comments
[ 2.4 ms ] story [ 71.3 ms ] threadI've talked on here in the past about how crazy it is that some of my customers had huge financial systems that were written in COBOL, quietly still plugging away.
The most important of which is a system which processes billions of dollars of transactions a day, and has done so since the early 70's (when they purchased the what was then off-the-shelf software). You can imagine what a maintenance nightmare it must be (both because it's old, very important, and written in a programming language which predates the cotton gin).
Assuming that they can't just rewrite the system (which might be a reality), they can certainly sit down a bunch of developers and have them learn COBOL. COBOL isn't even a particularly difficult language, we were writing pretty decently-sized programs after a few months in high school (and if the system is that old, it's probably not even using any fancy language features).
If only the military had some means by which they could take people without past exposure to some skillset and train them to be competent at that skillset. Maybe part of that training could involve having to climb over a wall, or some sort of rope course.
Which means fixing bugs means disassembly, hot patching, and keeping your fingers crossed. And if you're lucky and the place is incapable of learning, they'll then lose all documentation for the patch, too. Fixing anything is a nightmare. On the upside, if you can stomach it, there are nice consulting fees to be had.
(I'm not aware if the Pentagon lost source code - my experience is with civilian systems)
This sounds like something with extremely high overhead, that would benefit from scale. That is to say, instead of trying to reverse-engineer and patch each module every time you need to make a change, just:
1. run the entire project through a decompiler, making sure that each generated code file on its own recompiles to something that works when linked back together with the rest of the existing modules;
2. and then use that decompiled result as the new codebase, and start refactoring.
1) People work around it as much as possible.
2) When the pain is unbearable, you bring in a high-priced consultant, because your engineers say something about "recompile" and "refactor". Which you as VP don't understand, are scared of, and are not willing to spend the money on.
3) The consultant puts in a week or so to hack things together. It's much cheaper than the recompile/refactor thing, and so you as VP are a hero to your bosses.
4) Since the consultant is paid hourly, documentation seems kind of pricey. Let's skip that.
5) Lather. Rinse. Repeat.
I wish this was just Dilbert, but in quite a few companies, that's reality.
I also agree that the language is probably the least of their worries. I can understand reluctance to change a system when it works perfectly, but this obviously doesn't work perfectly. Sounds like a good project for challenge.gov IMO.
https://en.wikipedia.org/wiki/USS_Stark_(FFG-31)#Missile_att...
I remember reading about pay issues in the roman army when I was at the university. It hasn't changed in 2 000 years. Apparently the roman soldiers experience that their families back home didn't get their pay, and wives was given receipts where there was subtracted xx day for "leave" because their husbands was dead at the time.
We can only speculate if there is any motive and reason her, or only incompetence…
JPA was put together principally by the no-longer existent EDS, whose reputation for being able to handle large complex systems is well known in the industry.
It's worth mentioning that in the industry, smart cookies recognise that payroll is very much a wicked problem, and payroll for something as complicated as the armed forces is massively complex.
Less smart cookies do not recognise this and routinely massively underestimate the work involved and the support required.
Surely a team with a background in large systems could take a set of requirements and produce a working system in 12-18 months.
I think that's the key problem: nobody has a complete big-picture idea of what the program is doing. Sure, there are people in each service branch that are familiar with subsets of the overall technology, but I suspect that at the overlap of these subsets there is significant disagreement over interpretation & implementation that has never become a visible problem because nobody runs consistency checks.
Since compiling the mutually contradictory payroll policies into a non-probabilistic master policy would immediately shed light on these issues, it would be a political minefield. Every choice you make would be the wrong choice and nobody is simultaneously high enough on the totem pole to say "live with it" and low enough on the totem pole to actually implement it.
Everytime someone tries to start a system to fix pay they realize that they will need to change all the personnel systems to match, and figure that they'll go and fix the personnel systems too.
They tried that with DIHMRS, which actually did involve billions, and the project never even got finished it was such a C.F.
Why not eliminate the agency altogether? If we divide $1.36B by the number of active military members 1,429,995, that results in $951 / active military personnel.
If we assume payment issues happen to 10% of the military, then that means we can just institute a rule that would give every soldier up to $9,510 in free "payment error insurance."
But I guess having a government body of 12,000 people with the task of giving soldiers a hard time makes more sense to some federal bureaucrat.
Sounds like another scandal just waiting to happen.
The story pisses me off to no end. The fucking Pentagon pisses away TRILLIONS of dollars yet they can't even get deserving soldiers their just pay?
It's completely unacceptable!
Take money from the NSA to pay these people.
The author was a university professor who was urging his son not to join the military. One of the points he made is that if his son was injured the military would not take care of him. It was tragic ... the son ended up joining anyway and was killed in Iraq. I remember feeling like shit when that happened. Just kept thinking how awful the Dad must have felt.
I can't quite remember the fathers name .. but I'm pretty sure it was Scandinavian. I keep thinking Uwe S.....?
It often fails at that task, a Reuters investigation finds.
Ripe for disruption, says I.