> Phoenix project executives believed they could deliver a modernized payment system, customizing PeopleSoft’s off-the-shelf payroll package to follow 80,000 pay rules spanning 105 collective agreements with federal public-service unions.
Somehow I come away skeptical of the inevitable conclusion that Phoenix was doomed to fail and instead that perhaps they were hamstrung by architecture constraints dictated by assholes.
Yup, and with an equal amount of mindblowing-units-of-money spent, infrastructure projects all around me are still failing as well, or at least being modified (read: downsized), delayed and/or budget-inflated beyond recognition.
So, what's the point here, exactly? "Only licensed engineers as codified by (local!) law are allowed to do projects?" Nah, can't be it, their track record still has too many failures, sometimes even spectacularly explosive and/or implosive ones.
"Any public project should only follow Best Practices"? Sure... "And only make The People feel good"... Incoherent!
Ehhm, so, yeah, maybe things are just complicated, and we should focus more on the amount of effort we're prepared to put in, the competency (c.q. pay grade) of the staff we're willing to assign, and exactly how long we're willing to wait prior to conceding defeat?
> "Why worry about something that isn’t going to happen?”
Lots to break down in this article other than this initial quotation, but I find a lot of parallels in failing software projects, this attitude, and my recent hyper-fixation (seems to spark up again every few years), the sinking of the Titanic.
It was a combination of failures like this. Why was the captain going full speed ahead into a known ice field? Well, the boat can't sink and there (may have been) organizational pressure to arrive at a certain time in new york (aka, imaginary deadline must be met). Why wasn't there enough life jackets and boats for crew and passengers? Well, the boat can't sink anyway, why worry about something that isn't going to happen? Why train crew on how to deploy the life rafts and emergency procedures properly? Same reason. Why didn't the SS Californian rescue the ship? Well, the 3rd party Titanic telegraph operators had immense pressure to send telegrams to NY, and the chatter about the ice field got on their nerves and they mostly ignored it (misaligned priorities). If even a little caution and forward thinking was used, the death toll would have been drastically lower if not nearly nonexistent. It took 2 hours to sink, which is plenty of time to evacuate a boat of that size.
Same with software projects - they often fail over a period of multiple years and if you go back and look at how they went wrong, there often are numerous points and decisions made that could have reversed course, yet, often the opposite happens - management digs in even more. Project timelines are optimistic to the point of delusion and don't build in failure/setbacks into schedules or roadmaps at all. I've had to rescue one of these projects several years ago and it took a toll on me I'm pretty sure I carry to this day, I'm wildly cynical of "project management" as it relates to IT/devops.
I study and write quite a bit of tech history. IMHO from what I've learned over the last few years of this hobby, the primary issue is quite simple. While hardware folks study and learn from the successes and failures of past hardware, software folks do not. People do not regularly pull apart old systems for learning. Typically, software folks build new and every generation of software developers must relearn the same problems.
How do you study software history? Most of the lessons seem forever locked away behind corporate walls - any honest assessments made public will either end careers or start lawsuits
Software just feels so much more ephemeral than hardware. I haven't yet met a single 'old software enthusiast' in my life, yet there are so many enthusiasts for older hardware.
I am both a hardware and software enthusiast. Tons of DOS, Windows, and OS/2 software hanging around. While I don’t use them everyday, I do use them. From pre-Microsoft Visio to WordStar and MS Works for DOS, the applications are simple, powerful, and pleasing to use. While I don’t recommend anyone pull out Zenith 8bit and fire up COBOL-80 or LISP-80, they are interesting. Testing yourself in 64k is quite a challenge.
The retro community is huge and varied. If it exists, someone is really into it.
This is an interesting distinction, but it ignores the reasons software engineers do that.
First, hardware engineers are dealing with the same laws of physics every time. Materials have known properties etc.
Software: there are few laws of physics (mostly performance and asymptotic complexity). Most software isnt anywhere near those boundaries so you get to pretend they dont exist. If you get to invent your own physics each time, yeah the process is going to look very different.
For most generations of hardware, you’re correct, but not all. For example, high-k was invented to mitigate tunneling. Sometimes, as geometries shrink, the physics involved does change.
I think this is too simple. First of all, hardware people have high incentive to fully replace components and systems for many reasons. Replacement is also the only way they can fix major design mistakes.
Software people constantly do fix bugs and design mistakes. There is certainly no strong culture to document or dig up former mistakes made, but it's not like they don't learn from mistakes, it's just a constant process. In contrast to hardware, there is usually no point in time to retrospect. The incentives to rejuvenate systems are low and if considered often seem expensive. Software engineers self motivation is often ill-minded, new devs feeling uncomfortable with the existing system and calling for something "modern". But if the time comes to replace the "legacy" systems, then you are right, no one looks back at the former mistakes and the devs that know them, are probably long gone.
The question is whether we should ever replace an software system or focus more on gradual and active modernization. But the latter can be very hard, in hardware everything is defined, most of the time backed by standards, in software we usually don't have that, so complex interconnected systems rarely have sane upgrade paths.
IME, "Why systems fail" almost always boils down to a principal-agent problem. This is another way of expressing the Mungerism "show me the incentive, I'll show you the outcome".
Systems that "work" tend to have some way of correcting for or mitigating the principal agent problem by aligning incentives.
I'd also point out that hardware is a much older discipline, in terms of how long it's been operating at scale. It's had more time to formalize and crystallize. Intel is 56 years old. Google is 27.
> IT projects suffer from enough management hallucinations and delusions without AI adding to them.
Software is also incredibly hard, the human mind can understand the physical space very well but once we're deep into abstractions it simply struggles to keep up with it.
It is easier to explain how to build a house from scratch to virtually anyone than a mobile app/Excel.
New Zealand tried to do a new payroll system for teachers called Novopay which imploded spectacularly and is still creating grief. The system is now called EdPay (the government decided to take over the privately created system). The total cost of the debacle was north of $200M NZD. Somehow they managed to fail to replace a working system!
As someone that has seen technological solutions applied when they make no sense, I think the next revolution in business processes will be de-computerization. The trend has probably already started thank to one of the major cloud outages.
An endless succession of new tools, methodologies, and roles but failure persists because success is rooted in good judgment, wisdom, and common sense.
This has dot-com bubble written all over it. But there are some deeper issues.
First, we as a society should really be scrutinizing what we invest in. Trillions of dollars could end homelessness as a rounding error.
Second, real people are going to be punished for this as the layoffs go into overdrive, people lose their houses and people struggle to have enough to eat.
Third, the ultimate goal of all this investment is to displace people from the labor pool. People are annoying. They demand things like fair pay, safe working conditions and sick leave.
Who will buy the results of all this AI if there’s no one left with a job?
Lastly, the externalities of all this investment are indefensible. For example, air and water pollution and rising utility prices.
We’re bouldering towards a future with a few thousand wealthy people where everyone else lives in worker housing, owns nothing and is the next incarnation of brick kiln workers on wealthy estates.
So, I'm not a dev nor a project manager but I found this article very enlightening. At the risk of asking a stupid question and getting a RTFM or a LMGTFY can anyone provide any simple and practical examples of software successes at a big scale. I work at a hospital so healthcare specific would be ideal but I'll take anything.
FWIW I have read The Phoenix Project and it did help me get a better understanding of "Agile" and the DevOps mindset but since it's not something I apply in my work routinely it's hard to keep it fresh.
My goal is to try and install seeds of success in the small projects I work on and eventually ask questions to get people to think in a similar perspective.
I don't think you should focus on successful large projects. Generally you should consider that all big successes are outliers from a myriad of attempts. They have been lucky and you can't reproduce luck.
I'd like try to correct your course a bit.
DevOps is a trash concept, that had good intentions. But today it's just an industry cheatcode to fill three dev positions with a single one that is on pager duty. The good takeaways from it: Make people care that things work end to end. If Joe isn't caring about Bob's problems, something is off. Either with the process, or with the people.
Agile is a very loose term nowadays. Broadly spoken it's the opposite of making big up front plans and implement them in a big swipe. Agile wants to start small and improve it iteratively as needed. This tends to work in the industry, but the iterative time buckets have issues, some teams can move fast in 2 week cycles, others don't. The original agile movement also wanted to give back control and autonomy back to those who actually do stuff (devs and lower management). This is very well intended and highly functional, but is often buried or ignored in corporate environments. Autonomy is extremely valuable, it motivates people and fosters personal growth, but being backed by a skilled peers also creates psychological safety. One of the major complaints I hear about agile practices is that there are too many routines, meetings and other in person tasks with low value that keep you from working. This is really bad and in my perception was never intended, but companies love that shit. This part is about communication, make it easy for people to share and engage, while also keeping their focus hours high. Problems have to bubble up quickly and everyone should be motivated and able to help solving them. If you listen to agile hardliners, they will also tell you that software can't be reliably planned, you won't make deadlines, none of them, never. That is very true, but companies are unable to deal with it.
The biggest reason is developer ego. Devs see their code as artwork an extension of themselves, so it's really hard to have critical conversations about small things and they erupt into holy wars. Off hand:
* Formatting
* Style
* Conventions
* Patterns
* Using the latest frameworks or whats en-vogue
I think where I've seen results delivered effectively and consistently is where there is a universal style enforced, which removes the individualism from the codebase. Some devs will not thrive in that environment, but instead it makes the code a means-to-the-end, rather than being-the-end.
Software projects fail because humans fail. Humans are the drivers of everything in our world. All government, business, culture, etc... it's all just humans. You can have a perfect "process" or "tool" to do a thing, but if the human using it sucks, the result will suck. This means that the people involved are what determines if the thing will succeed or fail. So you have to have the best people, with the best motivations, to have a chance for success.
The only thing that seems to change this is consequences. Take a random person and just ask them to do something, and whether they do it or not is just based on what they personally want. But when there's a law that tells them to do it, and enforcement of consequences if they don't, suddenly that random person is doing what they're supposed to. A motivation to do the right thing. It's still not a guarantee, but more often than not they'll work to avoid the consequences.
Therefore if you want software projects to stop failing, create laws that enforce doing the things in the project to ensure it succeeds. Create consequences big enough that people will actually do what's necessary. Like a law, that says how to build a thing to ensure it works, and how to test it, and then an independent inspection to ensure it was done right. Do that throughout the process, and impose some kind of consequence if those things aren't done. (the more responsibility, the bigger the consequence, so there's motivation commensurate with impact)
That's how we manage other large-scale physical projects. Of course those aren't guaranteed to work; large-scale public works projects often go over-budget and over-time. But I think those have the same flaw, in that there isn't enough of a consequence for each part of the process to encourage humans to do the right thing.
> But I think those have the same flaw, in that there isn't enough of a consequence for each part of the process
If there was sufficient consequence for this stuff, no one would ever take on any risk. No large works would ever even be started because it would be either impossible or incredibly difficult to be completely sure everything will go to plan.
So instead we take a medium amount of caution and take on projects knowing it's possible for them to not work out or to go over budget.
Working on AI that helps to manage IT shops that learns from failure & success might be better for both results and culture than most IT management roles, a profession (painting an absurdly broad brush) that tends to attract a lot of miserable creatures.
managing software requirements and the corresponding changes to user/group/process behaviors is by far the hardest part of software development, and it is a task no one knows how to scale.
absent understanding, large companies engage in cargo cult behaviors: they create a sensible org chart, produce a gannt chart, have the coders start whacking code, presumably in 9 months a baby comes out.
> While hardware folks study and learn from the successes and failures of past hardware, software folks do not.
I guess that’s the real problem I have with SV’s endemic ageism.
I was personally offended, when I encountered it, myself, but that’s long past.
I just find it offensive, that experience is ignored, or even shunned.
I started in hardware, and we all had a reverence for our legacy. It did not prevent us from pursuing new/shiny, but we never ignored the lessons of the past.
The article is kind of dumb. eg it hangs its hat on the Phoenix payroll system, which
> Phoenix project executives believed they could deliver a modernized payment system, customizing PeopleSoft’s off-the-shelf payroll package to follow 80,000 pay rules spanning 105 collective agreements with federal public-service unions. It also was attempting to implement 34 human-resource system interfaces across 101 government agencies and departments required for sharing employee data.
So basically people -- none of them in IT, but rather working for the government -- built something extraordinarily complex (80k rules!), and then are like wow, it's unforeseen that would make anything downstream at least equally as complex. And then the article blames IT in general. When this data point tells us that replacing a business process that used to require (per [1]) 2,000 pay advisors to perform will be complex. While working in an organization that has shit the bed so thoroughly that paying its employees requires 2k people. For an organization of 290k, so 0.6% of headcount is spent on paying employees!
IT is complex, but incompetent people and incompetent orgs do not magically become competent when undertaking IT projects.
Also too, making extraordinarily complex things they shouting the word "computer" at them like you're playing D&D and it's a spell does not make them simple.
The lesson from “big software projects are still failing” isn’t that we need better methodologies, better project management, or stricter controls. The lesson is "don't do big software projects".
Software is not the same as building in the physical world where we get economies of scale.
Building 1,000 bridges will make the cost of the next incremental bridge cheaper due to a zillion factors, even if Bridge #1 is built from sticks (we'll learn standards, stable, fundamental engineering principles, predicable failure modes, etc.) we'll eventually reach a stable, repeatable, scalable approach to build bridges. They will very rarely (in modernity) catastrophically fail (yes, Tacoma Narrows happened but in properly functioning societies it's rare.)
Nobody will say "I want to build a bridge upside-down, out of paper clips and can withstand a 747 driving over it". Because that's physically impossible. But nothing's impossible in software.
Software isn't scalable in this way. It's not scalable because it doesn't have hard constraints (like the laws of physics) - so anything goes and can be in scope; and since writing and integrating large amounts of code is a communication exercise, suffers from diseconomies of scale.
Customers want the software to do exactly what they want and - within reason - no laws of physics are violated if you move a button or implement some business process.
Because everyone wants to keep working the way they want to work, no software project (even if it sounds the same) is the same. Your company's bespoke accounting software will be different than mine, even if we are direct competitors in the same market. Our business processes are different, org structures are different, sales processes are different, etc.. So they all build different accounting software, even if the fundamentals (GaaP, double-entry bookkeeping, etc.) are shared.
It's also the same reason why enterprise software sucks - do you think that a startup building expense management starts off being a giant mess of garbage? No! IT starts off simple and clean and beautiful because their initial customer base (startups) are beggars and cannot be choosers, so they adapt their process to the tool. But then larger companies come along with dissimilar requirements and, Expense Management SaaS Co. wins that deal by changing the product to work with whatever oddball requirements they have, and so on, until the product essentially is a bunch of config options and workflows that you have to build yourself.
(Interestingly, I think these products become asymptotically stuck - any feature you add or remove will make some of your customers happy and some of your customers mad, so the product can never get "better" globally).
We can have all the retrospectives and learnings we want but the goal - "Build big software" - is intractable, and as long as we keep trying to do that, we will inevitably fail. This is not a systems problem that we can fix.
The lesson is: "never build big software".
(Small software is stuff like Bezos' two pizza team w/APIs etc. - many small things make a big thing)
Every improvement will be moderated increased demands from management, crunch, pressure to release, "good enough", add this extra library that monetizes/spys on the customer etc
In the same way that hardware improvements are quickly gobbled up by more demanding software.
The people doing the programming will also be more removed technically.
I can do Python, Java , Kotlin. I can do a little C++ ,less C, and a lot less assembly.
On some of the infamous large public IT project failures, you just have to look at who gets the contract, how they work, and what their incentives are. (For example, don't hire management consulting partner smooth talkers, and their fleet of low-skilled seat-warmers, to do performative hours billing.)
It's also hard when the team actually cares, but there are skills you can learn. Early in my career, I got into solving some of the barriers to software project management (e.g., requirements analysis and otherwise understanding needs, sustainable architecture, work breakdown, estimation, general coordination, implementation technology).
But once you're a bit comfortable with the art and science of those, big new challenges are more about political and environment reality. It comes down to alignment and competence of: workers, internal team leadership, partners/vendors, customers, and investors/execs.
Discussing this is a little awkward, but maybe start with alignment, since most of the competence challenges are rooted in mis-alignments: never developing nor selecting for the skills that alignment would require.
Almost nobody who works in software development is a licensed professional engineer. Many are even self-taught, and that includes both ICs and managers. I'm not saying this is direct causation but I do think it odd that we are so utterly dependent on software for so many critical things and yet we basically YOLO its development compared to what we expect of the people who design our bridges, our chemicals, our airplanes, etc.
140 comments
[ 4.5 ms ] story [ 208 ms ] threadSomehow I come away skeptical of the inevitable conclusion that Phoenix was doomed to fail and instead that perhaps they were hamstrung by architecture constraints dictated by assholes.
So, what's the point here, exactly? "Only licensed engineers as codified by (local!) law are allowed to do projects?" Nah, can't be it, their track record still has too many failures, sometimes even spectacularly explosive and/or implosive ones.
"Any public project should only follow Best Practices"? Sure... "And only make The People feel good"... Incoherent!
Ehhm, so, yeah, maybe things are just complicated, and we should focus more on the amount of effort we're prepared to put in, the competency (c.q. pay grade) of the staff we're willing to assign, and exactly how long we're willing to wait prior to conceding defeat?
Lots to break down in this article other than this initial quotation, but I find a lot of parallels in failing software projects, this attitude, and my recent hyper-fixation (seems to spark up again every few years), the sinking of the Titanic.
It was a combination of failures like this. Why was the captain going full speed ahead into a known ice field? Well, the boat can't sink and there (may have been) organizational pressure to arrive at a certain time in new york (aka, imaginary deadline must be met). Why wasn't there enough life jackets and boats for crew and passengers? Well, the boat can't sink anyway, why worry about something that isn't going to happen? Why train crew on how to deploy the life rafts and emergency procedures properly? Same reason. Why didn't the SS Californian rescue the ship? Well, the 3rd party Titanic telegraph operators had immense pressure to send telegrams to NY, and the chatter about the ice field got on their nerves and they mostly ignored it (misaligned priorities). If even a little caution and forward thinking was used, the death toll would have been drastically lower if not nearly nonexistent. It took 2 hours to sink, which is plenty of time to evacuate a boat of that size.
Same with software projects - they often fail over a period of multiple years and if you go back and look at how they went wrong, there often are numerous points and decisions made that could have reversed course, yet, often the opposite happens - management digs in even more. Project timelines are optimistic to the point of delusion and don't build in failure/setbacks into schedules or roadmaps at all. I've had to rescue one of these projects several years ago and it took a toll on me I'm pretty sure I carry to this day, I'm wildly cynical of "project management" as it relates to IT/devops.
The retro community is huge and varied. If it exists, someone is really into it.
First, hardware engineers are dealing with the same laws of physics every time. Materials have known properties etc.
Software: there are few laws of physics (mostly performance and asymptotic complexity). Most software isnt anywhere near those boundaries so you get to pretend they dont exist. If you get to invent your own physics each time, yeah the process is going to look very different.
Systems that "work" tend to have some way of correcting for or mitigating the principal agent problem by aligning incentives.
I'd also point out that hardware is a much older discipline, in terms of how long it's been operating at scale. It's had more time to formalize and crystallize. Intel is 56 years old. Google is 27.
1. Enable grift to cronies
2. Promo-driven culture
3. Resume-oriented software architecture
Software is also incredibly hard, the human mind can understand the physical space very well but once we're deep into abstractions it simply struggles to keep up with it.
It is easier to explain how to build a house from scratch to virtually anyone than a mobile app/Excel.
Then in 2010s they spent $185M on a customized version of IBM's PeopleSoft that was managed directly by a government agency https://en.wikipedia.org/wiki/Phoenix_pay_system
And now in 2020s they are going to spend $385M integrating an existing SaaS made by https://en.wikipedia.org/wiki/Dayforce
That's probably one of the worst and longest software failures in history.
New Zealand tried to do a new payroll system for teachers called Novopay which imploded spectacularly and is still creating grief. The system is now called EdPay (the government decided to take over the privately created system). The total cost of the debacle was north of $200M NZD. Somehow they managed to fail to replace a working system!
Honest question.
First, we as a society should really be scrutinizing what we invest in. Trillions of dollars could end homelessness as a rounding error.
Second, real people are going to be punished for this as the layoffs go into overdrive, people lose their houses and people struggle to have enough to eat.
Third, the ultimate goal of all this investment is to displace people from the labor pool. People are annoying. They demand things like fair pay, safe working conditions and sick leave.
Who will buy the results of all this AI if there’s no one left with a job?
Lastly, the externalities of all this investment are indefensible. For example, air and water pollution and rising utility prices.
We’re bouldering towards a future with a few thousand wealthy people where everyone else lives in worker housing, owns nothing and is the next incarnation of brick kiln workers on wealthy estates.
FWIW I have read The Phoenix Project and it did help me get a better understanding of "Agile" and the DevOps mindset but since it's not something I apply in my work routinely it's hard to keep it fresh.
My goal is to try and install seeds of success in the small projects I work on and eventually ask questions to get people to think in a similar perspective.
I'd like try to correct your course a bit.
DevOps is a trash concept, that had good intentions. But today it's just an industry cheatcode to fill three dev positions with a single one that is on pager duty. The good takeaways from it: Make people care that things work end to end. If Joe isn't caring about Bob's problems, something is off. Either with the process, or with the people.
Agile is a very loose term nowadays. Broadly spoken it's the opposite of making big up front plans and implement them in a big swipe. Agile wants to start small and improve it iteratively as needed. This tends to work in the industry, but the iterative time buckets have issues, some teams can move fast in 2 week cycles, others don't. The original agile movement also wanted to give back control and autonomy back to those who actually do stuff (devs and lower management). This is very well intended and highly functional, but is often buried or ignored in corporate environments. Autonomy is extremely valuable, it motivates people and fosters personal growth, but being backed by a skilled peers also creates psychological safety. One of the major complaints I hear about agile practices is that there are too many routines, meetings and other in person tasks with low value that keep you from working. This is really bad and in my perception was never intended, but companies love that shit. This part is about communication, make it easy for people to share and engage, while also keeping their focus hours high. Problems have to bubble up quickly and everyone should be motivated and able to help solving them. If you listen to agile hardliners, they will also tell you that software can't be reliably planned, you won't make deadlines, none of them, never. That is very true, but companies are unable to deal with it.
* Formatting
* Style
* Conventions
* Patterns
* Using the latest frameworks or whats en-vogue
I think where I've seen results delivered effectively and consistently is where there is a universal style enforced, which removes the individualism from the codebase. Some devs will not thrive in that environment, but instead it makes the code a means-to-the-end, rather than being-the-end.
It's leadership and accountability (well, the lack of them).
The only thing that seems to change this is consequences. Take a random person and just ask them to do something, and whether they do it or not is just based on what they personally want. But when there's a law that tells them to do it, and enforcement of consequences if they don't, suddenly that random person is doing what they're supposed to. A motivation to do the right thing. It's still not a guarantee, but more often than not they'll work to avoid the consequences.
Therefore if you want software projects to stop failing, create laws that enforce doing the things in the project to ensure it succeeds. Create consequences big enough that people will actually do what's necessary. Like a law, that says how to build a thing to ensure it works, and how to test it, and then an independent inspection to ensure it was done right. Do that throughout the process, and impose some kind of consequence if those things aren't done. (the more responsibility, the bigger the consequence, so there's motivation commensurate with impact)
That's how we manage other large-scale physical projects. Of course those aren't guaranteed to work; large-scale public works projects often go over-budget and over-time. But I think those have the same flaw, in that there isn't enough of a consequence for each part of the process to encourage humans to do the right thing.
If there was sufficient consequence for this stuff, no one would ever take on any risk. No large works would ever even be started because it would be either impossible or incredibly difficult to be completely sure everything will go to plan.
So instead we take a medium amount of caution and take on projects knowing it's possible for them to not work out or to go over budget.
absent understanding, large companies engage in cargo cult behaviors: they create a sensible org chart, produce a gannt chart, have the coders start whacking code, presumably in 9 months a baby comes out.
every time, ugly baby
I guess that’s the real problem I have with SV’s endemic ageism.
I was personally offended, when I encountered it, myself, but that’s long past.
I just find it offensive, that experience is ignored, or even shunned.
I started in hardware, and we all had a reverence for our legacy. It did not prevent us from pursuing new/shiny, but we never ignored the lessons of the past.
> Phoenix project executives believed they could deliver a modernized payment system, customizing PeopleSoft’s off-the-shelf payroll package to follow 80,000 pay rules spanning 105 collective agreements with federal public-service unions. It also was attempting to implement 34 human-resource system interfaces across 101 government agencies and departments required for sharing employee data.
So basically people -- none of them in IT, but rather working for the government -- built something extraordinarily complex (80k rules!), and then are like wow, it's unforeseen that would make anything downstream at least equally as complex. And then the article blames IT in general. When this data point tells us that replacing a business process that used to require (per [1]) 2,000 pay advisors to perform will be complex. While working in an organization that has shit the bed so thoroughly that paying its employees requires 2k people. For an organization of 290k, so 0.6% of headcount is spent on paying employees!
IT is complex, but incompetent people and incompetent orgs do not magically become competent when undertaking IT projects.
Also too, making extraordinarily complex things they shouting the word "computer" at them like you're playing D&D and it's a spell does not make them simple.
[1] https://www.oag-bvg.gc.ca/internet/English/parl_oag_201711_0...
Software is not the same as building in the physical world where we get economies of scale.
Building 1,000 bridges will make the cost of the next incremental bridge cheaper due to a zillion factors, even if Bridge #1 is built from sticks (we'll learn standards, stable, fundamental engineering principles, predicable failure modes, etc.) we'll eventually reach a stable, repeatable, scalable approach to build bridges. They will very rarely (in modernity) catastrophically fail (yes, Tacoma Narrows happened but in properly functioning societies it's rare.)
Nobody will say "I want to build a bridge upside-down, out of paper clips and can withstand a 747 driving over it". Because that's physically impossible. But nothing's impossible in software.
Software isn't scalable in this way. It's not scalable because it doesn't have hard constraints (like the laws of physics) - so anything goes and can be in scope; and since writing and integrating large amounts of code is a communication exercise, suffers from diseconomies of scale.
Customers want the software to do exactly what they want and - within reason - no laws of physics are violated if you move a button or implement some business process.
Because everyone wants to keep working the way they want to work, no software project (even if it sounds the same) is the same. Your company's bespoke accounting software will be different than mine, even if we are direct competitors in the same market. Our business processes are different, org structures are different, sales processes are different, etc.. So they all build different accounting software, even if the fundamentals (GaaP, double-entry bookkeeping, etc.) are shared.
It's also the same reason why enterprise software sucks - do you think that a startup building expense management starts off being a giant mess of garbage? No! IT starts off simple and clean and beautiful because their initial customer base (startups) are beggars and cannot be choosers, so they adapt their process to the tool. But then larger companies come along with dissimilar requirements and, Expense Management SaaS Co. wins that deal by changing the product to work with whatever oddball requirements they have, and so on, until the product essentially is a bunch of config options and workflows that you have to build yourself.
(Interestingly, I think these products become asymptotically stuck - any feature you add or remove will make some of your customers happy and some of your customers mad, so the product can never get "better" globally).
We can have all the retrospectives and learnings we want but the goal - "Build big software" - is intractable, and as long as we keep trying to do that, we will inevitably fail. This is not a systems problem that we can fix.
The lesson is: "never build big software".
(Small software is stuff like Bezos' two pizza team w/APIs etc. - many small things make a big thing)
In the same way that hardware improvements are quickly gobbled up by more demanding software.
The people doing the programming will also be more removed technically. I can do Python, Java , Kotlin. I can do a little C++ ,less C, and a lot less assembly.
It's also hard when the team actually cares, but there are skills you can learn. Early in my career, I got into solving some of the barriers to software project management (e.g., requirements analysis and otherwise understanding needs, sustainable architecture, work breakdown, estimation, general coordination, implementation technology).
But once you're a bit comfortable with the art and science of those, big new challenges are more about political and environment reality. It comes down to alignment and competence of: workers, internal team leadership, partners/vendors, customers, and investors/execs.
Discussing this is a little awkward, but maybe start with alignment, since most of the competence challenges are rooted in mis-alignments: never developing nor selecting for the skills that alignment would require.