Apologies for bring in "AI" to a non-AI thread, but I really do think that things will be a game changer for institutional memory, both at recording it and recovering it. I don't personally use them but I have many coworkers that use AI tools to join meetings and get summaries/transcriptions aftward that they can read or query (also using AI). As people get more used to it, I would imagine that sort of thing becomes standard practice (regardless of whether or not it should, but that's a different topic)
I've heard this is part of why major infrastructure projects in America can be so expensive. A city builds one subway line, and everyone working on the project has no experience, so it takes a long time and is expensive. The expense convinces people to oppose any more projects, so the city doesn't build any public transit for a decade(s). By the time they're ready to build another line, all the experience has evaporated, so the new line takes a long time and is expensive. Repeat forever.
Not just that, it can be worse when different companies worked on different parts of the subway in the city. It results in no overall vision and desire to see subway network as a whole.
Not just that, it can be worse when different companies worked on different parts of the subway. It results in no overall vision and desire to see subway network as a whole to make it efficient and convenient to get from any part of the city to another.
This is also the reason Olkiluoto Nuclear Plant Unit 3[0] cost so much.
Nobody had built a nuclear reactor in ages. The last ones were from the 80's and this was a completely new technology (EPR). There was no institutional knowledge.
It didn't help that the French attitude to building was to "just slap it up real fast up north and use it as a reference to get REAL customers". They didn't figure in the fact that STUK (the Finnish radiation authority) is _really_ fucking good at what they do and don't cut any corners for any reason resulting in the French having to build many parts twice because the first attempt was subpar.
Oh, this is an area I know something about. I work on railway software in my day job.
Broadly speaking you are correct. Expertise like mine is rare and fleeting mostly because you can only really build it long term by working at a company which can convince international clients to take them on. Even large countries tend not to have more than a handful of trains being built on any given day of the week.
This is one place where having a business located in a nation long known for its relative neutrality, calmness and international trustworthiness can pay off. All of the Nordics are good at this, really.
> What happened next, you may not be surprised to hear, comes in different versions. The person who spotted that there might be a problem may have been a member of Her Majesty’s Constabulary…
>> While they were away, a passing policeman noticed an extraordinary whirlpool in the normally placid canal. He also noticed that the water level was falling. He rushed off to find the dredging gang. By the time they all returned, the canal had disappeared. It was then that realisation dawned. Jack and his men had pulled out the plug of the canal. One-and-a-half miles of waterway had gone down the drain.
> It may have been three anglers who raised the alarm, and given that they have names — Howard Poucher, Graham Boon and Pete Moxon — maybe that version’s true. Another telling says it wasn’t until the evening that
>> local police contacted Stuart Robinson, the British Waterways section inspector.
Perhaps tangentially related Re: “Chesterfield’s plug", Chesterton’s fence came to mind today while mulling over this sort of “forgetfulness” (which tends toward outright negligence) in my own circles.
This is often made worse as a result of hiring outside consultants. Firstly they don't have the institutional knowledge you have when starting a project, but they also aren't incentivised to properly document and hand over their knowledge at the end since that means less future work.
This is why a lot of government projects take so long, they don't see the value in keeping an in-house team of trained experts (see the difference in train line contruction costs in the UK compared to Spain), until you realised how good they were but you can't hire them back.
I am currently on a team of consultants. Ironically, most of us have more institutional knowledge than the client due to internal churn. Seems like every few years they try to cut our utilization in favor of some off-shore company that's "cheaper", the project blows up, then we have to jump in and save some middle manager's job.
Back when Anderson Consulting hadn't yet disgraced itself, a corp I worked for hired them for a cost-cutting project. "Find ways we can cut costs."
Blew my mind that 22-year olds, fresh out of good-brand universities (their "qualification"), were doing the research on how to cost-cut. Chesterton's fences all over the place were violated. It was sad watching the slow-moving disaster.
I suspect this is why it's good for the USA to be constantly at war. If you're only at war occasionally, you forget how to make war and can lose. If you're at war constantly, you'll remember how to do it.
Too much emphasis on documentation. It's people that matter.
If you build the sort of culture where people hang around, they will occasionally have time to tell each other the internal folklore. "When I started, an old guy told me about the plug under the canal".
People who work with software know this. Yeah, there are documents describing the system. No, reading them does not mean you understand the system.
Alas, it's an intangible, and doesn't get counted with the rest of the beans.
I had long term business relationship with a company, originating and developing a product for them.
From 50 - 1000 employees things worked very well. There was a great deal of continuity in the relationship. Lots of trust and flexibility in both directions. Our product quickly became the best available, by a long margin, and for a couple decades.
But after they passed about 1500 - 2000 employees they got more organized. A formalized organization and process system. Things quickly went downhill. As someone working from outside the company, their processes were incredibly disruptive and inefficient for me. Likewise, their turnover replaced a situation of working with long time friendly colleagues, who knew me very well, to working with people who had no idea what my positive reputation was, my track record of delivering quality without the hammer of conformance, etc.
The project's ambitious upward trajectory stalled. Even then it took about ten years to fall behind other players. But it never recovered. Today it operates deep in the shadows of others.
Virtually every employee I worked with was wonderful, inclined to be as supportive as restrictions allowed, etc. But the institutionalization smothered the organizations ability to operate with any flexibility, no matter how dysfunctional or value destroying the results.
The company became like someone who has permanently lost the ability to form new memories.
You can't build anything special with someone who keeps forgetting any context. I spent many years cycling between depression and resurrected determination trying. But finally gave up.
> The company became like someone who has permanently lost the ability to form new memories.
I like this. But reality is more like companies have Alzheimer's disease. They're losing memory on top of not being able to form new ones. Slowly at first and when people notice the symptoms it's already too late.
That story is very Chesterton Fence. If Chesterton was working in a canal instead of walking on a country path. There's a balance between preserving memory and maintaining and benefitting from that knowledge and choosing what not to remember.
When Kurt Cobain shot and killed himself in 1994, his widow went on TV to say that what he did was wrong. Cobain’s death then did not result in others’ killing themselves (known as the “Werther effect”). Robin William’s suicide 20 years later, however, did result in more deaths as the story spread widely.
But otherwise, I do agree that we should preserve institutional memory and that putting processes over people can lead to forgetting.
This is one of the biggest consequences of layoffs in corporations. There's this misconception that everything can and is "objectively" quantified, and thus layoffs targeting otherwise well-performing individuals are being done because this will quantifiably save the institution money and resources. Then something inevitably happens where someone they previously let go could've saved the cost of their employment and then some in damages, but the company is often too blind to realize this.
Thing is, I've seen this time and time again. A lot of us have, I suspect, seen this story repeatedly in our own current or prior organizations. Someone who worked for the company for a decade, or who had intricate knowledge of prior M&As, technology stacks, codebases, customers, and/or processes who was thrown out as a line on a spreadsheet.
I do my best to buck the trend in my work by documenting everything (the "bus problem", as I call it) I can and sharing it with my colleagues, but the continuous churn of M&As and software deprecation means that documentation is often discarded with old systems rather than reviewed and preserved, thus further erasing any lingering institutional memory.
To be fair, this issue isn't likely to kill a company outright on its own. Sure, it could lead to a serious problem and cost gobs of money, but it rarely kills a company or project outright in the process. Still, it's preventable harm just by keeping some additional persons around for knowledge or managing an organizational library of content. It's ultimately such a minor cost in the grand scheme of things that shareholders won't really care. $1m a year for a corporate library and a handful of staff to support it is peanuts on a multi-billion dollar enterprise balance sheet, and will almost certainly improve outcomes across the organization as a whole.
Or to put it far more simply: institutional memory is the fat on an animal. Cutting fat down to the bone leaves the animal weaker and vulnerable as a result, as it has no emergency stores of energy (or in this case, knowledge) to pull from and thus must cannibalize itself in times of crisis.
There is a (possibly apocryphal) story of cars being specified to understand a 100kmh air speed on the rear windscreen. 'Ridiculous, it can't reverse at more than 30kmh said the car designers' and ignored the spec. The first time new cars were transported on a train, all the rear windscreens blew in.
A long time ago I worked on a software product to try to record design decisions in the creation of long-lived artefacts, such as nuclear reactors. The idea being that engineers looking to make a change 20+ years later (when the original engineers had retired) would understand why something had been designed the way it had.
The project was not a success, despite some initial enthusiasm from some commercial sponsors. I think this was due to 2 main issues:
a) The software infrastructure of the days wasn't really up to it. This was just before wikis, intranets etc, which would have made everything a lot easier.
b) The engineers working on the design had no incentive to record the rationale of their decisions. It was extra work with no benefit for them (any benefit was by someone else, years down the line). In fact it could make it more likely for them to be held liable for a bad decision. And, in an age of cheap outsourcing, it could reduce their job security.
The second problem was by far the more important and I don't know how you get around it.
c) Work tends to proceed in drafts/iterations, with earlier initial ideas refined and processed over time. You can record the reasons behind the initial ideas, but eventually those ideas become implicit background and new ideas are diffs against them. It's hard to make all of that explicit and keep it up to date.
You'd think something like wikis would help here because you can see the history. But eventually as you move around in the design space you can make moves that seem obvious but cut across multiple dimensions simultaneously. And it's hard to note something like that well in a wiki commit.
A long form lab notebook (or doc) is often useful here.
It seems that a lot of businesses barely function. They're often stuffed with overpaid executives while the actual business wheezes along, barely managing to get its product out the door. More attention is usually paid to reducing competition, increasing one's moat, and restricting supply, so customers have little choice, as in the aerospace industry from this article.
1. Institutional memory does seem important. It feels like lots of government things are bad at this – big infrastructure projects tend to come in occasional bursts which means each time they are learning from scratch; Japan moves lots of civil servants around every few years which means that no one really remembers how to do things.
2. I think there is a negative side of this too, a kind of ‘institutional trauma’ where some bad memory can cripple an institution. Eg one reason Microsoft lost so much to Google in the early Internet was the memory of the late ’90s antitrust action making them less aggressive. Other companies can have one particular close shave which then causes them to focus too much on avoiding a repeat, a situation you also see writ small in tech teams.
3. I think a bit about production incidents in tech too here. When things are small and the systems are relatively new and they break a lot, this may be ok for the business and recovery can hopefully be fast because it is possible to quickly hypothesise / fix stupid problems. When most silly bugs have been squashed and systems are big and reliable, problems can snowball faster, the business may be more sad about them happening, people can’t understand the whole picture well enough to have good ideas, and the lower base rate of incidents means people will be more stressed or otherwise unable to focus on the actual problem
SpaceX does well on this front. There are four teams of workers at Boca Chica, working around the clock on 8-hour shifts that overlap. This helps continuity as well as sharing of knowhow.
I worked for a 100-year-old Japanese optical equipment manufacturer. It had a lot of institutional memory (detractors like to call it “tribal memory.” I guess, because it makes it sound more primitive).
Lots of “wise old men,” lots of rituals, and “we just do things this way” practices. Long apprenticeships, tons of training, and a metric shitton of paper trails.
But I could always get an explanation for why something was the way it was. Not many Chesterton’s Fences, and some damn interesting stories.
It could be maddening, especially for a software engineering manager (Yours Truly), but they got outstanding results. In fact, they suffered some real damage, when they tried to leave a lot of that behind, and have since returned to “the old ways.”
29 comments
[ 3.8 ms ] story [ 45.6 ms ] threadNobody had built a nuclear reactor in ages. The last ones were from the 80's and this was a completely new technology (EPR). There was no institutional knowledge.
It didn't help that the French attitude to building was to "just slap it up real fast up north and use it as a reference to get REAL customers". They didn't figure in the fact that STUK (the Finnish radiation authority) is _really_ fucking good at what they do and don't cut any corners for any reason resulting in the French having to build many parts twice because the first attempt was subpar.
[0] https://en.wikipedia.org/wiki/Olkiluoto_Nuclear_Power_Plant [1] https://en.wikipedia.org/wiki/Radiation_and_Nuclear_Safety_A...
Broadly speaking you are correct. Expertise like mine is rare and fleeting mostly because you can only really build it long term by working at a company which can convince international clients to take them on. Even large countries tend not to have more than a handful of trains being built on any given day of the week.
This is one place where having a business located in a nation long known for its relative neutrality, calmness and international trustworthiness can pay off. All of the Nordics are good at this, really.
> What happened next, you may not be surprised to hear, comes in different versions. The person who spotted that there might be a problem may have been a member of Her Majesty’s Constabulary…
>> While they were away, a passing policeman noticed an extraordinary whirlpool in the normally placid canal. He also noticed that the water level was falling. He rushed off to find the dredging gang. By the time they all returned, the canal had disappeared. It was then that realisation dawned. Jack and his men had pulled out the plug of the canal. One-and-a-half miles of waterway had gone down the drain.
> It may have been three anglers who raised the alarm, and given that they have names — Howard Poucher, Graham Boon and Pete Moxon — maybe that version’s true. Another telling says it wasn’t until the evening that
>> local police contacted Stuart Robinson, the British Waterways section inspector.
Institutional memory is not information or documents - it's people.
Every single real-world process has implicit knowledge. And you can't always capture that knowledge of paper.
But, many corporations seem to want to get rid of their most experienced people to save money and have better quarterly results for the stock market.
https://en.wikipedia.org/wiki/G._K._Chesterton#Chesterton's_...
Solid writing.
This is why a lot of government projects take so long, they don't see the value in keeping an in-house team of trained experts (see the difference in train line contruction costs in the UK compared to Spain), until you realised how good they were but you can't hire them back.
Blew my mind that 22-year olds, fresh out of good-brand universities (their "qualification"), were doing the research on how to cost-cut. Chesterton's fences all over the place were violated. It was sad watching the slow-moving disaster.
To some people, it looks like a lot of wasted money.
If you build the sort of culture where people hang around, they will occasionally have time to tell each other the internal folklore. "When I started, an old guy told me about the plug under the canal".
People who work with software know this. Yeah, there are documents describing the system. No, reading them does not mean you understand the system.
Alas, it's an intangible, and doesn't get counted with the rest of the beans.
From 50 - 1000 employees things worked very well. There was a great deal of continuity in the relationship. Lots of trust and flexibility in both directions. Our product quickly became the best available, by a long margin, and for a couple decades.
But after they passed about 1500 - 2000 employees they got more organized. A formalized organization and process system. Things quickly went downhill. As someone working from outside the company, their processes were incredibly disruptive and inefficient for me. Likewise, their turnover replaced a situation of working with long time friendly colleagues, who knew me very well, to working with people who had no idea what my positive reputation was, my track record of delivering quality without the hammer of conformance, etc.
The project's ambitious upward trajectory stalled. Even then it took about ten years to fall behind other players. But it never recovered. Today it operates deep in the shadows of others.
Virtually every employee I worked with was wonderful, inclined to be as supportive as restrictions allowed, etc. But the institutionalization smothered the organizations ability to operate with any flexibility, no matter how dysfunctional or value destroying the results.
The company became like someone who has permanently lost the ability to form new memories.
You can't build anything special with someone who keeps forgetting any context. I spent many years cycling between depression and resurrected determination trying. But finally gave up.
I like this. But reality is more like companies have Alzheimer's disease. They're losing memory on top of not being able to form new ones. Slowly at first and when people notice the symptoms it's already too late.
When Kurt Cobain shot and killed himself in 1994, his widow went on TV to say that what he did was wrong. Cobain’s death then did not result in others’ killing themselves (known as the “Werther effect”). Robin William’s suicide 20 years later, however, did result in more deaths as the story spread widely.
But otherwise, I do agree that we should preserve institutional memory and that putting processes over people can lead to forgetting.
Thing is, I've seen this time and time again. A lot of us have, I suspect, seen this story repeatedly in our own current or prior organizations. Someone who worked for the company for a decade, or who had intricate knowledge of prior M&As, technology stacks, codebases, customers, and/or processes who was thrown out as a line on a spreadsheet.
I do my best to buck the trend in my work by documenting everything (the "bus problem", as I call it) I can and sharing it with my colleagues, but the continuous churn of M&As and software deprecation means that documentation is often discarded with old systems rather than reviewed and preserved, thus further erasing any lingering institutional memory.
To be fair, this issue isn't likely to kill a company outright on its own. Sure, it could lead to a serious problem and cost gobs of money, but it rarely kills a company or project outright in the process. Still, it's preventable harm just by keeping some additional persons around for knowledge or managing an organizational library of content. It's ultimately such a minor cost in the grand scheme of things that shareholders won't really care. $1m a year for a corporate library and a handful of staff to support it is peanuts on a multi-billion dollar enterprise balance sheet, and will almost certainly improve outcomes across the organization as a whole.
Or to put it far more simply: institutional memory is the fat on an animal. Cutting fat down to the bone leaves the animal weaker and vulnerable as a result, as it has no emergency stores of energy (or in this case, knowledge) to pull from and thus must cannibalize itself in times of crisis.
A long time ago I worked on a software product to try to record design decisions in the creation of long-lived artefacts, such as nuclear reactors. The idea being that engineers looking to make a change 20+ years later (when the original engineers had retired) would understand why something had been designed the way it had.
The project was not a success, despite some initial enthusiasm from some commercial sponsors. I think this was due to 2 main issues:
a) The software infrastructure of the days wasn't really up to it. This was just before wikis, intranets etc, which would have made everything a lot easier.
b) The engineers working on the design had no incentive to record the rationale of their decisions. It was extra work with no benefit for them (any benefit was by someone else, years down the line). In fact it could make it more likely for them to be held liable for a bad decision. And, in an age of cheap outsourcing, it could reduce their job security.
The second problem was by far the more important and I don't know how you get around it.
c) Work tends to proceed in drafts/iterations, with earlier initial ideas refined and processed over time. You can record the reasons behind the initial ideas, but eventually those ideas become implicit background and new ideas are diffs against them. It's hard to make all of that explicit and keep it up to date.
You'd think something like wikis would help here because you can see the history. But eventually as you move around in the design space you can make moves that seem obvious but cut across multiple dimensions simultaneously. And it's hard to note something like that well in a wiki commit.
A long form lab notebook (or doc) is often useful here.
1. Institutional memory does seem important. It feels like lots of government things are bad at this – big infrastructure projects tend to come in occasional bursts which means each time they are learning from scratch; Japan moves lots of civil servants around every few years which means that no one really remembers how to do things.
2. I think there is a negative side of this too, a kind of ‘institutional trauma’ where some bad memory can cripple an institution. Eg one reason Microsoft lost so much to Google in the early Internet was the memory of the late ’90s antitrust action making them less aggressive. Other companies can have one particular close shave which then causes them to focus too much on avoiding a repeat, a situation you also see writ small in tech teams.
3. I think a bit about production incidents in tech too here. When things are small and the systems are relatively new and they break a lot, this may be ok for the business and recovery can hopefully be fast because it is possible to quickly hypothesise / fix stupid problems. When most silly bugs have been squashed and systems are big and reliable, problems can snowball faster, the business may be more sad about them happening, people can’t understand the whole picture well enough to have good ideas, and the lower base rate of incidents means people will be more stressed or otherwise unable to focus on the actual problem
Lots of “wise old men,” lots of rituals, and “we just do things this way” practices. Long apprenticeships, tons of training, and a metric shitton of paper trails.
But I could always get an explanation for why something was the way it was. Not many Chesterton’s Fences, and some damn interesting stories.
It could be maddening, especially for a software engineering manager (Yours Truly), but they got outstanding results. In fact, they suffered some real damage, when they tried to leave a lot of that behind, and have since returned to “the old ways.”
[1] https://web.archive.org/web/20111228105122/http://wrttn.in/0...
[2] https://news.ycombinator.com/item?id=42235452