These "invisible disasters" are an almost daily occurrence in software, even within a single sector or geography. Multi-hundreds million dollar projects that are either severely malformed when launched or that collapse entirely before being placed in the hands of a user. I wrote a little about them in a previous post. Have things improved?
Silly question.
https://news.ycombinator.com/item?id=17844712
And if you follow the link at the bottom of that post, you'll find what the thermocline consists of:
>There was upper Apple management, basking in the sun; the great mass of Apple employees below; and between the two, a constantly shifting layer of middle managers whose function appeared to be to keep the two groups from mixing.[0]
"Paula Vennells, who took over as Chief Executive in 2012 with, by all accounts, the best of intentions, found herself surrounded by a senior management team desperate to protect their own reputations and that of the Royal Mail. They told her that there simply couldn’t be problems with the IT system on the scale that was being suggested; it was much more likely that the sub-postmasters were colluding, lying about IT bugs in order to escape responsibility for what was at best incompetence and at worst theft."
Hanlon's razor would have been useful here. How could anyone honestly believe that mass collusion amongst a distributed and isolated group of people, that was also perfectly successful, was more likely than an IT failure?
She had a choice - either go against surrounding her "senior management team desperate to protect their own reputations and that of the Royal Mail" or blame those lowly sub-postmasters. It isn't about "honestly believing", it is what people to side with - your own who is just like you or the very different others.
>They told her that there simply couldn’t be problems with the IT system on the scale that was being suggested
and thus they provided her with plausible deniability. Like that Enron CEO who supposedly didn't know the financial details and relied on what is told by his CFO/etc.
The whole article sounds like a whitewash of senior management responsibility by blaming it on the lower ranks who reports "rosy picture". No wonder that the author seems to be a successful consultant - i can see how his expert opinion may be helpful for such a whitewashing by establishing the existence of the "thermocline of truth" in the specific situations what he gets brought into and thus is liked by the upper management.
As an extra example of "thermocline of truth" - that EBay CEO discussed again just yesterday who supposedly didn't know how his security chief with the gang were criminally harassing the blogger to carry out the CEO's order to “take her down”.
I think approaches like RICO or SOX or plain old army principle that the CO is responsible for whatever happens under his command are the examples of the tools to pierce the "thermoclines of truth".
The article doesn’t blame the underlings that report a “rosy picture”, but rather blames the culture that heavily incentivizes reporting a rosy picture regardless of the facts on the ground, and punished everything else. And the people with the most power over this culture are the ones at the top, making them the most culpable for the thermocline.
> How could anyone honestly believe that mass collusion amongst a distributed and isolated group of people
This is pure speculation and I don't want to sound like I'm supporting the Post Office and Fujitsu's behaviour, which was despicable.
I wondered if their previous systems were, internally, considered to be very flaky. You then get a new shiny system with a very different set of results and instead of the response being "why is the old system and new system disagreeing so badly" you get the response "hey, this is great, the new system has revealed wrong doing the old system was not able to".
IANAAccountant, but I have run systems that fed into G/L systems. If the G/L were set up halfway correctly, everyone in the executive suite would have known that Horizon was producing bad data. Most transactions at a post office would either have had effects on inventory (buying stamps or stationery) or on liabilities (if a customer paid to send a package, someone would have to actually deliver that package). "Duplicate" transactions would only escape notice for services provided immediately without reducing inventory, like a haircut. I doubt many people are getting their hair cut at the post office.
This whole episode calls to mind the authoritarian reaction to "impossible" hacks of the various chip'n'pin schemes rolled out in UK over the years. The credit cards were assumed to be perfect, so account holders were held responsible when they got hacked. [0] In general, individuals in conflict with giant organizations should have access to appeals outside those organizations.
> They are the inevitable consequence of an organisation that puts more emphasis on reputation than truth; that rewards good news and punishes bad; that has a leadership team disconnected from the delivery of the actual work; that instils fear and compliance with process into its employees, rather than a desire to do the right thing.
This is so endemic to all organizations.
When you shoot messengers, expect to receive fewer messages.
> When you shoot messengers, expect to receive fewer messages
Most modern organizations don't have pure messengers. The king is his messenger. That can blur the line between accountability and transparency. For example, in this case, the messengers were the senior managers partly responsible for the mess.
Good managers clearly delineate that boundary. Bringing bad news isn't punished. But making bad calls alone may be. In practice, this takes discipline to do.
Under "further reading", TFA links a more complete BBC piece [0], which documents that the sub-masters got screwed twice:
In December 2019, at the end of a long-running series of civil cases, the Post Office agreed to settle with 555 claimants. It accepted it had previously "got things wrong in [its] dealings with a number of postmasters", and agreed to pay £58m in damages. The claimants received a share of £12m, after legal fees were paid.
Those lawyers abused them worse than the Royal Mail did.
Although this piece refers to Post Office management a lot I seem to remember that one or more judges have laid blame at Fujitsu's door as well.
In my mind the behaviour of the Post Office was very, very poor but there are people within Fujitsu who had, and have, a much greater understanding of the modes of failure that might have resulted in the discrepancies that were being seen and yet, as I understand it, they went along with Post Office as they trashed people's reputations, livelihoods and threatened their freedom.
> The danger of the thermocline of truth is that, if you’re above it, it’s difficult to know it. Is the rosy picture of reality you’re hearing from subordinates genuine? Or are you heading for an iceberg that everyone can see but you?
Last time I gave an honest assessment of how far behind I was and why I got put on an "improvement plan". I've never been as exact since. I've been as behind as before but I don't call it out.
How can one eliminate a thermocline of truth from an organization? Ideas:
- Have every exec spend one day a week with leaf-node workers, just learning and asking questions.
- Rotate all middle managers regularly. When a project has problems, blame is shared by several managers, making it easier for the current manager to admit failure. When a manager rotates off a project, they lose their influence and ability to under-report problems. The new manager is incentivized to look closely for problems and over-report them.
- Do not allow middle managers to determine promotions and bonuses of managers and leaf-node workers. Have the execs do it, or hire someone specifically to do it. Without this influence, middle managers lose some ability to suppress bad news.
- Split your company into several smaller companies and eliminate all middle managers. This limits the size of the organization to about 1,000 people: CEO x 10 VPs x 10 managers x 10 leaf-node workers.
- Have execs and board members interview exiting leaf-node workers.
- Hire an outside company to provide an anonymous whistle-blowing service with big cash rewards for being the first to report false progress reports and covered-up problems. Does a company like this exist?
21 comments
[ 5.2 ms ] story [ 55.8 ms ] thread>There was upper Apple management, basking in the sun; the great mass of Apple employees below; and between the two, a constantly shifting layer of middle managers whose function appeared to be to keep the two groups from mixing.[0]
[0] http://andstillipersist.com/2006/08/guerrilla-programming-li...
Hanlon's razor would have been useful here. How could anyone honestly believe that mass collusion amongst a distributed and isolated group of people, that was also perfectly successful, was more likely than an IT failure?
>They told her that there simply couldn’t be problems with the IT system on the scale that was being suggested
and thus they provided her with plausible deniability. Like that Enron CEO who supposedly didn't know the financial details and relied on what is told by his CFO/etc.
The whole article sounds like a whitewash of senior management responsibility by blaming it on the lower ranks who reports "rosy picture". No wonder that the author seems to be a successful consultant - i can see how his expert opinion may be helpful for such a whitewashing by establishing the existence of the "thermocline of truth" in the specific situations what he gets brought into and thus is liked by the upper management.
As an extra example of "thermocline of truth" - that EBay CEO discussed again just yesterday who supposedly didn't know how his security chief with the gang were criminally harassing the blogger to carry out the CEO's order to “take her down”.
I think approaches like RICO or SOX or plain old army principle that the CO is responsible for whatever happens under his command are the examples of the tools to pierce the "thermoclines of truth".
This is pure speculation and I don't want to sound like I'm supporting the Post Office and Fujitsu's behaviour, which was despicable.
I wondered if their previous systems were, internally, considered to be very flaky. You then get a new shiny system with a very different set of results and instead of the response being "why is the old system and new system disagreeing so badly" you get the response "hey, this is great, the new system has revealed wrong doing the old system was not able to".
As I say it's a guess but I just wondered.
This whole episode calls to mind the authoritarian reaction to "impossible" hacks of the various chip'n'pin schemes rolled out in UK over the years. The credit cards were assumed to be perfect, so account holders were held responsible when they got hacked. [0] In general, individuals in conflict with giant organizations should have access to appeals outside those organizations.
[0] https://www.lightbluetouchpaper.org/2012/09/10/chip-and-skim...
This is so endemic to all organizations.
When you shoot messengers, expect to receive fewer messages.
Most modern organizations don't have pure messengers. The king is his messenger. That can blur the line between accountability and transparency. For example, in this case, the messengers were the senior managers partly responsible for the mess.
Good managers clearly delineate that boundary. Bringing bad news isn't punished. But making bad calls alone may be. In practice, this takes discipline to do.
In December 2019, at the end of a long-running series of civil cases, the Post Office agreed to settle with 555 claimants. It accepted it had previously "got things wrong in [its] dealings with a number of postmasters", and agreed to pay £58m in damages. The claimants received a share of £12m, after legal fees were paid.
Those lawyers abused them worse than the Royal Mail did.
[0] https://www.bbc.com/news/business-56718036
In my mind the behaviour of the Post Office was very, very poor but there are people within Fujitsu who had, and have, a much greater understanding of the modes of failure that might have resulted in the discrepancies that were being seen and yet, as I understand it, they went along with Post Office as they trashed people's reputations, livelihoods and threatened their freedom.
There's a simple heuristic:
Are you only hearing good news? That's bad news.
- Have every exec spend one day a week with leaf-node workers, just learning and asking questions.
- Rotate all middle managers regularly. When a project has problems, blame is shared by several managers, making it easier for the current manager to admit failure. When a manager rotates off a project, they lose their influence and ability to under-report problems. The new manager is incentivized to look closely for problems and over-report them.
- Do not allow middle managers to determine promotions and bonuses of managers and leaf-node workers. Have the execs do it, or hire someone specifically to do it. Without this influence, middle managers lose some ability to suppress bad news.
- Split your company into several smaller companies and eliminate all middle managers. This limits the size of the organization to about 1,000 people: CEO x 10 VPs x 10 managers x 10 leaf-node workers.
- Have execs and board members interview exiting leaf-node workers.
- Hire an outside company to provide an anonymous whistle-blowing service with big cash rewards for being the first to report false progress reports and covered-up problems. Does a company like this exist?