Organizations don’t choose whether or not to ask questions. People do. The behavior of an organization is the sum of the actions of the people in it. If you are a member of an organization, and you ask ‘how do we prevent this from happening again?’ Guess what? The organization just learned how to ask that question.
Now it just needs to learn how to answer it, and act on the answer.. but step one seems solvable.
If the person who is charged with resolving the issue is capable, there's no point in a senior manager knowing the details outside of professional curiosity.
Yes, if a director and SVP are discussing the technical details of an incident, I would be concerned -- they should be concerned with putting the right people and processes in place to make those decisions.
When "Something had gone wrong that shouldn't have" bubbles up that far, you have an organizational issue, not a technical issue.
The quote box at the start of the article is the essence of it, the rest is AI-speak dressing it up.
TLDR:
Some outage causes a meeting with an SVP. SVP preempts the discussion with the following:
“I know that if we get into the details, the reasons will be perfectly reasonable. You'll explain what happened, I'll understand why everyone made the decisions they made, and I'll empathise with you.
Then it'll happen again.
So I don't want the details. I want to know what we're changing.”
> Then I realised that "I don't want the details" wasn't being dismissive. The executive assumed that we were competent, and was saying "I already believe you. Now let's talk about what happens next".
Something about this feels wrong:
- If you trust them completely, you don't need to know what happens next either. Just trust them to do the right things, your leadership isn't required.
- If you don't trust them completely, how can you know if "what happens next" is appropriate without knowing the details?
Isn't this just reinforcing the idea that leadership doesn't need to have their feet on the ground?
The best leaders I've worked with were paying attention to things from the bottom-up as well as top-down.
Also, not every IC or EM has the same skills and experience. There may be certain categories of work that you can be 100% hands-off and trust them to just do it, and others where you really need to understand bottom-up. Part of being an eng leader is to know when to insert yourself and when to "trust the process".
You can trust someone to know the details and reasons for why something happened, and still want to provide the specific direction of "what are you doing to make sure it doesn't happen again". The organizational default is often to explain a fault (or worse, focus on blame) without specifically working to find a process improvement to prevent it from happening again.
> If you trust them completely, you don't need to know what happens next either. Just trust them to do the right things, your leadership isn't required.
> The best leaders I've worked with were paying attention to things from the bottom-up as well as top-down.
These two statements are at odds. You first assuming that lack of trust would be the only reason he needs to know what happens next, then go on to outline another reason he and the larger organization would both benefit from knowing more.
It's not just about trust. It's about knowing what your team is up to so you can provide them the resources necessary while also working to remove any obstacles in their way so they can get shit done. That's what good management does, it's just rare to see in low-trust organizations that treat everyone like a cog to be micromanaged because no one trusts anyone to get shit done of their own accord for whatever reason.
> These two statements are at odds. You first assuming that lack of trust would be the only reason he needs to know what happens next, then go on to outline another reason he and the larger organization would both benefit from knowing more.
But doesn't that same logic apply to whether they need to know the details?
No. Leaderships job in this context is to set expectations of what happens next and to drive to team towards a solution that meets those expectations. No amount of trust magically makes that expectation materialize or for it to be met.
So if I'm understanding you correctly, you're saying there's literally no reason for a leader to need to know the details ever unless they don't trust the people implementing the plan. I don't agree with that at all.
Leaders should not need to know every single detail (ie how this incident happens, or all the tiny process changes that are happening toward their visions). That is different than knowing nothing in detailed.
The baseline expectation in the original post also applies to the SVP: he knew enough to know that things aren't unreasonable to start with, this expectation might not apply to all companies.
An analogy that might make more sense is horizontal scalability vs. vertical scalability. A leader's ability to understand all the details of the proposal brought to them by the team inherently caps the organization's ability to scale up.
Offloading the detail onto the team allows the org to scale higher by allowing the leader to bring more teams/problems into the mix. But it also introduces the complexity of keeping those teams in close enough alignment that the overall effect is still positive.
This is by no means guaranteed to be positive, and there's a lot of details to get right for leadership in making this work. But they are different details than what each individual team was bringing to leadership.
Abstractions are helpful, in business management as much as in systems architecture.
The top-level comment that started this discussion was mentioning that a "leader" said "I don't need to know the details" but that they wanted to know what comes next. The comment I responded to claimed that there are valid reasons to need to know what comes next that don't stem from a lack of trust. I asked whether the same logic should apply to wanting to know the details (since to me it seems pretty reasonable that it should). The response said "no".
It seems like you're trying to rebut my argument by presenting my argument as being against a less strong version of the original claim rather than understanding it as being literally just a counterargument to the specific version of the claim I thought was too strong.
Leaders have the option of trusting subordinates, then operating at a higher level. However part of the job of a leader is to decide who to trust, with what. And why.
Sometimes that means, "Trust, but verify." The leader will spot check randomly to see that the whole is good.
Often, as here, that means, "Trust in some things, but not others." The leader in this case trusted the technical person to produce an accurate description of what happened. But didn't trust that same person to arrive at a solution that met the business of having a similar disaster not happen the next time.
But rather than undermine the tech person with, "I don't trust you to meet the business need," the leader communicated it with, "I trust you on the technical details, but here is the expectation for what you need to produce."
The statement I initially responded to was "You first assuming that lack of trust would be the only reason he needs to know what happens next, then go on to outline another reason he and the larger organization would both benefit from knowing more". I pointed out that the same logic seems like it would apply to the idea that there might be other reasons why the need to know the details. They said "no", and then a bunch of other stuff that you seem to think that I'm not reading. I'm reading it, and I'm reading what you said to add onto their comment, I just don't understand how you can claim their answer of "no" to my question doesn't mean "no" to the restatement of my question.
What do you think knowing the details will do for the person in the leadership position? It doesn't change what needs to be done, it doesn't change what has been done, all it usually does it put a monkey on the leaders back that's best kept with the team that created it.
Worst still it invites bike shedding most of the time, where folks sit in a room nitpicking pointless details that's cathartic for the people in charge, but once again, doesn't actually do anything productive.
> The statement I initially responded to was "You first assuming that lack of trust would be the only reason he needs to know what happens next, then go on to outline another reason he and the larger organization would both benefit from knowing more". I pointed out that the same logic seems like it would apply to the idea that there might be other reasons why a leader might need to know the details.
I'm genuinely confused why this is apparently not clear. Someone said that it's flawed logic to assume there aren't good reasons beyond a lack of trust to want to know next steps. I'm saying it seems reasonable there could be good reasons beyond a lack of trust to want to know the details of the current step. You're saying that I'm finding it difficult to "grok this concept", but I don't even know what concept you're talking about, because none of the responses seem to have been related to this, and I don't understand what anyone else could think I've been saying.
We are arguing because there is an IT maturity line that goes from chaos to corporate and the more like a startup the more it will be like chaos (by design because maturity is too restrictive). So horses for courses. But as the team grows the same lessons get learned everywhere you bring in ex-bigco to implement changes to be more mature. That will include incident management process that will make this situaton routine and not worth blogging about. Probably in a mature process leaders already read the summary and or postmortem if they felt they needed to.
In a large enough organization it is simply impossible for executives to be on the ground floor with everyone else. And not only is it impossible, it is essentially a waste of their time. They could maybe be maximally involved in only a few projects at a time.
I think its entirely contextual as to what the size of the org is and the scope of the projects. At a startup an "executive" or VP would legitimately be able to be working side by side with everyone else, but just by nature of the role being SVP I imagine this is a large enough company that this is no longer the case.
The role entirely morphs and their job becomes to enable teams, get blockers out of the way, fast track or assist things like permissions/processes, shoot down ideas that legitimately don't add business value or are not worth wasting existing engineering effort on.
And I imagine in this scenario the point wasn't for the executive to literally hear what happens next, it was to ensure the proper steps are being taken and make sure "somebody" is handling the what happens next part of things, and so they can report up through the chain or delegate tasks for exactly that "what happens next" task.
> In a large enough organization it is simply impossible for executives to be on the ground floor with everyone else. And not only is it impossible, it is essentially a waste of their time. They could maybe be maximally involved in only a few projects at a time.
This is a very strong argument that no organization should be allowed to get that big.
I get the sentiment, but this is the reason we have airplanes, MRI machines, and computers that fit in our pockets. Advanced technology simply isn't possible in an economy of only ~100 person organizations. And I say this as someone with a deep sense of unease with modernity and all of the diseases of civilization. We somehow have managed to create systems that support literally billions of humans and allow many of them to have a quality of life undreamed of even a few hundred years ago. I dislike big orgs as much as they next guy but we need to understand why they exist if we want to criticize and reform them.
Big organizations don't enable greater technological progress, which is why companies do 98% of their innovation before they get huge, and get bad at innovating once they are.
Big organizations enable scaling and associated economies of scale, so it's possible for Apple to sell millions of iPhones cheaper than if they were a smaller company. I don't think that tradeoff is worth it for any individual consumer (pay a little less to be much more abused by the company), and I don't think it's worth it for society.
Most of the big corporations in America aren't providing things as useful as MRI machines, they're providing net negative garbage like social media and advertising. Big corporations like United Healthcare Group are inflating the cost of medical care rather than providing economies of scale and reduced cost for their clients.
The US desperately needs to bring back antitrust and start breaking these massive companies into much smaller pieces.
Ironically, those same systems that enabled high QoL are also the ones involved in killing us, whether that's carcinogens everywhere, climate change, resistant bacteria, or even just plain old nukes (or soon drone swarms).
I wouldn't be alive if not for advanced medicine, but if the structures that saved my life are the same ones that constitute our own Great Filter, my life was most definitely not worth the trade. There are too many other people I love to think that.
You can build everything in the world with organizations at that size cooperating, but a society that values profit extraction will never organize itself in this manner. The problem isn't org size, it's profit motive.
There may be something common across many of the ground floor parts, such that even just knowing what's going on in a fraction of them, they are much further in their vibe-calibration of what ground floor is like than if they know zero.
They trust your ability to do a root cause analysis. They don’t trust that you feel empowered to do anything about it, so they are verifying that you actually do.
“Trust” is usually bounded. I trust my elementary aged son to get dressed for school without supervision. That doesn’t mean I trust him to start a campfire without supervision. A chef might trust that their new employee is a hard worker and well intentioned. That doesn’t mean they also trust them to create and serve original dishes.
Here you are recasting “I trust that, despite the bad outcome, everyone’s actions here were reasonable” as “I trust that you are going to solve this fully without my guidance”. The fact that the word “trust” appears in both of these statements does not make them equivalent. Trusting that people are capable and well intentioned does not mean trusting that they will accomplish the same thing without you.
Yup, several things are wrong with it even though it may be a good idea as a good shortcut on the part of the SVP
But it should be used and phrased better.
Used better:trust and only sometimes ask for the detials, like cutting the just-shuffled deck in a card game, sometimes the player to the dealer's right will just tap the deck instead of cutting it, in effect saying "I trust that shuffle". Sometimes, certainly enough to both keep the SVP well-grounded in the details, and to keep the team knowing the SVP is both watching and cares about the details, the SVP MUST ask for the details. Trust and verify.
In terms of phrasing, "I don't want the details" is all about the SVP sending the message s/he doesn't want to be bothered. The phrasing should be: "I trust you on the details on this one; let's go straight to what do we do to manage the next one of these?".
Notice leading with "I trust you on this", not "don't waste my time".
> If you trust them completely, you don't need to know what happens next either. Just trust them to do the right things, your leadership isn't required.
I work in a competent team and leadership is required.
A lot of times, people just need a gentle nudge, or an official blessing from an authority figure, to go ahead and make the change that is necessary. You can tell people to manage up, to fully own their work and their process, etc. But a lot of humans don’t want to step on others’ toes or be perceived as acting out of line. A worker on the assembly line might know the best way to make the line run more smoothly but it’s easy to think that’s “not my job” or “above my pay grade”. Sometimes all that’s needed is to tell them, “Yeah, go ahead and do it.”
Getting that to happen more automatically is one way out of this situation, but that tends to cause problems of its own, with blurry lines of responsibility and authority. So it works best with relatively flat organization structures, which aren’t used much in large organizations.
> But a lot of humans don’t want to step on others’ toes or be perceived as acting out of line.
It's conditioning. In all organizations above a certain size (> 100 people), stepping out of line receives an official reprimand. Someone always complains. Receive enough reprimands and the person is terminated. People learn to stay in line because they've been punished whenever they step out of it.
The best leaders know what they need to pay attention to. You cannot know everything. You can know any one thing (or perhaps several), but there are more things to know than you will have time to learn in your lifetime. The question is what is important for you to know, and what you leave to somebody else.
You will have to trust someone will figure out the details and come up with a reasonable solution. Sometimes you need to know that solution, sometimes in detail, sometimes just enough to know that they solved it. Sometimes you are the person who needs to figure out the details and make the solution. Knowing which you are is important.
I would agree with you, this just sets up a scenario where the SVP positions themselves as a leader, without having to understand something technical, and without making the decisions on how to change things. It isnt about trust, it is about performing accountability so the SVP can say they did their job.
I think the author of the article is going out of their way to be charitable to the SVP for no reason. That meeting was a reassertion of power, not a wise move made by a sage.
One reason is prioritization. It’s enough to know what the effects of the failure were, the likely cost of what needs to change, and what areas and processes the changes will affect, to decide if and when this should happen. Leadership still needs to understand these aspects on a conceptual, topical, and cost level to decide on the resource allocation (including time and schedule).
> - If you trust them completely, you don't need to know what happens next either. Just trust them to do the right things, your leadership isn't required.
You might need to know timescales so that you can inform upwards/outwards/other. You might need to make sure other resources that might be necessary are available. The planned "what next" and the proposed timescales might have impacts elsewhere that they haven't realised (and maybe even until now had no reason to be aware of). If they tell you the what next and there are no such complications, then they can carry on, otherwise there might need to be some changes to the plan or changes to other plans to allow time/resources to be available for this plan or just to stop potential near future conflicts.
The devs job is to impliment and advise how to attain the owners goals and priorities, not to set goals or priorities except at layers below, and in the service of, the c suite's directives.
I'm not saying c suite are unquestionable gods, I'm saying that there is no such trust-or-not dichotomy. It's 2 different things.
No matter how ignorant I am in some domain and no matter how knowlegeable some expert is I hire to do something for me in that domain, they can only tell me how to get what I want, or what's the closest that is humanly possible, or the costs of various conflicting priorities and compromises. They can't tell me what to want.
Maybe I DO actually want to burn my whole budget and 5 years of time on some facet that they and most people would say is not important and not worth it to the point of being irrational. Their job is to inform me not to decide for me. I can trust them to inform me honestly and with good judgment and deep knowledge, and then to impliment what I decide also honestly and with competent skill. And yet I still need to be the one who is informed, and then makes a decision and issues a directive.
It's 2 different things.
It's like Steve Jobs being totally unreasonable about tiny details of fit & finish in the hardware products. It makes no sense (to most people) to have custom ethernet jacks manufactured (back when it wasn't so effortless and flexible) just to get them a certain color or material feel, when off the shelf jacks already exist from multiple sources that work just fine for a 10th or 100th the cost, while already meeting a stack of compatibility standards and even safety regulations. No trustworthy engineer would do that. They would actually see it as violating that trust.
Asking what the OP was going to change is not an automatic nod of approval for his approach. I’ve been part of some large interventions spanning multiple quarters, that started from a simple conversation with an exec just like this. There are downstream effects to these decisions that affect many stakeholders and departments - so they cannot be made in a vacuum or delegated completely (I trust you, go ahead and fix whatever). Depends on the severity and scope of the issue of course. A simple deployment gone wrong is different from a fundamental issue with the architecture that is straining progress.
As an example, a major rehaul of a core system could push back product (new feature dev) timelines, that can affect customers and their releases, bottomline of the company, reputation, tax repercussions, if it’s an American public company then it’ll affect SOX compliance & audits, could lead to spicy investor/analyst conference calls, and so on.
First of all, he didn't say the executive "trusted them completely". Those are your words.
He said "The executive assumed that we were competent, and was saying "I already believe you. Now let's talk about what happens next".".
That can be true even if they don't trust them completely. Let's say he trusted them 80% and not 100%. Isn't that pretty much quite trusting anyway?
Besides, the executive can trust them even 100%, but still want them to give him the general picture. Among other things, it's his job to know and be able to report when asked.
The VP also needs to know if the fix will require some sort of resource reallocation, or if the timeline is long so they can deal with the other stakeholders, or whatever. These are all forward looking things that the details influence but the details aren't what matter.
Another explanation is that not wanting the details of operations is based on complete lack of trust, not in the their competence but in the way they would describe things. The boss doesn't want the details because they are sure the details would make it seem like the underling did the right thing and moreover, monitoring that detail would just give the underling leverage over them.
The boss wants to know about changes to avoid this but whatever it is, they'll push back on its extent and won't want to hear about details then either.
That’s not even true though, leadership is coordination, even if you trust someone else to do the right thing wrt their own problems that still may be globally suboptimal wrt the goals of the overall team / project.
A perfectly reasonable explanation is that this SVP needs to explain in a different room what steps are being taken to prevent it from occurring again.
Sometimes the real reason is "overscoped work", "no time to fix tech debt", "information silos", "people being protective rather than collaborative", "people overloaded and losing sleep" -- all problems the SVP needs to fix.
> If you trust them completely, you don't need to know what happens next either. Just trust them to do the right things, your leadership isn't required.
There are many reasons an executive needs to know what’s happening even if they’ve delegated the decision making authority.
Nobody can go to their boss and say “I don’t know what’s being done. I let so-and-so handle everything and they’ll figure it out”. Everyone has to know the big things that are happening.
Second, this is the step where the manager surveys the next steps for potential blockers or opportunities to accelerate. Maybe they notice that the plan could benefit from someone else’s help and they ask that person to contribute.
There’s a lot of overthinking happening in this thread. The “I don’t need to know the details” isn’t a literal statement that covers all details. It’s a way of saying “Skip forward to the part I need to hear about”.
It's crazy how far i had to scroll to find someone saying exactly what i was thinking - how come AI sloppy writing like this is showing up on Hn front page?
The important part is the action items are assigned an owner and executed.
I’ve done plenty of postmortems and many meetings where we go through the motions but “leadership” does not assign and address follow-ups. Most organizations pay lip service to reliability.
Actionable items go onto the backlog. The backlog grows from the top.
As a stopgap, a manual pre-flight check is added. The number of manual pre-flight checks becomes a ridiculous overhead but it gives a clear person to blame when something goes wrong.
I would not have said that. I would have asked for the details and tried to find gaps because, almost always, something could have been done. Then the question is how much risk reduction are you willing to pay for.
How about explain the problem and solution in 3 to 5 levels of details in a top-down hierarchical manner sort of like a decision tree. Let people decide how far they want to drill down.
We call this incident postmortem, part of a company process. After serious problems and firefighting, when the patches and hacks and people working 24 hour and it admins wake on a Sunday morning to do work, we sit down and try to figure out how this will _never_ happen again.
Learned this doing incident reviews. If the exec asks for details it usually means they don't believe you yet, so "skip to what changes" is actually the good outcome.
I suspect the author still misunderstands the SVP. If I were the SVP of engineering saying this to someone, especially someone who is in a non-core engineering role like product, I probably recognize that they have a tendency to verbally spew and I want them to focus on the risk mitigation and response plan. I’ve already talked to my engineering leader because they informed me immediately, and I’ve already read the incident details.
But also the post reads like it comes from an inexperienced/immature service org, so maybe they’re just sorting their operational response process out.
Love this idea; won't go over well at all with the other accountants I work with. MY CFO doesn't like graphs because they "hide too many details' and insists on everything presented to her being in a table. MY VP pulls out a calculator and re-does the math on tables in decks by hand to help him 'understand'.
A lot of leaders like to hide in the details because the bigger picture is a harder problem.
If you don’t know what happened how would you be able to tell if the changes are addressing the problem? If you don’t care about the problem being addressed why jump in a call in the first place?
I can't be the only person who thought this is a highly intelligent person being inadvertently gaslit by the SVP into thinking the exec is right to be dismissive with "I don't want the details" because introspection is common in intelligent people.
Had someone said that to me I'd have walked away after saying "fine I'll sort it'.
This illustrates that "trust" to do the job, and since they didn't want the details of the problem, they therefore don't need the specific solution description. This kind of response is also the same level of respect, it's either seen as trust in ability, or just downright disdain with plausible deniability for being rude.
This is a really really smart SVP. The answer to “how do we prevent this in the future” is to identify decisions that reduce that problem from happening again. This doesn’t have to be perfect. You can say “we are going to do X next time” and also say “but we don’t know how far X will work”. When the failure happens again, you can retire process X and move on to Y. The aim is to make a series of decisions that eventually get to the heart of the issue.
Everything else is a conversation that takes up space on a post-mortem or runbook.
An SVP of engineering should care about the details, this is the essence of leadership. When you don’t care about the details you are just a manager and worthless i.e. you should be replaced with a leader.
One thing that's often left out that tends to grind organizations to a halt is that continuous improvement processes tend to be additive.
You always add rules, and alerts, and so on. You need to revisit existing processes to and see what can be removed or replaced.
My other nit with these is a term that's abused a lot "Root Cause", most complex issues have many contributing factors, not a single root cause, and if you force the teams to find one, they will.
I dislike RCA (Root Cause Analysis) it puts you in the wrong mindset, CFA would be better (Contributing Factor Analysis).
Yeah I've seen this a lot. People freaking about issues that have small or moderate impact on deadlines, and very little to no revenue impact. And suddenly you're moving 50% slower because you have weeks of review processes to make sure nothing is missed.
I recently had a vendor CAPA rejected because the root cause was listed as "data entry error." Oh no, the root cause can't be data entry human error! It has to be that the deadline was too tight, or they were rushing, or they were distracted etc etc. I ask the vendor, were you distracted? No. Were you rushing? No. Just a simple typo. Update root cause to typo. Oh no the root cause can't be a typo it has to be the free text field in the form design that allows for typos. It's a process error that there's not an automated QC to catch these typos! Etc etc.
I think that's a problem domain in itself. Identify when failures that occur even when skilled people doing their jobs properly, adjust the environment so that they no longer happen.
It is understandably a completely different task compared to what those skilled people are specialised to do. You probably need a dedicated role to do it.
Perhaps you could call them a manager. Their job is to see the multiple parts of the system. They should ask for the Details of what happened so they can determine why the problem occurred.
Consider one of the problems listed in the article
>"The alert fired, but the on-call engineer had already dealt with twenty low-value alerts that evening".
The engineer can say they were busy, they didn't see the alert, that they are swamped with things they think are low-value. Someone else can say the alert fired. Each person involved may have their own perspective, with different ideas as to what the problem actually is.
It's easy when you see problem described in terms of what the solution is. Someone needs to figure that out, to do that they need the details.
This is similar to the approach in healthcare, at least my experience in pharmacy.
When an error occurs we find the root cause but in 99% of cases a change to the environment/process is required to avoid it in the future.
Humans are all imperfect, their competence will change hour to hour let alone day to day. But you can control a system and put checks in place (admittedly a human can still do the process wrong, but then you need to think how the process can be clearer).
No-blame culture is very effective at providing an open environment to share mistakes, learn, but most importantly avoid reoccurance.
Interesting. At our org we always do both. Root cause, what are we changing. I assumed that was standard practice.
If you just explain what happened and why, that's fine but...how are you going to make sure it doesn't happen again?
The problem I found more vexing as a manager was: how can you prevent this KIND of problem from occurring?
I'd have someone on my team make a technical error and xyz wouldn't work. Wed talk thru it, and they wouldn't make that exact mistake again. But there's literally 10k things that can go wrong in our system, so then a related mistake would happen later.
What they needed was improved pattern recognition vs if this / then that which comes from post mortems.
> At first I thought "I don't want the details" sounded dismissive.
Well that is because it was delivered by someone who was, in fact, being dismissive!
If they want to know what "we are changing" but don't want to know any detail as to why, even presented at an appropriate level, they are very likely looking for you to take all the risk of the decision and they are dismissing your need to have the necessary informed consent for it to become the we.
Now, if you're presenting that information at a level of detail that is truly irrelevant or deep, that is your problem to fix. But otherwise be very wary of this pattern of communication where "we" might mean "you".
If you have not seen it, watch Margin Call. When you get to the boardroom scene you'll see this play out, but you will also see the (pretty rapacious, direct) boss understand properly that the information he is getting from a low ranking employee is nevertheless his problem to understand, even if only at a 10,000 ft level.
Jared Cohen: Mr. Tuld, as I mentioned earlier, if you compare the figure at the top of page 13 --
Tuld: Jared, it's a little early for all that. Just speak to me in plain English.
Cohen: Okay --
Tuld: In fact, I'd like to speak to the guy who put this together. Mr. Sullivan, is it? Does he speak English?
Cohen: Sir?
Tuld: I'd like to speak with the analyst who seems to have stumbled across this mess.
Cohen: Certainly. That would be Peter Sullivan. Right here.
Tuld: Oh, Mr. Sullivan, you're here. Good morning. Maybe you could tell me what you think is going on here. And please, speak as you might to a young child or a golden retriever. It wasn't brains that got me here, I can assure you of that.
Peter Sullivan: Well, uh...sir, as you may or may not know, I work, here, for Mr. Rogers as an associate in the Risk Assessment and Management Office at MBS.
Tuld: Please. Just relax. Stand up. Tell us in a clear voice -- what is the nature of the problem?
I have been in rooms where it played out at least something like this, where the boss wants to make an attempt to share the burden of the detail, and I've been able to leave them feeling somewhat supported, or at least without feeling like I was being shafted.
And I've been in rooms with people who don't want to know the details but just want to know what "we" are going to fix — those guys have never been as good to work for. If they say "we" but leave you with the impression they may mean "you", it's bad.
If I'm to explain what needs to change, that will by definition address processes or lack thereof that led to the problem, no? Which is then explaining what happened and why in an indirect manner, so many of the proposed changes may not make sense without more context.
188 comments
[ 0.27 ms ] story [ 29.6 ms ] threadNow it just needs to learn how to answer it, and act on the answer.. but step one seems solvable.
When "Something had gone wrong that shouldn't have" bubbles up that far, you have an organizational issue, not a technical issue.
TLDR:
Some outage causes a meeting with an SVP. SVP preempts the discussion with the following:
“I know that if we get into the details, the reasons will be perfectly reasonable. You'll explain what happened, I'll understand why everyone made the decisions they made, and I'll empathise with you.
Then it'll happen again.
So I don't want the details. I want to know what we're changing.”
Something about this feels wrong:
- If you trust them completely, you don't need to know what happens next either. Just trust them to do the right things, your leadership isn't required.
- If you don't trust them completely, how can you know if "what happens next" is appropriate without knowing the details?
Isn't this just reinforcing the idea that leadership doesn't need to have their feet on the ground?
The best leaders I've worked with were paying attention to things from the bottom-up as well as top-down.
> If you trust them completely, you don't need to know what happens next either. Just trust them to do the right things, your leadership isn't required.
> The best leaders I've worked with were paying attention to things from the bottom-up as well as top-down.
These two statements are at odds. You first assuming that lack of trust would be the only reason he needs to know what happens next, then go on to outline another reason he and the larger organization would both benefit from knowing more.
It's not just about trust. It's about knowing what your team is up to so you can provide them the resources necessary while also working to remove any obstacles in their way so they can get shit done. That's what good management does, it's just rare to see in low-trust organizations that treat everyone like a cog to be micromanaged because no one trusts anyone to get shit done of their own accord for whatever reason.
But doesn't that same logic apply to whether they need to know the details?
No. Leaderships job in this context is to set expectations of what happens next and to drive to team towards a solution that meets those expectations. No amount of trust magically makes that expectation materialize or for it to be met.
The baseline expectation in the original post also applies to the SVP: he knew enough to know that things aren't unreasonable to start with, this expectation might not apply to all companies.
Offloading the detail onto the team allows the org to scale higher by allowing the leader to bring more teams/problems into the mix. But it also introduces the complexity of keeping those teams in close enough alignment that the overall effect is still positive.
This is by no means guaranteed to be positive, and there's a lot of details to get right for leadership in making this work. But they are different details than what each individual team was bringing to leadership.
Abstractions are helpful, in business management as much as in systems architecture.
It seems like you're trying to rebut my argument by presenting my argument as being against a less strong version of the original claim rather than understanding it as being literally just a counterargument to the specific version of the claim I thought was too strong.
Leaders have the option of trusting subordinates, then operating at a higher level. However part of the job of a leader is to decide who to trust, with what. And why.
Sometimes that means, "Trust, but verify." The leader will spot check randomly to see that the whole is good.
Often, as here, that means, "Trust in some things, but not others." The leader in this case trusted the technical person to produce an accurate description of what happened. But didn't trust that same person to arrive at a solution that met the business of having a similar disaster not happen the next time.
But rather than undermine the tech person with, "I don't trust you to meet the business need," the leader communicated it with, "I trust you on the technical details, but here is the expectation for what you need to produce."
Worst still it invites bike shedding most of the time, where folks sit in a room nitpicking pointless details that's cathartic for the people in charge, but once again, doesn't actually do anything productive.
> The statement I initially responded to was "You first assuming that lack of trust would be the only reason he needs to know what happens next, then go on to outline another reason he and the larger organization would both benefit from knowing more". I pointed out that the same logic seems like it would apply to the idea that there might be other reasons why a leader might need to know the details.
I'm genuinely confused why this is apparently not clear. Someone said that it's flawed logic to assume there aren't good reasons beyond a lack of trust to want to know next steps. I'm saying it seems reasonable there could be good reasons beyond a lack of trust to want to know the details of the current step. You're saying that I'm finding it difficult to "grok this concept", but I don't even know what concept you're talking about, because none of the responses seem to have been related to this, and I don't understand what anyone else could think I've been saying.
I think its entirely contextual as to what the size of the org is and the scope of the projects. At a startup an "executive" or VP would legitimately be able to be working side by side with everyone else, but just by nature of the role being SVP I imagine this is a large enough company that this is no longer the case.
The role entirely morphs and their job becomes to enable teams, get blockers out of the way, fast track or assist things like permissions/processes, shoot down ideas that legitimately don't add business value or are not worth wasting existing engineering effort on.
And I imagine in this scenario the point wasn't for the executive to literally hear what happens next, it was to ensure the proper steps are being taken and make sure "somebody" is handling the what happens next part of things, and so they can report up through the chain or delegate tasks for exactly that "what happens next" task.
This is a very strong argument that no organization should be allowed to get that big.
Big organizations enable scaling and associated economies of scale, so it's possible for Apple to sell millions of iPhones cheaper than if they were a smaller company. I don't think that tradeoff is worth it for any individual consumer (pay a little less to be much more abused by the company), and I don't think it's worth it for society.
Most of the big corporations in America aren't providing things as useful as MRI machines, they're providing net negative garbage like social media and advertising. Big corporations like United Healthcare Group are inflating the cost of medical care rather than providing economies of scale and reduced cost for their clients.
The US desperately needs to bring back antitrust and start breaking these massive companies into much smaller pieces.
I wouldn't be alive if not for advanced medicine, but if the structures that saved my life are the same ones that constitute our own Great Filter, my life was most definitely not worth the trade. There are too many other people I love to think that.
Here you are recasting “I trust that, despite the bad outcome, everyone’s actions here were reasonable” as “I trust that you are going to solve this fully without my guidance”. The fact that the word “trust” appears in both of these statements does not make them equivalent. Trusting that people are capable and well intentioned does not mean trusting that they will accomplish the same thing without you.
But it should be used and phrased better.
Used better:trust and only sometimes ask for the detials, like cutting the just-shuffled deck in a card game, sometimes the player to the dealer's right will just tap the deck instead of cutting it, in effect saying "I trust that shuffle". Sometimes, certainly enough to both keep the SVP well-grounded in the details, and to keep the team knowing the SVP is both watching and cares about the details, the SVP MUST ask for the details. Trust and verify.
In terms of phrasing, "I don't want the details" is all about the SVP sending the message s/he doesn't want to be bothered. The phrasing should be: "I trust you on the details on this one; let's go straight to what do we do to manage the next one of these?".
Notice leading with "I trust you on this", not "don't waste my time".
I work in a competent team and leadership is required.
Getting that to happen more automatically is one way out of this situation, but that tends to cause problems of its own, with blurry lines of responsibility and authority. So it works best with relatively flat organization structures, which aren’t used much in large organizations.
It's conditioning. In all organizations above a certain size (> 100 people), stepping out of line receives an official reprimand. Someone always complains. Receive enough reprimands and the person is terminated. People learn to stay in line because they've been punished whenever they step out of it.
So you can elide some details when you talk to leadership, investors, and regulators, but you can't elide everything.
You will have to trust someone will figure out the details and come up with a reasonable solution. Sometimes you need to know that solution, sometimes in detail, sometimes just enough to know that they solved it. Sometimes you are the person who needs to figure out the details and make the solution. Knowing which you are is important.
I think the author of the article is going out of their way to be charitable to the SVP for no reason. That meeting was a reassertion of power, not a wise move made by a sage.
You might need to know timescales so that you can inform upwards/outwards/other. You might need to make sure other resources that might be necessary are available. The planned "what next" and the proposed timescales might have impacts elsewhere that they haven't realised (and maybe even until now had no reason to be aware of). If they tell you the what next and there are no such complications, then they can carry on, otherwise there might need to be some changes to the plan or changes to other plans to allow time/resources to be available for this plan or just to stop potential near future conflicts.
Trust, but verify. And perhaps even assist.
I'm not saying c suite are unquestionable gods, I'm saying that there is no such trust-or-not dichotomy. It's 2 different things.
No matter how ignorant I am in some domain and no matter how knowlegeable some expert is I hire to do something for me in that domain, they can only tell me how to get what I want, or what's the closest that is humanly possible, or the costs of various conflicting priorities and compromises. They can't tell me what to want.
Maybe I DO actually want to burn my whole budget and 5 years of time on some facet that they and most people would say is not important and not worth it to the point of being irrational. Their job is to inform me not to decide for me. I can trust them to inform me honestly and with good judgment and deep knowledge, and then to impliment what I decide also honestly and with competent skill. And yet I still need to be the one who is informed, and then makes a decision and issues a directive.
It's 2 different things.
It's like Steve Jobs being totally unreasonable about tiny details of fit & finish in the hardware products. It makes no sense (to most people) to have custom ethernet jacks manufactured (back when it wasn't so effortless and flexible) just to get them a certain color or material feel, when off the shelf jacks already exist from multiple sources that work just fine for a 10th or 100th the cost, while already meeting a stack of compatibility standards and even safety regulations. No trustworthy engineer would do that. They would actually see it as violating that trust.
As an example, a major rehaul of a core system could push back product (new feature dev) timelines, that can affect customers and their releases, bottomline of the company, reputation, tax repercussions, if it’s an American public company then it’ll affect SOX compliance & audits, could lead to spicy investor/analyst conference calls, and so on.
I've worked with plenty of competent people. Only a few were so good I trusted them completely.
First of all, he didn't say the executive "trusted them completely". Those are your words.
He said "The executive assumed that we were competent, and was saying "I already believe you. Now let's talk about what happens next".".
That can be true even if they don't trust them completely. Let's say he trusted them 80% and not 100%. Isn't that pretty much quite trusting anyway?
Besides, the executive can trust them even 100%, but still want them to give him the general picture. Among other things, it's his job to know and be able to report when asked.
Another explanation is that not wanting the details of operations is based on complete lack of trust, not in the their competence but in the way they would describe things. The boss doesn't want the details because they are sure the details would make it seem like the underling did the right thing and moreover, monitoring that detail would just give the underling leverage over them.
The boss wants to know about changes to avoid this but whatever it is, they'll push back on its extent and won't want to hear about details then either.
There are many reasons an executive needs to know what’s happening even if they’ve delegated the decision making authority.
Nobody can go to their boss and say “I don’t know what’s being done. I let so-and-so handle everything and they’ll figure it out”. Everyone has to know the big things that are happening.
Second, this is the step where the manager surveys the next steps for potential blockers or opportunities to accelerate. Maybe they notice that the plan could benefit from someone else’s help and they ask that person to contribute.
There’s a lot of overthinking happening in this thread. The “I don’t need to know the details” isn’t a literal statement that covers all details. It’s a way of saying “Skip forward to the part I need to hear about”.
[1] realized
Good article write-up, I didn't need to read more than one paragraph to get all the information from the whole page.
I do like the term "organizational folklore" though. Actually properly documenting processes & expectations is important (and rare).
1. timeline of events
2. impact
3. 5 whys — the details of how and why things happend
4. actionable items
I've never seen a post-mortem without actionable items.
I’ve done plenty of postmortems and many meetings where we go through the motions but “leadership” does not assign and address follow-ups. Most organizations pay lip service to reliability.
As a stopgap, a manual pre-flight check is added. The number of manual pre-flight checks becomes a ridiculous overhead but it gives a clear person to blame when something goes wrong.
But also the post reads like it comes from an inexperienced/immature service org, so maybe they’re just sorting their operational response process out.
A lot of leaders like to hide in the details because the bigger picture is a harder problem.
People need to tell their stories. They need to be heard and have their work and its difficulty respected.
That's what went wrong. Everything else is an abuse victim rationalizing their abuse.
I can't be the only person who thought this is a highly intelligent person being inadvertently gaslit by the SVP into thinking the exec is right to be dismissive with "I don't want the details" because introspection is common in intelligent people.
Had someone said that to me I'd have walked away after saying "fine I'll sort it'.
This illustrates that "trust" to do the job, and since they didn't want the details of the problem, they therefore don't need the specific solution description. This kind of response is also the same level of respect, it's either seen as trust in ability, or just downright disdain with plausible deniability for being rude.
I'm not so sure this means "I trust you."
Everything else is a conversation that takes up space on a post-mortem or runbook.
You always add rules, and alerts, and so on. You need to revisit existing processes to and see what can be removed or replaced.
My other nit with these is a term that's abused a lot "Root Cause", most complex issues have many contributing factors, not a single root cause, and if you force the teams to find one, they will.
I dislike RCA (Root Cause Analysis) it puts you in the wrong mindset, CFA would be better (Contributing Factor Analysis).
It is understandably a completely different task compared to what those skilled people are specialised to do. You probably need a dedicated role to do it.
Perhaps you could call them a manager. Their job is to see the multiple parts of the system. They should ask for the Details of what happened so they can determine why the problem occurred.
Consider one of the problems listed in the article
>"The alert fired, but the on-call engineer had already dealt with twenty low-value alerts that evening".
The engineer can say they were busy, they didn't see the alert, that they are swamped with things they think are low-value. Someone else can say the alert fired. Each person involved may have their own perspective, with different ideas as to what the problem actually is.
It's easy when you see problem described in terms of what the solution is. Someone needs to figure that out, to do that they need the details.
When an error occurs we find the root cause but in 99% of cases a change to the environment/process is required to avoid it in the future.
Humans are all imperfect, their competence will change hour to hour let alone day to day. But you can control a system and put checks in place (admittedly a human can still do the process wrong, but then you need to think how the process can be clearer).
No-blame culture is very effective at providing an open environment to share mistakes, learn, but most importantly avoid reoccurance.
If you just explain what happened and why, that's fine but...how are you going to make sure it doesn't happen again?
The problem I found more vexing as a manager was: how can you prevent this KIND of problem from occurring?
I'd have someone on my team make a technical error and xyz wouldn't work. Wed talk thru it, and they wouldn't make that exact mistake again. But there's literally 10k things that can go wrong in our system, so then a related mistake would happen later.
What they needed was improved pattern recognition vs if this / then that which comes from post mortems.
Well that is because it was delivered by someone who was, in fact, being dismissive!
If they want to know what "we are changing" but don't want to know any detail as to why, even presented at an appropriate level, they are very likely looking for you to take all the risk of the decision and they are dismissing your need to have the necessary informed consent for it to become the we.
Now, if you're presenting that information at a level of detail that is truly irrelevant or deep, that is your problem to fix. But otherwise be very wary of this pattern of communication where "we" might mean "you".
If you have not seen it, watch Margin Call. When you get to the boardroom scene you'll see this play out, but you will also see the (pretty rapacious, direct) boss understand properly that the information he is getting from a low ranking employee is nevertheless his problem to understand, even if only at a 10,000 ft level.
I have been in rooms where it played out at least something like this, where the boss wants to make an attempt to share the burden of the detail, and I've been able to leave them feeling somewhat supported, or at least without feeling like I was being shafted.And I've been in rooms with people who don't want to know the details but just want to know what "we" are going to fix — those guys have never been as good to work for. If they say "we" but leave you with the impression they may mean "you", it's bad.