It's a long-winded argument for using checklists, later to be turned into a book and a major motion picture. (Long-winded, but worth a read, assuming you can get past the paywall, if you haven't read The Checklist Manifesto.)
Short version. Use checklists. Even intelligent and highly trained people such as pilots use them. We used them in intensive care and surgery and avoided many basic errors as a result.
I'd recommend not buying the book it's basically the above expanded to 200+ pages.
This probably got posted in reaction to the news that the author, Atul Gawande, was announced to be the CEO of a new healthcare company. https://news.ycombinator.com/item?id=17355346
"The checklists provided two main benefits, Pronovost observed. First, they helped with memory recall, especially with mundane matters that are easily overlooked in patients undergoing more drastic events. . . ."
Indeed, benefits of checklists can be huge. this reminds me of a US Army study I read about some time ago which examined the effects of sleep deprivation and stress in demanding environments.
What they found was that, in general, the high-level strategic capabilities stayed intact -- the people could still analyze complex situations, make and execute battle plans, etc.
What was lost was the smaller, mundane functions, which could add up to critical failures, such as forgetting to refill one's water can or refuel the vehicle before going out, etc.
I've always hated my bad memory, and I learned a long while ago that simply trying to remember through sheer force of concentration did not work. It was as hopeless as praying for rain. Only external memory tools and strategies planned in advance can solve it for me.
I have to build a system to keep me from forgetting, because if I leave it up to my future self alone then he will forget 100% of the time.
In his book, The Checklist Manifesto, Gawande reports that at the end of a big checklist pilot project for the World Health Organization (IIRC), participating surgeons were surveyed.
• One of the survey questions was, in essence: Would the use of checklists improve your surgical practice? Most of the surgeons responded no. This was consistent with the resistance that Gawande encountered: I don't need no stinking checklist; I'm a good surgeon!
• But the more-telling survey question was this: Would you want another surgeon to use a checklist if operating on you or one of your family members? Something like 93% of the surgeons surveyed responded ... yes.
Wow. That's such a supreme indicator of overconfidence--the belief that I and I alone am so competent that I don't need safeguards.
I would not have hoped that a high percentage of surgeons have that trait. I wonder if I am guilty of it as well. Something to examine in the mirror in our future endeavors.
I like to use the term Dunning-Kruger instead of over-confidence or hubris to describe it. Everyone thinks they demonstrate above average skill. During the 2007-2009 financial crisis every single financial services firm justified their executive bonuses by claiming their personnel were above average. Very few car drivers consider themselves below average.
Before germ theory gained acceptance or was even understood, physicians and surgeons considered themselves above hand washing as a gentleman's hands are always clean. Ignaz Semmelweis discovered that washing hands and tools in chlorine solution reduced childbirth deaths. But he was unable to convince others.
In modern times doctors have answered a survey on the effect of pharmaceutical representatives visit. The overwhelming majority responded that they are not swayed by drug saleswomen, but that other docs are susceptible to choosing a prescription drug because of salesmanship instead of only efficacy & risk.
Dunning-Kruger is something more specific, that people with lower skill in an area overestimate their skill, because they are unable to distinguish competence in that area.
It's not the Dunning-Kruger effect because it's overconfidence expressed by highly trained, highly competent experts about their own performance compared to their peers. If anything it's "skilled and too aware of it" rather than "unskilled and unaware of it".
>I like to use the term Dunning-Kruger instead of over-confidence or hubris to describe it. Everyone thinks they demonstrate above average skill.
On HN, everyone thinks that everyone but themselves are affected by Dunning Kruger. In fact, calling out Dunning Kruger in others is a symptom of someone under the very same effect.
Good points. I feel like in the surgeon's case it's subtly but significantly different, though; these people aren't ignorant amateurs, they're highly trained professionals who do actual work on human bodies, with all the problem-solving skills and knowledge that implies. By all accounts they should be able to accurately assess their own abilities, as opposed to people affected by Dunning-Kruger. I think it's important to be mindful of hubris among the competent as well, perhaps more so because they are the ones actually doing the work in the end.
The effects you talk about definitely exist and I feel like a lot of those are arrogance, or possibly even our fundamental inability to observe without bias unless we take very careful precautions ahead of time. Perhaps doctors otherwise used to proper experimental control and thinking scientifically didn't apply the same precautions ahead of meeting with sales reps, because that appeared to be out of context of practicing medicine at the time, even though it is still part of the process. Apparently they are affected by bias without being aware of it. This is wholly conjecture on my part.
Just today, I was thinking the same thing, but about compile-time type checking. "You don't need it, because you're a good programmer? Great. But not everyone on your team is as wonderful as you. Do you want to be able to catch their errors at compile time?"
I chalk this up more to management and not individual egos. Incentive structures configured by the vast majority of management in most organizations do not reward checklist development and more importantly checklist maintenance.
Bob: I'm developing and testing a checklist.
Management: Why is it taking sooo long!?!?? Why aren't you doing <<revenue/KPI-enhancing activity>>?
Versus:
Bob: I'm selling truckloads of surgeries.
Management: W000t! My manager will be ecstatic!
There is also a free rider problem with checklist development. Lots of workplaces love to somewhat or all-out pit employees against each other. Unless the entire team is creating and maintaining the checklist, the subset of the team creating/maintaining the checklist will always over time come out looking worse on KPI's that almost invariably never measure these kinds of activities.
The subset of the team that doesn't spend the time on creating/maintaining the checklist come out looking better on KPI's over time, and the checklists increase their relative standing against their checklist-creating peers.
Fix the incentive structures, and this problem fixes itself.
I have a dumbass problem where after I've driven away from home I can't remember if I locked the front door or closed the garage door so I turn around and go back home to check. I've managed to fix this by saying out loud, "door locked - check" and "garage door down - check" when I've done it. Now I just need a checklist to remind me to do a checklist.
It might sound dumb to you but your solution is used by Japanese train engineers before leaving at a station and trains in Japan are really efficient and on-schedule.
I am an avid checklist user. I have a number of complex processes which are repeated monthly, quarterly or yearly. When the office is calm and I am feeling good I don't need them and find myself ticking them off having already done the stage. When I am feeling less-good, the office is noisy and I keep being interrupted the check-lists are the reason I succeed.
Yet still I find almost complete resistance to them by colleagues. They seem to think it is somehow demeaning to reduce their work to a fixed process. Perhaps it takes the mystique out of it, and gives the idea that anyone could do it.
Only if the job really could be done by anybody. Pilots use checklists for basically everything, but I don't think there's a widespread belief that the pilot could now be replaced by J. Random Passenger. The point of a checklist isn't necessarily to completely remove any thought or training from a process, but to ensure that every critical step is performed in the proper order.
Spot on. My checklist is not so easy to follow even by people with equivalent qualifications. It works as a aide-memoir for me, but anyone else would have to learn my profession and look-up the procedure for each task.
Checklist doctors should follow:
1. Wash their hands with soap.
2. Clean the patient’s skin with chlorhexidine antiseptic.
3. Put sterile drapes over the entire patient.
4. Wear a sterile mask, hat, gown and gloves.
5. Put a sterile dressing over the catheter site.
I think a key element of checklists, and checklist discipline, is coordination between members of a team. What checklists allow you to do is have accountability across differences in experience/authority/expertise and also track a process over time.
In my work as a naval aviator (caveat, I'm still a student), we use checklists in the cockpit to coordinate all sorts of functions such as engine start, takeoff, landing and emergencies. These ensure that we start the engine the same way, verify we are ready for landing, and also handle emergency procedures in an approved way. When there are rank and experience differences in the cockpit these checklists help equalize the disparity, and let less experienced or junior members hold senior members accountable. Before each flight we also follow a briefing checklist, and that includes human factors such as any possible issues relating to illness/medication/stress/alcohol/fatigue/eating. Doing so ensures that there is always a place for people to vocalize anything that the team should know.
Previously I worked on a Coast Guard cutter, and we used similar checklists to ensure we were prepared to get underway, enter port, or complete a complex evolution like tow another vessel. For these checklists the main value was in verifying between multiple changes of team personnel that everything was being completed to meet the timeline. The checklist served as documentation that the person before you had collected the necessary information, made proper notifications, and started/finished processes.
It has baffled me when I hear that medical personnel dont use checklists for these exact reasons. They perform time critical, high stress procedures that have a fairly documented set of steps. They require a team of personnel that have different scopes of expertise, experience, and authority. There are frequently changes in the team's composition over the duration of the patients care, and it seems like the patient will be handed off between different teams that specialize in their own respective areas of patient care. For us in the military it is pretty easy for authority to mandate we use checklists, and the buy in come from accident investigations revealing that checklists were ignored or skipped. Is there an equivalent medical authority that can pass this information down, or does this have to emerge independently from each hospital system?
Does anyone know of a good cross-platform checklist app? There's a million and one todo list apps but I can't seem to find a good checklist one. I use my phone's memo app for checklists but it doesn't help me with staying consistent over time, and doesn't scale as I use the same app for note taking, ideas, a scratch space and so on.
Oh, and if you're thinking "I don't need no stinkin' checklists, what would you even use them for?" here's what I have currently:
- Code review key checks
- Code review smells and anti-patterns
- Out-the-door checks (got my keys, entry pass, grabbed lunch from fridge etc)
- Gym gear list
- Traits to work on a la Benjamin Franklin
- Self-care list for when I'm stressed/anxious (relief suggestions e.g. drink a glass of water, go for a walk, mindfulness, organise workload on paper, etc)
- Carry lists for a couple hobbies
I would likely have more but the poor form factor limits how practical it is to keep many lists.
it's not so much the text formatting as the ability to set reminders, physically check off the list and have a record per-day/instance/whatever of completing the list, that kind of thing. The main reason I use checklists so much is because my memory is extremely bad, so prompts make a huge difference to me.
A few thoughts from someone who has used checklists extensively for many years:
1. Use a spreadsheet. The flexibility to change the structure as you go, and to create a 2-dimensional checklist is great. All the other power of a spreadsheet is at your fingertips, including automation.
2. Visualization is extremely valuable, and I think necessary. If you have to read through the text of the checklist to understand the current state, it's too cumbersome (other than for small lists) and hard to process. If spreadsheet cells change color based on status, and if their location indicates, via visual memory, what they represent, then now you have a dashboard which tells you the state of the project at a glance.
3. Human beings are terrible at following and updating checklists. I don't know why - it seems easy in concept - but I've never met anyone who could do it without forcing themselves. And everyone starts instinctively skipping steps after awhile. Keep it as simple as possible, with as little detail and as few updates as are necessary to keep things going.
People are generically bad at reliably setting states. The problem isn't so much that someone failed to do something (an action) but that they failed to notice that the action was not done (the state was wrong).
Often times people forget the "check" part of the term "checklist" and treat them as a list of things they must do. The result is less reliable because it is easy to skip a step, even if you end up ticking a box. A checklist involves two steps, first doing the thing (possibly from memory) and then consulting the checklist to see that the appropriate state is set.
This is a particular issue for programmers as programmers are constantly involved with lists of actions that are performed in sequence. As a result there is a tendency to let the list run the human in a way similar to how a computer runs a program. Computers are way better at states than humans are however...
My coworker just fixed an unset state of mine— I checked the box, but did not perform the related action, three weeks ago.
Thankfully, Trello keeps a log of most changes, so it was easy to see where in the process my eyes had glazed over. We’ll get around to automating those steps someday, too.
After reading this article when it came out, I ended up reading the book "Checklist Manifesto" and have incorporated checklists into my professional and even personal life. I highly recommend it.
One question (for you or anyone else with relevant experience, really): the more negative reviews on Amazon say that the book is mostly a collection of anecdotal evidence about how great checklists are.
Assuming this is true, and that one is already sold on how useful a checklist is, can you suggest more practical hands-on books on how to build effective checklists?
(If you think that the reviews are wrong, and the book provides actual practical advice, this would also be useful information, thanks).
46 comments
[ 3.3 ms ] story [ 122 ms ] threadI'd recommend not buying the book it's basically the above expanded to 200+ pages.
Edit: Why was this downvoted?
This probably got posted in reaction to the news that the author, Atul Gawande, was announced to be the CEO of a new healthcare company. https://news.ycombinator.com/item?id=17355346
https://www.newyorker.com/magazine/2009/06/01/the-cost-conun...
https://www.newyorker.com/magazine/2015/05/11/overkill-atul-...
Indeed, benefits of checklists can be huge. this reminds me of a US Army study I read about some time ago which examined the effects of sleep deprivation and stress in demanding environments.
What they found was that, in general, the high-level strategic capabilities stayed intact -- the people could still analyze complex situations, make and execute battle plans, etc.
What was lost was the smaller, mundane functions, which could add up to critical failures, such as forgetting to refill one's water can or refuel the vehicle before going out, etc.
Interesting lesson
I have to build a system to keep me from forgetting, because if I leave it up to my future self alone then he will forget 100% of the time.
• One of the survey questions was, in essence: Would the use of checklists improve your surgical practice? Most of the surgeons responded no. This was consistent with the resistance that Gawande encountered: I don't need no stinking checklist; I'm a good surgeon!
• But the more-telling survey question was this: Would you want another surgeon to use a checklist if operating on you or one of your family members? Something like 93% of the surgeons surveyed responded ... yes.
I would not have hoped that a high percentage of surgeons have that trait. I wonder if I am guilty of it as well. Something to examine in the mirror in our future endeavors.
Before germ theory gained acceptance or was even understood, physicians and surgeons considered themselves above hand washing as a gentleman's hands are always clean. Ignaz Semmelweis discovered that washing hands and tools in chlorine solution reduced childbirth deaths. But he was unable to convince others.
In modern times doctors have answered a survey on the effect of pharmaceutical representatives visit. The overwhelming majority responded that they are not swayed by drug saleswomen, but that other docs are susceptible to choosing a prescription drug because of salesmanship instead of only efficacy & risk.
[1]: https://en.wikipedia.org/wiki/Illusory_superiority
Dunning-Kruger is something more specific, that people with lower skill in an area overestimate their skill, because they are unable to distinguish competence in that area.
On HN, everyone thinks that everyone but themselves are affected by Dunning Kruger. In fact, calling out Dunning Kruger in others is a symptom of someone under the very same effect.
The effects you talk about definitely exist and I feel like a lot of those are arrogance, or possibly even our fundamental inability to observe without bias unless we take very careful precautions ahead of time. Perhaps doctors otherwise used to proper experimental control and thinking scientifically didn't apply the same precautions ahead of meeting with sales reps, because that appeared to be out of context of practicing medicine at the time, even though it is still part of the process. Apparently they are affected by bias without being aware of it. This is wholly conjecture on my part.
Bob: I'm developing and testing a checklist.
Management: Why is it taking sooo long!?!?? Why aren't you doing <<revenue/KPI-enhancing activity>>?
Versus:
Bob: I'm selling truckloads of surgeries.
Management: W000t! My manager will be ecstatic!
There is also a free rider problem with checklist development. Lots of workplaces love to somewhat or all-out pit employees against each other. Unless the entire team is creating and maintaining the checklist, the subset of the team creating/maintaining the checklist will always over time come out looking worse on KPI's that almost invariably never measure these kinds of activities.
The subset of the team that doesn't spend the time on creating/maintaining the checklist come out looking better on KPI's over time, and the checklists increase their relative standing against their checklist-creating peers.
Fix the incentive structures, and this problem fixes itself.
https://www.allaboutlean.com/pointing-and-calling/
66 °F ≈ 18.9 °C
+10 °F ≈ +5.6 °C
98.6 °F = 37.2 °C
Yet still I find almost complete resistance to them by colleagues. They seem to think it is somehow demeaning to reduce their work to a fixed process. Perhaps it takes the mystique out of it, and gives the idea that anyone could do it.
If you simplify your job, everybody could do it, and the salary and job security would go down.
In my work as a naval aviator (caveat, I'm still a student), we use checklists in the cockpit to coordinate all sorts of functions such as engine start, takeoff, landing and emergencies. These ensure that we start the engine the same way, verify we are ready for landing, and also handle emergency procedures in an approved way. When there are rank and experience differences in the cockpit these checklists help equalize the disparity, and let less experienced or junior members hold senior members accountable. Before each flight we also follow a briefing checklist, and that includes human factors such as any possible issues relating to illness/medication/stress/alcohol/fatigue/eating. Doing so ensures that there is always a place for people to vocalize anything that the team should know.
Previously I worked on a Coast Guard cutter, and we used similar checklists to ensure we were prepared to get underway, enter port, or complete a complex evolution like tow another vessel. For these checklists the main value was in verifying between multiple changes of team personnel that everything was being completed to meet the timeline. The checklist served as documentation that the person before you had collected the necessary information, made proper notifications, and started/finished processes.
It has baffled me when I hear that medical personnel dont use checklists for these exact reasons. They perform time critical, high stress procedures that have a fairly documented set of steps. They require a team of personnel that have different scopes of expertise, experience, and authority. There are frequently changes in the team's composition over the duration of the patients care, and it seems like the patient will be handed off between different teams that specialize in their own respective areas of patient care. For us in the military it is pretty easy for authority to mandate we use checklists, and the buy in come from accident investigations revealing that checklists were ignored or skipped. Is there an equivalent medical authority that can pass this information down, or does this have to emerge independently from each hospital system?
Oh, and if you're thinking "I don't need no stinkin' checklists, what would you even use them for?" here's what I have currently:
- Code review key checks
- Code review smells and anti-patterns
- Out-the-door checks (got my keys, entry pass, grabbed lunch from fridge etc)
- Gym gear list
- Traits to work on a la Benjamin Franklin
- Self-care list for when I'm stressed/anxious (relief suggestions e.g. drink a glass of water, go for a walk, mindfulness, organise workload on paper, etc)
- Carry lists for a couple hobbies
I would likely have more but the poor form factor limits how practical it is to keep many lists.
1. Use a spreadsheet. The flexibility to change the structure as you go, and to create a 2-dimensional checklist is great. All the other power of a spreadsheet is at your fingertips, including automation.
2. Visualization is extremely valuable, and I think necessary. If you have to read through the text of the checklist to understand the current state, it's too cumbersome (other than for small lists) and hard to process. If spreadsheet cells change color based on status, and if their location indicates, via visual memory, what they represent, then now you have a dashboard which tells you the state of the project at a glance.
3. Human beings are terrible at following and updating checklists. I don't know why - it seems easy in concept - but I've never met anyone who could do it without forcing themselves. And everyone starts instinctively skipping steps after awhile. Keep it as simple as possible, with as little detail and as few updates as are necessary to keep things going.
Often times people forget the "check" part of the term "checklist" and treat them as a list of things they must do. The result is less reliable because it is easy to skip a step, even if you end up ticking a box. A checklist involves two steps, first doing the thing (possibly from memory) and then consulting the checklist to see that the appropriate state is set.
This is a particular issue for programmers as programmers are constantly involved with lists of actions that are performed in sequence. As a result there is a tendency to let the list run the human in a way similar to how a computer runs a program. Computers are way better at states than humans are however...
Thankfully, Trello keeps a log of most changes, so it was easy to see where in the process my eyes had glazed over. We’ll get around to automating those steps someday, too.