I did a 3 month contract gig with a local robotics firm a while back. I developed a few cool features for them, but the reason they fell in love with me was that I worked up extensive testing checklists. During testing I'd be the one with the checklist hardcopy (printed out from the internal wiki) in my lap and a walky talky in one hand.
"Range safety officer, do we have your OK? Safety driver, do we have your OK? Safety driver, throw the switch to manual. Safety driver, is steeting calibrated? Remote operator, unpark the robot. Remote operator, confirm feature X is turned off. Safety driver, confirm vehicle feature Y is on..."
Without the checklist, a trial-and-error run through of a certain feature took 3 hours. With the checklist we could do it in 3 minutes.
I came back from an intense demo of the robot to third parties and decided to decompress in my woodworking shop...and next to my lathe was a checklist I had written up a while earlier on how to make a certain Christmas stocking stuffer
I think it is a completely personality issue. Just like their desk, some people like to keep things tidy and organized and know whats next all the time, while some people seem to thrive in a bit of messiness, creativity and improvisation and getting things done first.
When it comes to jobs that require originality and creativity, a checklist is not going to be productive. In a job that is more operational, predictive and repetitive, then a checklist would help.
When it comes to software engineering, the job is semi-structural (you are bound by certain conventions, languages, computer science principles), and semi-creative (you can use your imagination in a lot of places).
For QA types, a checklist is a must.
For pure new feature programing, a step by step checklist is basically a creativity killer. Crap in, crap out.
I personally distaste checklist, and try to avoid artificial constrains as much as I can.
I often use lists in the development of a new feature, and do not consider it a creativity killer. Start with an insight, and break it down into simple steps or components. If this list is crap, it is obvious at that point and can be scrapped and the process can be tried again. For me the creative aspect of a feature is the original insight, and it is usually easy to judge a proposed implementation against how well it achieves that insight.
I think you are mis-assessing the value of a checklist for creative work. I think medicine involves a considerable amount of creative problem solving for example. And yet the use of checklists to manage basic techniques has an enormous positive impact on patient outcomes.
More fundamentally, not every aspect of a task or project requires creativity or needs to be reinvented each time. To the extent that there are good practices it speeds you up to encode them in a checklist so you can free your mind to focus on the creative aspects.
Doctors were quizzed, "Do you want to use a checklist?", they answered, "No.". "Do you want someone who is operating on you to use a checklist?", they answered, "Yes.".
If you read the book you'll find that almost everyone hates checklists but when used they cut down on mistakes and human error. The surprising discovery is not that they work, but the reluctance of people to use them...
"And checklists lack flexibility. They might be useful for simple procedures like central line insertion, but they are hardly a panacea for the myriad ills of modern medicine. Patients are too varied, their physiologies too diverse and our knowledge still too limited."
The problem is that everyone (including doctors) overestimates both their ability to diagnose based on instinct, and the percentage of patients who deviate from the norm.
For heart attacks, they found that a 3 step checklist was X% effective in diagnosing a heart attack, while the doctor alone was Y%, and X was much higher than Y. Yes, sometimes the doctor would use his expertise to catch something the checklist would miss, but over time the math was with the checklist.
I agree, but the real solution is either a branching checklist for the situations where there are only a few possible norm deviations, or an honest to god Expert System when there are potentially very many norm deviations.
Having said that, I'm pleased that this review mentions unintended consequences and the limits of this approach. Checklists are not by any means a panacea.
If you're interested in the benefits and limitations of checklists, you should definitely pick up the book. It's a quick read, and there's practically a whole chapter dedicated to the time he spent interviewing the guy at Boeing who's responsible for developing checklists for aircraft procedures - both normal and emergency. He goes into more detail about what does and what doesn't make a good checklist, as well as where they can and can't be applied effectively.
I've been more interested in checklists after hearing Charlie Munger repeatedly espouse them:
"...You’ve got all the tools. And you’ve got to have one more trick. You’ve got to use those tools checklist-style, because you’ll miss a lot if you just hope that the right tool is going to pop up unaided whenever you need it. But if you’ve got a full list of tools, and go through them in your mind, checklist-style, you will find a lot of answers that you won’t find any other way."
That's a great talk, but I sort of took away the opposite from it. Munger mentions checklists and algorithms a couple of times, but it's clear that what he is really talking about are heuristics ("I was using some rough algorithms that work pretty well in a great many complex systems").
The problem with checklists is that they don't allow the possibility that the checklist may be wrong. Heuristics are in this sense the opposite of algorithms. It's the reliance on algorithms that leads to the sort of false precision that Munger is complaining about:
When I talk about this false precision, this great hope for reliable, precise formulas, I am reminded of Arthur Laffer.... His trouble is his craving for false precision, which is not an adult way of dealing with his subject matter.
12 comments
[ 4.4 ms ] story [ 44.7 ms ] thread"Range safety officer, do we have your OK? Safety driver, do we have your OK? Safety driver, throw the switch to manual. Safety driver, is steeting calibrated? Remote operator, unpark the robot. Remote operator, confirm feature X is turned off. Safety driver, confirm vehicle feature Y is on..."
Without the checklist, a trial-and-error run through of a certain feature took 3 hours. With the checklist we could do it in 3 minutes.
I came back from an intense demo of the robot to third parties and decided to decompress in my woodworking shop...and next to my lathe was a checklist I had written up a while earlier on how to make a certain Christmas stocking stuffer
http://smartflix.com/projects/3
...which is when I realized that there's something about checklist that permeates my personality!
When it comes to jobs that require originality and creativity, a checklist is not going to be productive. In a job that is more operational, predictive and repetitive, then a checklist would help.
When it comes to software engineering, the job is semi-structural (you are bound by certain conventions, languages, computer science principles), and semi-creative (you can use your imagination in a lot of places).
For QA types, a checklist is a must. For pure new feature programing, a step by step checklist is basically a creativity killer. Crap in, crap out.
I personally distaste checklist, and try to avoid artificial constrains as much as I can.
More fundamentally, not every aspect of a task or project requires creativity or needs to be reinvented each time. To the extent that there are good practices it speeds you up to encode them in a checklist so you can free your mind to focus on the creative aspects.
If you read the book you'll find that almost everyone hates checklists but when used they cut down on mistakes and human error. The surprising discovery is not that they work, but the reluctance of people to use them...
The problem is that everyone (including doctors) overestimates both their ability to diagnose based on instinct, and the percentage of patients who deviate from the norm.
For heart attacks, they found that a 3 step checklist was X% effective in diagnosing a heart attack, while the doctor alone was Y%, and X was much higher than Y. Yes, sometimes the doctor would use his expertise to catch something the checklist would miss, but over time the math was with the checklist.
Having said that, I'm pleased that this review mentions unintended consequences and the limits of this approach. Checklists are not by any means a panacea.
"...You’ve got all the tools. And you’ve got to have one more trick. You’ve got to use those tools checklist-style, because you’ll miss a lot if you just hope that the right tool is going to pop up unaided whenever you need it. But if you’ve got a full list of tools, and go through them in your mind, checklist-style, you will find a lot of answers that you won’t find any other way."
http://www.tilsonfunds.com/MungerUCSBspeech.pdf
The problem with checklists is that they don't allow the possibility that the checklist may be wrong. Heuristics are in this sense the opposite of algorithms. It's the reliance on algorithms that leads to the sort of false precision that Munger is complaining about:
When I talk about this false precision, this great hope for reliable, precise formulas, I am reminded of Arthur Laffer.... His trouble is his craving for false precision, which is not an adult way of dealing with his subject matter.