AFAIK some fields distinguish "Pilot error", caused by someone's skill as a pilot, and "Human error", caused by the fact that you are not made of silicon.
You can't prevent all error just by manipulating the inputs. For one thing, you can't fully control all the inputs.
Human error is a useful approximation because it basically says people are unreliable, and will probably be so for life.
The point isn't to blame individuals, but to built systems that are resistant to errors, and do not have behavioral single points of failure where one action can make a mess you can't easily fix.
Most of society is really deep in a "Shit happens, move on" mindset, and their solution to any human error is usually "Be more careful", which often becomes fairly meaningless, as one often can't slow down or reduce distractions.
In more reliability focused thinking, the response to human error might be to specifically train for avoiding that mistake, if you think you can, to remove the preconditions of the mistake, to add checklist procedures, or to remove the ability of the mistake to cause harm.
The study of human error has been incredibly important in my life. As of yet, nobody has been able to teach me how to stop making mistakes, day in, and day out.
But I can often predict them, sometimes prevent them, and usually mitigate them, by modeling them as background noise that's just always there, like as if my intentions and my actual thoughts were on opposite ends of a noisy dial up link.
This is of course already done (to some extent) in aviation. Human error is indeed a start of investigation - there is extensive psychological profiling for pilots, why they did what they did given the circumstances, how much sleep did they get (often - not enough) and what function did they optimize (e.g. their salary bonus). One more thing is we have a co-pilot that can mitigate errors of the captain. Sadly, we don't (yet) for driving (and timescale of decision making is much smaller typically).
The 9000 series is the most reliable computer ever made.
No 9000 computer has ever made a mistake or distorted information.
We are all, by any practical definition of the words, foolproof and incapable of error.
> human error is an unfortunately effective way at ending an
investigation
If you can't find a scapegoat, senior executives will line up to throw
themselves on a grenade in return for a golden parachute and
preferential place at the revolving door of professional management
networks.
Why? Because the kind of contextual root cause analysis the author is
(in good faith) promoting here is the last thing anybody wants in
companies that cut corners and push risk externalities on to the
world.
> we human beings are physical beings, you can think of us as machines
Nice postulate let's see where that leads...
It's certainly dehumanising (and therefore fails multiple essential
requirements for real-world use) but let's entertain the idea that we
are simply little functional utility maximisers.
> how did the operator's g function score a higher value for pushing A
over B?
All we have to do is take into account every single variable and
contextual factor of the workplace, training, remuneration, budgets,
policy, corporate strategy, markets, competition...
But there's a fly in the ointment. Are corporations not "legal
persons"? So we must also consider them "as machines", right?
They cannot be 'blamed', but are merely little profit maximising
functional automatons. And by dint of the same logic, we must look to
their environment, contextual variables, national and global politics
and so on, in which governments are simply little utility maximising
machines with their own values.
While not an infinite regress, it is an analytic process that pushes
all causation out to a nebulous margin which, most importantly, cannot
be changed. Therefore all failures, no matter how many lives or
billions of dollars are lost, is ultimately "just the way the world
is".
In our hearts most of us already know this. But perfomative justice,
and the entire edifice of rational scientific management, rests on
finding local culprits after no more than "5 why?" iterations.
> Companies are easy to reprogram compared to people
Is that true? I'm not so sure.
Surely the difficulty is a multiple of the number of employees.
Unless you imagine that by writing some new policies and issuing
edicts one can, like Capt. Picard simply "Make It So". In reality
institutional culture can be notoriously hard to reset.
Change/transformation is a big deal when you have more than a roomful
of people.
Well, I'm thinking that companies are mostly based on rules and processes. Humans are a lot of things, independent and can follow processes, but they are also flesh and bone - there's a lot about how humans work that we can't change by "reprogramming". It's hard to make people twice as perceptive about (for example) people walking behind their car when reversing. Because it's about our perception and senses, not rules.
The most reliable companies out there have aggressively adopted practices that treat incidents as expected and "human error" as a symptom of a (correctable) systems failure.
Somewhat counterintuitively these high-reliability companies have MORE incidents than less reliable organizations that haven't removed the stigma around incident reporting because their goal is to learn from them and catch them early vs. punishing the unlucky individual who happened to step on whatever systemic land-mine exploded that day.
Looks like one of Dekker's books is already listed in this post, but another one worth checking out is his "Field Guide to Understanding Human Error"[1] which is a very approachable book focused on the aviation industry and the learnings (especially post WWII) that have made that industry so safe.
If you're working on this at your own company (especially if you're in a supervisor / executive position) it's incredibly powerful and impactful to be the incident champion who works to make incident response open and accessible across the org. So much catastrophic failure comes as a result of hiding the early signs due to fear retaliation or embarrassment.
Also worth checking out: we[2] hosted a mini conference on Incident Response earlier this year with lots of great videos from folks who have worked in this space for decades about everything from culture to practices: https://www.irconf.io/
"No Human Error" is a bad approach for most real world context. It could be useful in an academic or simulation approach.
As other commentors have pointed out, Sidney Dekker has written some very good books on this topic. I'll point out two approaches or mental tools that he teaches.
1: "Slip" vs "Mistake". A slip is an non-intentional error: hitting the wrong switch accidentally, forgetting to click a button, etc. In general you could help prevent these with simple UI tweaks. A mistake is an error in or with intention. You meant to do something, and you had in mind what you wanted to do, but you made a miscalculation or had a misunderstanding of the situation. Changing workflows and adding context can help with these. This is a gross oversimplification, listen to his talks online and buy the book if you're interested.
2: Understand hindsight bias and blame. Hindsight bias is absolutely pervasive in human thinking after an incident. After an incident you have much more time and information to process than the operators involved. If you really want to prevent future incidents you must understand the information, tools, and mental context of the operator at the time of the incident.
Blame is a box to hide complexity in. People are afraid of the tiger in the forest. Complexity is hard to talk about. After an incident you want to feel safe and OK. If you can put the blame somewhere, you have hunted down the tiger and you can feel safe. But, if you are simply blaming a person, then another person may make the exact same mistake. This is very hard to talk about because in many cases there are legal and moral implications. When you take a systemic approach to safety and preventing human error, leave your morals at the door, and do not mix legal and engineering discussions during the investigation. You can pick up your morals when you go home for the day and be disgusted on your own time. Having an overly legalistic and punitive system can delay system improvements. Again, this is super oversimplified, there is a universe of nuance in this subject.
> Under this paradigm, if one of the contributing factors in an incident was the user pushing “A” instead of “B”, we ask “how did the operator’s g function score a higher value for pushing A over B”? There’s no concept of “error” in this model. Instead, we can explore the individual’s context and history to get a better understanding of how their g function valued A over B. We accomplish this by talking to them.
To me, the interesting thing about this model is that the "g function" is almost entirely unconscious. The conscious mind is like the tip of the iceberg, with the vast bulk of processing happening out-of-sight in the unconscious mind. Most of the time "human error" is actually a deliberate act on the part of the unconscious mind which is following a kind of program to achieve some goal or benefit. Sometimes people really do just plain make mistakes, usually due to fatigue or hunger or bad lighting or whatever. But most behaviour is deliberate, even when we don't know it. In fact, the conscious experience of making decisions is an illusion: brain scans show that your brain makes a decision and then a few milliseconds later "you" have the experience of "making" it. In other words, your "free will" is an illusion. (Although that doesn't rule out the possibility that your whole system, conscious and unconscious together, has free will!)
There are two wrinkles: 1) the illusion of "free will" and "human error" is an extraordinarily deep component of almost all human societies (basically only a few Buddhist ashrams might be excluded, on good days.) And 2) the unconscious mind is not isolated in each human being, it's a "distributed OS" in computer terms, or in Jungian terms the unconscious mind is "collective". (There's nothing "woo-woo" about this. It's not telepathy, just high-bandwidth communication. E.g. take a video of folks in a cafe or something and slow it down or speed it up, it's pretty easy to see (at different time scales) the unconscious synchronization of strangers who are consciously ignoring each other. Or watch people walking together, typically their gaits will sync, even when their legs are different lengths. "Group mind" is the norm.)
Anyway, all this to say that the model described in the post can be very powerful if you really run with it, but one should be aware of where it eventually leads. Once you open the box and look inside, you take on responsibility for dealing with the results.
What do you think about someone who drives under the influence and kills someone else?
The weird thing about the blameless postmortems / no human error movement is that taken to its logical conclusion it seems to produce no personal responsibility.
The extreme other side of the pendulum where management is just looking for scapegoats to blame is obviously poor, but the optimum (like damn near everything else) probably lies in the middle someplace.
I think it’s important to acknowledge that human error is in part systems design error.
The recent Atlassian error was human error. Or was it? My boss at YouTube used to write shell scripts in lane duck mode. By default they output a dry run. Why does he do this? Preventing human error. That’s a systems design to eliminate the vast majority of misuse of a shell scrip.
If Atlassian had implemented a similar scheme, especially for a dangerous script they wouldn’t have accidentally deleted hundreds of client records.
Had they planned for data recovery and practices it they could have written partial data restore scripts at their leisure rather than when 300 clients were down.
Had they split functionally of safe scrips away from dangerous scripts, had they described dangerous behavior in the script, warned or required confirmation, had they trained users in proper use of dangerous scripts, had they documented proper processes…
Human error just means you designed a system that can be misused. It’s not reasonable to design for eliminating all error but it’s certainly possible to prioritize risk and plan for avoiding high risk errors.
As the next step in our everlasting quest to transform people into dehumanized drones, we should start modeling them as machines minimizing a cost function, doing away with such outdated notions as "making mistakes". To err is human, as they say.
I thought this post was going to show me something profound but it’s clearly a surface level idea without any real knowledge of the topic. Human error comes in many forms and from a safety perspective we want to identify what impacts human error will have and then work to reduce the risk associated with each one to as low as is reasonably practicable.
Process design, interface design, organisation and manning, ergonomics etc all play a role.
It is inevitable that a human will make an error, with a well designed system it should be very very hard for them to do so.
18 comments
[ 3.2 ms ] story [ 54.3 ms ] threadYou can't prevent all error just by manipulating the inputs. For one thing, you can't fully control all the inputs.
Human error is a useful approximation because it basically says people are unreliable, and will probably be so for life.
The point isn't to blame individuals, but to built systems that are resistant to errors, and do not have behavioral single points of failure where one action can make a mess you can't easily fix.
Most of society is really deep in a "Shit happens, move on" mindset, and their solution to any human error is usually "Be more careful", which often becomes fairly meaningless, as one often can't slow down or reduce distractions.
In more reliability focused thinking, the response to human error might be to specifically train for avoiding that mistake, if you think you can, to remove the preconditions of the mistake, to add checklist procedures, or to remove the ability of the mistake to cause harm.
The study of human error has been incredibly important in my life. As of yet, nobody has been able to teach me how to stop making mistakes, day in, and day out.
But I can often predict them, sometimes prevent them, and usually mitigate them, by modeling them as background noise that's just always there, like as if my intentions and my actual thoughts were on opposite ends of a noisy dial up link.
... and politics.
.margorp dab a nevig saw eh ,rorre na ekam t'ndid laH
If you can't find a scapegoat, senior executives will line up to throw themselves on a grenade in return for a golden parachute and preferential place at the revolving door of professional management networks.
Why? Because the kind of contextual root cause analysis the author is (in good faith) promoting here is the last thing anybody wants in companies that cut corners and push risk externalities on to the world.
> we human beings are physical beings, you can think of us as machines
Nice postulate let's see where that leads...
It's certainly dehumanising (and therefore fails multiple essential requirements for real-world use) but let's entertain the idea that we are simply little functional utility maximisers.
> how did the operator's g function score a higher value for pushing A over B?
All we have to do is take into account every single variable and contextual factor of the workplace, training, remuneration, budgets, policy, corporate strategy, markets, competition...
But there's a fly in the ointment. Are corporations not "legal persons"? So we must also consider them "as machines", right?
They cannot be 'blamed', but are merely little profit maximising functional automatons. And by dint of the same logic, we must look to their environment, contextual variables, national and global politics and so on, in which governments are simply little utility maximising machines with their own values.
While not an infinite regress, it is an analytic process that pushes all causation out to a nebulous margin which, most importantly, cannot be changed. Therefore all failures, no matter how many lives or billions of dollars are lost, is ultimately "just the way the world is".
In our hearts most of us already know this. But perfomative justice, and the entire edifice of rational scientific management, rests on finding local culprits after no more than "5 why?" iterations.
Is that true? I'm not so sure.
Surely the difficulty is a multiple of the number of employees. Unless you imagine that by writing some new policies and issuing edicts one can, like Capt. Picard simply "Make It So". In reality institutional culture can be notoriously hard to reset. Change/transformation is a big deal when you have more than a roomful of people.
Somewhat counterintuitively these high-reliability companies have MORE incidents than less reliable organizations that haven't removed the stigma around incident reporting because their goal is to learn from them and catch them early vs. punishing the unlucky individual who happened to step on whatever systemic land-mine exploded that day.
Looks like one of Dekker's books is already listed in this post, but another one worth checking out is his "Field Guide to Understanding Human Error"[1] which is a very approachable book focused on the aviation industry and the learnings (especially post WWII) that have made that industry so safe.
If you're working on this at your own company (especially if you're in a supervisor / executive position) it's incredibly powerful and impactful to be the incident champion who works to make incident response open and accessible across the org. So much catastrophic failure comes as a result of hiding the early signs due to fear retaliation or embarrassment.
Also worth checking out: we[2] hosted a mini conference on Incident Response earlier this year with lots of great videos from folks who have worked in this space for decades about everything from culture to practices: https://www.irconf.io/
[1] https://www.amazon.com/Field-Guide-Understanding-Human-Error...
[2] shameless plug for https://kintaba.com, my startup in this space
As other commentors have pointed out, Sidney Dekker has written some very good books on this topic. I'll point out two approaches or mental tools that he teaches.
1: "Slip" vs "Mistake". A slip is an non-intentional error: hitting the wrong switch accidentally, forgetting to click a button, etc. In general you could help prevent these with simple UI tweaks. A mistake is an error in or with intention. You meant to do something, and you had in mind what you wanted to do, but you made a miscalculation or had a misunderstanding of the situation. Changing workflows and adding context can help with these. This is a gross oversimplification, listen to his talks online and buy the book if you're interested.
2: Understand hindsight bias and blame. Hindsight bias is absolutely pervasive in human thinking after an incident. After an incident you have much more time and information to process than the operators involved. If you really want to prevent future incidents you must understand the information, tools, and mental context of the operator at the time of the incident.
Blame is a box to hide complexity in. People are afraid of the tiger in the forest. Complexity is hard to talk about. After an incident you want to feel safe and OK. If you can put the blame somewhere, you have hunted down the tiger and you can feel safe. But, if you are simply blaming a person, then another person may make the exact same mistake. This is very hard to talk about because in many cases there are legal and moral implications. When you take a systemic approach to safety and preventing human error, leave your morals at the door, and do not mix legal and engineering discussions during the investigation. You can pick up your morals when you go home for the day and be disgusted on your own time. Having an overly legalistic and punitive system can delay system improvements. Again, this is super oversimplified, there is a universe of nuance in this subject.
To me, the interesting thing about this model is that the "g function" is almost entirely unconscious. The conscious mind is like the tip of the iceberg, with the vast bulk of processing happening out-of-sight in the unconscious mind. Most of the time "human error" is actually a deliberate act on the part of the unconscious mind which is following a kind of program to achieve some goal or benefit. Sometimes people really do just plain make mistakes, usually due to fatigue or hunger or bad lighting or whatever. But most behaviour is deliberate, even when we don't know it. In fact, the conscious experience of making decisions is an illusion: brain scans show that your brain makes a decision and then a few milliseconds later "you" have the experience of "making" it. In other words, your "free will" is an illusion. (Although that doesn't rule out the possibility that your whole system, conscious and unconscious together, has free will!)
There are two wrinkles: 1) the illusion of "free will" and "human error" is an extraordinarily deep component of almost all human societies (basically only a few Buddhist ashrams might be excluded, on good days.) And 2) the unconscious mind is not isolated in each human being, it's a "distributed OS" in computer terms, or in Jungian terms the unconscious mind is "collective". (There's nothing "woo-woo" about this. It's not telepathy, just high-bandwidth communication. E.g. take a video of folks in a cafe or something and slow it down or speed it up, it's pretty easy to see (at different time scales) the unconscious synchronization of strangers who are consciously ignoring each other. Or watch people walking together, typically their gaits will sync, even when their legs are different lengths. "Group mind" is the norm.)
Anyway, all this to say that the model described in the post can be very powerful if you really run with it, but one should be aware of where it eventually leads. Once you open the box and look inside, you take on responsibility for dealing with the results.
The weird thing about the blameless postmortems / no human error movement is that taken to its logical conclusion it seems to produce no personal responsibility.
The extreme other side of the pendulum where management is just looking for scapegoats to blame is obviously poor, but the optimum (like damn near everything else) probably lies in the middle someplace.
This eliminates both growth and progress.
If Atlassian had implemented a similar scheme, especially for a dangerous script they wouldn’t have accidentally deleted hundreds of client records. Had they planned for data recovery and practices it they could have written partial data restore scripts at their leisure rather than when 300 clients were down.
Had they split functionally of safe scrips away from dangerous scripts, had they described dangerous behavior in the script, warned or required confirmation, had they trained users in proper use of dangerous scripts, had they documented proper processes…
Human error just means you designed a system that can be misused. It’s not reasonable to design for eliminating all error but it’s certainly possible to prioritize risk and plan for avoiding high risk errors.
Process design, interface design, organisation and manning, ergonomics etc all play a role.
It is inevitable that a human will make an error, with a well designed system it should be very very hard for them to do so.