I have a cousin in law enforcement, so yes I can, but it’s still maddening. The formula is essentially: a problem arises. Everyone covers their asses with red tape, and new rules and regulations are born, and never die. It’s a classic, yet very difficult-to-manage bureaupathology. The alternative is just that a friend of someone in power makes the tools.
A friend of mine works at a state facility where, as he describes it, it's impossible to be a manager without constantly breaking rules, because many contradict one another or violate the laws of reality. Result is that management is a nasty snake-pit where everyone knows which rules everyone else is breaking and they'll collectively point their fingers at anyone who tries to rock the boat (including, and perhaps especially, improving things), quickly getting them fired (or at least transferred somewhere they'd rather not be).
Perhaps significantly, management positions aren't represented by the state workers' union, which is the only thing (barely) insulating the lower level workers from that insanity.
It may seem like micromanagement, but small seemingly meaningless deviations from procedure can add up to big problems, particularly when dealing with space.
Doesn't sound like the tool in question had any real impact on that accident (so to speak). I'm guessing if you drop a 3-foot torque wrench through a fuel tank, it does just as much damage as a 3-foot socket wrench. The fact that they grabbed the wrong tool was entirely incidental to damaging the missile.
Also, it takes an extremely precise sequence of events to cause a nuclear detonation. A nearby explosion is not enough to do it, as you can obviously see here.
tldr; The socket didn't properly fit the wrench and they had to manually hold the socket on the end of the wrench. One man dropped the socket, which fell through a small gap between the walkway and missile. The torque wrench was large enough that it would not have slipped through this gap if/when dropped.
And the larger problem is one of knowledge asymmetry.
If you're dealing with technology so complicated that one person cannot simultaneously design and repair / maintain it, then whoever performs the latter doesn't possess all the knowledge of the former.
Maybe they used an flammable alloy under certain conditions for engineering reasons.
Most metal doesn't burn. But do you want repair / maintenance techs making that assumption?
Multiple by 100,000 parts.
PS: A more realistic example was one of the Skunk Works projects. I forget the details, but either tools of special metallic composition or wooden tools had to be used to avoid damaging the flight characteristics of the aircraft.
Was it really possible that the fuel explosion could have also detonated the warhead? I was under the impression that these nukes had fusing apparatus that made it difficult/impossible for them to detonate if not explicitly armed and in the correct scenario.
Actually one of the books written about that incident, "Command and Control," has a lot of interesting detail about the different types of warhead fusing schemes that were implemented at various times.
I can't remember which fusing system was on that warhead (though the book describes it in detail) but I think it was one of the more sophisticated kinds than the simple "one point safety" fuses and that the warhead was basically never in danger of exploding.
Risky if you do not want to be blown up too or at least irradiated. Decommissioning does involve removal of nuclear material first. Missile fuel is likely burned up as it is highly toxic.
> Early in the morning (Friday, September 19), a two-man PTS investigation team entered the silo. Because their vapor detectors indicated an explosive atmosphere, the two were ordered to evacuate. The team was then ordered to reenter the silo to turn on an exhaust fan. Senior Airman David Livingston reentered the silo to carry out the order and shortly thereafter, at about 3:00 a.m., the hypergolic fuel exploded.
Livingston died as a result of that explosion.
This seems tragic to me. I can't believe someone would "order" him back into the silo to turn on an exhaust fan. First, starting a fan motor inside of an explosive environment can be dangerous. Second, how could someone have such disregard for another person's life, that they send him into a room full of explosive gas?
Perhaps the stakes seemed so high, they felt the only way to stop the warhead from scattering radiation in a resulting explosion was to exhaust the silo.
> This seems tragic to me. I can't believe someone would "order" him back into the silo to turn on an exhaust fan. First, starting a fan motor inside of an explosive environment can be dangerous. Second, how could someone have such disregard for another person's life, that they send him into a room full of explosive gas?
This is the military: it’s sad but not out of scope to order someone to do something dangerous, or even likely fatal, in order to accomplish an objective. Consider “you stay here with this machine gun while the rest of us go back the way we came.”
> This seems tragic to me. I can't believe someone would "order" him back into the silo to turn on an exhaust fan. First, starting a fan motor inside of an explosive environment can be dangerous. Second, how could someone have such disregard for another person's life, that they send him into a room full of explosive gas?
Ordering people to take potentially fatal risks to avoid greater harms is not merely recurrent within the military, it's fundamental to it.
That doesn't mean that there can't be room to debate whether a particular risk ordered was warranted, but one absolutely should not be shocked that a military superior ordered a subordinate to do something that involves a potentially fatal risk.
Indeed. And it doesn't work. Look at the sheer number of near world destroying accidents our military have managed to muster over the years.
I was actually working for a company that was a subcontractor to the USAF in the distant past. There is little room for individual creativity and ingenious problem solving in such a process heavy environment. It's actually like trying to work on an engineering project with Siri. The outcome of this is that only the problems allowed to be solved by the process are solved and none of the risks not identified by the process are reported, because it's too much paperwork already if you find a new one and you're not going to want your job to get any harder.
Ergo, all unidentified risks get taken because they are attractive solutions to many problems. Looking at the spanner incident pointed out by two sibling comments, the "dont carry heavy tools or anchor then when working on the big explodey thing" rule was not enforced and double-checked but hundreds of minor rules with precisely no benefit were being applied. Someone knew of that risk but didn't put the mitigation in place, probably because it was too hard.
Now military rigour and all of that doesn't like to be seen to be a turd sieve of a process so what happens is the answer is to add more processes, not look at the fundamental problem in the approach.
Overall, you need to focus on the key places where risk is considered high, but for most things, meh. Listening to people and making sure risk is managed effectively is a win.
> Indeed. And it doesn't work. Look at the sheer number of near world destroying accidents our military have managed to muster over the years.
What is that number? And do we know that the number would be less without these types of regulations?
> Looking at the spanner incident pointed out by two sibling comments, the "dont carry heavy tools or anchor then when working on the big explodey thing" rule was not enforced and double-checked
My understanding from listening to the podcast episode (this morning) is that the tool-anchoring rules were only added after that particular disaster.
Could you imagine betting your multi-million dollar satellite on whether the contractor used the approved torque wrench, or something they found at Harbor Freight? Hell, who needs a torque wrench? I've been doing this long enough I can do it by feel.
Using the right tool gets us to the moon. Complaining about how micromanagement stifles one's creative vision is how we get the shit show we call "software" today. Somewhere in the middle I'm sure there's a happy medium.
Not only can I imagine it I’ve lived it and I think it was great.
First: you are misstating the constraint: a procedure was defined by the manufacturer (in this case SpaceX) and they committed to the customer that they would use that procedure.
Second: in my case it was pharmaceutical manufacture and we were giving these drugs to patients for an approval study. We had defined a reproducible procedure that gave reproducible results. Deciding the change the procedure would require proving that the result of the change was the same as before said change. As a person who takes medication I am glad for GMP (the equivalent of ISO9000 in pharma manufacture).
In the case of an expensive rocket that launches something likely unrepairable after launch (and perhaps irreplaceable) I want to know that the mfr has figured out as many of the risk factors as possible and how to mitigate them.
When working on large complicated systems the best engineering approach is often to define a process (tools included) and then always repeat the same way. You’d be surprised how many things can go wrong when you stop acting repeatably. It will definitely seems like a waste of time until the first time you blow up a half billion dollar payload because someone unknowingly trashed an important component using the wrong wrench.
Don't they typically test rockets in outdoors test ranges? And aren't rockets normally supposed to be able to operate outdoors?
Of course there is going to be dirt, rocks and yes, even fecal matter. What are they supposed to do, blow their hot exaust gases into a climate-controlled cleanroom?
I worked on AWACS back in the early 2000s and we had an adversarial dynamic with our QA team. We would see them as 'just getting in the way', but I can now say, after working as a software developer, that without a strong QA (backed up by military laws), our planes would be flying with a lot of 'solutions' that would lead to unnecessary deaths.
35 comments
[ 345 ms ] story [ 795 ms ] threadCould you imagine being so micromanaged that you'd literally need the Federal Government to change a manual to approve a tool you need?
Perhaps significantly, management positions aren't represented by the state workers' union, which is the only thing (barely) insulating the lower level workers from that insanity.
It may seem like micromanagement, but small seemingly meaningless deviations from procedure can add up to big problems, particularly when dealing with space.
Also, it takes an extremely precise sequence of events to cause a nuclear detonation. A nearby explosion is not enough to do it, as you can obviously see here.
tldr; The socket didn't properly fit the wrench and they had to manually hold the socket on the end of the wrench. One man dropped the socket, which fell through a small gap between the walkway and missile. The torque wrench was large enough that it would not have slipped through this gap if/when dropped.
If you're dealing with technology so complicated that one person cannot simultaneously design and repair / maintain it, then whoever performs the latter doesn't possess all the knowledge of the former.
Maybe they used an flammable alloy under certain conditions for engineering reasons.
Most metal doesn't burn. But do you want repair / maintenance techs making that assumption?
Multiple by 100,000 parts.
PS: A more realistic example was one of the Skunk Works projects. I forget the details, but either tools of special metallic composition or wooden tools had to be used to avoid damaging the flight characteristics of the aircraft.
I can't remember which fusing system was on that warhead (though the book describes it in detail) but I think it was one of the more sophisticated kinds than the simple "one point safety" fuses and that the warhead was basically never in danger of exploding.
Livingston died as a result of that explosion.
This seems tragic to me. I can't believe someone would "order" him back into the silo to turn on an exhaust fan. First, starting a fan motor inside of an explosive environment can be dangerous. Second, how could someone have such disregard for another person's life, that they send him into a room full of explosive gas?
Perhaps the stakes seemed so high, they felt the only way to stop the warhead from scattering radiation in a resulting explosion was to exhaust the silo.
This is the military: it’s sad but not out of scope to order someone to do something dangerous, or even likely fatal, in order to accomplish an objective. Consider “you stay here with this machine gun while the rest of us go back the way we came.”
Ordering people to take potentially fatal risks to avoid greater harms is not merely recurrent within the military, it's fundamental to it.
That doesn't mean that there can't be room to debate whether a particular risk ordered was warranted, but one absolutely should not be shocked that a military superior ordered a subordinate to do something that involves a potentially fatal risk.
https://www.thisamericanlife.org/radio-archives/episode/634/...
> In 1980, deep in a nuclear missile silo in Arkansas, a simple human error nearly caused the destruction of a giant portion of the Midwest.
Spoiler: at a critical point, it involves a technician improvising with the wrong tool.
I was actually working for a company that was a subcontractor to the USAF in the distant past. There is little room for individual creativity and ingenious problem solving in such a process heavy environment. It's actually like trying to work on an engineering project with Siri. The outcome of this is that only the problems allowed to be solved by the process are solved and none of the risks not identified by the process are reported, because it's too much paperwork already if you find a new one and you're not going to want your job to get any harder.
Ergo, all unidentified risks get taken because they are attractive solutions to many problems. Looking at the spanner incident pointed out by two sibling comments, the "dont carry heavy tools or anchor then when working on the big explodey thing" rule was not enforced and double-checked but hundreds of minor rules with precisely no benefit were being applied. Someone knew of that risk but didn't put the mitigation in place, probably because it was too hard.
Now military rigour and all of that doesn't like to be seen to be a turd sieve of a process so what happens is the answer is to add more processes, not look at the fundamental problem in the approach.
Overall, you need to focus on the key places where risk is considered high, but for most things, meh. Listening to people and making sure risk is managed effectively is a win.
What is that number? And do we know that the number would be less without these types of regulations?
> Looking at the spanner incident pointed out by two sibling comments, the "dont carry heavy tools or anchor then when working on the big explodey thing" rule was not enforced and double-checked
My understanding from listening to the podcast episode (this morning) is that the tool-anchoring rules were only added after that particular disaster.
https://en.wikipedia.org/wiki/List_of_military_nuclear_accid...
I guess it depends on your definition of "near world destroying accidents".
Using the right tool gets us to the moon. Complaining about how micromanagement stifles one's creative vision is how we get the shit show we call "software" today. Somewhere in the middle I'm sure there's a happy medium.
First: you are misstating the constraint: a procedure was defined by the manufacturer (in this case SpaceX) and they committed to the customer that they would use that procedure.
Second: in my case it was pharmaceutical manufacture and we were giving these drugs to patients for an approval study. We had defined a reproducible procedure that gave reproducible results. Deciding the change the procedure would require proving that the result of the change was the same as before said change. As a person who takes medication I am glad for GMP (the equivalent of ISO9000 in pharma manufacture).
In the case of an expensive rocket that launches something likely unrepairable after launch (and perhaps irreplaceable) I want to know that the mfr has figured out as many of the risk factors as possible and how to mitigate them.
Of course there is going to be dirt, rocks and yes, even fecal matter. What are they supposed to do, blow their hot exaust gases into a climate-controlled cleanroom?
https://en.wikipedia.org/wiki/Foreign_object_damage
I prefer to swap words around in cases like this to make it unambiguous, for example, “soda bottles and poop...”