Those claims that it was maintenance is a bit bizarre considering these were brand new craft, and that panel isn't exactly a wear item. (ofc, some wear with each pressurization cycle, but I digress)
It'd be different if it was an engine issue on a 20+ year old plane.
But still, this isn't anything like the prior MAX issue - and frankly could have just as easily happened to any manufacturer like Airbus.
I don't think it's worth bashing Boeing much about, I doubt they're too happy about it either.
One major difference I recall back when working in the industry is that Airbus didn't rely on contract manufacturing like Boeing adopted after the McDonnell Douglas merger. Spirit was just Boeing Wichita until they divested the operation only to immediately contract work with the new owner at a discount.
The Boeing MDD merger was before my time. Airbus sure does rely on contract manufacture for everything from wings over fuselage sections to doors so. Often using the same suppliers Boeing does, the market for these tier 1 manufacturing suppliers is small.
One of the most disappointing things about this whole issue has been a kind of "fanboy"-ism that's been a bit shocking. Fanboy-ism is for the "We hate Apple", or the "Emacs is better than vi" level arguments.
There is no room for fanboy-ism in airliner design. It's a business that should be approached in a manner that is at once, sober and serious. If there are problems with this airliner, let's get them all on the table and start working through them one by one. I thought this is what the FAA and Boeing were meant to do? (Even in the face of monetary losses). But the mounting evidence of a rubber-stamp approval regime is concerning.
We really do need to have cooler more dispassionate heads take things back in hand here. I think a move away from the money-men, the fanboys, and the doomsday alarmists is overdue at this point.
I took a second to check your links, and as I guessed, at least two of those would be unfairly characterized as "apologists." The fact your bad faith, ad hom post is here at the top of HN is a pretty damning indictment of the current userbase's sobriety when it comes to engineering issues and precise reading of claims.
None of what you posted were apologising for Boeing.
One was rightly pointing out that we should investigate first before jumping to any conclusions and the other two were pointing out that Alaskan had maintenance failures which is also true.
My assumption is the decompression event would have put the velocity of that decompression on the door/bolts and they are going to be weaker - and the addition of a window into a plug/door like this adds structural mass to add to that which is being sucked out via the decompression putting them over spec load for that area - but since it was a plug, the surrounding structure was still sound - just that thing couldnt handle the load.
EDIT: Maybe the bolts have a really high shearing tolerance, but not as good of a tensile pull?
I don't think there were any modern cases of airliner windows just being removed like that. They're fairly strong and consist of multiple layers of acrylic that are sandwiched between other components, so the entire window assembly would need to get torn out. Even then, it doesn't seem realistic that the difference in pressure would be big enough to instantly rip a properly-attached door clean off. Of course, only the experts can tell for sure, but my money's on a failure of the whole structure.
Look at causes of explosive decompression - look at the mass increase to add windows' struct infra to weight and mass to a plane, look at explosive bolts for emergencies
* Explosive decompression is the most dangerous, and can cause severe injuries or death. It is caused by a sudden and catastrophic failure of the pressure vessel, such as a window blowing out or a structural failure.
* Rapid decompression is less dangerous than explosive decompression, but can still cause serious injuries. It is caused by a more gradual loss of pressure, such as a door being left open or a seal failing.he cowling on the window failed. It was due to poor quality control for the extra mass around the window housings on plugs - the extra mass due to the pull and (IN ADDITION) the patsy ois on bolts.
They dont use explosive bolts in aircraft plugs, apparently, but yeah - this wasnt just bolts. Its a procedural impact, installation impact AND faulty materials QA in componentry.
My bet is on faulty adhesives or faulty QA on bolts.
I am pretty sure BOLTS in a plug door are a ruse UNLESS one can prove that window assembly procedures are either by the same company, the same process, or the same QA from multi-vendors.
Rule that out first.
An explosive decompression will not happen unless a single core seal will break....
Look T THE PASSENGER MANIFEST FOR THIS FLIGHT... look at MH370...
And if you want a mind F - look at Facebooks Husband who died in that flight to mexico....
If you look at the similarities, large numbers of high-tech company execs were taken out when those two crashed....
Who was on this manifest?
AI execs? Who lived.... So just give a list of professions of the passenger manifest. (Yes, aware no deaths - who could have? Who were to folks that missed that flight?)
If a singular bolt was faulty - it wouldnt have been an explosive decomp
It would have been a leak.
Explosive at 10K feet is FF attack level "first priority"
(PS they already ruled IN that they use differing contractors to produce certain parts such as plug doors.
Lets not forget about Wellstone ,and Poland and so many others. Would you like a Helo ride over the beautiful Amazon?
You are a MORON of Aerial-Assassination, as it isnt your thing. Hope you dont get in a small plane anytime soon - why do you think billionaires dont employ you to watch over their planes, Helios, etc...you not-minion.
>>If a singular bolt was faulty - it wouldnt have been an explosive decomp
You dont know this about me. My Grandfather was the primary nuke designer behind Hanford.
Handed me my first engineering design book before I was 6 years old.
I drew every single design in that book before I was 7.
When I went into HighSchool (I am smoothing over a bunch of details) I had already drawn that book left-to-right.) (ABC's Of Autocad} and I went back and forth.
I am 2nd in the nation of the Cad Olympympics.
Might sound stupid - but imagine that in 1992.
3 Hour test finished in 35 minutes;
I designed the datacenters you feed off you dorks.
I was the principle designer for so many you touched.
I fucking hate how much so many people dont get cred.
Just look at Tiny. - We couldnt ever jump out of a plane without Tiny. A 5' woman fearless who invented the parachute....
Look at Kent (the mips designer) - youll never know about him - but why we use the word "fabric" on networking... (that you may never know... It was he and me on ... (he had a sexual deviency proclivity that was bad for computing vcs) - but when he was at MIPS we designed slot level rack computing in 1996
(If anyone wants to refute this, they can refute my claims of being aware of googles early mother board designs,etc...
I found this video illuminating. The name is confusing because these aren’t plug style doors. They can open out like they did here without passing through components https://youtu.be/maLBGFYl9_o?si=Km8qfkEv2YLHDY2G
They're not plug-style doors. They open outward. That's the problem.
Opening outward is mechanically riskier, but more likely to be useful in an emergency.
The drill for plug-style over-wing exits: Remove cover from upper handle, grab upper and lower handle, yank upper handle, pull door inside, have 15Kg or more of door land on your lap, turn big clunky door sideways, pitch through hole so it's outside and out of the way, climb through hole. Training video.[1] Few passengers are likely to get that drill right in an emergency, and the flight attendants are in the wrong place to do it.
They're only a plug when they're in the locked position. These bolts on the door plug prevent the plug from sliding into a position where it can come out.
Unfortunately consumers don't have much choice, the airline can change which type operates a flight at any time.
However longer term if Boeing's reputation keeps getting worse that can motivate buying decisions by the airlines. That change will be a very slow one, because fleet changes cost a lot of money in training crews, changing maintenance etc.
For example KLM is changing their short-haul fleet from the 737 NG to the A320neo family, that will take many years to completely roll out.
Most airlines do have MAXs. Delta does not. However, they have ordered some MAXs. So they will have MAXs in the future. That will take quite some time however.
So if it's really that big of an issue for a person, then just fly Delta for the time being.
Delta doesn't have any currently, although they ordered a bunch of the MAX family that will be delivered from 2025 - 2029. It will probably be impossible to avoid them, long-term.
Maybe when airlines start advertising that they don’t use the new 737s.
The more likely scenario is that the Chinese or Europeans will ground the new 737s until some very expensive fix is put into place and not accept any workarounds. That would probably make a difference.
There are mutual recognition agreements between the regulators and if the europeans try something like that, the americans can create counter measures to even the field in a mutually undesired spiral. At this moment, the biggest enemy of Boeing is Boeing itself.
It is more like "our cooperation is more valuable than temporary monetary advantages which could be negated and which could leave everyone worse overall". I'm not sure why deliberately misinterpret my comment, but you are quite wrong. The first 737max precedent is quite instructive for the relationship between the two regulators.
I’m not sure I understand what you are getting at.
I’m assuming that the US is less likely to demand costly changes to aircraft produced by Boeing because of the implications it would have for their own economy. I wouldn’t trust France to preemptive ground a series of Airbus planes either.
See, that, grounding of Airbus olanes, is exactly what EASA (which is European and not French) does. Spontaniously, I can think of fractures in the wings of the A380 when that happened.
"The European Union Aviation Safety Agency (EASA) said it had also adopted the FAA’s Emergency Airworthiness Directive (EAD) to ground this particular. ...
configuration of Boeing 737-9.
“EASA took the decision to adopt the FAA’s EAD despite the fact that - to the agency's knowledge and also on the basis of statements from the FAA and Boeing - no airline in an EASA member state currently operates an aircraft in the relevant configuration,” said EASA in a statement"
I doubt ever, consumers have been proven to select the cheapest airline ticket. The average once-a-year flier likely has no idea what aircraft they are even flying on.
That's not 1% of fliers representing half the flights though, I think it's more like 10% - it's saying that 1% of the entire world population causes it and only 11% of the world actually flies once a year.
I'd be willing to pay a premium not to fly on such an aircraft but not too big, as even though they aren't quite as safe as they should be, my impression is that the risk is still fairly low. Something on the order of 5%
Statistically speaking it's not worth worrying about. In 2021 there were less than 0.001 deaths per 100M passenger miles. Assuming these aircraft are 10x more dangerous than the alternative, that works out to 0.01 deaths per 100M passenger miles, or 0.00000026 deaths NY to SF flight. For an average american aged 38.9 years[2] with 40 years of life left[3], that works out to a loss of life of 5.4 minutes. I don't think most americans would worry over this, especially compared to other far more deadly threats like obesity.
>Statistically speaking it's not worth worrying about.
Those statistics don't really work out at all. They take number of miles and deaths, and nothing else. Let's say flown miles and driven miles are exactly the same for arguments sake. If the number of miles flown on planes were traveled by 4 million people, whereas the number of miles traveled by car were from 4 billion people, your likelihood of dying in a plane crash is FAR higher than by car if you're one of the 4 million who fly on a plane.
Miles and deaths alone are pretty poor metrics. There's also the part where I may only drive once or twice a week vs. the average which takes into account people who commute daily. So my odds of dying in a car crash are still lower. The frequency of travel seems just as pertinent as the miles traveled.
> They take number of miles and deaths, and nothing else. Let's say flown miles and driven miles are exactly the same for arguments sake. If the number of miles flown on planes were traveled by 4 million people, whereas the number of miles traveled by car were from 4 billion people, your likelihood of dying in a plane crash is FAR higher than by car if you're one of the 4 million who fly on a plane.
It's passenger miles, so the number of passengers is already factored in.
A better metric IMHO is "crashes involving at least one fatality" per flight. For large aircraft the fatalities are often everyone on board or nearly so. So people care about "what is the probability that the flight I'm getting on has a fatal accident?" This is largely independent of number of passengers or miles flown.
I don't get it, is your point that the "deaths per 100M passenger miles" is underestimating the likelihood of death? Doesn't the metric already normalize for number of passengers on a plane?
It's obfuscating it behind enormous numbers. Most accidents happen on takeoff or landing, so per-flight statistics are ideal for them. Some accidents happen in flight, so I'd maybe include hours in flight somehow. But miles and people counts don't serve any purpose.
Does this matter for the average flyer? Sure, if you're in Hawaii and flying daily for island hopping purposes you should probably estimate your risk as being higher, but the average flyer is still going to have average risk per miles flown (by definition).
People don't change their behavior much when things like this happen. When it comes around to the next big travel weekend, people will continue to book on whatever airline gets them to the right place at the right time for the right price.
Even when people have a direct ability to change what vehicle they travel on, like with car sales, safety incidents/features/records/etc have little impact to no on sales.
This is one of the reasons why transportation safety is regulated rather than being left up to market forces. Market forces don't really have an impact until the safety risk reaches some extreme levels.
As far as I can see, there's no real way to communicate this preference to airlines. I won't book a flight on a MAX, but the airline can change the airframe at the last second and it's highly situational how willing/able I am to just refuse to fly if they swap in a MAX.
I've communicated it albeit very indirectly. When looking at holiday packages for this summer I looked up the route to see if it's a 737 max before booking.
The typical person does not have the information about these failures, won't prioritize better odds of survival over short-term savings, and do not have enough money to choose alternatives anyway.
People are very bad at managing risk. Even with all this, a 737-NG is infinitely safer than getting in a car, and yet people drive every day. Getting into your sedan is several orders of magnitude riskier than a 737. You're far more likely to die en route to the airport.
Obviously that doesn't excuse Boeing, I'm just saying, by the numbers.
[edit] The US has 1.5 deaths per 100M passenger-miles driven. [1]
The 737 NG has hit 10,000,000 flight hours 5 year into operating. That should be around 500,000,000 miles flown. About 180 seats per plane makes that ~90B seat-miles. That's 0.0038 deaths per 100M seat-miles flown.
So the 737-NG models are about 400X safer per mile than a car.
The median American lives 17mi from an airport, so the safety break-even point is 13,600mi flown on a 737-NG, assuming your destination is also a median 17mi away from the airport. If you're flying less than 13,600mi on a given 737 one-way trip, you're more likely to die getting to and from the airport.
I think HN really overestimates the impact of events like this with the general public.
The very first A320 demonstration flight crashed, killing 3 people, allegedly due to software issues in the new fly-by-wire system, and then involved allegations that Airbus had tampered with the investigation. The demonstration flight was mostly journalists on the plane. [0]
The A330 crashed on one of its certification flights due to an issue with the autopilot, killing all 7 onboard. [1]
Both went on to be some of the all-time best selling aircraft, and meanwhile today in this thread you have people touting Airbus as a paragon of safety. Humans have a short memory.
I mention these two specifically because they're examples of high-profile failures attributed due to issues with the aircraft early in their lifecycle, but basically every single popular aircraft type (except the B777 and A350) have had serious issues and loss of life due to manufacturing defects, but still went on to be overall successes. Hell, people still flew on the DC-10 even after its _numerous_ issues like AA 191..
Or it means the torque specs were a little too low, or too high and stripped, or the bolt threading was defective/spec’d wrong. Or bad metallurgy. Or the spec didn’t specify which order to tighten the bolts, so which direction you go causes different outcomes.
Lots of stuff is assembled consistently and carefully “wrong”, but as specified.
Some part of the process -- anywhere from specs to assembly-- had some cost cutting that led to the outcome. The root cause of this problem is something financially motivated.
While I don’t disagree with you that it is possible, it isn’t a very “engineering” approach to declare that is the case without doing a root cause analysis. Stating it as fact is as bad as the MBAs…
Cost cutting could have been a factor. Or the root cause might have been something entirely different.
It's quite possible to choose to spend more money on a process or method you believe is higher quality, but still discover it has some specific problem that the previous cheaper version didn't.
That lock nut may not have lined up correctly for some reason and was backed off a bit to get the pin in? That could make it torqued under spec. But also, it doesn't look like that bolt clamps anything so it may not matter if it was a little "loose"..
Are you looking at the center nut in the photo that's cotter pinned in place, or the two over on the right side that appear to be "loosely threaded on at best, washer rattling along the shaft"?
The center one doesn't concern me. The two that aren't even finger tight do.
I was and missed everyone was talking about those other bolts in this particular case.
Those are much more problematic as the remaining tight bolts could fail leading to a cascade. Though, would expect the flange to still be connected to the post in the aftermath images and the upper pins .. how did they get out of the track?
I wish we had close up pics of all the bits in the airframe opening.
It could also mean their torque wrenches are not being properly calibrated and traced, and have become out of spec. Out of college I worked as a mechanical engineer at a torque testing firm. We made devices with high quality torque sensors which are calibrated against NIST certified weights every 3-6 months, and those sensors are supposed to be used to check every single torque wrench both before and after each work shift to ensure they are accurate. Someone could choose to ignore that process, if they were sloppily cutting costs.
Those don’t sound very reassuring either. What’s to say that this is the only bolt with incorrect torque spec, or if this part is bad why not others, and if bad parts are not being caught during quality control then what other parts will be next to fail prematurely? And if there are unexpected vibrations in that area, why wasn’t it discovered during test flights? What other vibration issues would they not have accounted for if they weren’t capable of detecting this one?
To be fair, I read recently that another issue with MAX 7 was discovered through flight testing. The issue is that now Boeing wants to exempt plane from safety rules…
What's crazy to me is that these things do not roll of the factory line in large numbers. If it was a car plant or some other line where large numbers come off per shift, you could find that possibly one operator for one shift set their torque wrench to the wrong setting causing the finished items for that shift to be suspect. But seeing as not one plane rolls of the floor per shift, this is much more systemic like possibly the documentation was wrong or similar where it is persistently done incorrectly. Or maybe just the one plane for that one day that the worker incorrectly set their torque wrench that day.
It can be opposite of your interpretation too. If you make 100,000 cars in a year, each step is done at least 100,000 times and it's justified to have a single wrench, with single non-adjustable torque value, and automated datalogging of applied value for a particular step. Or even a robotic process. Maybe even computer vision to check fastener length.
If you make 10 or 500 planes in a year, you have to rely on multiple people to use multiple torque wrenches, with multiple attachments, to follow multiple procedures to assemble multiple planes using fasteners of multiple lengths, nearby multiple other processes and people. There is a big emphasis on procedures and traceability but there are still so many potential failure points that can go undiscovered for a long time.
Well they left a ladder in the tail of a plane during assembly in the not distant past. It's not like just a couple screws or washers were left behind to rattle around.
An ex worked on a QA-type project related to production for them. It makes me wonder if some of the issues were more fundamental than that project could ever have addressed.
When a junior software engineer drops a table in production, we don’t say the junior engineer was sloppy. We say the protections and process in place were bad.
I don’t like how the sentence in this parent comment blames individuals for process problems.
It's a cliche thing to say, but if it was a known issue and management papered over it - as the alternative is incompetent engineering instead of just cost-center-managed engineering - someone should go jail?
Nah. When you have an engineering culture problem, putting people in jail is not going to solve anything. It will scare employees, sure, but it won't fix your culture.
I believe someone at Boeing actually was criminally prosecuted in connection with the earlier MAX 8 problems, but, as in all these situations, it is seriously unlikely that that fixed any actual problems. Most likely, somebody felt they needed a neck to wring and found a convenient scapegoat.
It seems there's a sense of malaise falling over everything and everyone in the USA. People at large simply cannot be bothered to care about anything, including torque specs for freakin' AIRLINERS
But when you have a quota, then you must ignore better qualified applicants in order to satisfy the quota. It's a mathematical fact.
I am not saying this is the reason why Boeing is turning to shit. But it's an interesting observation.
I myself am a "minority/POC" and I'm also extremely competent at my job. So I'm not saying that minorities are incompetent, because I am not incompetent.
When I interview candidates I give zero fucks about their colour/gender/whatever. If they are morons then I won't hire them.
It’s a current Fox News talking point. Don’t have a link at hand currently, but one host in a recent segment drew an explicit line between this disaster and DEI initiatives in general, with zero evidence. (As always, it was under the guise of “just asking questions.”)
I know of a pretty famous car tuner in the US (I won't get too specific for various reasons) and despite most people in my community going to him for service over the years, multiple people have come out and proved he rarely if ever torques anything to spec. For the longest time he even allowed customers to watch in the shop space as he worked on their cars and he'd hand-torque many things that the car manual was very clear needed to be specifically torqued for safety and operational reasons.
The fact that people would share this and it didn't curb the amount of business and referrals he got just proved to me what you've said for the longest time.
People don't like to be troubled with details and they'd rather be ignorant of them.
How many times are the torque specs available to me when working on my car? I have no idea what random bolts are supposed to be so they get the good n tight click.
Buy a service manual for your vehicle? I have yet to see an important fastener without a specification (torque and pattern).
If you are a professional mechanic, it's your responsibility to obtain the specifications and follow them. This is especially true when it comes to a licensed and certified A&P mechanic and not your neighborhood shadetree mechanic.
To be fair, manufacturers are trying to discontinue the ability to 'buy' a service manual and make you pay for x amount of time of access. And IIRC they don't format the page into something that's offline friendly.
It's still doable to get the full manual off, just not easy anymore.
If you don't know what you are doing then better stay away from anything critical. There are 'stretch' bolts, bolts that are definitely not to be overstressed or they'll weaken and bolts that will cause damage to whatever they're bolted into if over torqued. Every workshop manual and every workshop process has specs for all fasteners.
It gets worse when bolts and part are of different materials (say: steel bolt in aluminum part), that's a very nice recipe for trouble (also in the long run, not just during assembly because oxidization will almost certainly occur on the interface and oxidized metal takes up more room than clean metal so unless you very explicitly protect against it the fasteners won't come out without massive damage to the part).
The vast majority of fasteners on your car do not depend on being torqued precisely. Basically anywhere you need to worry about clearance (e.g. bearings). If it's a matter of ensuring nothing comes loose you'll have things like threadlocker or a cotter (split) pin.
I've worked in stage lighting and general construction at different parts of my career, and I never saw these being used by anyone despite them being very much required for a proper job.
WAGO clamp terminals are a godsend here - no need to take care about wire nuts going loose, screw terminals being too loose or too tight (leading to fracture), ferrules being properly crimped... just insert the cable, lower the lever, and off you go. Unfortunately, standard DIN fuses still come with only screw terminals.
My beef is that torque specs are for factory assembly. If you’re using new parts, sure, but once they’re old/re-used parts with some corrosion due to dissimilar metals, age or salt belt, you’re flying blind and probably undertorquing if going by the book.
Given the amount of (soft) aluminum on aircraft, (737 is 80% aluminum), it would be insane to not precisely consider torque.
Why would the softness matter for torque, if there has been some shift or compression since original install the torque should still ensure the correct pressure between the fitting, no?
A major concern is presumably over-torquing a screw and causing partial failure of threads in the material. This could be subtle and difficult to detect, but present as a problem over time.
e.g. any slight boogering of the threads of a used fastener could translate into rotational torque that doesn't end up being converted to a clamping force.
(or in more extreme instances, some fasteners are torque to yield, and change shape after their first use, and must be replaced. But I'll presume that's not what they were talking about here)
Skipping that part is even more ubiquitous than failure to properly torque. It takes a lot of effort to buy the correct bolts, only specialists have them to hand for a given make. Even dealers might not have every bolt. So it means you need to procure many more parts you would otherwise (the part you're replacing, plus 20 specific bolts with part numbers).. Most mechanics don't bother unless it's something really big like cylinder head bolts or an axle nut.
That is probably valid reasoning when talking about a neighbourhood mechanic dealing with all brands but it does not hold for an aircraft maintenance shop. Skipping this step makes them liable for any problems caused by failure of related parts so I assume [1] they keep to the book in this respect. Anyone here who has experience with one of these shops want to chime in? When I work on machinery I tend to follow the rules when dealing with stretch bolts since these do tend to fail far more often when re-torqued but for normal (non-stretch) bolts I do often reuse them if they're in good enough shape. I work mostly on farming/forestry equipment and our personal vehicles though, not on airplanes.
When I was in high school I helped a lot of my friends fix issues caused by quick tire and lube shops. Lots of snapped, stripped, and rounded fasteners.
I imagine at least part of this is that specs (and documentation generally) just suck now. It's nigh impossible to sort out which bits are CYA legalese, and which bits are "no, actually, do this or else <terrible outcome>".
That obviously isn't the problem with airplane manufacturing, and maybe not for car mechanics either. But it's totally endemic in the consumer world.
"Do not operate while driving" on car HUDs. "Do not consume if pregnant" on perfectly safe OTC medications. "Do not continue to ride after a crash" on bike frames.
It's not surprising most of this is just ignored now -- there's no information content. The documentation is nothing more than a list of things for which the manufacturer would like not to be liable, and the marginal cost of adding to that list is ~0. It will grow until we run out of room in the manual / space on the packaging.
"Store between 68 and 75 F." Or what? Is that a "must follow or else death", or a "it may reduce efficacy 0.5%" or a "we've never run a sufficiently powerful study under any other conditions, but there's no theoretical reason it should matter"? It matters quite a bit to me which!
I don't see how we can hope to have good-faith communication under such a heavy threat of litigation. I would not be surprised if /that/ turns out to be relevant to the Boeing issue, even if the rest is unrelated.
Culture problem as we hire people of a generation who were not educated or engrained with responsibility they avoid doing the right thing and this happens.
Get this generationalist BS out of here. It isn't supported by the facts. Airline safety has improved with each generation. There were many more plane crashes in the 1970s-1980s when boomers were in their prime. Five times as many!
Heh, I remember back then when they said "Airplane crashes come in threes", while I don't think that's exactly true, the number of large fatal crashes was much higher.
I don't think it's generational per se, but I've noticed a declining level of concern for safety in product design compared to even like 20 years ago. I think people got used to the idea that everything is generally very safe and now there's less of an emphasis, so given enough time we'll start seeing dangerous products again and the cycle will start anew.
I agree. We need way younger CEOs (and legislators!). This current crop is perfectly fine rushing QA to ship defective products as long as it lines their own pockets. In industries where failures won't directly kill people they actually plan the obsolescence to happen.
Leadership from a more environmentally-conscious and empathetic Gen Z would definitely have safer products that last longer.
Also an issue, short term success does not equal long term success. Lots of people in the c-suite management are short sighted making 1 year goals and not more.
For example a company of 400 people laying off vital talent is literally shooting themselves in the foot. If I was a CEO I would take a cut till I could course correct. Losing talent with a layoff is way more expensive.
You end up paying three to six months for that talent to not work. This is a huge problem.
> People at large simply cannot be bothered to care about anything, including torque specs for freakin' AIRLINERS
It may have not been operator error with the torque wrench. Maybe the torque wrench itself was miscalibrated. Maybe the bolt had a flaw in the metal. It seems too early to make conclusions.
A bolt from an aircraft flap control unit fractured in the threaded region of the shank near the shoulder with the head upon installation after a major service. A metallurgical investigation was carried out to identify the cause of failure. The bolt was manufactured from cadmium-plated, high-strength steel. Material checks carried out on the bolt showed that it conformed to the required specification and was found to have an approximate ultimate tensile strength of 1380 MPa.
The fracture surface of the failed bolt was examined using SEM to identify the mode of fracture and determine if pre-existing defects were present that could account for the unexpected failure. The fracture surface exhibited two distinct modes of failure. [...]
The embrittlement in this case was attributed to the cadmium plating, which is applied to the bolts to provide corrosion protection to the steel. Hydrogen is evolved during the plating process, which becomes absorbed by the steel. The cadmium plating acts as a barrier to hydrogen diffusion at ambient temperature so that the hydrogen becomes ‘trapped’ in the steel. In high strength steels (>1100 MPa) this leads to embrittlement. To overcome this problem, high strength steel fasteners, which have been cadmium-plated, are baked at 175–205°C for 24 hours to allow hydrogen to diffuse through the cadmium. In this case, failure of the bolts was caused by insufficient baking after plating, which gave rise to hydrogen embrittlement.
I share much of the sentiment of the OP, but I wonder if it is simply my perspective changing as I take on more and more responsibilities. Maybe things have always kind of been this way.
I would be hesitant to say it's all going to hell in a handbasket because that tends to be one of those self-fulfiling prophecies. I prefer to look at things as "challenging, but workable".
>People at large simply cannot be bothered to care about anything
People care deeply about a lot of things, just not this. And why should they? When you have every business decision made either by private equity or public companies with a focus only on the next 90 days it turns out everything goes to shit. Wages are stagnant. Wealth inequality is massive. Our society is sick. Issues like this will only increase. Our rail ecosystem is experiencing the same issues for the same reason.
Train travel tend to be much more comfortable than bus, car or plane. Especially if the plane takes as long time due to airports being located outside the city, security hazel, lots of stress at the airport and so on.
The train cost more, but you enter the train in the city centre, sit down and relax, go to the restaurant car and eat a nice dinner, relax som more and you arrive in the destination city.
This works better in continental Europe, where you have shorter distances and high speed trains between the capitols, often making the train faster from city centre to city centre than air travel. Traveling from Chicago to SF by train is pleasant, but take a couple days, not a couple of hours.
I'd pay extra to take the train for shorter trips. The entire experience is better, from not passing through a security theater production to the larger seats. I've never had a train sitting on the tarmac for hours, either, my plane trips frequently involve waits and delays.
I took the Amtrak a couple times from DC through WVA and have had several delays trying to get from VA to KY due to priority being given to the freight trains. I remember one trip having a one or two hour delay in the middle of no where while we were side tracked.
It is much, much more comfortable than a plane though.
I kinda knew I'd get called out on that, but, yeah, definitely more comfortable. You can even get up instead of, "everyone has to stay seated because any minute now we will be ready." Some trains have a food car. It's great.
Not OP, but as a simple physics exercise, the fact that it's cheaper to put a metal tube flying in the air instead of hurling it down a pair of low-friction steel rails firmly attached to the ground to get from the same point A to the same point B is mind-boggling. It becomes even more so when you consider that I was easily able to find a ~$500 first class flight for the same route that is non-stop for just 1.5 hours of flight time, so flying is going to be a much shorter door to door time for half the price and probably a better experience to boot.
I haven't ridden Amtrak in a long time, but I would be absolutely shocked if the first class experience on the Acella is as nice, much less worth a premium compared to flying. Regardless, there's pretty much nothing they could offer to make it worth the price premium.
Presumably the "with layover" is something crazy like a Delta flight that goes via Atlanta. You can get some pretty crazy routing sometimes if you pick the wrong combination of airports and carrier. United has a non-stop with weekday first class tickets for ~$500, so I think the point stands.
Those prices are extreme, I regularly travel 1st class by train from Göteborg/Sweden to Amsterdam/the Netherlands at about €85 for a single with ICE/EC/IC, a ~15 hour trip.
What is the reason behind this thought? I know that most people think like this. Train should be cheaper than air.
I guess it has to do with planes replacing plane, hence plane must be better, hence more expensive.
But the reality is that the plane won because of two things: speed and cost. Air is free. Miles of miles of railroad is very expensive to build and maintain.
Trains are a bit cheaper than planes. But they travel slowly. So cost per mile per passenger is lower for a plane.
Even when subsidized, the cost of rail travel is often more expensive per mile than air travel.
So when Europe and Asia now are building a lot of high speed rail, it's not to make it cheaper, but to lower carbon footprint and to make a more pleasant experience.
Amtrak and plane tickets are priced differently. For each class of ticket, Amtrak has X seats at price A, Y seats at price B > A, Z seats at price C > B, etc. When you check prices or buy a ticket, it will offer you the cheapest remaining ticket in that class. If you book Amtrak early, you'll find it is significantly cheaper than a plane ticket. Wait until a week until departure, and usually the airplane will cost less.
You're catastrophizing a bit. This isn't an America problem it's a Boeing problem.
American air travel is safer than it's ever been. Going years without a single fatality is now the norm, not an exception. The airlines are well run and do care deeply about safety. The regulators aren't perfect but generally do a good job. The NTSB is world-class.
Boeing will continue to lose marketshare to Airbus and feel some pain which is appropriate. Eventually they'll get their act together or they will continue to suffer real consequences.
Working conditions in the USA have come to a point where many don't care because the company doesn't care about them. People are fired at will after working for 20+ years with zero shits given. There is zero loyalty and in return workers just don't care either.
There are only so many checks and balances you can build in until this situation will catch up with you.
Doesn't amazon have such a huge turn over that they are running out of possible candidate to replace those workers?
Being pushed to deliver more and more packages in shorter time or inspect train cars in less time then required why would one care if a packages gets tossed or a rail car is not properly inspected?
In most jobs the only consequence is to be fired which is already possible for no reason what so ever in many states.
Sorry, but habe you ever worked in an aero assembly plant? You're making it sound like it's just like any other warehouse sweatshops job where you need to pee in a bottle while you work to meet your quotas when it's the furthest thing possible from that.
You definitely have enough time to work at leisurely pace and assemble every part right. There's not much rushing going on there. If workers get away with doing dodgy work you have a QA issue.
The report from Al Jazeera in 2014[1] about Boeing's assembly line for the 787 was pretty damning, workers were complaining exactly about quality control issues, Cynthia Cole was at Boeing for 32 years as an engineer and she mentions in the report she wouldn't fly on the 787.
So conditions at the plant in North Charleston haven't looked very good for almost 10 years...
In 1966, Boeing had 142k employees and thousands more employees working for them indirectly through suppliers. By 1970 they had less than 39k employees.
I commented on another thread[1] but I feel like it would apply here:
> My theory is that this is not limited to Boeing or even aircraft design, it's a much deeper and systemic problem affecting all kinds of fields. We've had a lot of industrial accidents lately.
> When aircraft manufacturing was an emerging industry there were tons of undocumented safety margins and "slack" in the design and production pipeline.
Over time, the beancounters start optimizing stuff, so these undocumented safety margins are eroded in the name of efficiency/profit (and sometimes even documented safety margins too).
> Furthermore, workers back in the day had a much better life when it comes to purchasing power (especially when it comes to property), and so could actually "give more fucks" about the job than they do now which is a compounding factor. You used to get a lot of implicit quality assurance back then which you don't get now.
> We've now reached a stage where these undocumented safety margins have been eroded enough that it actually starts to cause issues, and the safeguards that are supposed to catch them aren't good enough, either due to 1) they've never been good enough but just weren't really needed before or 2) they too have been eroded in the same way for the same reason.
The 'malaise' is the rot coming from uncurbed late-stage capitalism. Just about every single company only cares about short-term forecasts which leads to enshittification because there is no long-term planning or allowance for things like quality control.
Privately-owned companies or worker-owned companies are more resilient to this problem because the nature of their existence means a stronger focus on long term goals. Though there are still issues with publicly owned companies exerting enough influence to acquire and subsequently destroy private companies. For example, Rite-Aid acquiring Bartell Drugs then going bankrupt and proceeding to shutter most Bartell Drug stores leaving Seattle with a deficit of pharmacies.
“Show me the incentive and I will show you the outcome.”
Welcome to the outcome. When companies look at employees and employee training as an expense, and when the only way to improve your salary is to go to a new company, expect to see these companies to be hollowed out shells of untrained people.
This is reassuring news to me, as someone who has opted for Airbus over Boeing (and even paid extra for it) whenever given the choice since the first 737-MAX crashes in 2017.
It's "reassuring" in the sense that I'm reassured I've been making the right decision... but yeah, "vindicating" is probably a more appropriate choice of word :)
Id feel mostly vindicated (which is worse - no one likes that guy), but also reassured. Reassured that I'm not crazy!
The decision OP took might have been extreme and costly (time wise and effort mostly). People around him might have lightly whispered to him to be reasonable, or "trust the experts". This wears on you eventually and you start to doubt your own reasonableness.
Boeing starts leaving bolts unfastened and all of a sudden you remember the almost weekly stream of pre-pandemic HN articles with new material on Boeing's fall from excellence.
Boeing's marketing dept. was capitalizing that everyone had forgotten about the MAX. The few of us who remembered it maybe weirdos, but we're nut nuts!
Could AI help firms like Boeing perform more robust and automated safety checks? I'm curious about how much an issue like this can be chalked up to human error vs. poor, semi-automated QA.
This is hilarious, but I think we might be being a bit unfair.
Why couldn't you use something a camera on eyeglasses while doing the work correctly to fine tune a multimodal model, and then infer to a user wearing the same glasses? Audio reply saying "nope, try again."
You would need multiple frames per second, so not today, but not that far into the future, right?
edit: Zero-shot, I just took photos of me "fixing things" around the house, and here is what ChatGPT told me. It does not suck. Do this with fine tuning, many frames per second, and what am I missing?
If a missing some major required component aside from finetuning and frames per sec, which will require a few years for everything to be fast enough... please let me know what it is.
Wait, oh it will be understanding of time... the sequencing of the frames. That is missing for now, right?
I think you're giving it too much credit for solving examples that are obviously wrong. Not only a non-expert but even most children know those pairings don't go together.
Given that the discussion in this post is around torquing to the right specifications, I don't know if just fine tuning is enough. It might need more serious training on videos of assembly. Even then, can it distinguish between the right torque or not from video?
I think it's interesting, but you should deliberately try to make it fail to see where the edges are. Like hold a wrench of the obviously wrong size next to a nut and ask if that will work. Force perspective so they look the same size. See if it will prevent you from mixing potentially dangerous chemical combinations. Will it warn you to wear protective eyewear when using a circular saw?
Oh definitely, I was giving it some softballs. Maybe I am also giving too much credit to what fine tuning can achieve.
But in a manufacturing environment, if one labelled wrench sizes with different colors, then things like noticing wrench size would be easier. Also, I imagined that the camera would of known (same) geometry during training and inference in a Boeing implementation.
I also wonder if multimodal LLMs are blowing my mind unnecessarily. It really feels like a huge leap to me though.
They are a pretty huge leap! I just think for manufacturing and assembly where it is critical to get specific values right we cannot rely on neural nets in their current form.
I think there are digital wrenches that will record the torque they applied, you don't need to mess around with colors. But at that point, it's sort of like, why bother teaching the AI to notice if people are doing it right, stick a camera on the wrench and QR sticker on the bolt. Make a more formal verification process that can be guaranteed to match specifications.
The reason why it's bad is that this is an overcomplicated process that introduces potentially unreliable bells and whistles for little upside. At the end of the day, there aren't fundamental issues with a trained employee following a checklist. Machine learning is extremely powerful but tons of issues can be solved in a straightforward algorithmic fashion, so we should really be using it where it could make a big difference.
Thanks. I get it, and generally agree with your point here. Paying for good people and giving them enough time to get the job done is generally enough. I was just trying to be fair to op's question as devil's advocate. I don't think it should just be dismissed as a joke. I have always been a huge cryptocurrency skeptic, and I just don't see the same trajectory for ML.
So back in the day, Expert Systems were a big thing at Boeing. Searching Boeing's job openings today, it's a still a word used in hiring, but I don't understand what they mean by it in terms of manufacturing. Do you have any idea what they mean by that today?
Yeah, I understand your sentiment. I hope I didn't appear to technologically conservative in my comment, it is kind of a minor pushback to people promoting generative AI as a panacea that makes every problem ever easier.
I can't tell you about what Boeing does with expert systems. I've never been employed by them and I never really looked into it or where they use them. It's especially unclear because Boeing does a lot more than just pure manufacturing.
Neural networks for quality assessment aren't necessarily unheard of. I'm not sure how they'd be using them in this case, though?
Maybe something like a camera monitoring that they are using the correct tool at various parts of assembly? But I'm not sure how feasible this would be at airplane levels of volume.
I bet it'll be common in car assembly at some point, though.
It is not so clear if the loose bolts are on the door or if they're the bolts that attach the door to the frame. Also given that the door that fell was just found in one piece this makes me suspect the latter
But regardless, not looking good for Boeing
(*door but of course it's more a plug than a real door, just using the term for ease of understanding)
I found it interesting that Boeing did proactively tell airlines to inspect 737 MAXs for a possible loose bolt in a different part of the plane (rudder section) at least 8 days before the January 5th event. Example story: https://www.reuters.com/business/aerospace-defense/boeing-ur...
Unfortunately, Boeing did not know they had other issues with the plug door bolts.
The problem was not really the software in isolation, but that pilots expected the 737 NG to behave exactly like the old version - because Boeing decided it was too expensive to retrain pilots.
The expense for retraining pilots falls on the airline.
Retraining has its own problems. No matter how well retraining is done, pilots still make mistakes from doing the right thing for the previous plane that is the wrong thing for the one they are currently flying.
Adjusting airplanes to fly the same way is a major safety advantage.
Arguably, Boeing hit the uncanny safety valley -- similar enough so that pilots and airlines relaxed, but different enough so that relaxation ultimately killed people.
The emergency procedure for runaway trim was the same for both aircraft types, and was not followed. After the first crash, an Emergency Airworthiness Directive was issued to all MAX pilots reiterating the procedure, which was not followed in the second crash, as well as not reacting to an overspeed warning.
Unreported by the media, there was another MAX incident before the first crash. The crew had no knowledge of MCAS, but did follow the emergency runaway trim procedure, and continued the flight and landed safely.
"Runaway stab trim". It is a memory item, every pilot should be able to perform it from memory.
Turn off the motor, and the trim is manual. There is a crank right there in the cockpit. If it is too hard to turn, change aircraft configuration to reduce the forces required to. Pilot know how to do this. This pilot stuff, they understand the forces on the flight controls and what impacts them.
Boeing made an engineering mistake. The pilots also made an operational mistake. Unfortunately, both mistakes at the same time were fatal.
I pray that pilot training has improved. And that Boeing has made systems level changes to the aircraft that will preclude it happening in the future.
And that is how aviation becomes safer every year; at a significant cost of customers lives.
The fatal accident count is higher for GA, but I didn't normalize against flight hours or flights, just glanced at it.
I'm sure there's been a study somewhere that attempts to untangle all the factors that differ between commercial carriers and GA, to see which safety is most sensitive to -- continuous highly professional maintenance, highly trained and experienced crew, rigorous airliner certification regime, etc.
> The probable cause of this accident was the inappropriate response by the first officer as the pilot flying to an inadvertent activation of the go-around mode, which led to his spatial disorientation and nose-down control inputs that placed the airplane in a steep descent from which the crew did not recover.
The electric trim switches override MCAS. This was explained in the Emergency Airworthiness Directive sent to all MAX pilots after the first crash.
Also, overspeeding the airplane makes it much harder to turn the manual trim wheel. The cockpit voice recorder on the EA flight recorded the overspeed warning horn, which the crew did nothing about (they were at full power, should have pulled the throttles back).
The LA crew restored normal trim twenty-five times before crashing. What they never did was turn it off after restoring normal trim.
If a pilot can't be expected to maintain the pitch of a plane on takeoff, he has no business flying ANYTHING.
What Boeing did (and is STILL doing) is expect pilots to know or remember obscure NON-PILOTAGE (and in the case of MCAS, BURIED) trivia to prevent disaster.
Now... what's the more-responsible approach? Expect pilots to pilot, or expect them to recall an ever-growing list of workarounds to incompetent system design?
The whole MCAS was just unnecessary feature (bug fix).
Without it the plane would have worked just fine. The pilots would just have had to go some amount of training scenarios to get the certification on how the MAX plain flies.
Exactly. Unless the "upward-pitching tendency" under high power is extreme, any competent pilot should be able to keep the plane's attitude as desired.
The problem was software that prioritized input from a fauly external sensor, over pilot control, and literally crashed planes directly into the ground. At a certain step in the sequence it was not physically possible for a pilot to pull hard enough on the control element to counteract the software. Could they have disabled the system? Only if they could figure out the specific software trying to crash the plane.
Is that what you meant by "the problem wasn't the software?" Because the pilots should have been trained to unplug the computer to stop it from crashing the plane?
Even more ridiculous, Boeing offered a second source of truth option, but marked it as an upcharge, which the airlines in question rejected. "No thanks, no need for a second AoA sensor, one is none is probably fine!"
Additionally, two feels like a really strange number. I would think three for a tiebreaker would be standard for any sensor with that much impact (no pun intended).
Yes, two would be very ill-advised. I think there was some incident where a plane had only two pitot tubes and of course this caused problems. Or... I may have simply misremembered pitot tubes instead of AOA sensors in the Boeing case.
>> Because the pilots should have been trained to unplug the computer to stop it from crashing the plane?
Yes.
The fault lies with the airlines because I don't for a second believe they didn't put pressure on Boeing to get the MAX certified without mandating retraining.
And then once that was done, didn't dig into the details too hard about what changes were made.
I have a low tolerance for 'I set up all the conditions and incentives to encourage you to break the law... but you should take all the blame when it explodes.'
At some point, the customer has to take some responsibility for what they asked for.
It’s easier to blame Boeing because they made the damn thing its documentation. We know for a fact they are at fault. Some or all of the airlines may or may not have put pressure on Boeing.
This wasn’t related to autopilot and they removed mention of the MCAS system from the documentation to support the main selling point of the 737 MAX, which was that existing 737 pilots would be able to switch easily without recertification. They knew that they’d lose most sales to Airbus if the aircraft were compared on their merits so they were banking hard on their huge pool of certified pilots as the competitive edge.
If you listen to podcasts, these two episodes of Causality are excellent:
Ha, playing hardball! I wonder whether you’d find pilots who are Boeing loyalists who’d take offense, or if those guys are even madder at the current management for letting them down.
Pilots should (are supposed to) disable the auto-trim if it's doing something uncommanded/unexpected. Runaway trim can happen for reasons other than faulty software. MCAS was a new factor and they should have been told about it, I don't dispute that at all.
The manufacturer company put in larger engines than the aircraft is designed for. And they did it to avoid all the homologation licences and design costs involved in bringing a new aircraft to market with the appropriate tolerances, and to compete with another company's aircraft in time (Loss of sales).
They introduced MCAS in the aircraft for to balance by software a hardware issue, a big design negligent issue which can lead to stalling. It is beyond to trim an aircraft, and because of this there is a big difference in the scale of the values that the algorithm manages from a trimming.
It is not my field, but I think it is not a simple factor, and that it should not be put this over the Pilots like if it were a normal aircraft that received a simple update. Every pilot flying that plane should have been warned that it was not a classic plane with a classic update.
If this type of behaviour by aircraft manufacturers becomes the norm, costs over safety, we as passengers will suffer it, as other passengers unfortunately suffered it, while they blame the Pilots. In addition that nowadays the China's aircraft manufacturing industry wants to enter global market. Some days ago I read they want permission (homologations approvals) for to enter in the European Union.
PS: They also cut costs retiring backup sensors, delegating responsibility for a vital system due the MCAS to the buyer as if it was an unimportant feature; disaster was the order of the day. And the spending cuts were not limited to that, as we have seen in recent days.
> They introduced MCAS in the aircraft for to balance by software a hardware issue, a big design negligent issue which can lead to stalling.
> Every pilot flying that plane should have been warned that it was not a classic plane with a classic update.
I was mean,
> They introduced MCAS to use software to attempt to balance an aerodynamically unbalanced aircraft with a high stall tendency, in order to avoid designing a new aircraft.
> Any pilot flying that aircraft should have been warned that it was a plane that didn't want to fly aerodynamically, with software forcing it to fly without backed redundancy. It was not mere trimming.
Here we are again, this misconception just won't die.
In the 737 MAX, the only way to disable auto-trim also disables powered trim (the thumb buttons). As grand parent says, at a certain step in the sequence it was not physically possible for a pilot to trim the plain back to stability manually. It simply can't be done.
In the 737 ng, there was a button to do just that. That would have been useful.
And that's even ignoring the fact that all symptoms were very different from those present in a runaway trim situation as described in the manual and learned by the pilots.
The thumb buttons would override MCAS. But then you'd have to disable the trim motors and trim manually (by hand-cranking a wheel). That part was not clearly understood by the pilots, because they were not told about MCAS.
How are pilots expected to disable a malfunctioning MCAS in an emergency, and balance manually by trial and error the aerodynamic extravagances of the angle of attack of such unbalanced aircraft in the middle of procedures?
The user of the parent comment is remarking about time.
The aircraft can be certified without MCAS?
By what I read, MCAS is there for to avoid entering into an aerodynamic stall when the aircraft is approaching a high angle of attack, due it's using larger motors for what classical 737 was designed for. It's balancing an unbalanced aircraft using software to repeatedly adjust the horizontal stabilizer.
It is not my field, but I'm not even sure if it should be called to trim, it sounds like a euphemism for what's going on.
> Software QA when actually practiced is more advanced now than airline QA.
...eh, I think "when actually practiced" is doing a lot of carrying there.
What do you mean by "actually practiced".
Outside of the aerospace and healthcare industries,
I'm not sure there are many software shops that are doing QA to a level I would like to trust anyone's life with.
what does advanced mean when comparing things so unlike from each other?
also software is the least likely comparison I would have made; software quality is a shit-show on a general level, and the vast public is quite aware of this every time a subway timeboard blue-screens or gets frozen on an AMI screen, or the POS machine that they're forced to interact with at work does something equally as stupid.
Nah, in the software world, the truth is QA is where the people who can't get jobs as programmers end up. I've seen testers go on to become programmers, but I've never seen a programmer become a tester. Maybe it's different for real-time or life-critical systems, sure, but I can confidently say this is how it is in web development.
I work in the defense industry, it’s very much like the aerospace industry in that we deal with human life as a consequence of our work. We have software QA departments that operate very much like manufacturing or aerospace QA.
Software QA provides nothing of value to software development; having it as a dedicated function works against the overtly stated goals of the function and counterintuitively acts to degrade quality within software by mandating strict top down process and brittle end-to-end testing.
Although Software QA is intended to be an independent verification body that provides engineering organizations with tools and resources, in practice they function as a moral crumple zone [1] within the complex socio-technical defense industrial system, being one of the groups that the finger will be pointed to when something goes wrong and absorb shock to the business in the event of a failure. As a result they have a strong incentive to highly systematize their work with specific process steps, to shield them from liability, which can be applied generically to all projects.
Good software teams build quality into projects by introducing continuous integration, unit testing, creating feedback, and tightening these feedback loops. This acts to find problems quickly and resolve them quickly. Software QAs need for high level, top down, generic systemization requires them to work against these principles in practice. Bespoke project specific checks, such as unit testing, is not viewed as contributing to the final product and is discouraged by leadership who see it as waste.
To give an example of how these dynamics destroy quality in software. I once found a bug in software on a piece of test equipment where a logarithmic search function was not operating on a strictly sorted list. When I pointed this out to my leadership I was told that if we changed any part of code, it would require a new FQT, which would be too expensive to conduct and was not in the budget. Although the bug would have been trivial to solve, and was clearly wrong and would not provide any benefits by remaining in the test equipment software, the process required for changes prevented solving the issue.
It sounds like they are calling something QA but using it as a liability shield. It makes sense that you are upset about that, but naming something QA and having it do something else doesn't mean that QA as an effort is bad. It means that the people doing that are being deceptive.
Fair point, you are correct in your inference that there are some bad actors in my workplace. However, I’ll argue that the fundamental dynamics of bifurcating the responsibility of quality from software leads to a steady state where all QA departments end up as a liability shield given enough time.
This is driven by Pournelle's iron law of bureaucracy [1], which says that people who promote the bureaucracy rather than the mission of the bureaucracy will get promoted within the organization and come to dominate its decision making.
For example, in schools, administrators make more money than teachers. This is despite both groups having similar levels of education and intelligence. The reason for this is that administrators know the laws and regulations of the environment they’re working in and ensure the continuity of the organization. Despite not directly contributing to the organization’s stated mission of education, they are in charge of the organization and take more benefits from it.
Software QA has similar dynamics. A QA department may start out making good faith contributions to the organization. Eventually there are product failures, eventually leadership needs a scapegoat to show they’re doing something, and eventually QA takes the blame. People get moved, demoted, or fired. QA realizes its risk, and takes steps to mitigate it. They create a highly systematized workflow and process, adopt or introduce standards. Then assert that following process equates to good outcomes. When bad outcomes occur, they point to their strict adherence to following process as evidence of innocence.
If the process does not support the work or mission, that is a cost they are happy to impose on other functions to deal with. This is the final state until a system disruption happens.
Manufacturing process builds identical widgets using standard equipment. Widgets are inspected to confirm they are within spec. Frequentist statistics are used to determine when widgets are consistently out of spec. When this happens, equipment is inspected and repaired as a corrective action. This process is well defined and linear.
Software produces bespoke, non-standard widgets to address domain specific needs. At the end of the day, software developers are defining a process for machines to follow. If you want to control quality in the software development , aside from having perfect domain knowledge for a particular project, the only way to do it do it is to define an arbitrary process for developers to follow and track adherence to it. This may have no impact, or be a hindrance. It will never add value because it will never be abstract enough to be appropriate for every domain.
I have seen a case of Software QA taking a very different shape, so I'd like to argue that the outcome you describe is not intrinsic to software QA, but rather to company culture.
The case I'm talking about does not have a separate QA department, but QA people as part of every software team. If a product fails, that team is responsible, so software devs are in the same boat as QA. They focus on learning from these failures, so no scapegoat is needed. Process does get followed, but not as a defense mechanism, but because not doing so introduces noise that is an obstacle to improvement. In case of bad outcomes, people do point out that they followed process because then it is clear that the process is involved in the failure and should be improved.
Unfortunately, companies with that kind of culture are rare.
I can see integrated QA working for teams because QA personnel would understand project specific constraints and degrees of freedom and tailor solutions in a way that top-down QA cannot.
However, there are situations where less QA may be needed, for periods of time, such as when PRs may be low. QA may be seen as overhead by management, and something to reduce. This will lead to QA shared between teams and a push for standardization and top-down process deployment to minimize complexity for these personnel will develop. Complexity to manage the QA personnel will be shifted to development teams.
This situation absolutely is controlled by company culture. A culture that neither values QA nor development will do this. A company under financial strain will do that. Companies wax and wane constantly.
It's like saying communism isnt the problem, but that it s how every single group attempted to implement it that should be blamed.
Sure, maybe, but if nobody ever can implement the theoretical utopia, maybe we should talk of things humans can do instead and ditch the unimplementable idea.
QA cannot be done by a separate team the way you dream: it will always be a political buffer zone staffed by the cheapest half-competent people you can find, expulsing good people into dev or management. Or you merge it into dev/solution design.
The reason is simple: just like contract law, you only care about quality once you are in trouble and need to reverse back the source of the issue to give to the client a post mortem. Otherwise, you care first about velocity, or $ input/hr of effort.
> Good software teams build quality into projects by introducing continuous integration, unit testing, creating feedback, and tightening these feedback loops.
Agreed, for good software teams.
I would content that most software teams at most companies are not good.
Which is to ask, with an average to bad software team is it better to have integrated or separate QA?
Does it make sense to degrade the performance of good software teams because bad software teams exist?
Ideally we’d always have good software teams, but in the real world sometimes you have to build software with bad teams.
Leaders have options, they can do things like reduce scope, increase budget, increase schedule, or full on abandon or cancel the project. These are all options available to leaders, but they require tradeoffs and decisions to be made on a project by project basis.
It is scalable to have a strict process that everyone has to follow, then impose a watchdog to enforce it on a wide scale. It may not be better to have separate QA, but it is easier for those in charge.
It makes the most sense to me to match the org structure to the teams you have.
If I'm trying to build something with undertrained, demoralized, underpaid engineers... it's not optimal to use methods intended for self-motivated, high-performance teams.
And nothing says there must be company-wide mandates. Maybe this area gets a formal, independent QA team, but this other area doesn't.
My experience just doesn't bear out that collapsing the QA function into development always leads to better outcomes.
I've seen the opposite happen too often, and QA be the sole bulwark between idiocy and customers.
> Does it make sense to degrade the performance of good software teams because bad software teams exist?
Consider the classic statistic "most drivers think they are above average".
I posit that the same is true of software teams, almost every team will self-assess as above average, i.e. good. Those teams will then imagine that, being good, they build quality into the process and very little verification QA is done.
I have worked as a software consultant for 15 years now. I've worked with at least 40 separate software teams in that time. Every single team manager would pep talk with "this is the best team I've ever seen". Some of this is obviously blowing smoke to get people to work harder and feel good. But over the years I've had candid conversations with managers and realized that most of the time the genuinely think their team is really good, truly top 10-20%.
Here's the rub. Being a consultant, I'm almost always brought in by higher level management because something is going horribly wrong. The team can't deliver quickly. The software they deliver is bug ridden. They routinely deliver the wrong software (i.e. incorrect interpretation of requirements.)
Often times these problems are not only the fault of the development team, management has issues too. But in every single case, the development team is in dire straits. They have continuous integration sure, and unit tests, and nightly builds, and lots of green check marks. But the unit tests test that the test works. The stress tests have no reality based basis for expected load. The continuous integration system builds software but it can't be deployed in that form for x, y & z reasons, so production has a special build system, etc...
In 15 years I have never once encountered a team that would not benefit from a QA team doing boring, old school, black box manual testing. And the teams that most adamantly refuse to accept that reality are precisely those that think they are really top tier because they have 90+% unit test coverage, use agile and do nightly builds.
So, my question is, do you (I don't mean the specific "you" here, rather everyone should ask themselves this, all the time) think that most bad software teams know they are bad? Including the one you are part of? Would it really hurt to have some ye olde QA, just in case, you know, you are actually just average? :)
I'm curious: in your many years of being a consultant to these bad teams, where the manager really thought they were top 20%, did you get a chance to talk to the rank-and-file team members, and did they paint a very different picture of the team health and software quality than their manager?
Also, did you run across any orgs where they basically refused to use a process like Agile, and instead just did ad-hoc coding, insisting that this was the best way since it worked just fine for them back when they were a 5-person startup?
Not parent, but in my experience as a consultant working with bad teams, the rank and file were 'doing the job.'
You usually had a few personality archetypes:
- The most technical dev on the team, always with a chip on their shoulder and serious personality issues, who had decided to settle for this job for (reasons)
- The vastly undertrained dev who was trying to keep up with the rest of the team, but would eventually be found out and tossed, usually to blame for a major issue
- The earnest and surprisingly competent meek dev, who presumably didn't have enough confidence to apply to a better job, but easily could have made it on merit, work ethic, and skill
- The over-confident dev who read a bit of SDLC practice, and could see every tree while missing the forest
The key is that, aside from the incompetent person, they had all always been working there for awhile. Consequently, there wasn't good or bad health and quality: there was just "the system" (at that company) and dealing with it.
And none of these folks ever worked at 5-person startups. ;) I think it was definitely more an issue of SDLC "unknown unknowns" they should be doing, than willful decisions not to.
> I'm curious: in your many years of being a consultant to these bad teams, where the manager really thought they were top 20%, did you get a chance to talk to the rank-and-file team members, and did they paint a very different picture of the team health and software quality than their manager?
Yes, generally I join teams and work as an engineer or sometimes as a team lead, so I'm talking to all the team members.
Most start up teams are composed of junior developers, often pretty smart people. Usually 5 or fewer years of experience. Many times these are people who have already accomplished stuff they didn't think they could do. So that generally means that yes they think pretty highly of themselves. To a degree it is quite justifiable, they tend to be very accomplished but in a narrow domain. Unfortunately they don't realize that their technical accomplishments in a specific field does not mean that they are experts everywhere. Their managers understand that these are smart people and assume again that this is therefore a good team.
Non start ups that I join are usually just plain dysfunctional.
> Also, did you run across any orgs where they basically refused to use a process like Agile, and instead just did ad-hoc coding, insisting that this was the best way since it worked just fine for them back when they were a 5-person startup?
Usually more the opposite. In my experience I come across teams that are sure they must not need any help because they follow all the rules in Scrum and have great code coverage metrics.
It is really common to see this kind of thing. I call it "the proxy endpoint fallacy". It can crop up anywhere that there is something that can be measured. In that example, it would be confusing adherence to Scrum with having a working SDLC or perhaps confusing code coverage metrics with the objective of having bug-free releases.
This isn't a software only fallacy. In politics, GDP is often confused with societal well-being. Always be wary of your metrics and change them as required to keep you tracking your actual goals.
Depending on the shape of the distribution, most drivers could be above average. Average doesn't imply 50th percentile, that's what the median is for. A minority of tremendously poor drivers could certainly mean that most drivers are in fact better than average, in the same way that my friends on average have more friends than I do.
If your devs aren't good what are the chances of your QA team being good enough to make up for their short comings? The dynamics laid out by the parent comment will just hit even harder. Your best bet is to enforce basic practices like continuous integration, coverage goals and maybe a coverage ratchet as a merge gate. Training and education on areas were the team is weak is also a must.
About 2000, Software QA (and almost all traditional QA activities) were changed. The focus was on process over inspection.
"Design in quality, do not inspect it into the product"
Suppliers (to include software) were expected to manage the quality of the product they provided; the purchaser would focus on how they managed the process, not in the compliance of every part.
This had a chance until software process was tossed in the name of "agile".
Here's a stupid question: How do you know your process is good unless you inspect it?
"Hey Bob I know you're a competent engineer, but don't worry about specifying a certain type of bolt or loctite, the untrained assembly personnel will figure it out. I'm sure they won't let 200 people die in a plane crash."
I recall a bug I was involved with at a telecoms equipment market in the early 2000s. The bug only showed up in our biggest base stations in high load situations. We diagnosed the bug, and there were a couple of parts to it. Sloppy software design in an optional hardware module (no state machine) was one part - and was fixed. But there was another underlying issue in the way message queues were handled.
Anyhow, the fix for this was created and written. But we never got to put it into production. The reason: the company didn't have a lab test facility that could put a sufficient load on the software to prove it. Even though we were getting field failures because of this issue that were getting a bad rep, we couldn't fix it because even though the old code was known to be buggy, we couldn't prove the new code. So the process said we couldn't ship it.
Another way of looking at that is that within the ability to test, the implementations were indistinguishable, so the process mandated that the older implementation must be used. I wonder if they would have explicitly specified age as a metric if this was considered when designing the process.
> We have software QA departments that operate very much like manufacturing or aerospace QA.
I don’t work in this industry, but this seems fairly ridiculous on its face: software is not at all like manufacturing.
In manufacturing, there’s a design and a manufacturing process, and a critical function of QA is ensuring that the manufactured produce is manufactured to spec.
With software, the software is written, compiled, and then repeatedly copied. And something should verify that it’s copied correctly, but this is straightforward and boring.
So software QA ought to be much more like the kind of validation that happens when designing hardware, not like the kind of testing and validation that happens as products are manufactured.
Ideally there should be a solid spec written and then qa can test against the spec. Maybe there is somewhere that does write solid specs, including accounting for corner cases, but in my 25 years working professionally in the industry, I’ve never seen it.
The only complete and precise specification of software is the code itself. If some other form of specification was complete, we would be able to auto-generate the code.
This is beside the point. The code specifies what the product _is_, not what it "should" be. If you ask for a word processor and I deliver a perfectly bug-free and feature-complete calculator would you really believe it lived up to spec?
This is also beside the point. I think both of you are trying to warn against the dangers that lie on both sides of this coin: people can invest too heavily in a specification and waste an enormous amount of time, and people can immediately jump into coding and code something that does not do what it was intended to do. Like with all things in life, there’s a balance between these two extremes that’s correct.
You need some level of specification so you know what you’re building, but you have to keep in mind that the final code defines what the behavior truly is. Sometimes, that behavior unintentionally becomes part of the specification because users begin to rely on it.
I do like the fact that you both used hyperbole to succinctly illustrate the dangers of veering too far in either direction though :)
A (human language) specification is simply _enough_ information about a system that a human can figure out the intention of the author. The smarter and more context-rich the human, the simpler the specification can be. The dumber and less context-rich the human, the closer the specification needs to be to code.
It's asymptotic. By the time you reach a human who is as dumb as an actual computer, the specification _is_ the code.
I work in MedTech. We do this. A design has to be reviewed by QA, and is then tested, and the test is reviewed again. So just to counter the narrative, there are companies that do that, and it is working. In other jobs I also saw the cargo cult of QA. But in some industries it is just crucial, otherwise the pressure is too high to cut corners to implement something. It is a good mechanism to counter the need to move fast and break things.
I work in software for CPU design/verification. Even here, where in theory there should be a rock-solid spec, there's not. There's a 12,000 page architectural specification, which is very helpful for specifying all the end-user visible state. But the microarchitectural specification is scattered all over different PDFs, visio docs, excel sheets, and sometimes the only spec is the RTL code itself.
I think that's why people always tell each other to not take things at face value.
Of course there is a big difference between sw and hw QA, in the thing that they test, and how they test them.
But they are also very similar. Any QA department has to think about ways that things can go wrong, and what things to test for, how to test, which testing methods, which standards to handle, keeping certifications, etc. During testing you also need to keep reevaluating if you actually are catching each problem/bug and how to implement changes in your company that decreases the amount of problems or increase the amount that you catch.
I think in that way there's a lot of overlap in thinking about business processes and how to identify problems with them.
Of course once a specific binary gets tested and approved by QA it shouldn't matter if it gets copied or whatever as long as you make sure its the same binary (by a checksum for example).
But still making sure that errors don't reach the customer, is vital in any QA. If errors does happen, QA is the department that can make sure that it doesn't happen again. And ofc be able to proof in court that you did your due diligence if something does happen.
> Good software teams build quality into projects by introducing continuous integration, unit testing, creating feedback, and tightening these feedback loops.
No. Good software teams are led by competent, technical management. Managers who aren't afraid to get down into the dirty details. Managers who aren't afraid to roll up their sleeves and write code if they need to.
The process doesn't matter. The management of what is or is not important does. Agile is just one process out of many.
Imagine an accounting team led by someone who never did accounting in their life: "Just make the numbers work out! I don't care how you do it! My bonus is at stake!"
Why not both? Am I missing something? You can have feedback loops and CI and all that, "good craftsmanship" or "good practices" (not "best" practices because those often suck hah), where of course opinions vary on the details of that -- and then someone who is also good at the craft who spends more or most time on helping the rest work together, i.e. manage/lead them.
I’d bet the children would come out better simply because they have parents who are likely multi-disciplined as a group. A disparite group will (almost) always come up with better results than a homogeneous one (at least in my experience)
> Good software teams are led by competent, technical management.
...or perhaps with no managers at all. I'm less and less convinced of the importance of management in engineering except to give investors an illusion of control.
I sort of agree, and I do think it’s possible depending on the team. But unfortunately developers can be too opinionated and get focused on low priority things.
Sigh... This myth that the only people who can competently manage developers are other developers has been floating round for decades.
For some reason, developers seem remarkably blind to the skills other roles and disciplines require. Only a developer can do that, everyone else is basically useless fluff. Maybe it's a form of arrogance or just deep unself-awareness.
Let's apply your reasoning to medicine. I'm sure you would be completely fine with managers telling your surgeon what parts of the surgery can be 'optimized away'.
Hahah, indeed. So have you seen a law department in a company headed by someone who doesn't come from law background? How about a finance department headed by some schmuck who doesn't know anything about finance?
I've seen plenty of departments managed by people who don't come from the background of the department. My current boss is extremely good and came from a different discipline.
Although I don't deny it can help to have the background, it is not necessary to be a good manager of something. Also seen plenty of good techies promoted to management and failing badly.
This is a lie, and you know it. Even a mere idea about lawyers being managed by a non-lawyer would be laughed at. Same with finance, nobody would be stupid enough to even try it
I disagree with you. You are stating it, but you are not giving reasons. Managers who weren’t developers tend to not be able to manage the team. They can’t help with or understand the technical decisions made. The non-technical managers tend to be project managers just focused on dates.
We may be talking about different levels of management here. A manager should not be making technical decisions, they should have team leads and architects who do that. It's their job to manage the team, interface with the business, prioritise work and give cover to the team so they can get on with it.
I guess if you have a manager who is making technical decisions, they are really a hybrid manager/contributor role. Maybe that works better in smaller organisations.
Then what exactly is the non technical manager's added value?
He has no experience to lead the team in high pressure situations. Like production being down.
He can't truly have a first person understanding of the work of the people who he manages. He has to rely upon others to tell him who's good and who's bad. That sets up a pecking order.
He can't help or mentor engineers with design decisions, or provide a historical context.
He doesn't understand the technology so there's an immediate communication and knowledge barrier that has to be overcome between him and his directs.
He doesn't feel the pain of a bad decision, because he's not coding it, and he can't emphasize with them since he doesn't code.
He tends to push feature development without fixing technical debt. Again that's pain he personally doesn't feel.
To me, the role you are describing is a principal engineer or team lead, not a manager.
Simply not true however that a good manager can't lead the team in a high pressure situation. I'd say that exactly what a good manager could do well. Obviously they won't be making overtly technical decisions, that's what you are for. They can make business decisions, provide cover, get resources, communicate to other stakeholders... All the bits that need doing but would be a huge hassle for the techies who are trying to fix the issue.
Not everything of value is technical. That you don't see the value is either because you have great managers shielding from having to deal with all that, or you have always worked in a place that combines management and technical responsibilities (which I never have except for very small companies).
I wouldn't call what I'm describing as product management, although it's possible they could do general management too.
I'm not going to argue with the general thrust of your comment, which I think is insightful as to how incentives can compromise objectives. But...
> To give an example of how these dynamics destroy quality in software. I once found a bug in software on a piece of test equipment where a logarithmic search function was not operating on a strictly sorted list. When I pointed this out to my leadership I was told that if we changed any part of code, it would require a new FQT, which would be too expensive to conduct and was not in the budget. Although the bug would have been trivial to solve, and was clearly wrong and would not provide any benefits by remaining in the test equipment software, the process required for changes prevented solving the issue.
I've seen this happen where it was a bad thing, but also where it was a good thing.
It's all about risk.
What risk does the software defect pose to the mission? What risk is inherent in making any change to the software? Noting that even trivial changes can be fat-fingered and thus are a source of risk. I've seen it go wrong this way: a seemingly trivial change was made, but the developer accidentally checked an extra file into source control, causing a further defect.
And then: what is the cost of mitigating these risks? Maybe the software defect is as trivial as its fix. Maybe an acceptable fix would be to write up a workaround in the documentation.
I don't think it's always wrong to say no to fixing issues. I also don't think it's always right that a separate QA department contributes nothing to the organization, even if they act as a handbrake on the software developers (sometimes, precisely because they do that). Human factors are real.
I think you're sort of misunderstanding the role of QA.
You think that QA is a liability shield, but that is only a side effect of the work that they actually do.
The task of QA is exactly that: an entity that tries to assure that the quality is up to some standard.
Even in favourable conditions mistakes happen, so how do you make sure as a company that not 1 in every 100 product are faulty and tarnishes the good reputation that your company has spent so much time and money on to build? You hire a QA to make sure problems get caught before delivery.
But if all humans make mistakes, and QA is human, how do you make sure that the QA doesn't make a mistake? A never ending chain of QAs expecting each other?
No of course not. One thing that helps with reducing errors is to have a rigid protocol that is followed to the letter everytime. Pilots, for example, have a preflight checklist that they have to run every time they operate the plane.
The rigid protocol of QA teams is therefore an essential part of their jobs.
Although from your standpoint as a developer it might seem strange that QA is 'preventing' you from fixing a bug, it is actually very reasonable.
Especially since you work in the defence industry, I hope you understand that it is very important that the software that operates radars, planes, missiles, bombs, etc is working exactly as expected. Understandably there is a great deal of effort made to assure that when those things are needed they work exactly to spec.
So in your example it is probably very reasonable that any change you make needs to go through some rigorous process. The fact that it 'only' was about test equipment, doesn't matter because test equipment is just as, if not more important as the stuff it tests.
The reason why QA has the side-effect of being a 'liability shield' is that it gives companies the ability to argue (and proof) after the fact that the company did their due diligence in making sure that the product was to spec.
Especially certification is basically to get an external organisation to approve your QA. In that case if you get sued you can rightfully claim that you did everything that was legally asked of you, and if there is blame, then it is the certifying company using insufficient standards.
Is it plausible that Boeing has "learned" from software/startup/venture-capital culture with regards to tolerating higher risk to minimize costs?
I suspect it's rather a case of parallel evolution between McDonnell Douglas brass and software startup culture, since cost-cutting culture goes back many decades (remember "Chainsaw" Al Dunlap[1] ?) — but I wonder if there's a more direct influence.
In lots of ways, the "learning" there would just be "capitalism".
It's inherently short-sighted unless forced to do otherwise by legislation. Cutting small corners pays off A LOT until the hammer falls, so there's a massive advantage to doing it / you need to do it if competition is doing it, or you eventually shut down as they take all your business.
It's inherently a race to the bottom. Sometimes that's a net gain, sometimes it isn't.
Yep, Chernobyl being a prime example. Or Komarov's failed re-entry after complaining about the design faults of the vehicle long before launch. Then there was the more uhhm run of the mill backyard blast furnace campaign which contributed to misallocation of workforce which then led to mass starvation.
There are many many more examples. I find it so tiresome to see young people just use capitalism as a catch all for the failure of something. It's such a lazy and uninformed argument.
I'm not carte blanche defending capitalism - its a mixed bag but it sure outpaces the competing systems put forward to date. It does need some stronger safeguards against industry self regulation - that has a bad track record.
I think we're on the same page. Economic systems need failsafes so that they don't suffer from positive feedback loops.
What anti-capitalist sympathizers, in my view, don't realize is that this is due to people being in the loop. These economic systems are merely vehicles, some better than others, but the conductors are people, be they communists or capitalists. At least with capitalisms there is a delayed regulator (negative feedback) in communism it's up to the system to decide if it needs to modify itself.
All major economic systems of all major national economies over the last century have perverse incentives. It’s not a capitalist thing.
Other systems had incentives such as, get it running by such and such date or have yourself and relatives sent to inhospitable place. So people rushed flawed designs into production.
That said, upper management at Boeing needs a shake-up. People need to get fired. They need to do what Intel is trying and that is to get more engineers in charge, or at least grant them veto power on designs.
> All major economic systems of all major national economies over the last century have perverse incentives. It’s not a capitalist thing.
It should be a lesson against dogmatic pursuit of absolutes: capitalism comes in a wide range of flavors, and the worst is if it’s completely unrestrained. Communism produced worse and worse results the further it got from any sort of public accountability, etc.
The two problems that I see is that the concept of nuance is somewhat at odds with having a simple concept to teach kids at school, and there’s always a group which is more motivated to game the system than the average person who really just wants to hang out with their friends, raise a family, etc. rather than play political games. Boeing didn’t start it by any means but they’ve benefited enormously from decades of reduced oversight and elevated pay driven by a sort of cartoon libertarianism where letting people get enormously rich will motivate them to build great things unfettered by “red tape”.
> All major economic systems of all major national economies over the last century have perverse incentives. It’s not a capitalist thing.
They have, but post-Thatcher neoliberal capitalism has taken the existing perverse incentives and made them exponentially worse. We're on a course heading straight to feudalism, just with fancy titles with legal rights replaced by economic might.
That adage is okay, but for it to work not everything can be forgiven — there actually has to be an expectation to be held responsible towards acting on good faith.
Boeing Airliners are much safer now than before they merged with McDonnel-Douglas. (Because basically all airliners are.) And I say that as a regular Boeing critiquer.
Here's a Netflix documentary (in the wake of the MCAS crashes) that alleges that after the merger with McDonnell Douglas, the culture of the firm changed. Previously dominated by engineers, it was now dominated by MBAs with a focus on profit and shareholder value.
"With impressive clarity, Downfall: The Case Against Boeing reveals corporate corruption that's enraging in its callousness and frightening in its scope."
Would you pay at least 2x for your software to have couple more nines of reliability? I’m gonna guess that “no”. At places where it costs $$$ to have bugs shipped to the end customer (e.g phones) or where there’re regulatory requirements they still have dedicated qa.
Which is what i said in second part of my comment. For most software businesses the cost of shipping a bug is trivial and/or poorly measured so due to McNamara fallacy it is readily exchanged for well measured cost of having a functioning qa team
I’m regularly critical of Boeing Defense (particularly space contracts where I’m a huge Boeing skeptic), but I think people are pretty off base if they think Boeing is just completely incompetent.
Airliner safety is insanely good. Just vast seas of competence, but when there’s a super rare failure, the incorrect impression people get is that Boeing (or Airbus) is just full of incompetency. Almost nothing that humans do is held to the same standard. Not spaceflight, not software, not healthcare, and certainly not automotive.
Flying a 737 Max with a bad door and without the fix to the angle of attack sensor is probably still better per mile than driving. In spite of going at 10 times the speed and miles above the Earth.
You can almost argue it’s held to a higher standard than it should, slowing development of cleaner aviation (and therefore killing more people in the future due to tertiary effects of climate change, etc).
It kind of annoys me when comment sections are filled with people talking about how incompetent Boeing is. It feels like out of shape slobs on their La-Z-boy chairs talking about how incompetent or slow some professional sports players are. Like, airliner safety is just a totally different league than almost anyone else plays in. On the worst day, their better than almost anyone else is on their best.
Because I dug it up for another comment, commercial carriers operating under Part 121 (roughly: scheduled passenger and cargo operation) had 4 fatal incidents in the last 10 years. [0]
Totalling 6 deaths.
In 10 years of US commercial carrier aviation.
One of those was literally 'the engine exploded and threw part of the turbine into the cabin (and also shredded some of the wing)'!!
Which resulted in 1 person dying and a successful landing.
Ya but your sample size is way too small to measure the death rate. Aircraft deaths are rare, but flying is too.
The two MAX 8s that fell from the sky were 100% Boeing's fault and could have happened in the US. If 5% of airline traffic is in the US you can renormalize those hundreds of dead and you get dozens dead.
We know US pilots have been warning about the same issues that led to the deathly crashes later but were ignored. The thing is, one part of US commercial aviation being so safe is a lot of pilots responsible for the jet airliners are ex-military. Someone mentioned Southwest Airlines Flight 1380, yup, captain Tammie Jo Shults was one of the first Navy female fighter pilots. Miracle on the Hudson? Sully Sullenberger was an Air Force captain and training officer. Civilian training, no matter how good, is just no replacement for military training and experience.
I can't find specific numbers but estimates say about one in three has a military background. That's an awful lot.
> I can't find specific numbers but estimates say about one in three has a military background. That's an awful lot.
Not surprising given that pilot training is really really expensive. Airlines love former military pilots because they are a significantly lower financial risk for them. Put them into type rating and off they go, it's rare that one ends up as a dud.
Let's assume American pilots are gods. They were shouting that their crafts were unsafe.
No matter how good they are and how prescient, that doesn't help them if the aircraft computer decides it's stalling, forces a nose down and they cant fight the controls.
But, even if we assume omnipotence from these American pilot gods, and assume they can fly outside the bird and Superman-style catch it, they are still only 30% of American pilots. Just another population to normalize out.
> forces a nose down and they cant fight the controls.
But that's not what happened. According to every report, it is possible to take back control, it's just very much not intuitive and the situation was confusing.
According to the Seattle Times
> However on both accident flights, the angle-of-attack sensor failure set off multiple alerts causing distraction and confusion from the moment of takeoff, even before MCAS kicked in.
> On the Ethiopian Airlines flight, for example, a “stick shaker” noisily vibrated the pilot’s control column throughout the flight, warning the plane was in danger of a stall, which it wasn’t; a computerized voice repeating a loud “Don’t sink!” warned that the jet was too close to the ground; a “clacker” making a very loud clicking sound signaled the jet was going too fast; and multiple warning lights told the crew that the speed, altitude and other readings on their instruments were unreliable.
> I manually positioned the thrust levers ASAP. This resolved the
threat
Then there's
> B737 MAX First Officer reported that the aircraft pitched nose down after engaging
autopilot on departure. Autopilot was disconnected and flight continued to
destination
> I called "descending" just prior to the GPWS sounding "don't sink, don't
sink." The Captain immediately disconnected the autopilot and pitched into a climb
Another
> Takeoff and climb in light to moderate turbulence. After flaps 1 to "up" and
above clean "MASI up speed" with LNAV engaged I looked at and engaged A
Autopilot. As I was returning to my PFD (Primary Flight Display) PM (Pilot
Monitoring) called "DESCENDING" followed by almost an immediate: "DONT SINK
DONT SINK!"
I immediately disconnected AP (Autopilot) (it WAS engaged as we got full horn
etc.) and resumed climb
> It kind of annoys me when comment sections are filled with people talking about how incompetent Boeing is. It feels like out of shape slobs on their La-Z-boy chairs talking about how incompetent or slow some professional sports players are.
People do this with everything though, and air travel induces a large amount of fear in the populace. Not only are we not generally comfortable flying in the air for obvious reasons, but when it happens almost everyone has to concede control to a few people in the cockpit and on the ground. Driving, even if exponentially more dangerous, affords the illusion of control of one's outcome, given driving or having someone you know driving, and control over the vehicle maintenance, etc, as well as familiarity with the control and mechanism of the vehicle. These things don't exist with airplanes for the vast majority of people.
So, you can see why there is a need to find a human component to air travel problems, because that is something one can fix (fire the incompetent people, fine them, whatever), as opposed to all of the other things which must be accepted or rejected entirely.
It is entirely in line with human nature to do this, regardless of its accuracy or effectiveness.
Being better then driving shouldn't be the standart. Specially driving in the US.
Flying isn't safer then trains I would assume.
Flying has the advantage of being seperated from almost everything else. Most accidents happen when there is mixed traffic, specially cars operated by people with minimal training.
> Airliner safety is insanely good. Just vast seas of competence, but when there’s a super rare failure, the incorrect impression people get is that Boeing (or Airbus) is just full of incompetency. Almost nothing that humans do is held to the same standard. Not spaceflight, not software, not healthcare, and certainly not automotive.
And there's good reasons for that. Spaceflight actually is regulated pretty strictly (partially, because any spaceworthy rocket is effectively a missile), and space pilots and tourists both sign up for such missions fully knowing that they will have a very significant chance of dying one way or another - there simply hasn't been enough human spaceflight activity to work out and understand all the failure modes, unlike with other forms of transportation.
Humans, unlike birds, aren't naturally wired to travel by air... they need to be able to trust their lives to a significantly higher degree to someone else behaving like they should, because unlike in a car they have zero control (or the illusion of control) in an aircraft.
Additionally, the inherent security risk of an airliner is very high: what is a widebody airplane at its core? Hundreds of tons of weight, a decent portion of which is fuel, propelled at near-supersonic speed, and only two people in control of it. Anything goes bonkers and you can get thousands of people killed and injured (see 9/11).
In contrast, cars, even trucks, have way less capability to cause damage simply because they weigh so much less. The only thing that comes close is railways, and hell I don't get what the US is doing there, there's barely any regulation compared to European standards (see the videos I linked at https://news.ycombinator.com/item?id=38725988).
Cockpit resource management is also something a lot of industries can learn from. As well as human error analysis. How an error came to be is often much more interesting then the personal shortcomings of the person who caused it.
At it's best, software QA and related methods should be equal to airline manufacture.
Think of railway signalling systems, control-by-wire bits of modern cars, medical equipment, etc. Where the design of the software is formally proven, and the implementation verified to ensure it fits the design.
of course, but most of us aren't working on products where a quality problem would kill hundreds of people. Having aircraft-level QA would be plain silly, you don't expect that level of quality from most other industry like eg guitar manufacturing, do you?
Nope. It is most probably caused by operational stress - rudder assembly is moving, fuselage is also working (compression and decompression cycles on take off and landing, thermal expansion and compression). I bet they don't just put red Loctite on it to keep it from getting loose. My bet is design flaws, not manufacturing or QA.
EDIT: I saw the pictures of bolts with pins and bolts without pins. The ones with pins cannot get loose, the others can. Let's see what happened.
I don't know about you, but in my industry, "QA" also means extensive testing to ensure that part/assembly/etc doesn't break with expected operations. So, yeah, from where I'm standing, this was a QA problem. Something did not get checked or tested as it probably should have.
The 4 restraining nuts and bolts on the door have a cotter pin like mechanism to prevent them from loosening. If assembled correctly they cannot loosen unless the pin fails.
Bolts are most likely tightened with a torque wrench or a gun that is set to a torque spec. Over tightening a bolt is as bad as a loose bolt.
I speculate these passed QA from Boeing because they might have been correctly torqued to the spec. What happens in field is hard to understand. One possibility is vicinity to the engine can cause extreme vibrations, these can make them loose.
Other possibility is the maintenance side of things - maybe a badly calibrated torque wrench could be the reason.
Mechanical systems are not inherently immutable.
I would expect lock wire or some other method of ensuring the bolt does not un-torque itself. Especially for bolts that are not required to be removed past final assembly...
Given everything I've seen so far, I'd bet good money that what happened here was miscommunication between Spirit and Boeing. Spirit started out locking down the plug, then Boeing asked them to just loosely attach it[1] so Boeing could yank the plug for interior/wiring/AC/paint, then someone at Boeing forgot about the "loosely". So now, they get in a hurry (maybe the AC/interior didn't need any access to work on, which makes sense for this MAX variant, it wouldn't need as many hatches to pull wire) and it went down the Renton line as if the plug was fully installed. It's enough to pass high blow inspection and other inspections, but then over time that "shipment config" attachment vibrated out, and pop goes the plug.
Almost certainly systemic issue though, so that sucks. Sucks real bad.
They need to get a Tiger Team or whatever together to look at everything with a shipment config, and make sure those "ship kits" don't leak into the real actual airplane configuration. This is . . ok, this is really manufacturing 101 stuff, but well, things happen.
I'm in the industry, but haven't touched the MAX, so take this with a grain of salt.
They are not related. Probably different types of bolts, for sure different stress types. Rudder assembly is a moving part, these false door panels are not.
It's not a control surface, but it is a "moving part." That's what's baffling to me, that they spent a lot of effort building this hinge and pin roller system, and designed the door to hinge open up to 15 degrees.
It makes me wonder if there's maintenance procedures that at some point would require the operation of that door to successfully complete. Otherwise, the mechanism itself seems so incredibly overwrought, with lots of additional bolts, castle nuts, retaining pins, and even sprung hinges at the bottom.
Does anyone know why this "plug-type non-plug door" is built this way?
It needs to be usable depending on how many passengers the interior is configured for.
So it has all of the door bits there. Maybe some parts like the emergency escape slide are not installed.
e: I should be clear that it's not usable as an emergency exit, as configured by Alaska. However the operator could choose to activate it later and install a usable exit.
This is not correct. To the passengers, this just looks like another seat next to a window with a plug installed. It's not a door.
If there was a reconfiguration to a seating standard that required the extra exit, the plug would be removed and a proper door would be installed, with the associated interior pieces.
> To the passengers, this just looks like another seat next to a window with a plug installed. It's not a door.
This is true.
However there's still common hardware in there to allow the plug to be installed and maintained. This is why it's a complicated set of kit vs just bolting in a permanent fixture.
Where did you see that? My understanding is that it's an optional plug door that's used to assist with interior installation. Once the interior is done, it's bolted shut and interior paneling is installed over top. From the inside, you can't tell it's there.
Ahh. Well, in the case of the Alaska flight, it's a plug door and not used as an exit. It's pinned in place with large pin that has a bolt, a castle nut and cotter pin which lock the pin.
If you are correct, then the implication is that the concern extends beyond door plugs for MAX-9 737s to all emergency exit doors on all models of aircraft sharing this design. This is somewhat reminiscent of the huge problem with the 688 (Los Angeles) class submarines, where the discovery of a faulty weld that had passed inspections raised doubts about all welds.
This is not true. It's designed to be opened when inspecting the fuselage for corrosion or stress cracks at the opening. To open it you have to remove the interior plastic panels and undo the 4 bolts that this accident is about
I recall a running joke from my childhood - from a former communist East European counry - about a certain car saying you should finish the assembly at home after purchase, tightening the screws before first use. Despite being a famously poor quality car - even in the sloppy East European practices - that supposed to be a joke not to follow suit!
My grandfather worked for Braniff as a mechanic, and I've heard stories where very strange things would happen that definitely gave merit to some sort of sabotage being a likely explanation. From very specialized tools would be missing not from just one bay, but from all of the bays to other issues that would cause regular maintenance from being able to be completed in a timely turn causing more and more planes to be taken out of service. Lots of things went wrong with that airline, but some of the things just makes you scratch your chin in wonder if it might not be possible.
I really doubt it is but important to keep in mind that foreign governments running active espionage campaigns against Boeing. The French have dozens of agents. This is from a conversation I had with a person who works in a large counter espionage group at Boeing. This was corporate espionage not defense but I'm sure there's a group for that.
There were only 2 possible causes of the incident, either the bolts had been replaced with bolts made from an inappropriate material, so they broke, or the bolts had not been assembled correctly, and they became loose.
It appears that the latter is what happened.
Perhaps those who did the assembly of the doors at Boeing did not use the right kind of washers that are needed to prevent the unscrewing of the bolts, or they did not apply the correct torque to the bolts.
It is extremely surprising if such trivial errors can happen during the assembly of an aircraft.
No matter the real root cause, surely multiple layers of precautions must have failed to allow this to happen, right? So given that's a foregone conclusion, how can we discredit a hypothesis based on that?
- "bad parts from vendor"
- "bad programming on torque wrench from MRP"
- "insufficient training on process"
- "tooling was programmed correctly but bad sensors"
- "lube mislabeled for 3 days"
This is absolutely not some simple thing. This is why people don't take us seriously as engineers.
For loose bolts, incorrect assembly is by far the most likely cause.
It can be a design problem if the bolts and associated parts like washers have been substituted recently or if the assembly instructions have been changed recently, e.g. by specifying a different torque. If any such engineering change has happened recently, then that would be the likely culprit.
A resonance problem of the aircraft body as supposed by another poster seems extremely unlikely as that would have required significant recent changes to the aircraft body, which did not happen.
> There were only 2 possible causes of the incident, either the bolts had been replaced with bolts made from an inappropriate material, so they broke, or the bolts had not been assembled correctly, and they became loose.
Or the ‘correct’ assembly instructions weren’t actually correct, or the design isn’t good enough (e.g. it uses too few bolts, or didn’t check the strength of the bolts when they get cold), or the bolts were of the correct material, but designed too thin, etc.
There's at least 1 million root causes of the incident and while I have my own predictions, this industry is a great example of checking all possibilities before jumping to "2 possible causes".
It's really unlikely that washers are a root cause (spring/split washers are useless and are definitely not used for this type of apllication). "Correct torque" is not trivial to determine and maybe it was calculated right or wrong based on right or wrong materials/conditions or maybe conditions were changed or maybe...
I am 99.9% sure a procedure was followed and a torque or other measurement applied and was documented. This is probably not a trivial error but a cascading failure in design and/or documention and/or integration.
Workers are trained and qualified to sign of on their work. Some work is double checked by a co-worker. Some is again checked by QA. FAL work, and loose bolts falling under FOD absolutely, are checked by QA. Or at least should be.
No idea how this can happen, Boeing really has to get its shit together.
This is SOP in manufacturing to indicate that a bolt has been torqued to whatever torque specified in the assembly drawing; as an added bonus, if the mark is applied properly, you can also tell at a glance whether the bolt has worked itself loose.
Unlike on a rollercoaster, these bolts are hidden within the fuselage behind the cabin panelling so a regular visual inspection to see if they've been loosening wouldn't be practical. During assembly the bolts should have been torqued and then checked and the bolts would only be reinspected when the panelling is removed after 3 years or so.
Yes aircraft fasteners are often torque marked, and almost all safety critical fasteners have an additional preload locking mechanism. Often these fasteners use mechanical locking features such as cotter pins, safety wire, or safety
cable.
Even on light aircraft some bolts will have a safety wire through the head to prevent it from turning. You might even see these on bolts holding the propeller on.
> I've seen on Toyotas and also at Universal Studios bolts on rides are marked with paint presumably to indicate that the nut hasn't backed out.
This is also standard procedure for electrical maintenance inside distribution equipment. Every lug/termination is torqued to spec and then marked to indicate it has been torqued down. This should be done regularly to prevent any loose terminations from causing an arc flash.
If a guy holding the drill CAN cause this issue, that means there's ALSO a drill calibration issue, a bolt inspection issue, an inspection recordkeeping and doublechecking issue, etc.
Humans make mistakes. The whole point of modern manufacturing is to make products better than any human can make them, by layering processes and procedures to catch those mistakes before they get out the door, and continually improve the processes to catch ever more.
In The Design of Everyday Things Donald Norman mentions the swiss-cheese model, whereby multiple layers in a process need to align correctly for a mistake to lead all the way to an incident: https://en.m.wikipedia.org/wiki/Swiss_cheese_model
Keep seeing this take (just an installation issue), over and over and over again.
I think it's much worse than a design flaw.
Even very carefully-engineered systems can have flaws. Engineering flaws, once identified, can be engineered around, managed, or corrected.
I'm much less comfortable with the idea that the assembly plant for these planes could be a random-critical-failure generator based on how the employees handle a torque wrench.
If this turns out to be the assembly issue it seems to be, it hints at a deeper, potentially cultural, issue at Boeing. One that could affect all their planes, because why would only be the B737 MAX FAL be sloppy?
Boeing already warned airlines on january 5 (before the door issue) to check for lose bolts in a different section of the plane (the rudder). This is not a door plug issue.
It's not really about the door plug per se, it's about how Boeing's entire manufacturing, design and QA process could allow such an elementary mistake to escape notice in the first place.
If they got this wrong, what other bolts didn't they tighten properly? This question is especially relevant since just a few weeks ago, Boeing issued an airworthiness directive about loose bolts in the rudder system.
In any reasonable aerospace organization, the discovery of a systematic problem with the rudder system assembly should have prompted an audit of other processes. Why did that audit not uncover this issue with the plug door?
Presumably, the answer is that Boeing's processes are so incomplete or and/or unreliable so that, even when being given a hint at what kind of a problem to look for, they can't find other instances of the same problem themselves.
It raises the question of what other systems on the plane have problems, and whether Boeing is even capable of confidently answering this question at this point, or whether we just have to wait and see what other parts start falling off.
The crazy thing is that since the old Max fiasco several of us believed that the priorities of the Boeing company were completely wrong: they had decided to cut so many corners to increase profit to the point of negligence. Doing That, in an airplane is just crazy.
Their manufacturing process is just kaput. And there is no force that will make them fix it until their planes start falling mid flight in pieces.
I find it unfathomable that the FAA is basically sitting watching with their arms crossed. They should fine the heck out of Boeing and ground all their planes. But american protectionism is strong in this one.
Metrics that don't give employees enough time to accurately complete their tasks? Management whose first and last priorities are quarterly financial projections? Transitioning from external oversight to self-reporting?
I do hope they can track down the cause, whatever it might be.
However, the same photo shows other critical bolts that hold the whole hinge on the door are loose, and there’s not meant to be a pin on those.
I’m curious how the decision is made whether to include that pin in the design. Did they idiot-proof the maintenance of the plug, but not the initial installation?
> As part of the production process, Spirit builds fuselages for 737s and sends them by train with the special door assembly “semi-rigged,” one of the people said.
> “They are fitted but not completed," the person said.
> At its Renton, Washington, plant, Boeing typically removes the pop-out, or non-functioning, door and uses the gap to load interiors. Then, the part is put back and the installation in completed. Finally, the hull is pressurized to 150% to make sure everything is working correctly, the person said.
I can imagine a diffusion of responsibility as to whose job it is supposed to be to tighten those hinge bolts. Spirit is installing the plug in a “semi-rigged” state. Boeing is removing the plug to load the interior, then reinstalling it. I’d hazard a guess that Boeing is not removing the hinges, because the plug can be removed without doing so. What if, when reinstalling the plug, Boeing workers just redo the stuff they removed? They tighten the vertical movement arrestor bolt, put the pin through it, and believe they have done their job? If they never messed with those other hinge bolts, they don’t bother tightening them?
> A design problem probably would have shown up in an older one.
Maybe, unless it required really specific conditions that have only occurred for this one or also required some maintenance snafu that also had nothing to do with Boeing assembly QA. We don’t have enough data to establish anything as the most probable explanation yet.
You're talking out of your ass. Watch this documentary about whistleblower complaints as far back as the 737 NG line as they observed workers beating structural parts into place with hammers. Clearly, things haven't improved since then. https://youtu.be/IaWdEtANi-0
I’m no MBA or aerospace engineer, but why sub-contract the fuselage? I get subbing engines, since engine designs can be used on/adapted to other (possibly non-Boeing) airframes, hence opening up economies of scale, etc. But the fuselage is most of the plane by mass and integral to the design of everything else. If you don’t like making fuselages, maybe you should get out of the aircraft business. Was this just a union-busting thing?
Engines are weirder. My understanding is that the airline specs out what the engine requirements are, and any engine manufacturer is allowed to try to meet that requirement.
Customers get to decide which engine to fit and negotiate for them separately from the aircraft.
Sometimes there's only one choice, but other times gives you a chance to have the same (or similar) engine to maintain across your fleet (or avoid that to avoid a total grounding) and whatever other reasons to choose one over the other.
At least that's what the A380 tour in Toulouse said. Kinda makes sense to at least keep the big aircraft builders from throwing too much of their weight around.
Yes, and to lower compensation because Boeing’s benefits are probably better than Spirits, and so if Boeing wants to pay the people that make the fuselage less, then having them be employed by someone else helps them pass non discrimination testing for pre tax benefits offered to Boeing’s higher compensated employees.
Also, lower unemployment insurance premiums if they want to reduce the amount of labor they buy, since Spirit would be the one laying people off.
"Loose bolts" is not a ton of detail. It could have nothing to do with any of the bolts discussed in that video.
I'm having a hard time imagining how this failure could occur from just those bolts "needing tightening". They are lock bolts with pins and appear to take shear forces and provide no clamping functionality. Even if the bolts were "loose", or not torqued to spec, how would they come all the way undone? Then the bolts, under shear, work their way out completely? And isn't the lift spring forcing the top pins into the upper part of the track? On top of that the curve of the track appears to be such that outward force on the door would actually cause the pins to go into the upper part of the track.
IDK, seems like there is something else going on. Different bolts maybe.
The bolts themselves don't take the entirety of the shear forces. They provide tension and therefore friction between the mounting surfaces which is where the shear is handled. By being loose, there is no friction to handle shear and also the item can gain additional momentum (more distance to travel) during shakes and impacts which might increase the impact force above the rated strengths strength.
I'm not an aerospace engineer, but it seems those hinge bolts might be better replaced with rivets --- if the hinge isn't frequently removed from the door.
Since the aircraft with the incident was extremely new, it is pretty certain that the bolts have not been properly assembled at Boeing, so the problem is with their assembly procedures.
The fuselage of this plane is not built by Boeing. It could be an issue with the contractor, Spirit AeroSystems, who builds it. It could be an issue with some other final configuration procedure. We'll need to wait for the investigation to conclude.
Not sure that would help - the Boeing subcontractor that likely played a role in the defective parts, Spirit Aero, was previously part of Boeing and spun off in 2005.
If Boeing has insufficient competition, they don't have to care because their customers have limited alternatives. If they don't have to care then they don't have to care if their subcontractors care. They also have limited fear of regulators because the US government will be wary about damaging the country's sole supplier in an important industry.
Break them up and it's easier for both the market and the government to punish bad behavior.
Make five of them that each have full rights to the existing company's designs and split their existing physical plant between them. Does one of the new companies have the only plant that makes a certain type of component? That's fine, customers can buy from them like they buy from the existing Boeing, but now the other sister companies have the incentive to build another plant to compete for that business, and the first company has the incentive to keep prices competitive to deter them from doing that. For five years they have to sell to each other for cost, giving them enough time to either negotiate with each other or build new facilities.
They all have the rights to the existing designs, and can create derivatives, but they don't automatically get the rights to the others' derivatives, so they have the incentive to specialize or develop new designs to find a niche, for the same reason the Boeing 737 isn't an exact clone of the Airbus A320 or vice versa.
You realize how type ratings work yeah? An airline wants to fly a common model. They wanted a 737. They would buy a worse plane for twice as much (see: 737 max) over a brand new clean sheet model that’s incompatible with their pilots.
Type ratings work that way because there are so few types of planes, and because the incumbents like it that way. "Oh no, I guess it's impossible for anyone else to produce a new aircraft model because nobody's qualified to fly it, now we'll just have to rest on our laurels forever."
If there were more competitors, they'd lobby to change how the regulations work so they didn't effectively destroy the market for new aircraft types.
The FAA is susceptible to Boeing, the huge conglomerate and defense contractor, not any random airline that could go out of business without anybody really caring. But the status quo is good for Boeing because they don't have to pay to design new stuff, or at least they're ambivalent to it. Change that and see what happens.
All top 5 defense contractors preferably. And do the same in the EU too. Monopoly (effectively) clearly failed western defense efforts as the war shows now.
I think you would be mistaken. The car’s engines probably have more in common than the two planes’ engines (low-bypass turbofans in the original, vs. latest high-bypass on the newest ones; engines so different that they changed the plane’s handling enough to require the maligned MCAS). Avionics are entirely different and arguably much more vital to the planes function than a car’s instruments. The interiors are also basically not comparable (sure they have seats but that’s about it).
Deliberate or not, the NHTSA and automakers have found a symbiotic relationship where mandatory safety requirements become integral to a planned obsolescence sales cycle. Cars are more reliable and longer lasting than ever, but safety features mean many people feel compelled to buy a new car much sooner than they might have 30 or 40 years ago. Automakers negotiate with the NHTSA to set mandatory safety feature timelines so that features can work their way from luxury models down over the course of several years. By the time a features becomes mandatory, it's already been milked as a sales differentiator, and trickled down into most if not all car models.
I don't know if such a state of affairs is possible with the aircraft or airline industries. Those industries are too rational, whereas the expense of safety features in the consumer automobile market is in part quietly covered by arguably irrational consumer spending habits.
>Such a crappy aircraft - can we just get rid of these already?
It was a beautiful aircraft, destroyed by years of management comittees.
Boeing needs to be broken up and reorganized, full stop. Congress has to act. It is simply too important and vital of a company to US strategic interests to be left in this state.
>Maybe Apple, or Elon Musk, or some other person or company with a spare hundred billion could spin up a decent aircraft company?
SV has tried and failed time and again to realize the enormity of undertaking that is commercial aviation. There are less than 10 countries in the world capable of producing and marketing a competitive modern civil airliner. And of those, a vast majority are state nationalized conglomerates, or in the case of Airbus multi-country efforts. It takes almost the entirety of a country's aviation workforce and industrial capacity to maintain just one of these companies. Boeing cannot be replaced. It has to be fixed.
Elon Musk doesn't have a spare hundred billion. Apple is one of the only entities that does, but it's not really their kind of market. Institutional customers that are going to expect gradual iteration and thorough specifications that allow independent third party repairs etc.
That's more like Microsoft's kind of market, but uh... don't give them any ideas.
Probably not going to happen. If someone was interested, they could have had it easy if they bought up Bombardier's CSeries aviation unit before Airbus did.
> Boeing needs to be broken up and reorganized, full stop. Congress has to act. It is simply too important and vital of a company to US strategic interests to be left in this state.
The MD merger that led to the shift of Boeing management culture was due to the government intervening and forcing a marriage between the two companies because MD was too important for vital defense interests.
Very interesting video that shows a clear picture of how the plug is held in place. Some interesting spots in the video, 8:44 shows how the upper two locking bolts lock the roller pins into the upper door tracks, and 10:00 show the lower two locking bolts onto the sliding hinge posts. 13:10 shows how the door blew cleanly off, with very little damage to the roller pins and lower hinge posts. Some comments say the same plug have been used on the 737-900 which didn't have this issue.
My armchair speculative guess: there are only 4 bolts effectively holding the door onto the roller pins and lower hinges. Somehow I don't think it's a simple case of someone forgetting to tighten the bolts; since they're using castle nuts, forgetting to torque them down would leave behind extra cotter pins. Those bolts look positively tiny, probably no more than M12 diameter, and are subject to intense shearing forces. In the case of the upper roller pin with locking bolt, they are effectively two cylinders perpendicular and on top of each other, which causes extreme compressive forces to be concentrated on one tiny spot on the bolt. My guess is they cheaped out and switched to an inappropriate/softer bolt, which sheared and/or corroded. If one shears, the load quickly spreads to the remaining 3 bolts which all also shear off. This theory would still jive with OPs article about UA; "loose bolts" may not necessarily mean "untightened nuts and bad QA", but rather, signs that the locking bolts are all beginning to bend or shear.
Shear forces on four bolts, isn't that the same thing that caused the disaster in that hotel with the suspended walkway in the 1980s? With more than a hundred deaths. Is this not a basic case in engineering school, like THERAC-25? Not a licensed PE.
Longitudinal forces on the bolts there, not shear. A change during construction doubled the force on the nuts and the bolts went through weak weld joints in beams. (Also not a PE.)
Most of the pressure of the door should be supported by the stop pads, rather than the bolts. The force on the bolts should generally be from vibrations, not from pressure differential pushing the door out.
The 4 bolts are there to restrain vertical motion of the door which I presume would not involve large forces. They are meant to keep the door on the 4 cylindrical posts and 12 stop pads and it is those posts and pads that take the pressure differential forces on the door and keep it from blowing out. They are way beefier than the bolts and I believe they are still intact on the accident plane.
do some of these bolts also carry the weight of the door?
what if the springs on the lower hinges (that are supposed to keep an upward pressure on the door, supporting its weight) are weaker than necessary or somehow failed and went unnoticed... would that put too much burden on the bolts and other fastening mechanisms and cause them to fail in turn over time?
Oh wow. I made another comment about how important tension in fasteners is, to provide a clamping load and therefore friction between two faces. I was thinking that there must have been a process failure. Fasteners often shear off if there is not enough tension and therefore friction. But in this case it looks like there is not any clamping load between parts for at least some of the fasteners, so those fasteners basically act only as pins. There are other "stop pad" parts that constrain movement of the plug but as far as I can tell they must be compliant/flexible parts. The video mentions "roller pins" which for a door would be used with a cam mechanism. I think that stop pads plus roller pins/door/cam mechanism holds the door securely, but stop pads plus plug/pins does not.
On paper all 6 degrees of freedom may be constrained with the plug but in principle compliant pads may allow shear forces and/or rattling on the fasteners and they eventually fail.
I will eat my hat if the eventual required solution doesn't involve some kind of mating cup/cone or wedge shaped parts to more positively constrain the plug.
The recurring question is that it was only 2 months old so likely had those bolts underneath the roller thing was not fastened properly or the whole concept is flawed and this one simply had some bad turbulence/rough landings which created the perfect pressure on them.
I am not familiar enough with that incident but I wonder how fast the plane was going and how close it was to actually being airborne, that seems like it must have been quite the impact but given that all crew survived there must be some factor that I'm missing. Blind luck either way.
i'm not sure are you being sarcastic, there were no fatalities on the airplane. but once motorcycle and another time a firetruck crashed with the plane and people died.
Half the A320neo fleet needs to be grounded for 250-300 days over the next year due to engine manufacturing issues affecting their wear (which tbf is entirely Pratt & Whitney's fault). Airbus settled a $2bn suit last year with Qatar Airways over A350 paint jobs so bad they grounded them and refused to take scheduled deliveries. You'll hear more about Boeing than the rest of the industry put together because this is HN...
I mean, it is. The Pratt & Whitney engine issue which is going to ground half the MAX's rival's aircraft for a year because a manufacturing fault makes the engine turbine blades liable to crack garnered two threads and zero comments (Boeing got more for a fun story about paper planes!). Airbus being found at fault but acquitted of involuntary manslaughter by a Paris court last year for the AF447 crash (a 2009 mid air stall with a few MCAS parallels) passed without comment. HN is not a place for aviation news
> Airbus being found at fault but acquitted of involuntary manslaughter by a Paris court last year for the AF447 crash (a 2009 mid air stall with a few MCAS parallels) passed without comment
“Company isn’t found criminally negligent in case where it pretty clearly wasn’t being criminally negligent” isn’t.. news.
I’d be _really_ curious to hear what you think the MCAS parallels are (besides both cases involving a sensor malfunction) if you think that’s news.
Ah...because HN famously never has any interest in tech litigation and definitely won't have anything to say if a court finds in Boeing's failure in several years time!
The parallels are fairly obvious: AoA sensors malfunctioned, the situation was recoverable but the pilots were confused by conflicting and absent cockpit feedback and lack of relevant training, the OEM initially placed the blame entirely on the pilot but the problem was resolved with a tech remedy. Plus a whole lot of scope for speculation about Airbus regulatory capture of EASA and whether a first incident should have lead to grounding etc. Sure, with AF447 the issue was sensors having a (known) proneness to systematic failure rather than lack of redundancy and the plane plummeted because a stabilisation system disengaged at the worst possible time. They're obviously also not exactly the same, and the Qantas Flight 72 (different software subsystem input conflict automatically pitches nose down) near miss was a closer analogue, but they're all related to critical software handling edge cases and how guidance and UX might have mitigated issues. But as I said, you won't get much of a picture of the aviation industry from HN.
That’s pretty incredible. After a heavy maintenance breakdown, what percentage of the plane gets replaced? Or is everything already on a routine maintenance schedule so the intention is to visually identify unexpectedly failing parts?
IIRC every bolt's torque value needs to be logged any time it's (re)fastened. At least that's true for aircraft engine maintenance, not sure if it applies to the assembly process of the airframe, I'd sure hope so!
I'm not sure what the standard is in the aviation industry, but things like data-logging torque wrenches exist, and are used in other industrial applications.
The Max 9 has been in service since 2018 though, I think if it was a design issue it would have occurred before now. This seems like a Manufacturing QA “escape” to me. Some bolt was either not installed properly or not tightened to the proper spec, QA did not catch it
> The Max 9 has been in service since 2018 though, I think if it was a design issue it would have occurred before now.
That's a bit of a leap. If the bolts are e.g. too soft and have worn out in use, that would be a design issue that would only show up after a period of use.
Does wiring bolts actually help? I feel like if a bolt is going to come loose enough such that the wire is all that is holding it in then you have a big problem already. The wire is just delaying the inevitable.
A bolt is either fastened or it isn’t. If the wire is all that’s left to hold it, then it isn’t fastened and your parts are being held together with a very weak piece of wire.
The safety wire comes in to prevent a loose bolt from gently unscrewing itself further and completely disassembling itself (which can happen with vibration).
You're much better off with a loose bolt, than a missing one. It'll hopefully make itself known somehow (maybe even just visual inspection) while still remaining somewhat attached, which is better than nothing.
Thanks for the explanation. Does that mean a correctly wired bold should have wires loose enough to allow a loose bolt to give itself away — either through rattling or becoming visibly undone?
When I searched for images of wired bolts just now I saw many examples where the wire is done up quite tightly, which doesn’t seem right.
There's different kinds of bolts that get wired, with different kinds of goals and results.
My only experience is wiring bolts according to the rules for club racing motorcycles.
Some things, like oil sump drain bolts for example, wiring them will stop them backing out so far as to fall our, and while it will probably weep oil if it loosened enough to take ups the writing slack, it wouldn't dump all the oil out quickly.
Other things, like axle bolts, if it backs out enough to take up the wire slack, it'll have lost its clamping pressure and that's bad, but not immediately catastrophic. The major axle loads are all in shear so the wheel won't fall off, it'll just get more floppy than it's supposed to be. Long term it'd be bad and cause wear to the point that something would eventually fail, but it'd get you to the end of the race most likely, and hopefully the rider will have felt something odd and gone looking or a post race inspection would have caught it.
Happened to me once when I was racing an R6 6 years ago. I didn't torque the rear wheel nut, but had put my safety wire clip back through the axle so the nut couldn't back off. Got a warmup lap in and started in on a fast lap during a practice session, and it was immediately obvious something was wrong. Limped back into the pits and found the loose nut. I said a quick prayer to the wiring rulebook (and the people who crashed before that rulebook existed), and inspected everything on the bike.
"The five aircraft were delivered to United between November 2022 and September 2023, according to ch-aviation, and would likely not have been through a heavy maintenance C check that occurs every 4,000 to 6,000 hours or two to three years."
772 comments
[ 3.0 ms ] story [ 390 ms ] threadIt'd be different if it was an engine issue on a 20+ year old plane.
But still, this isn't anything like the prior MAX issue - and frankly could have just as easily happened to any manufacturer like Airbus.
I don't think it's worth bashing Boeing much about, I doubt they're too happy about it either.
Sure, other than being with a variant of the same model of plane built by the same manufacturer, this latest issue is nothing like the prior issue.
The previous issue:
* Was primarily a software issue, not a hardware issue
* Was caused by the design of the craft, not an issue with construction
* Was known by Boeing in advance and ignored, not discovered later
So yes, the two issues aren't very much alike.
There is no room for fanboy-ism in airliner design. It's a business that should be approached in a manner that is at once, sober and serious. If there are problems with this airliner, let's get them all on the table and start working through them one by one. I thought this is what the FAA and Boeing were meant to do? (Even in the face of monetary losses). But the mounting evidence of a rubber-stamp approval regime is concerning.
We really do need to have cooler more dispassionate heads take things back in hand here. I think a move away from the money-men, the fanboys, and the doomsday alarmists is overdue at this point.
One was rightly pointing out that we should investigate first before jumping to any conclusions and the other two were pointing out that Alaskan had maintenance failures which is also true.
And is anything actually proven yet ?
EDIT: Maybe the bolts have a really high shearing tolerance, but not as good of a tensile pull?
Look at causes of explosive decompression - look at the mass increase to add windows' struct infra to weight and mass to a plane, look at explosive bolts for emergencies
* Explosive decompression is the most dangerous, and can cause severe injuries or death. It is caused by a sudden and catastrophic failure of the pressure vessel, such as a window blowing out or a structural failure.
* Rapid decompression is less dangerous than explosive decompression, but can still cause serious injuries. It is caused by a more gradual loss of pressure, such as a door being left open or a seal failing.he cowling on the window failed. It was due to poor quality control for the extra mass around the window housings on plugs - the extra mass due to the pull and (IN ADDITION) the patsy ois on bolts.
They dont use explosive bolts in aircraft plugs, apparently, but yeah - this wasnt just bolts. Its a procedural impact, installation impact AND faulty materials QA in componentry.
My bet is on faulty adhesives or faulty QA on bolts.
I am pretty sure BOLTS in a plug door are a ruse UNLESS one can prove that window assembly procedures are either by the same company, the same process, or the same QA from multi-vendors.
Rule that out first.
An explosive decompression will not happen unless a single core seal will break....
Look T THE PASSENGER MANIFEST FOR THIS FLIGHT... look at MH370...
And if you want a mind F - look at Facebooks Husband who died in that flight to mexico....
If you look at the similarities, large numbers of high-tech company execs were taken out when those two crashed....
Who was on this manifest?
AI execs? Who lived.... So just give a list of professions of the passenger manifest. (Yes, aware no deaths - who could have? Who were to folks that missed that flight?)
If a singular bolt was faulty - it wouldnt have been an explosive decomp
It would have been a leak.
Explosive at 10K feet is FF attack level "first priority"
(PS they already ruled IN that they use differing contractors to produce certain parts such as plug doors.
How secure are those?
You are a MORON of Aerial-Assassination, as it isnt your thing. Hope you dont get in a small plane anytime soon - why do you think billionaires dont employ you to watch over their planes, Helios, etc...you not-minion.
You dont know this about me. My Grandfather was the primary nuke designer behind Hanford.
Handed me my first engineering design book before I was 6 years old.
I drew every single design in that book before I was 7.
When I went into HighSchool (I am smoothing over a bunch of details) I had already drawn that book left-to-right.) (ABC's Of Autocad} and I went back and forth.
I am 2nd in the nation of the Cad Olympympics.
Might sound stupid - but imagine that in 1992.
3 Hour test finished in 35 minutes;
I designed the datacenters you feed off you dorks.
I was the principle designer for so many you touched.
I fucking hate how much so many people dont get cred.
Just look at Tiny. - We couldnt ever jump out of a plane without Tiny. A 5' woman fearless who invented the parachute....
Look at Kent (the mips designer) - youll never know about him - but why we use the word "fabric" on networking... (that you may never know... It was he and me on ... (he had a sexual deviency proclivity that was bad for computing vcs) - but when he was at MIPS we designed slot level rack computing in 1996
(If anyone wants to refute this, they can refute my claims of being aware of googles early mother board designs,etc...
We will go with your over-valuated opinion on [who gives a shit]
Opening outward is mechanically riskier, but more likely to be useful in an emergency. The drill for plug-style over-wing exits: Remove cover from upper handle, grab upper and lower handle, yank upper handle, pull door inside, have 15Kg or more of door land on your lap, turn big clunky door sideways, pitch through hole so it's outside and out of the way, climb through hole. Training video.[1] Few passengers are likely to get that drill right in an emergency, and the flight attendants are in the wrong place to do it.
[1] https://www.youtube.com/watch?v=RCFzEg-t_Bk
However longer term if Boeing's reputation keeps getting worse that can motivate buying decisions by the airlines. That change will be a very slow one, because fleet changes cost a lot of money in training crews, changing maintenance etc.
For example KLM is changing their short-haul fleet from the 737 NG to the A320neo family, that will take many years to completely roll out.
So if it's really that big of an issue for a person, then just fly Delta for the time being.
https://en.m.wikipedia.org/wiki/Colgan_Air_Flight_3407
Still impressive.
https://en.wikipedia.org/wiki/PenAir_Flight_3296
The more likely scenario is that the Chinese or Europeans will ground the new 737s until some very expensive fix is put into place and not accept any workarounds. That would probably make a difference.
FYI: https://qz.com/1569865/china-grounds-737-max-8-after-ethiopi...
I’m assuming that the US is less likely to demand costly changes to aircraft produced by Boeing because of the implications it would have for their own economy. I wouldn’t trust France to preemptive ground a series of Airbus planes either.
"The European Union Aviation Safety Agency (EASA) said it had also adopted the FAA’s Emergency Airworthiness Directive (EAD) to ground this particular. ...
configuration of Boeing 737-9. “EASA took the decision to adopt the FAA’s EAD despite the fact that - to the agency's knowledge and also on the basis of statements from the FAA and Boeing - no airline in an EASA member state currently operates an aircraft in the relevant configuration,” said EASA in a statement"
* Ryanair (124 planes out of 575)
* TUI (40 planes out of 134)
* Turkish (25 planes out of 400)
* Norwegian (18 planes out of 81)
* Icelandair (3 planes out of 47)
and aside 787s, they haven't ordered any more Boeings
[1] https://injuryfacts.nsc.org/home-and-community/safety-topics...
[2] https://www.census.gov/newsroom/press-releases/2023/populati...
[3] https://www.ssa.gov/oact/STATS/table4c6.html
Those statistics don't really work out at all. They take number of miles and deaths, and nothing else. Let's say flown miles and driven miles are exactly the same for arguments sake. If the number of miles flown on planes were traveled by 4 million people, whereas the number of miles traveled by car were from 4 billion people, your likelihood of dying in a plane crash is FAR higher than by car if you're one of the 4 million who fly on a plane.
Miles and deaths alone are pretty poor metrics. There's also the part where I may only drive once or twice a week vs. the average which takes into account people who commute daily. So my odds of dying in a car crash are still lower. The frequency of travel seems just as pertinent as the miles traveled.
It's passenger miles, so the number of passengers is already factored in.
Even when people have a direct ability to change what vehicle they travel on, like with car sales, safety incidents/features/records/etc have little impact to no on sales.
This is one of the reasons why transportation safety is regulated rather than being left up to market forces. Market forces don't really have an impact until the safety risk reaches some extreme levels.
It wasn't, it was an A320 as I expected.
Obviously that doesn't excuse Boeing, I'm just saying, by the numbers.
[edit] The US has 1.5 deaths per 100M passenger-miles driven. [1]
The 737 NG has hit 10,000,000 flight hours 5 year into operating. That should be around 500,000,000 miles flown. About 180 seats per plane makes that ~90B seat-miles. That's 0.0038 deaths per 100M seat-miles flown.
So the 737-NG models are about 400X safer per mile than a car.
The median American lives 17mi from an airport, so the safety break-even point is 13,600mi flown on a 737-NG, assuming your destination is also a median 17mi away from the airport. If you're flying less than 13,600mi on a given 737 one-way trip, you're more likely to die getting to and from the airport.
Unless my math is off. This is just napkin math.
[1] https://injuryfacts.nsc.org/motor-vehicle/historical-fatalit...
The very first A320 demonstration flight crashed, killing 3 people, allegedly due to software issues in the new fly-by-wire system, and then involved allegations that Airbus had tampered with the investigation. The demonstration flight was mostly journalists on the plane. [0]
The A330 crashed on one of its certification flights due to an issue with the autopilot, killing all 7 onboard. [1]
Both went on to be some of the all-time best selling aircraft, and meanwhile today in this thread you have people touting Airbus as a paragon of safety. Humans have a short memory.
0: https://en.wikipedia.org/wiki/Air_France_Flight_296Q
1: https://en.wikipedia.org/wiki/Airbus_Industrie_Flight_129
I mention these two specifically because they're examples of high-profile failures attributed due to issues with the aircraft early in their lifecycle, but basically every single popular aircraft type (except the B777 and A350) have had serious issues and loss of life due to manufacturing defects, but still went on to be overall successes. Hell, people still flew on the DC-10 even after its _numerous_ issues like AA 191..
It means people were careless and sloppy during assembly.
Think of it like seeing "one cockroach in your kitchen." It's not one cockroach. You just haven't opened up the rest of the walls.
Lots of stuff is assembled consistently and carefully “wrong”, but as specified.
Cost cutting could have been a factor. Or the root cause might have been something entirely different.
It's quite possible to choose to spend more money on a process or method you believe is higher quality, but still discover it has some specific problem that the previous cheaper version didn't.
https://nitter.net/ByERussell/status/1744460136855294106
I think this should stick out to an assembly person as not quite right.
Complicates the issue of who is at fault; where/when were they loosened?
The center one doesn't concern me. The two that aren't even finger tight do.
Those are much more problematic as the remaining tight bolts could fail leading to a cascade. Though, would expect the flange to still be connected to the post in the aftermath images and the upper pins .. how did they get out of the track?
I wish we had close up pics of all the bits in the airframe opening.
Maybe. Could also be incorrect torque specs, bad parts, unexpected vibration in that area, etc.
Edit: Yes, none of these are good either. Just saying there are many possibilities.
If you make 10 or 500 planes in a year, you have to rely on multiple people to use multiple torque wrenches, with multiple attachments, to follow multiple procedures to assemble multiple planes using fasteners of multiple lengths, nearby multiple other processes and people. There is a big emphasis on procedures and traceability but there are still so many potential failure points that can go undiscovered for a long time.
An ex worked on a QA-type project related to production for them. It makes me wonder if some of the issues were more fundamental than that project could ever have addressed.
https://www.nytimes.com/2019/06/28/business/boeing-787-dream...
In aerospace this is dead people.
Hospital system failing to parse patient's allergies can kill.
MCAS
Therac-25
That said, the stuffs I normally work in (or rather what I allow myself to work in) fall in to the reboot and apologize category.
I don’t like how the sentence in this parent comment blames individuals for process problems.
Boeing has a responsibility to make sure its workers do their jobs as needed.
>> "Initial reports from our technicians indicate some loose hardware was visible on some aircraft."
Friendly reminder that 'some' can also mean most or all.
https://www.travelweekly.com/Travel-News/Airline-News/Omnibu...
I believe someone at Boeing actually was criminally prosecuted in connection with the earlier MAX 8 problems, but, as in all these situations, it is seriously unlikely that that fixed any actual problems. Most likely, somebody felt they needed a neck to wring and found a convenient scapegoat.
But when you have a quota, then you must ignore better qualified applicants in order to satisfy the quota. It's a mathematical fact.
I am not saying this is the reason why Boeing is turning to shit. But it's an interesting observation.
I myself am a "minority/POC" and I'm also extremely competent at my job. So I'm not saying that minorities are incompetent, because I am not incompetent.
When I interview candidates I give zero fucks about their colour/gender/whatever. If they are morons then I won't hire them.
"Please don't use Hacker News for political or ideological battle. It tramples curiosity."
https://news.ycombinator.com/newsguidelines.html
The fact that people would share this and it didn't curb the amount of business and referrals he got just proved to me what you've said for the longest time.
People don't like to be troubled with details and they'd rather be ignorant of them.
If you are a professional mechanic, it's your responsibility to obtain the specifications and follow them. This is especially true when it comes to a licensed and certified A&P mechanic and not your neighborhood shadetree mechanic.
It's still doable to get the full manual off, just not easy anymore.
It gets worse when bolts and part are of different materials (say: steel bolt in aluminum part), that's a very nice recipe for trouble (also in the long run, not just during assembly because oxidization will almost certainly occur on the interface and oxidized metal takes up more room than clean metal so unless you very explicitly protect against it the fasteners won't come out without massive damage to the part).
WAGO clamp terminals are a godsend here - no need to take care about wire nuts going loose, screw terminals being too loose or too tight (leading to fracture), ferrules being properly crimped... just insert the cable, lower the lever, and off you go. Unfortunately, standard DIN fuses still come with only screw terminals.
You would definitely hope aerospace assembly would be immune to this widespread phenomenon though.
Given the amount of (soft) aluminum on aircraft, (737 is 80% aluminum), it would be insane to not precisely consider torque.
Steel on steel is more forgiving.
Much easier to strip aluminum threading if you over torque. Steel tends to just bend (and bend back better).
Aluminum also fractures while steel bends/stretch back and forth better.
(This is mainly from my experience working on bicycles, and a little auto)
e.g. any slight boogering of the threads of a used fastener could translate into rotational torque that doesn't end up being converted to a clamping force.
(or in more extreme instances, some fasteners are torque to yield, and change shape after their first use, and must be replaced. But I'll presume that's not what they were talking about here)
Most service manuals require new bolts, fasteners and washers for torque-critical parts.
[1] which makes an ass (out of) u (and) me
That obviously isn't the problem with airplane manufacturing, and maybe not for car mechanics either. But it's totally endemic in the consumer world.
"Do not operate while driving" on car HUDs. "Do not consume if pregnant" on perfectly safe OTC medications. "Do not continue to ride after a crash" on bike frames.
It's not surprising most of this is just ignored now -- there's no information content. The documentation is nothing more than a list of things for which the manufacturer would like not to be liable, and the marginal cost of adding to that list is ~0. It will grow until we run out of room in the manual / space on the packaging.
"Store between 68 and 75 F." Or what? Is that a "must follow or else death", or a "it may reduce efficacy 0.5%" or a "we've never run a sufficiently powerful study under any other conditions, but there's no theoretical reason it should matter"? It matters quite a bit to me which!
I don't see how we can hope to have good-faith communication under such a heavy threat of litigation. I would not be surprised if /that/ turns out to be relevant to the Boeing issue, even if the rest is unrelated.
Leadership from a more environmentally-conscious and empathetic Gen Z would definitely have safer products that last longer.
For example a company of 400 people laying off vital talent is literally shooting themselves in the foot. If I was a CEO I would take a cut till I could course correct. Losing talent with a layoff is way more expensive.
You end up paying three to six months for that talent to not work. This is a huge problem.
It may have not been operator error with the torque wrench. Maybe the torque wrench itself was miscalibrated. Maybe the bolt had a flaw in the metal. It seems too early to make conclusions.
An example of flawed bolts on aircraft in an excerpt from https://www.sciencedirect.com/science/article/pii/S136970210...:
A bolt from an aircraft flap control unit fractured in the threaded region of the shank near the shoulder with the head upon installation after a major service. A metallurgical investigation was carried out to identify the cause of failure. The bolt was manufactured from cadmium-plated, high-strength steel. Material checks carried out on the bolt showed that it conformed to the required specification and was found to have an approximate ultimate tensile strength of 1380 MPa.
The fracture surface of the failed bolt was examined using SEM to identify the mode of fracture and determine if pre-existing defects were present that could account for the unexpected failure. The fracture surface exhibited two distinct modes of failure. [...]
The embrittlement in this case was attributed to the cadmium plating, which is applied to the bolts to provide corrosion protection to the steel. Hydrogen is evolved during the plating process, which becomes absorbed by the steel. The cadmium plating acts as a barrier to hydrogen diffusion at ambient temperature so that the hydrogen becomes ‘trapped’ in the steel. In high strength steels (>1100 MPa) this leads to embrittlement. To overcome this problem, high strength steel fasteners, which have been cadmium-plated, are baked at 175–205°C for 24 hours to allow hydrogen to diffuse through the cadmium. In this case, failure of the bolts was caused by insufficient baking after plating, which gave rise to hydrogen embrittlement.
In this specific case, I would agree.
I share much of the sentiment of the OP, but I wonder if it is simply my perspective changing as I take on more and more responsibilities. Maybe things have always kind of been this way.
I would be hesitant to say it's all going to hell in a handbasket because that tends to be one of those self-fulfiling prophecies. I prefer to look at things as "challenging, but workable".
People care deeply about a lot of things, just not this. And why should they? When you have every business decision made either by private equity or public companies with a focus only on the next 90 days it turns out everything goes to shit. Wages are stagnant. Wealth inequality is massive. Our society is sick. Issues like this will only increase. Our rail ecosystem is experiencing the same issues for the same reason.
$1000 for a first class ticket from DC to BOS. 6.5 hours.
Same flight first class with Delta this week is $850. Also 6 hours with layover.
Blows my mind first class rail travel is as much as first class air travel.
In my experience, in the UK, taking a train is more expensive than taking a flight (or driving, for that matter).
The train cost more, but you enter the train in the city centre, sit down and relax, go to the restaurant car and eat a nice dinner, relax som more and you arrive in the destination city.
This works better in continental Europe, where you have shorter distances and high speed trains between the capitols, often making the train faster from city centre to city centre than air travel. Traveling from Chicago to SF by train is pleasant, but take a couple days, not a couple of hours.
I took the Amtrak a couple times from DC through WVA and have had several delays trying to get from VA to KY due to priority being given to the freight trains. I remember one trip having a one or two hour delay in the middle of no where while we were side tracked.
It is much, much more comfortable than a plane though.
Also IME Amtrak's prices are surprisingly reasonable. More like a place around the corner than like an airport food court.
I haven't ridden Amtrak in a long time, but I would be absolutely shocked if the first class experience on the Acella is as nice, much less worth a premium compared to flying. Regardless, there's pretty much nothing they could offer to make it worth the price premium.
I guess it has to do with planes replacing plane, hence plane must be better, hence more expensive.
But the reality is that the plane won because of two things: speed and cost. Air is free. Miles of miles of railroad is very expensive to build and maintain.
Trains are a bit cheaper than planes. But they travel slowly. So cost per mile per passenger is lower for a plane.
Even when subsidized, the cost of rail travel is often more expensive per mile than air travel.
So when Europe and Asia now are building a lot of high speed rail, it's not to make it cheaper, but to lower carbon footprint and to make a more pleasant experience.
American air travel is safer than it's ever been. Going years without a single fatality is now the norm, not an exception. The airlines are well run and do care deeply about safety. The regulators aren't perfect but generally do a good job. The NTSB is world-class.
Boeing will continue to lose marketshare to Airbus and feel some pain which is appropriate. Eventually they'll get their act together or they will continue to suffer real consequences.
There are only so many checks and balances you can build in until this situation will catch up with you.
Doesn't amazon have such a huge turn over that they are running out of possible candidate to replace those workers?
Being pushed to deliver more and more packages in shorter time or inspect train cars in less time then required why would one care if a packages gets tossed or a rail car is not properly inspected?
In most jobs the only consequence is to be fired which is already possible for no reason what so ever in many states.
You definitely have enough time to work at leisurely pace and assemble every part right. There's not much rushing going on there. If workers get away with doing dodgy work you have a QA issue.
So conditions at the plant in North Charleston haven't looked very good for almost 10 years...
[1] https://www.aljazeera.com/program/investigations/2014/7/20/t...
Let's not pretend layoffs are new.
If not you've got things like a reduction in manufacture of ICBMs to consider.
https://en.wikipedia.org/wiki/LGM-30_Minuteman
> My theory is that this is not limited to Boeing or even aircraft design, it's a much deeper and systemic problem affecting all kinds of fields. We've had a lot of industrial accidents lately.
> When aircraft manufacturing was an emerging industry there were tons of undocumented safety margins and "slack" in the design and production pipeline. Over time, the beancounters start optimizing stuff, so these undocumented safety margins are eroded in the name of efficiency/profit (and sometimes even documented safety margins too).
> Furthermore, workers back in the day had a much better life when it comes to purchasing power (especially when it comes to property), and so could actually "give more fucks" about the job than they do now which is a compounding factor. You used to get a lot of implicit quality assurance back then which you don't get now.
> We've now reached a stage where these undocumented safety margins have been eroded enough that it actually starts to cause issues, and the safeguards that are supposed to catch them aren't good enough, either due to 1) they've never been good enough but just weren't really needed before or 2) they too have been eroded in the same way for the same reason.
[1]: https://news.ycombinator.com/item?id=38896457
Privately-owned companies or worker-owned companies are more resilient to this problem because the nature of their existence means a stronger focus on long term goals. Though there are still issues with publicly owned companies exerting enough influence to acquire and subsequently destroy private companies. For example, Rite-Aid acquiring Bartell Drugs then going bankrupt and proceeding to shutter most Bartell Drug stores leaving Seattle with a deficit of pharmacies.
Welcome to the outcome. When companies look at employees and employee training as an expense, and when the only way to improve your salary is to go to a new company, expect to see these companies to be hollowed out shells of untrained people.
can be reassuring if it removed the doubt that he was making a senseless decision. while still being the opposite about flying the thing.
The decision OP took might have been extreme and costly (time wise and effort mostly). People around him might have lightly whispered to him to be reasonable, or "trust the experts". This wears on you eventually and you start to doubt your own reasonableness.
Boeing starts leaving bolts unfastened and all of a sudden you remember the almost weekly stream of pre-pandemic HN articles with new material on Boeing's fall from excellence.
Boeing's marketing dept. was capitalizing that everyone had forgotten about the MAX. The few of us who remembered it maybe weirdos, but we're nut nuts!
Why couldn't you use something a camera on eyeglasses while doing the work correctly to fine tune a multimodal model, and then infer to a user wearing the same glasses? Audio reply saying "nope, try again."
You would need multiple frames per second, so not today, but not that far into the future, right?
edit: Zero-shot, I just took photos of me "fixing things" around the house, and here is what ChatGPT told me. It does not suck. Do this with fine tuning, many frames per second, and what am I missing?
https://imgur.com/a/H1eSShH
https://imgur.com/a/YXxz8uL
If a missing some major required component aside from finetuning and frames per sec, which will require a few years for everything to be fast enough... please let me know what it is.
Wait, oh it will be understanding of time... the sequencing of the frames. That is missing for now, right?
Given that the discussion in this post is around torquing to the right specifications, I don't know if just fine tuning is enough. It might need more serious training on videos of assembly. Even then, can it distinguish between the right torque or not from video?
I think it's interesting, but you should deliberately try to make it fail to see where the edges are. Like hold a wrench of the obviously wrong size next to a nut and ask if that will work. Force perspective so they look the same size. See if it will prevent you from mixing potentially dangerous chemical combinations. Will it warn you to wear protective eyewear when using a circular saw?
But in a manufacturing environment, if one labelled wrench sizes with different colors, then things like noticing wrench size would be easier. Also, I imagined that the camera would of known (same) geometry during training and inference in a Boeing implementation.
I also wonder if multimodal LLMs are blowing my mind unnecessarily. It really feels like a huge leap to me though.
I think there are digital wrenches that will record the torque they applied, you don't need to mess around with colors. But at that point, it's sort of like, why bother teaching the AI to notice if people are doing it right, stick a camera on the wrench and QR sticker on the bolt. Make a more formal verification process that can be guaranteed to match specifications.
So back in the day, Expert Systems were a big thing at Boeing. Searching Boeing's job openings today, it's a still a word used in hiring, but I don't understand what they mean by it in terms of manufacturing. Do you have any idea what they mean by that today?
I can't tell you about what Boeing does with expert systems. I've never been employed by them and I never really looked into it or where they use them. It's especially unclear because Boeing does a lot more than just pure manufacturing.
Maybe something like a camera monitoring that they are using the correct tool at various parts of assembly? But I'm not sure how feasible this would be at airplane levels of volume.
I bet it'll be common in car assembly at some point, though.
But regardless, not looking good for Boeing
(*door but of course it's more a plug than a real door, just using the term for ease of understanding)
Edit: https://twitter.com/ByERussell/status/1744460136855294106?t=...
Unfortunately, Boeing did not know they had other issues with the plug door bolts.
Especially since most shops have pretty much tossed professional career QA out the window.
Retraining has its own problems. No matter how well retraining is done, pilots still make mistakes from doing the right thing for the previous plane that is the wrong thing for the one they are currently flying.
Adjusting airplanes to fly the same way is a major safety advantage.
Unreported by the media, there was another MAX incident before the first crash. The crew had no knowledge of MCAS, but did follow the emergency runaway trim procedure, and continued the flight and landed safely.
Turn off the motor, and the trim is manual. There is a crank right there in the cockpit. If it is too hard to turn, change aircraft configuration to reduce the forces required to. Pilot know how to do this. This pilot stuff, they understand the forces on the flight controls and what impacts them.
Boeing made an engineering mistake. The pilots also made an operational mistake. Unfortunately, both mistakes at the same time were fatal.
I pray that pilot training has improved. And that Boeing has made systems level changes to the aircraft that will preclude it happening in the future.
And that is how aviation becomes safer every year; at a significant cost of customers lives.
"Significant" might be inaccurate.
It looks like FAA Part 121 accidents over the last 10 years with fatalities have been... 4. [0]
For a total of 6 fatalities.
[0] https://www.ntsb.gov/Pages/AviationQueryV2.aspx; 2018 (1 passenger fatality) https://www.ntsb.gov/investigations/Pages/DCA18MA142.aspx ; 2019 (3 crew fatalities, cargo flight) https://www.ntsb.gov/investigations/Pages/DCA19MA086.aspx and (1 passenger fatality) https://www.ntsb.gov/investigations/Pages/DCA20MA002.aspx ; 2022 (1 ramp fatality) https://data.ntsb.gov/carol-repgen/api/Aviation/ReportMain/G...
I'm sure there's been a study somewhere that attempts to untangle all the factors that differ between commercial carriers and GA, to see which safety is most sensitive to -- continuous highly professional maintenance, highly trained and experienced crew, rigorous airliner certification regime, etc.
> The probable cause of this accident was the inappropriate response by the first officer as the pilot flying to an inadvertent activation of the go-around mode, which led to his spatial disorientation and nose-down control inputs that placed the airplane in a steep descent from which the crew did not recover.
Also, overspeeding the airplane makes it much harder to turn the manual trim wheel. The cockpit voice recorder on the EA flight recorded the overspeed warning horn, which the crew did nothing about (they were at full power, should have pulled the throttles back).
The LA crew restored normal trim twenty-five times before crashing. What they never did was turn it off after restoring normal trim.
What Boeing did (and is STILL doing) is expect pilots to know or remember obscure NON-PILOTAGE (and in the case of MCAS, BURIED) trivia to prevent disaster.
Now... what's the more-responsible approach? Expect pilots to pilot, or expect them to recall an ever-growing list of workarounds to incompetent system design?
Is that what you meant by "the problem wasn't the software?" Because the pilots should have been trained to unplug the computer to stop it from crashing the plane?
Afaic, the fault apportionment was Boeing documentation > airlines >> pilots > Boeing technical design.
Edit: Also, how does the fault lie with the airlines? Boeing didn’t document the existence of MCAS in the flight manual or training materials.
>> Because the pilots should have been trained to unplug the computer to stop it from crashing the plane?
Yes.
The fault lies with the airlines because I don't for a second believe they didn't put pressure on Boeing to get the MAX certified without mandating retraining.
And then once that was done, didn't dig into the details too hard about what changes were made.
I have a low tolerance for 'I set up all the conditions and incentives to encourage you to break the law... but you should take all the blame when it explodes.'
At some point, the customer has to take some responsibility for what they asked for.
If you listen to podcasts, these two episodes of Causality are excellent:
https://engineered.network/causality/episode-33-737-max/
https://engineered.network/causality/episode-50-737-max-ethi...
Back when I flew regularly before covid, I was tempted to create a bunch of these and hand them out to the flight crew for the flights I flew on.
They introduced MCAS in the aircraft for to balance by software a hardware issue, a big design negligent issue which can lead to stalling. It is beyond to trim an aircraft, and because of this there is a big difference in the scale of the values that the algorithm manages from a trimming.
It is not my field, but I think it is not a simple factor, and that it should not be put this over the Pilots like if it were a normal aircraft that received a simple update. Every pilot flying that plane should have been warned that it was not a classic plane with a classic update.
If this type of behaviour by aircraft manufacturers becomes the norm, costs over safety, we as passengers will suffer it, as other passengers unfortunately suffered it, while they blame the Pilots. In addition that nowadays the China's aircraft manufacturing industry wants to enter global market. Some days ago I read they want permission (homologations approvals) for to enter in the European Union.
PS: They also cut costs retiring backup sensors, delegating responsibility for a vital system due the MCAS to the buyer as if it was an unimportant feature; disaster was the order of the day. And the spending cuts were not limited to that, as we have seen in recent days.
> They introduced MCAS in the aircraft for to balance by software a hardware issue, a big design negligent issue which can lead to stalling.
> Every pilot flying that plane should have been warned that it was not a classic plane with a classic update.
I was mean,
> They introduced MCAS to use software to attempt to balance an aerodynamically unbalanced aircraft with a high stall tendency, in order to avoid designing a new aircraft.
> Any pilot flying that aircraft should have been warned that it was a plane that didn't want to fly aerodynamically, with software forcing it to fly without backed redundancy. It was not mere trimming.
In the 737 MAX, the only way to disable auto-trim also disables powered trim (the thumb buttons). As grand parent says, at a certain step in the sequence it was not physically possible for a pilot to trim the plain back to stability manually. It simply can't be done.
In the 737 ng, there was a button to do just that. That would have been useful.
And that's even ignoring the fact that all symptoms were very different from those present in a runaway trim situation as described in the manual and learned by the pilots.
The user of the parent comment is remarking about time.
The aircraft can be certified without MCAS?
By what I read, MCAS is there for to avoid entering into an aerodynamic stall when the aircraft is approaching a high angle of attack, due it's using larger motors for what classical 737 was designed for. It's balancing an unbalanced aircraft using software to repeatedly adjust the horizontal stabilizer.
It is not my field, but I'm not even sure if it should be called to trim, it sounds like a euphemism for what's going on.
Software QA when actually practiced is more advanced now than airline QA.
...eh, I think "when actually practiced" is doing a lot of carrying there.
What do you mean by "actually practiced".
Outside of the aerospace and healthcare industries, I'm not sure there are many software shops that are doing QA to a level I would like to trust anyone's life with.
also software is the least likely comparison I would have made; software quality is a shit-show on a general level, and the vast public is quite aware of this every time a subway timeboard blue-screens or gets frozen on an AMI screen, or the POS machine that they're forced to interact with at work does something equally as stupid.
Software QA provides nothing of value to software development; having it as a dedicated function works against the overtly stated goals of the function and counterintuitively acts to degrade quality within software by mandating strict top down process and brittle end-to-end testing.
Although Software QA is intended to be an independent verification body that provides engineering organizations with tools and resources, in practice they function as a moral crumple zone [1] within the complex socio-technical defense industrial system, being one of the groups that the finger will be pointed to when something goes wrong and absorb shock to the business in the event of a failure. As a result they have a strong incentive to highly systematize their work with specific process steps, to shield them from liability, which can be applied generically to all projects.
Good software teams build quality into projects by introducing continuous integration, unit testing, creating feedback, and tightening these feedback loops. This acts to find problems quickly and resolve them quickly. Software QAs need for high level, top down, generic systemization requires them to work against these principles in practice. Bespoke project specific checks, such as unit testing, is not viewed as contributing to the final product and is discouraged by leadership who see it as waste.
To give an example of how these dynamics destroy quality in software. I once found a bug in software on a piece of test equipment where a logarithmic search function was not operating on a strictly sorted list. When I pointed this out to my leadership I was told that if we changed any part of code, it would require a new FQT, which would be too expensive to conduct and was not in the budget. Although the bug would have been trivial to solve, and was clearly wrong and would not provide any benefits by remaining in the test equipment software, the process required for changes prevented solving the issue.
[1] https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2757236
This is driven by Pournelle's iron law of bureaucracy [1], which says that people who promote the bureaucracy rather than the mission of the bureaucracy will get promoted within the organization and come to dominate its decision making.
For example, in schools, administrators make more money than teachers. This is despite both groups having similar levels of education and intelligence. The reason for this is that administrators know the laws and regulations of the environment they’re working in and ensure the continuity of the organization. Despite not directly contributing to the organization’s stated mission of education, they are in charge of the organization and take more benefits from it.
Software QA has similar dynamics. A QA department may start out making good faith contributions to the organization. Eventually there are product failures, eventually leadership needs a scapegoat to show they’re doing something, and eventually QA takes the blame. People get moved, demoted, or fired. QA realizes its risk, and takes steps to mitigate it. They create a highly systematized workflow and process, adopt or introduce standards. Then assert that following process equates to good outcomes. When bad outcomes occur, they point to their strict adherence to following process as evidence of innocence.
If the process does not support the work or mission, that is a cost they are happy to impose on other functions to deal with. This is the final state until a system disruption happens.
[1] https://en.m.wikipedia.org/wiki/Jerry_Pournelle
An equivalent software QA to building planes would be to verify a known process with existing tooling.
Manufacturing process builds identical widgets using standard equipment. Widgets are inspected to confirm they are within spec. Frequentist statistics are used to determine when widgets are consistently out of spec. When this happens, equipment is inspected and repaired as a corrective action. This process is well defined and linear.
Software produces bespoke, non-standard widgets to address domain specific needs. At the end of the day, software developers are defining a process for machines to follow. If you want to control quality in the software development , aside from having perfect domain knowledge for a particular project, the only way to do it do it is to define an arbitrary process for developers to follow and track adherence to it. This may have no impact, or be a hindrance. It will never add value because it will never be abstract enough to be appropriate for every domain.
The case I'm talking about does not have a separate QA department, but QA people as part of every software team. If a product fails, that team is responsible, so software devs are in the same boat as QA. They focus on learning from these failures, so no scapegoat is needed. Process does get followed, but not as a defense mechanism, but because not doing so introduces noise that is an obstacle to improvement. In case of bad outcomes, people do point out that they followed process because then it is clear that the process is involved in the failure and should be improved.
Unfortunately, companies with that kind of culture are rare.
However, there are situations where less QA may be needed, for periods of time, such as when PRs may be low. QA may be seen as overhead by management, and something to reduce. This will lead to QA shared between teams and a push for standardization and top-down process deployment to minimize complexity for these personnel will develop. Complexity to manage the QA personnel will be shifted to development teams.
This situation absolutely is controlled by company culture. A culture that neither values QA nor development will do this. A company under financial strain will do that. Companies wax and wane constantly.
Sure, maybe, but if nobody ever can implement the theoretical utopia, maybe we should talk of things humans can do instead and ditch the unimplementable idea.
QA cannot be done by a separate team the way you dream: it will always be a political buffer zone staffed by the cheapest half-competent people you can find, expulsing good people into dev or management. Or you merge it into dev/solution design.
The reason is simple: just like contract law, you only care about quality once you are in trouble and need to reverse back the source of the issue to give to the client a post mortem. Otherwise, you care first about velocity, or $ input/hr of effort.
Agreed, for good software teams.
I would content that most software teams at most companies are not good.
Which is to ask, with an average to bad software team is it better to have integrated or separate QA?
Ideally we’d always have good software teams, but in the real world sometimes you have to build software with bad teams.
Leaders have options, they can do things like reduce scope, increase budget, increase schedule, or full on abandon or cancel the project. These are all options available to leaders, but they require tradeoffs and decisions to be made on a project by project basis.
It is scalable to have a strict process that everyone has to follow, then impose a watchdog to enforce it on a wide scale. It may not be better to have separate QA, but it is easier for those in charge.
If I'm trying to build something with undertrained, demoralized, underpaid engineers... it's not optimal to use methods intended for self-motivated, high-performance teams.
And nothing says there must be company-wide mandates. Maybe this area gets a formal, independent QA team, but this other area doesn't.
My experience just doesn't bear out that collapsing the QA function into development always leads to better outcomes.
I've seen the opposite happen too often, and QA be the sole bulwark between idiocy and customers.
Here's a real, perhaps unexpected counterpoint. Say you have a good software team. How do they build good software with bad management?
Consider the classic statistic "most drivers think they are above average".
I posit that the same is true of software teams, almost every team will self-assess as above average, i.e. good. Those teams will then imagine that, being good, they build quality into the process and very little verification QA is done.
I have worked as a software consultant for 15 years now. I've worked with at least 40 separate software teams in that time. Every single team manager would pep talk with "this is the best team I've ever seen". Some of this is obviously blowing smoke to get people to work harder and feel good. But over the years I've had candid conversations with managers and realized that most of the time the genuinely think their team is really good, truly top 10-20%.
Here's the rub. Being a consultant, I'm almost always brought in by higher level management because something is going horribly wrong. The team can't deliver quickly. The software they deliver is bug ridden. They routinely deliver the wrong software (i.e. incorrect interpretation of requirements.)
Often times these problems are not only the fault of the development team, management has issues too. But in every single case, the development team is in dire straits. They have continuous integration sure, and unit tests, and nightly builds, and lots of green check marks. But the unit tests test that the test works. The stress tests have no reality based basis for expected load. The continuous integration system builds software but it can't be deployed in that form for x, y & z reasons, so production has a special build system, etc...
In 15 years I have never once encountered a team that would not benefit from a QA team doing boring, old school, black box manual testing. And the teams that most adamantly refuse to accept that reality are precisely those that think they are really top tier because they have 90+% unit test coverage, use agile and do nightly builds.
So, my question is, do you (I don't mean the specific "you" here, rather everyone should ask themselves this, all the time) think that most bad software teams know they are bad? Including the one you are part of? Would it really hurt to have some ye olde QA, just in case, you know, you are actually just average? :)
Also, did you run across any orgs where they basically refused to use a process like Agile, and instead just did ad-hoc coding, insisting that this was the best way since it worked just fine for them back when they were a 5-person startup?
You usually had a few personality archetypes:
- The most technical dev on the team, always with a chip on their shoulder and serious personality issues, who had decided to settle for this job for (reasons)
- The vastly undertrained dev who was trying to keep up with the rest of the team, but would eventually be found out and tossed, usually to blame for a major issue
- The earnest and surprisingly competent meek dev, who presumably didn't have enough confidence to apply to a better job, but easily could have made it on merit, work ethic, and skill
- The over-confident dev who read a bit of SDLC practice, and could see every tree while missing the forest
The key is that, aside from the incompetent person, they had all always been working there for awhile. Consequently, there wasn't good or bad health and quality: there was just "the system" (at that company) and dealing with it.
And none of these folks ever worked at 5-person startups. ;) I think it was definitely more an issue of SDLC "unknown unknowns" they should be doing, than willful decisions not to.
Yes, generally I join teams and work as an engineer or sometimes as a team lead, so I'm talking to all the team members.
Most start up teams are composed of junior developers, often pretty smart people. Usually 5 or fewer years of experience. Many times these are people who have already accomplished stuff they didn't think they could do. So that generally means that yes they think pretty highly of themselves. To a degree it is quite justifiable, they tend to be very accomplished but in a narrow domain. Unfortunately they don't realize that their technical accomplishments in a specific field does not mean that they are experts everywhere. Their managers understand that these are smart people and assume again that this is therefore a good team.
Non start ups that I join are usually just plain dysfunctional.
> Also, did you run across any orgs where they basically refused to use a process like Agile, and instead just did ad-hoc coding, insisting that this was the best way since it worked just fine for them back when they were a 5-person startup?
Usually more the opposite. In my experience I come across teams that are sure they must not need any help because they follow all the rules in Scrum and have great code coverage metrics.
It is really common to see this kind of thing. I call it "the proxy endpoint fallacy". It can crop up anywhere that there is something that can be measured. In that example, it would be confusing adherence to Scrum with having a working SDLC or perhaps confusing code coverage metrics with the objective of having bug-free releases.
This isn't a software only fallacy. In politics, GDP is often confused with societal well-being. Always be wary of your metrics and change them as required to keep you tracking your actual goals.
Suppliers (to include software) were expected to manage the quality of the product they provided; the purchaser would focus on how they managed the process, not in the compliance of every part.
This had a chance until software process was tossed in the name of "agile".
"Hey Bob I know you're a competent engineer, but don't worry about specifying a certain type of bolt or loctite, the untrained assembly personnel will figure it out. I'm sure they won't let 200 people die in a plane crash."
Anyhow, the fix for this was created and written. But we never got to put it into production. The reason: the company didn't have a lab test facility that could put a sufficient load on the software to prove it. Even though we were getting field failures because of this issue that were getting a bad rep, we couldn't fix it because even though the old code was known to be buggy, we couldn't prove the new code. So the process said we couldn't ship it.
I don’t work in this industry, but this seems fairly ridiculous on its face: software is not at all like manufacturing.
In manufacturing, there’s a design and a manufacturing process, and a critical function of QA is ensuring that the manufactured produce is manufactured to spec.
With software, the software is written, compiled, and then repeatedly copied. And something should verify that it’s copied correctly, but this is straightforward and boring.
So software QA ought to be much more like the kind of validation that happens when designing hardware, not like the kind of testing and validation that happens as products are manufactured.
You need some level of specification so you know what you’re building, but you have to keep in mind that the final code defines what the behavior truly is. Sometimes, that behavior unintentionally becomes part of the specification because users begin to rely on it.
I do like the fact that you both used hyperbole to succinctly illustrate the dangers of veering too far in either direction though :)
It's asymptotic. By the time you reach a human who is as dumb as an actual computer, the specification _is_ the code.
I think that's why people always tell each other to not take things at face value.
Of course there is a big difference between sw and hw QA, in the thing that they test, and how they test them.
But they are also very similar. Any QA department has to think about ways that things can go wrong, and what things to test for, how to test, which testing methods, which standards to handle, keeping certifications, etc. During testing you also need to keep reevaluating if you actually are catching each problem/bug and how to implement changes in your company that decreases the amount of problems or increase the amount that you catch.
I think in that way there's a lot of overlap in thinking about business processes and how to identify problems with them.
Of course once a specific binary gets tested and approved by QA it shouldn't matter if it gets copied or whatever as long as you make sure its the same binary (by a checksum for example).
But still making sure that errors don't reach the customer, is vital in any QA. If errors does happen, QA is the department that can make sure that it doesn't happen again. And ofc be able to proof in court that you did your due diligence if something does happen.
No. Good software teams are led by competent, technical management. Managers who aren't afraid to get down into the dirty details. Managers who aren't afraid to roll up their sleeves and write code if they need to.
The process doesn't matter. The management of what is or is not important does. Agile is just one process out of many.
Imagine an accounting team led by someone who never did accounting in their life: "Just make the numbers work out! I don't care how you do it! My bonus is at stake!"
Let's say you have a bunch of school children and architects create a skyscraper. I've given both groups the process to design a skyscraper.
So in both cases, I should end up with a safe building?
...or perhaps with no managers at all. I'm less and less convinced of the importance of management in engineering except to give investors an illusion of control.
For some reason, developers seem remarkably blind to the skills other roles and disciplines require. Only a developer can do that, everyone else is basically useless fluff. Maybe it's a form of arrogance or just deep unself-awareness.
Your contention is that the surgeon should be running the hospital.
Although I don't deny it can help to have the background, it is not necessary to be a good manager of something. Also seen plenty of good techies promoted to management and failing badly.
I stand by what I said, although my experience is in the technical domain, not finance or law. Maybe those departments are different, I don't know.
I guess if you have a manager who is making technical decisions, they are really a hybrid manager/contributor role. Maybe that works better in smaller organisations.
He has no experience to lead the team in high pressure situations. Like production being down.
He can't truly have a first person understanding of the work of the people who he manages. He has to rely upon others to tell him who's good and who's bad. That sets up a pecking order.
He can't help or mentor engineers with design decisions, or provide a historical context.
He doesn't understand the technology so there's an immediate communication and knowledge barrier that has to be overcome between him and his directs.
He doesn't feel the pain of a bad decision, because he's not coding it, and he can't emphasize with them since he doesn't code.
He tends to push feature development without fixing technical debt. Again that's pain he personally doesn't feel.
Simply not true however that a good manager can't lead the team in a high pressure situation. I'd say that exactly what a good manager could do well. Obviously they won't be making overtly technical decisions, that's what you are for. They can make business decisions, provide cover, get resources, communicate to other stakeholders... All the bits that need doing but would be a huge hassle for the techies who are trying to fix the issue.
I wouldn't call what I'm describing as product management, although it's possible they could do general management too.
The head of accounting should be an accountant, not a surgeon.
And even at the executive level of a hospital, you would want people who have spent their careers in healthcare, rather than, say, architecture.
"Nah, I don't need to slouch anymore in this moral crumple zone!"
"We need this new feature in our program!"
"If we implement this it means the management fell in a moral crumple zone"
"What seat would you have to have for your flight?"
"Anywhere, but not in the moral crumple zone, please"
Them: “Please review this design.”
Me: “Ok, sure, when do you plan to start coding?”
Them: “Oh, it’s already in beta.”
Me: “So you can’t do anything with my feedback, but you’ll say I reviewed it?”
Them: “Well…”
Me: “You’re putting me in a moral crumple zone here!”
> To give an example of how these dynamics destroy quality in software. I once found a bug in software on a piece of test equipment where a logarithmic search function was not operating on a strictly sorted list. When I pointed this out to my leadership I was told that if we changed any part of code, it would require a new FQT, which would be too expensive to conduct and was not in the budget. Although the bug would have been trivial to solve, and was clearly wrong and would not provide any benefits by remaining in the test equipment software, the process required for changes prevented solving the issue.
I've seen this happen where it was a bad thing, but also where it was a good thing.
It's all about risk.
What risk does the software defect pose to the mission? What risk is inherent in making any change to the software? Noting that even trivial changes can be fat-fingered and thus are a source of risk. I've seen it go wrong this way: a seemingly trivial change was made, but the developer accidentally checked an extra file into source control, causing a further defect.
And then: what is the cost of mitigating these risks? Maybe the software defect is as trivial as its fix. Maybe an acceptable fix would be to write up a workaround in the documentation.
I don't think it's always wrong to say no to fixing issues. I also don't think it's always right that a separate QA department contributes nothing to the organization, even if they act as a handbrake on the software developers (sometimes, precisely because they do that). Human factors are real.
You think that QA is a liability shield, but that is only a side effect of the work that they actually do.
The task of QA is exactly that: an entity that tries to assure that the quality is up to some standard. Even in favourable conditions mistakes happen, so how do you make sure as a company that not 1 in every 100 product are faulty and tarnishes the good reputation that your company has spent so much time and money on to build? You hire a QA to make sure problems get caught before delivery.
But if all humans make mistakes, and QA is human, how do you make sure that the QA doesn't make a mistake? A never ending chain of QAs expecting each other?
No of course not. One thing that helps with reducing errors is to have a rigid protocol that is followed to the letter everytime. Pilots, for example, have a preflight checklist that they have to run every time they operate the plane.
The rigid protocol of QA teams is therefore an essential part of their jobs.
Although from your standpoint as a developer it might seem strange that QA is 'preventing' you from fixing a bug, it is actually very reasonable.
Especially since you work in the defence industry, I hope you understand that it is very important that the software that operates radars, planes, missiles, bombs, etc is working exactly as expected. Understandably there is a great deal of effort made to assure that when those things are needed they work exactly to spec.
So in your example it is probably very reasonable that any change you make needs to go through some rigorous process. The fact that it 'only' was about test equipment, doesn't matter because test equipment is just as, if not more important as the stuff it tests.
The reason why QA has the side-effect of being a 'liability shield' is that it gives companies the ability to argue (and proof) after the fact that the company did their due diligence in making sure that the product was to spec.
Especially certification is basically to get an external organisation to approve your QA. In that case if you get sued you can rightfully claim that you did everything that was legally asked of you, and if there is blame, then it is the certifying company using insufficient standards.
I suspect it's rather a case of parallel evolution between McDonnell Douglas brass and software startup culture, since cost-cutting culture goes back many decades (remember "Chainsaw" Al Dunlap[1] ?) — but I wonder if there's a more direct influence.
[1] https://en.wikipedia.org/wiki/Albert_J._Dunlap
It's inherently short-sighted unless forced to do otherwise by legislation. Cutting small corners pays off A LOT until the hammer falls, so there's a massive advantage to doing it / you need to do it if competition is doing it, or you eventually shut down as they take all your business.
It's inherently a race to the bottom. Sometimes that's a net gain, sometimes it isn't.
I'm not carte blanche defending capitalism - its a mixed bag but it sure outpaces the competing systems put forward to date. It does need some stronger safeguards against industry self regulation - that has a bad track record.
What anti-capitalist sympathizers, in my view, don't realize is that this is due to people being in the loop. These economic systems are merely vehicles, some better than others, but the conductors are people, be they communists or capitalists. At least with capitalisms there is a delayed regulator (negative feedback) in communism it's up to the system to decide if it needs to modify itself.
Other systems had incentives such as, get it running by such and such date or have yourself and relatives sent to inhospitable place. So people rushed flawed designs into production.
That said, upper management at Boeing needs a shake-up. People need to get fired. They need to do what Intel is trying and that is to get more engineers in charge, or at least grant them veto power on designs.
It should be a lesson against dogmatic pursuit of absolutes: capitalism comes in a wide range of flavors, and the worst is if it’s completely unrestrained. Communism produced worse and worse results the further it got from any sort of public accountability, etc.
The two problems that I see is that the concept of nuance is somewhat at odds with having a simple concept to teach kids at school, and there’s always a group which is more motivated to game the system than the average person who really just wants to hang out with their friends, raise a family, etc. rather than play political games. Boeing didn’t start it by any means but they’ve benefited enormously from decades of reduced oversight and elevated pay driven by a sort of cartoon libertarianism where letting people get enormously rich will motivate them to build great things unfettered by “red tape”.
They have, but post-Thatcher neoliberal capitalism has taken the existing perverse incentives and made them exponentially worse. We're on a course heading straight to feudalism, just with fancy titles with legal rights replaced by economic might.
Ironically, I believe it was Grace Hopper who said it... Whoops.
"With impressive clarity, Downfall: The Case Against Boeing reveals corporate corruption that's enraging in its callousness and frightening in its scope."
https://en.wikipedia.org/wiki/Downfall:_The_Case_Against_Boe... https://www.netflix.com/hk-en/title/81272421
1% of 10000 is 100.
.01% is 1.
If someone came up to me and said, "Hey I can save you 99% of expected costs with 1% of your profit.", I might go for it.
I’m regularly critical of Boeing Defense (particularly space contracts where I’m a huge Boeing skeptic), but I think people are pretty off base if they think Boeing is just completely incompetent.
Airliner safety is insanely good. Just vast seas of competence, but when there’s a super rare failure, the incorrect impression people get is that Boeing (or Airbus) is just full of incompetency. Almost nothing that humans do is held to the same standard. Not spaceflight, not software, not healthcare, and certainly not automotive.
Flying a 737 Max with a bad door and without the fix to the angle of attack sensor is probably still better per mile than driving. In spite of going at 10 times the speed and miles above the Earth.
You can almost argue it’s held to a higher standard than it should, slowing development of cleaner aviation (and therefore killing more people in the future due to tertiary effects of climate change, etc).
It kind of annoys me when comment sections are filled with people talking about how incompetent Boeing is. It feels like out of shape slobs on their La-Z-boy chairs talking about how incompetent or slow some professional sports players are. Like, airliner safety is just a totally different league than almost anyone else plays in. On the worst day, their better than almost anyone else is on their best.
Because I dug it up for another comment, commercial carriers operating under Part 121 (roughly: scheduled passenger and cargo operation) had 4 fatal incidents in the last 10 years. [0]
Totalling 6 deaths.
In 10 years of US commercial carrier aviation.
One of those was literally 'the engine exploded and threw part of the turbine into the cabin (and also shredded some of the wing)'!!
Which resulted in 1 person dying and a successful landing.
[0] https://news.ycombinator.com/item?id=38921664
Okay, awesome. But how much of that was luck with the 737 Max that they didn't crash on US soil by US airlines?
The two MAX 8s that fell from the sky were 100% Boeing's fault and could have happened in the US. If 5% of airline traffic is in the US you can renormalize those hundreds of dead and you get dozens dead.
I can't find specific numbers but estimates say about one in three has a military background. That's an awful lot.
Not surprising given that pilot training is really really expensive. Airlines love former military pilots because they are a significantly lower financial risk for them. Put them into type rating and off they go, it's rare that one ends up as a dud.
No matter how good they are and how prescient, that doesn't help them if the aircraft computer decides it's stalling, forces a nose down and they cant fight the controls.
But, even if we assume omnipotence from these American pilot gods, and assume they can fly outside the bird and Superman-style catch it, they are still only 30% of American pilots. Just another population to normalize out.
But that's not what happened. According to every report, it is possible to take back control, it's just very much not intuitive and the situation was confusing.
According to the Seattle Times
> However on both accident flights, the angle-of-attack sensor failure set off multiple alerts causing distraction and confusion from the moment of takeoff, even before MCAS kicked in.
> On the Ethiopian Airlines flight, for example, a “stick shaker” noisily vibrated the pilot’s control column throughout the flight, warning the plane was in danger of a stall, which it wasn’t; a computerized voice repeating a loud “Don’t sink!” warned that the jet was too close to the ground; a “clacker” making a very loud clicking sound signaled the jet was going too fast; and multiple warning lights told the crew that the speed, altitude and other readings on their instruments were unreliable.
You can find some of the US reports complaining about MCAS here https://s3.documentcloud.org/documents/5766398/ASRS-Reports-... and it includes
> I manually positioned the thrust levers ASAP. This resolved the threat
Then there's
> B737 MAX First Officer reported that the aircraft pitched nose down after engaging autopilot on departure. Autopilot was disconnected and flight continued to destination
> I called "descending" just prior to the GPWS sounding "don't sink, don't sink." The Captain immediately disconnected the autopilot and pitched into a climb
Another
> Takeoff and climb in light to moderate turbulence. After flaps 1 to "up" and above clean "MASI up speed" with LNAV engaged I looked at and engaged A Autopilot. As I was returning to my PFD (Primary Flight Display) PM (Pilot Monitoring) called "DESCENDING" followed by almost an immediate: "DONT SINK DONT SINK!" I immediately disconnected AP (Autopilot) (it WAS engaged as we got full horn etc.) and resumed climb
People do this with everything though, and air travel induces a large amount of fear in the populace. Not only are we not generally comfortable flying in the air for obvious reasons, but when it happens almost everyone has to concede control to a few people in the cockpit and on the ground. Driving, even if exponentially more dangerous, affords the illusion of control of one's outcome, given driving or having someone you know driving, and control over the vehicle maintenance, etc, as well as familiarity with the control and mechanism of the vehicle. These things don't exist with airplanes for the vast majority of people.
So, you can see why there is a need to find a human component to air travel problems, because that is something one can fix (fire the incompetent people, fine them, whatever), as opposed to all of the other things which must be accepted or rejected entirely.
It is entirely in line with human nature to do this, regardless of its accuracy or effectiveness.
Flying isn't safer then trains I would assume.
Flying has the advantage of being seperated from almost everything else. Most accidents happen when there is mixed traffic, specially cars operated by people with minimal training.
And there's good reasons for that. Spaceflight actually is regulated pretty strictly (partially, because any spaceworthy rocket is effectively a missile), and space pilots and tourists both sign up for such missions fully knowing that they will have a very significant chance of dying one way or another - there simply hasn't been enough human spaceflight activity to work out and understand all the failure modes, unlike with other forms of transportation.
Humans, unlike birds, aren't naturally wired to travel by air... they need to be able to trust their lives to a significantly higher degree to someone else behaving like they should, because unlike in a car they have zero control (or the illusion of control) in an aircraft.
Additionally, the inherent security risk of an airliner is very high: what is a widebody airplane at its core? Hundreds of tons of weight, a decent portion of which is fuel, propelled at near-supersonic speed, and only two people in control of it. Anything goes bonkers and you can get thousands of people killed and injured (see 9/11).
In contrast, cars, even trucks, have way less capability to cause damage simply because they weigh so much less. The only thing that comes close is railways, and hell I don't get what the US is doing there, there's barely any regulation compared to European standards (see the videos I linked at https://news.ycombinator.com/item?id=38725988).
Think of railway signalling systems, control-by-wire bits of modern cars, medical equipment, etc. Where the design of the software is formally proven, and the implementation verified to ensure it fits the design.
EDIT: I saw the pictures of bolts with pins and bolts without pins. The ones with pins cannot get loose, the others can. Let's see what happened.
Sure, it failed, and it isn't perfect.
But planes have had a long track record of being absurdly safe.
Almost certainly systemic issue though, so that sucks. Sucks real bad.
They need to get a Tiger Team or whatever together to look at everything with a shipment config, and make sure those "ship kits" don't leak into the real actual airplane configuration. This is . . ok, this is really manufacturing 101 stuff, but well, things happen.
I'm in the industry, but haven't touched the MAX, so take this with a grain of salt.
[1] Basically a "shipping" or train configuration
It's not a control surface, but it is a "moving part." That's what's baffling to me, that they spent a lot of effort building this hinge and pin roller system, and designed the door to hinge open up to 15 degrees.
It makes me wonder if there's maintenance procedures that at some point would require the operation of that door to successfully complete. Otherwise, the mechanism itself seems so incredibly overwrought, with lots of additional bolts, castle nuts, retaining pins, and even sprung hinges at the bottom.
Does anyone know why this "plug-type non-plug door" is built this way?
It needs to be usable depending on how many passengers the interior is configured for.
So it has all of the door bits there. Maybe some parts like the emergency escape slide are not installed.
e: I should be clear that it's not usable as an emergency exit, as configured by Alaska. However the operator could choose to activate it later and install a usable exit.
If there was a reconfiguration to a seating standard that required the extra exit, the plug would be removed and a proper door would be installed, with the associated interior pieces.
This is true.
However there's still common hardware in there to allow the plug to be installed and maintained. This is why it's a complicated set of kit vs just bolting in a permanent fixture.
Alaskan airlines chose a 178 passenger configuration for their 737-9, and so are not required to have a mid-cabin exit door.
Lion Air's chosen to go with a 221 passenger configuration, and so are required to have an operating door.
Obviously changing up the number of seats isn't done on-demand, you'd need to go for a refit/maintenance cycles.
But if Alaskan decided to change density, or sold the aircraft to someone else who decided to change density - then they could go and do this.
[1] https://youtu.be/nw4eQGAmXQ0?t=305 "The Boeing 737 Technical Channel"
If you are correct, then the implication is that the concern extends beyond door plugs for MAX-9 737s to all emergency exit doors on all models of aircraft sharing this design. This is somewhat reminiscent of the huge problem with the 688 (Los Angeles) class submarines, where the discovery of a faulty weld that had passed inspections raised doubts about all welds.
It appears that the latter is what happened.
Perhaps those who did the assembly of the doors at Boeing did not use the right kind of washers that are needed to prevent the unscrewing of the bolts, or they did not apply the correct torque to the bolts.
It is extremely surprising if such trivial errors can happen during the assembly of an aircraft.
The latter seems more likely.
- "bad parts from vendor" - "bad programming on torque wrench from MRP" - "insufficient training on process" - "tooling was programmed correctly but bad sensors" - "lube mislabeled for 3 days"
This is absolutely not some simple thing. This is why people don't take us seriously as engineers.
It can be a design problem if the bolts and associated parts like washers have been substituted recently or if the assembly instructions have been changed recently, e.g. by specifying a different torque. If any such engineering change has happened recently, then that would be the likely culprit.
A resonance problem of the aircraft body as supposed by another poster seems extremely unlikely as that would have required significant recent changes to the aircraft body, which did not happen.
I wouldn't be so certain it's installation error or mfr defect of the fasteners...
Or the ‘correct’ assembly instructions weren’t actually correct, or the design isn’t good enough (e.g. it uses too few bolts, or didn’t check the strength of the bolts when they get cold), or the bolts were of the correct material, but designed too thin, etc.
It's really unlikely that washers are a root cause (spring/split washers are useless and are definitely not used for this type of apllication). "Correct torque" is not trivial to determine and maybe it was calculated right or wrong based on right or wrong materials/conditions or maybe conditions were changed or maybe...
I am 99.9% sure a procedure was followed and a torque or other measurement applied and was documented. This is probably not a trivial error but a cascading failure in design and/or documention and/or integration.
Are airliners marked in this way? Is checking the bolts simply a visual inspection, or do the inspectors need to get out a torque wrench?
No idea how this can happen, Boeing really has to get its shit together.
https://www.youtube.com/watch?v=maLBGFYl9_o&t=523s
This is also standard procedure for electrical maintenance inside distribution equipment. Every lug/termination is torqued to spec and then marked to indicate it has been torqued down. This should be done regularly to prevent any loose terminations from causing an arc flash.
Humans make mistakes. The whole point of modern manufacturing is to make products better than any human can make them, by layering processes and procedures to catch those mistakes before they get out the door, and continually improve the processes to catch ever more.
I think it's much worse than a design flaw.
Even very carefully-engineered systems can have flaws. Engineering flaws, once identified, can be engineered around, managed, or corrected.
I'm much less comfortable with the idea that the assembly plant for these planes could be a random-critical-failure generator based on how the employees handle a torque wrench.
If they got this wrong, what other bolts didn't they tighten properly? This question is especially relevant since just a few weeks ago, Boeing issued an airworthiness directive about loose bolts in the rudder system.
In any reasonable aerospace organization, the discovery of a systematic problem with the rudder system assembly should have prompted an audit of other processes. Why did that audit not uncover this issue with the plug door?
Presumably, the answer is that Boeing's processes are so incomplete or and/or unreliable so that, even when being given a hint at what kind of a problem to look for, they can't find other instances of the same problem themselves.
It raises the question of what other systems on the plane have problems, and whether Boeing is even capable of confidently answering this question at this point, or whether we just have to wait and see what other parts start falling off.
Their manufacturing process is just kaput. And there is no force that will make them fix it until their planes start falling mid flight in pieces.
I find it unfathomable that the FAA is basically sitting watching with their arms crossed. They should fine the heck out of Boeing and ground all their planes. But american protectionism is strong in this one.
Grounding every plane Boeing has ever built would be quite an overreaction and would be disastrous for both passenger and cargo transportation.
Hopefully they can track down the cause.
I do hope they can track down the cause, whatever it might be.
> Safety and quality were taking a second seat to schedule and cost.
"most scrutinized world leader in history"
Some of the bolts that would be loosened when the plug is opened during maintenance have a pin to prevent them from turning. That pin is present in this photo: https://x.com/byerussell/status/1744460136855294106?s=46&t=s...
However, the same photo shows other critical bolts that hold the whole hinge on the door are loose, and there’s not meant to be a pin on those.
I’m curious how the decision is made whether to include that pin in the design. Did they idiot-proof the maintenance of the plug, but not the initial installation?
Some quotes from this article: https://www.reuters.com/business/aerospace-defense/spirit-ae...
> As part of the production process, Spirit builds fuselages for 737s and sends them by train with the special door assembly “semi-rigged,” one of the people said.
> “They are fitted but not completed," the person said.
> At its Renton, Washington, plant, Boeing typically removes the pop-out, or non-functioning, door and uses the gap to load interiors. Then, the part is put back and the installation in completed. Finally, the hull is pressurized to 150% to make sure everything is working correctly, the person said.
I can imagine a diffusion of responsibility as to whose job it is supposed to be to tighten those hinge bolts. Spirit is installing the plug in a “semi-rigged” state. Boeing is removing the plug to load the interior, then reinstalling it. I’d hazard a guess that Boeing is not removing the hinges, because the plug can be removed without doing so. What if, when reinstalling the plug, Boeing workers just redo the stuff they removed? They tighten the vertical movement arrestor bolt, put the pin through it, and believe they have done their job? If they never messed with those other hinge bolts, they don’t bother tightening them?
Bolts can be loosened by the physical and thermal stresses of use if there is a design flaw. So it’s not evidently an assembly QA problem.
Maybe, unless it required really specific conditions that have only occurred for this one or also required some maintenance snafu that also had nothing to do with Boeing assembly QA. We don’t have enough data to establish anything as the most probable explanation yet.
The discussion about snowman hole issues and other problems is really alarming.
Customers get to decide which engine to fit and negotiate for them separately from the aircraft.
Sometimes there's only one choice, but other times gives you a chance to have the same (or similar) engine to maintain across your fleet (or avoid that to avoid a total grounding) and whatever other reasons to choose one over the other.
At least that's what the A380 tour in Toulouse said. Kinda makes sense to at least keep the big aircraft builders from throwing too much of their weight around.
Yes, and to lower compensation because Boeing’s benefits are probably better than Spirits, and so if Boeing wants to pay the people that make the fuselage less, then having them be employed by someone else helps them pass non discrimination testing for pre tax benefits offered to Boeing’s higher compensated employees.
Also, lower unemployment insurance premiums if they want to reduce the amount of labor they buy, since Spirit would be the one laying people off.
https://en.wikipedia.org/wiki/Castellated_nut
https://en.wikipedia.org/wiki/Positive_locking_device
I'm having a hard time imagining how this failure could occur from just those bolts "needing tightening". They are lock bolts with pins and appear to take shear forces and provide no clamping functionality. Even if the bolts were "loose", or not torqued to spec, how would they come all the way undone? Then the bolts, under shear, work their way out completely? And isn't the lift spring forcing the top pins into the upper part of the track? On top of that the curve of the track appears to be such that outward force on the door would actually cause the pins to go into the upper part of the track.
IDK, seems like there is something else going on. Different bolts maybe.
Break them up and it's easier for both the market and the government to punish bad behavior.
They all have the rights to the existing designs, and can create derivatives, but they don't automatically get the rights to the others' derivatives, so they have the incentive to specialize or develop new designs to find a niche, for the same reason the Boeing 737 isn't an exact clone of the Airbus A320 or vice versa.
If there were more competitors, they'd lobby to change how the regulations work so they didn't effectively destroy the market for new aircraft types.
Car platforms change a lot more, a lot more often, IMHO.
I don't know if such a state of affairs is possible with the aircraft or airline industries. Those industries are too rational, whereas the expense of safety features in the consumer automobile market is in part quietly covered by arguably irrational consumer spending habits.
It was a beautiful aircraft, destroyed by years of management comittees.
Boeing needs to be broken up and reorganized, full stop. Congress has to act. It is simply too important and vital of a company to US strategic interests to be left in this state.
SV has tried and failed time and again to realize the enormity of undertaking that is commercial aviation. There are less than 10 countries in the world capable of producing and marketing a competitive modern civil airliner. And of those, a vast majority are state nationalized conglomerates, or in the case of Airbus multi-country efforts. It takes almost the entirety of a country's aviation workforce and industrial capacity to maintain just one of these companies. Boeing cannot be replaced. It has to be fixed.
That's more like Microsoft's kind of market, but uh... don't give them any ideas.
The MD merger that led to the shift of Boeing management culture was due to the government intervening and forcing a marriage between the two companies because MD was too important for vital defense interests.
My armchair speculative guess: there are only 4 bolts effectively holding the door onto the roller pins and lower hinges. Somehow I don't think it's a simple case of someone forgetting to tighten the bolts; since they're using castle nuts, forgetting to torque them down would leave behind extra cotter pins. Those bolts look positively tiny, probably no more than M12 diameter, and are subject to intense shearing forces. In the case of the upper roller pin with locking bolt, they are effectively two cylinders perpendicular and on top of each other, which causes extreme compressive forces to be concentrated on one tiny spot on the bolt. My guess is they cheaped out and switched to an inappropriate/softer bolt, which sheared and/or corroded. If one shears, the load quickly spreads to the remaining 3 bolts which all also shear off. This theory would still jive with OPs article about UA; "loose bolts" may not necessarily mean "untightened nuts and bad QA", but rather, signs that the locking bolts are all beginning to bend or shear.
Longitudinal forces on the bolts there, not shear. A change during construction doubled the force on the nuts and the bolts went through weak weld joints in beams. (Also not a PE.)
Wow
or if some other mechanism like the springs on hinges below grew weaker over time .. would that in turn put too much weight on the bolts?
what if the springs on the lower hinges (that are supposed to keep an upward pressure on the door, supporting its weight) are weaker than necessary or somehow failed and went unnoticed... would that put too much burden on the bolts and other fastening mechanisms and cause them to fail in turn over time?
On paper all 6 degrees of freedom may be constrained with the plug but in principle compliant pads may allow shear forces and/or rattling on the fasteners and they eventually fail.
I will eat my hat if the eventual required solution doesn't involve some kind of mating cup/cone or wedge shaped parts to more positively constrain the plug.
https://en.wikipedia.org/wiki/Airbus_A320neo_family
A320neo is the direct competitor to the MAX. So yeah, they're doing pretty well.
Crew and passengers were unharmed after a collision with a firetruck crossing the runway while the aircraft was doing its take-off roll.
edit: 235 Kph!
https://www.youtube.com/watch?v=FBEE7bzatKk&t=47s
two 10ths of a second later and it would have been an entirely different story. Ugh.
> You'll hear more about Boeing than the rest of the industry put together because this is HN...
That's not it.
I mean, it is. The Pratt & Whitney engine issue which is going to ground half the MAX's rival's aircraft for a year because a manufacturing fault makes the engine turbine blades liable to crack garnered two threads and zero comments (Boeing got more for a fun story about paper planes!). Airbus being found at fault but acquitted of involuntary manslaughter by a Paris court last year for the AF447 crash (a 2009 mid air stall with a few MCAS parallels) passed without comment. HN is not a place for aviation news
“Company isn’t found criminally negligent in case where it pretty clearly wasn’t being criminally negligent” isn’t.. news.
I’d be _really_ curious to hear what you think the MCAS parallels are (besides both cases involving a sensor malfunction) if you think that’s news.
The parallels are fairly obvious: AoA sensors malfunctioned, the situation was recoverable but the pilots were confused by conflicting and absent cockpit feedback and lack of relevant training, the OEM initially placed the blame entirely on the pilot but the problem was resolved with a tech remedy. Plus a whole lot of scope for speculation about Airbus regulatory capture of EASA and whether a first incident should have lead to grounding etc. Sure, with AF447 the issue was sensors having a (known) proneness to systematic failure rather than lack of redundancy and the plane plummeted because a stabilisation system disengaged at the worst possible time. They're obviously also not exactly the same, and the Qantas Flight 72 (different software subsystem input conflict automatically pitches nose down) near miss was a closer analogue, but they're all related to critical software handling edge cases and how guidance and UX might have mitigated issues. But as I said, you won't get much of a picture of the aviation industry from HN.
https://www.bloomberg.com/opinion/articles/2024-01-08/alaska...
Fortunately, most airlines in my country use Airbus planes.
That's a bit of a leap. If the bolts are e.g. too soft and have worn out in use, that would be a design issue that would only show up after a period of use.
So how did this happen if they we checked so recently?
A bolt is either fastened or it isn’t. If the wire is all that’s left to hold it, then it isn’t fastened and your parts are being held together with a very weak piece of wire.
https://fastenerengineering.com/what-is-a-castellated-nut-an...
You're much better off with a loose bolt, than a missing one. It'll hopefully make itself known somehow (maybe even just visual inspection) while still remaining somewhat attached, which is better than nothing.
When I searched for images of wired bolts just now I saw many examples where the wire is done up quite tightly, which doesn’t seem right.
If the painted seal is broken or misaligned, you know the nut has moved.
There's different kinds of bolts that get wired, with different kinds of goals and results.
My only experience is wiring bolts according to the rules for club racing motorcycles.
Some things, like oil sump drain bolts for example, wiring them will stop them backing out so far as to fall our, and while it will probably weep oil if it loosened enough to take ups the writing slack, it wouldn't dump all the oil out quickly.
Other things, like axle bolts, if it backs out enough to take up the wire slack, it'll have lost its clamping pressure and that's bad, but not immediately catastrophic. The major axle loads are all in shear so the wheel won't fall off, it'll just get more floppy than it's supposed to be. Long term it'd be bad and cause wear to the point that something would eventually fail, but it'd get you to the end of the race most likely, and hopefully the rider will have felt something odd and gone looking or a post race inspection would have caught it.
Most likely weren't checked since the planes left the factory yes.
To quote from: https://theaircurrent.com/feed/dispatches/united-finds-loose...
"The five aircraft were delivered to United between November 2022 and September 2023, according to ch-aviation, and would likely not have been through a heavy maintenance C check that occurs every 4,000 to 6,000 hours or two to three years."