It sure doesn't look good to have two software issues very late in the product development cycle, on a live test. While the first one would have been averted by a human crew, the second one would not be so benign.
Software is hard. Embedded software doubly so, but... Come on... This should not be happening to Boeing.
One of the takeaways of the MAX debate maybe was "yes, this should be happening to Boeing" as a result of specific steps to weaken the engineering leadership in the company from the top down. If that's the case we can only hope it makes for the case study.
Kind of yes, but this is a different part of Boeing. If I were one of the astronauts who'll fly Starliner, eventually, I'd not be nearly as polite as the formal communications suggest. I'd probably be reading code, sitting next to an engineer, and asking them to explain to me what the code does.
The thruster map error should have been detected in simulation. If it wasn't, the simulator is also faulty.
I'll assume the thruster map was only slightly wrong and that whatever simulator runs they did with it were very subtly off.
If you listen to the NASA update [1] you'll hear the audit team state that if Boeing was following the procedures that they stated they were doing they would have found the flaw at 4 separate places. Let's just say the auditors tone of voice says it all.
There are specific requirements to assure the fidelity of simulators so this still should have been caught if the design process was robust enough I think
> specific steps to weaken the engineering leadership
What in the world are you talking about? There’s been no reporting of anything like that. The MAX engineers failed all by themselves. No management told them to design MCAS in a brain dead fashion. Nobody “weakened” engineering leadership.
When management makes statements like "we don't need [to pay for] senior engineers here, these are mature products... ", and clearly putting finance first among corporate goals, the natural result is that engineering dies a death by a thousand cuts...
Muilenberg became CEO in 2015. The McDouglas thing happened in '97. He inherited an existing management structure that was already primarily interested in profit over all other concerns, especially engineering.
Moreover, the 737 Max program ran from 2011-2015. The CEO at the time was an MBA.
I think maybe the difference here is the fact that SpaceX probably treats software as a first class citizen; just like Tesla does, and is likely to have a modern view on how to build and evolve software. Additionally, software is a key component for them that is very much determining the abilities of the hardware. Tesla even ships pure software features that they charge money for. That's unheard off in the car market. And I would expect, Elon Musk to drive a similar strategy in SpaceX.
I suspect Boeing maybe treats software the same way as they've always done, just one of many boxes that need to be ticked in a long waterfall like process. They just throw more people at it hoping it gets done on whatever time line they imagine it needs to be done. They are likely to be very conservative with respect to tools, technologies, and practices (i.e. not much has changed there in decades).
The fact that SpaceX is able to come up with a new design for Starship and can then get it and the software working for several successful hovers & hops in around 9 months, tells me they are able to iterate quickly and adapt to quite major design changes. They launch a lot of rockets and they seem to be learning a lot every time they do this. I don't see how they'd be able to do this without very solid software development practices. I'd expect lots of CI & test harnasses, probably lots of emulators and other tools, and I have a hunch they probably use a lot of static code analysis etc. to prevent preventable bugs from happening.
Any engineer or tech at SpaceX is likely worth their weight in gold at any other company due to the company's perceived high bar. Additionally, I don't hear complaints about their pay either, which is likely extremely good.
No one is locking them into cages and telling them they can't leave. I would bet most everyone who is there strongly believes in the mission.
EDIT: Eh. They aren't paid fantastically but not slave wages either. Additionally this is spread out over three very different cities.
> No one is locking them into cages and telling them they can't leave. I would bet most everyone who is there strongly believes in the mission.
That's true for a lot of other companies where people here have advocated the employees would benefit from unionization.
For the record, I am conflicted on engineer unionization, and my original statement was intended to be neutral.
As to this specific example, on one hand I understand the drive that comes from having a real mission (as opposed to an MBA-concocted "mission") and how a team who share that drive can execute with amazing speed; conversely, I have seen how poorly teams of clock-punchers putting in the bare minimum can perform.
On the other hand, I have also seen "the mission" used as a means to guilt trip people into overwork or accepting substandard compensation.
Of the two, I lean towards the SpaceX approach being the better one. In the absolute I don't particularly like either of them.
> EDIT: Eh. They aren't paid fantastically but not slave wages either. Additionally this is spread out over three very different cities.
Those salaries look to be in line with what equivalent engineers at Raytheon, Lockheed, and Boeing are making in those same locations, maybe a bit higher. Thus, if the WLB reports from SpaceX are to be believed, their per-hour pay is much lower. It's hard to tell, though, without a proper breakdown by years of experience. Also, there are certainly teams within those companies where the engineers put in just as much time and effort as SpaceX engineers. Boeing is the only one I am aware of that has unionized engineers, though.
Eh, I wouldn't say that there's a perceived high bar at SpaceX or Tesla. At this point they're mostly known (within my corner of the aerospace world anyway) for working their employees to the bone. Very few people I know who have worked for SpaceX have lasted more than a year or two there.
Logic says that would negatively impact the quality of the software. If you are working 50+ hours a week in software development, a lot of what you do will be done when you are too tired to properly reason about what you are doing.
Not that it's impossible, but you'll need tooling that prevents you from making the mistakes you'll unavoidably make when tired.
Sorry but anyone who has worked in a professional high-tech environment should be aware of that an 8 hour days is not productive. You NEED to spend you entire day on this, you mind always needs to be on it to work on that level.
Einstein would certainly not have solved relativity by saying "Uh I guess from 9 to 5 I will work on this with a one hour break in between and then I am heading off to the fun stuff called leisure time".
No... Only if you think about these problems day in and day out you can work at in the top percentiles and this is the kind of people that get hired there.
This is getting cause and effect totally screwed up.
People want to work on cool stuff (in the sense of rockets or in the sense of making the world better, or both). They will accept lower pay (per hour) to do that. Therefore, working on cool stuff demands longer hours and lower pay compared to less glamorous things. It's not an inherent requirement of the field; it's a side effect of human preferences.
Engineering companies have capital expenses that eclipse what a software shop has to deal with. The rock stars can't demand such a large chunk of revenue.
just checked the project i'm working on, its close to 1 million lines of code without considering dependencies and we are just 3 devs, not that hard, if they have a decent team (as i would expect), a thorough test environment and processes focused on security this should not happen. They are probably rushing or outsourcing parts of it like everything else they do.
More than 5 years for sure, the thing is ancient. Quite big Ruby CRM with some IOT on the side. How did you find 25 Ruby devs? I though we were a rarity.
Well, it really depends on what the project does. Most general purpose programming has looser requirements (both on specification and implementation) than critical projects. I have the sense you're comparing apples and oranges. Have a look at ex. MISRA https://en.wikipedia.org/wiki/MISRA_C to have an idea of the strictness applied.
Hint: if you're using any methodology from the "agile" umbrella, it's not even in the same league. Specification and verification processes are arguably the most time-consuming part, not maintaining the code by itself.
Checked it, it's nice to have a standard linter, but then even at web development we use those. I know C/C++, embed systems and orbital dynamics will be harder but I would expect them to have the whole mission tested and dry-ran before, at the very least. I'd bet they were just rushed to deliver and focused on a working features checklist instead of thorough testing.
I you think MISRA-C is just "a standard linter" I don't think you've really grasped what it requires. MISRA-C compliant code requires a pretty fundamental shift in thinking.
There are some good common sense recommendations on the general development and then there are basically linter rules. The rules may be more restrictive than others but makes sense for a language such as C and the application is being used. But all this rules can be implemented by a linter and a short checklist. http://caxapa.ru/thumbs/468328/misra-c-2004.pdf
Agile has nothing to do with this. I don't want to dox myself, but I currently work on a spacecraft flight software team and we use a lot of agile principles in our development process.
I have recently worked on software testing for a roughly analagous project to Starliner. The answer is thorough simulations and ground testing. Flight code should have essentially 100% coverage in unit tests as well.
You can do a pretty decent simulation of a space mission on a powerful cluster. You then plug in your simulated spacecraft and run the real flight software. Reentry is easily the most critical phase of flight for a crewed spacecraft, and they should have absolutely tested and caught this flaw in simulation. Ground testing could have also checked on the actual spacecraft that the correct valves were actuated when commanded to.
If I were to speculate, this suggests a pretty large disconnect between the software and hardware engineering teams. The software team should have been able to identify that the wrong valves were being actuated.
I am just wondering if all these issues are not linked to a same error that would be: we loaded an old version of the software on the spacecraft.
It can be quite common in the space industry to load non-flight or old versions late in the AIT phase in order to test on other modules, or even just to officially close old actions. But before delivering everything we would check all the versions, and make sure that everything is as expected and most importantly, as described. If that step was screwed up, it is bad news for the QA, but that would explain why the errors could be found and fixed quickly: it meant just going through all the non-conformances and change logs and understand what was different between the version used, and the version that should have been used.
The incredible thing about this is that when NASA gave the contract to both Boeing and SpaceX, SpaceX was supposed to be the risky new startup and Boeing was supposed to be the safe "nobody ever got fired for choosing IBM" backup option. How times have changed.
Are you shilling against SpaceX, or was that worded slightly weird, or what?
Bug-free is a ridiculous theoretical concept, where there are humans at work, there are bugs.
As for 'perfect development culture', same thing. Ridiculous concept. There is always something to improve on.
But would I personally, with my limited knowledge about Boeing and SpaceX, but with the catastrophic fails at Boeing regarding the 737 MAX in mind, judge SpaceX the more competent company in terms of management engineering skills, efficiency, to get a product as bug-free as they can without putting money before safety concerns? OF COURSE. There is no question.
I may be misremembering but if I recall correctly SpaceX hasn't had a failure in a launch abort test. Only in a static firing test (in regards to dragon 2).
It's not about blowing stuff up. Everyone blows stuff up in aerospace. It's about engineering culture: learning from everything, folding back into product and process, and improving every revision so that failure never recurs. It's the only way to make money in that business.
Boeing used to be that culture, but has abandoned its 100 years of wisdom. Now, instead of learning from a mistake, they use marketing, blame, and PR to steer the balance sheet, not the product. They try to influence regulators instead of physics.
Really that pattern (technically antipattern) shows why things degraded the way they are. If the "default option" is taken as granted there is no incentive for it to stay adequate let alone improve and advance like it should and a "profiteer" temptation to monetize the fixed demand grows until a breaking point is reached.
Which is why in any smart organization should ironically fire someone for "choosing IBM because nobody got fired for it". Not because it is the standard but because the butt covering over actual quality is a self fufilling prophecy of trouble. Essentially choosing IBM in itself is fine so long as you do at least a surface evaluation to be sure it is actually a better choice instead of cover which subverts the actual root function.
Also why anti-monopoly laws need to be reevaluated, and improved to ensure nothing counts as a market with fewer than 10 practical competitors at any given moment, and probably more like 100 for real competition (or some similar set of thresholds to ensure flexibility).
We've sacrificed meaningful constant progress at the altar of relatively speaking small economies of scale. That's a win... for the first few years; and a loss for the hundreds thereafter.
There's nothing necessarily wrong with a segment where there's essentially one one or two providers; some things may be natural monopolies. But if so - those suppliers are essentially governments of their fiefdoms, and deserve similar restrictions, including most particularly, no incentive-distorting profit motive.
Should the government prop up 10 competitors in every industry? Is there a geographical threshold? If there is one plumber in a small town should the government prop up more or punish the one guy who wants to serve that market? Does market size matter? What if I discovered a super niche market that no one serves should we assume this market needs 10-100 competitors?
Off the top of my head I have 100s of other questions and problems with your idea. I think picking a constant number to apply throughout the entire economy is going to be fraught with edge cases.
The government doesn't need to "prop up" competitors. It just needs to prevent mergers and takeovers that limit the number of market players to an unworkably small number.
Yes, market size matters and the problem only really applies to companies who provide strategic infrastructure and services and are large enough to dictate terms to government, to customers, and to competitors, in ways that distort and limit their market freedoms.
None of this is mysterious or hard to understand, and it's been the foundation of antitrust legislation since Standard Oil.
That's easier said than done. Often times, refusing to allow a merger will mean that one of the competitors will just go out of business, and avoiding that will require propping up a competitor.
The result of that is a merger on the cheap. The surviving companies get to acquire talent, patents, trade secrets at a far cheaper rate. This would essentially kill off any company's chance of merging if it wasn't relatively healthy.
Right, I was just responding to the parent's assertion that preventing a monopoly doesn't require propping up competitors. Sometimes it does require that, and we should be conscious of the trade off we'd make in that situation.
In the 1960s you had several passenger airline companies based in the U.S. such as Boeing, Lockheed, and McDonald Douglas.
Europeans got seriously spooked because they were afraid that if Italian Airlines flew Italian planes and such, fragmented markets would prevent any European company from being a viable player in the long term.
So they collaborated on the Concorde, and then the Airbus A300, consolidating the European aerospace industry quite successfully.
It costs a lot of money to train a pilot to fly a new aircraft, and on top of that you have mechanics, ground handlers, air traffic controllers and many other skilled people whose operations revolve it.
Thus there is some public and industry benefit to consolidation, but at the moment the competitive landscape looks very different for widebody (B787, A350) airliners where new technology is making it to to the public as opposed to narrowbody (B737, A320) airliners where the youngest design is 35 years old.
(Some of that parallels the lack of competition in the U.S. airline market which is dominated by narrowbody airliners as opposed to international flights on which you are likely to ride on a widebody. If you're going to compress people in all three axes to get them in a 737, why bother thanking them for flying with you?)
Antitrust legislation used to block mergers of companies that were 1000x smaller than standard oil and controlled a much smaller share of the market. How antitrust is interpreted and enforced is very much a moving target.
I wonder how much of that was locality vs shifting standards. I recall antitrust receiving a defacto softening of standards by going from business impact to consumer impact (there are some things to debate there since effectively any merger which makes them more competitive would technically count some).
Essentially locality means Exxon and Mobil can merge but not the only two gas stations in town. Speaking of which I remember a few gas station chain mergers resulting in closing of gas stations at an intersection not because they were redundant but laws against the same owner having separate gas stations within a certain distance.
You historical knowlage is just wrong here. In fact the 'foundation' of anti-trust has changed a lot since Standard Oil and the Progressive 'Trust Busters'.
We have since realized that Anti-Trust in the Progressive period basically was applied with very little oversight and it was not applied consistantly. It was basically a way for politicans to go after the competition of their funders. This part of Anti-Trust history is often ignored in the glorifited 'Progessives fight evil capitalists'-story that is now so popular with the modern progressives.
Under the new rules, that the judges (specifically Richard Posner) came up with you actually have to show consumers were harmed. And since then all Anti-Trust cases are judged according to that criteria. And unsurprisingly the amount of Anti-Trust conviction has gone to basically nothing.
Standard Oil being case and point, during Standard Oil the oil price consistantly went down and the quality improved. The people that ran amoke against it were not in fact the poor consumers but rather local oil companies that were getting their ass handed to them by Standard Oil. They simply could not compete with the price or the quality.
You can't have 100 aerospace companies doing big projects; the economy just isn't that large.
For very mature industries, I believe studies have found that the optimal number of competitors is usually 3 large companies. For immature industries with a lot of change and innovation, that number is generally higher, but as the industry matures, players either die off or merge. A good example of this is the American auto industry.
One specific example of antitrust that wasn't handled properly was the Bombardier C Series and Boeing petitioning it for dumping. It resulted in one fewer competitor in the market because it forced Bombardier to sell the plane to Airbus.
I think that there will/can be competitors that spring out from the lower end. The trick is to prevent the oligopolists/monopolists from squashing them through manipulation of the legal system or other underhanded tactics.
The counter argument to this is that the auto industry runs on razor thin margins and is littered with failed companies. Hell, the entire industry almost went belly up in the late 00s. Just look at how much cash had to be funneled into Tesla, which is a small market luxury brand.
I would disagree with that. For a market like that 3 competitors in one country is actually probebly close to maximal efficnecy. There is such a unification in basic car concepts that adding more providers really doesn't add much.
Specially if there is also tons of international competition as well.
When there was a real need for a new company, because of EV shift, a new one happened. If it was a real oligopoly such could not be done. In fact the Big 3 attempt at using their market power against Tesla have mostly been marketing oppertunites for Tesla.
Don't forget, there's a LOT more than 3 competitors in the auto market. There's 4 in America now (Ford, GM, Fiat/Chrysler, Tesla), but that doesn't count all the foreign automakers, many of whom have manufacturing plants right here in the US: Honda, Toyota, BMW, VW, etc. Just looking at cars on the road, the "American" automakers don't even appear to have a majority of sales.
Generally I would say this is a good measure for companies that serve individuals. Person A shops around and buys a car that most meets their needs/budget. This goes out the window once you enter the world of government contracts, especially large scale ones. You currently have a few large companies; Boeing, Lockheed, Northrop, etc. that land all the major contracts. The landing of these contracts though is for the most part based on government lobbying, the assurance of a nice private sector job post government retirement and a large helping of 'this is the way we have always done it'. This has recently led to debacle after debacle where the end user (the US taxpayer) is left holding the bag for massive global corporations that continue to profit. When there is an incident large enough to cause people to sit up and take notice, the ceo of the company gets a massive golden parachute and the company says 'see we took care of it. wink wink'. Look at the F-35, Littoral Battleship, and the subject of this article.
If the government started bidding out smaller real hardware contracts to new companies, we would see a rise in new competitors and an accompanying rise in quality and roi. To do that though lobbying would have to be severely curtailed. Probably not going to happen.
>If the government started bidding out smaller real hardware contracts to new companies, we would see a rise in new competitors and an accompanying rise in quality and roi.
Do you have any evidence that this would actually work?
Small companies generally don't have the resources to do large project. How is a small company going to build a state-of-the-art fighter jet? It's pretty much impossible. Look at other countries to see how they do it with small companies: they don't. The European nations basically have only a few large companies, the Chinese are the same, and so are the Russians.
The reasons America had a lot more aerospace companies in decades past are 1) planes were a lot simpler, 2) they were wiling to accept higher risk back then (some test planes crashing and killing the pilot was OK), and 3) the defense budget was enormous because there was a Cold War and the US's economy was gigantic thanks to being the only industrial power left standing after WWII. #3 is probably the biggest factor, and isn't going to happen again.
As for the stuff about golden parachutes, that's endemic to US companies in general, not just contractors. I don't see how it really affects competition much since everyone does it, except that it adds costs to American companies that, for instance, Chinese companies do not have, so it's probably bad in the long term for American competitiveness, but that really doesn't affect this argument for competitiveness between US government contractors (since they don't compete with foreign companies anyway). The bit about "a nice private sector job post government retirement" is pretty much the same: it's bad for American competitiveness, but doesn't affect things much internally, except that it probably makes upstart companies more competitive as far as costs go, but since acquiring contracts is very political it actually probably hurts upstarts since they're not connected the way the incumbents are.
Also, I should add they government actually does farm out smaller contracts to smaller companies. They even have a policy I think which causes them to do this more than they probably should, in an effort to help small companies. They don't farm out fighter jets to small companies, of course, but there are a bunch of processes that cause them to farm out small things to small companies.
The other thing that's important to understand is that most small companies do NOT want government contracts, if they're not already in that business. It's not like there's small companies that would be happy to bid on these things but are being prevented. In reality, government contracting is a giant pain in the ass, so the companies that do it are ones that are interested in it, and are good at it: their processes are set up to handle that kind of business model. Bidding on contracts is a pretty arduous thing and requires a lot of specialized skill and knowledge and special staff, so companies that aren't in that sector just don't bother.
Sorry to reply twice, but I think there's more here to address.
First, the thing about cars doesn't work even for individuals, because it's not comparable. Buying an all-new spacecraft or aircraft isn't anything like buying a car. Cars are engineered and designed for a market, and manufactured in quantities of tens to hundreds of thousands or more. This is more like getting an extremely custom house built for yourself (think something like a Frank Lloyd Wright house, not a subdivision house): you have to find someone who can architect it, and someone to build it. Neither of these is easy because the house is entirely unique and not even like other houses. It's not something you can just go pay someone to do and not think much about, like you can with a car: you have to be actively involved in the design and construction process.
>If the government started bidding out smaller real hardware contracts to new companies
What companies have these abilities? Would you have a super-custom mansion designed and built by some random guy you met on a street corner? Of course not. You have to find someone who can actually do the work, and you have to be careful because a bunch of people will claim they can do it, but they really can't, so you can't accept their bids, and that means you have to do a lot of work in verifying they're capable of delivering.
Building a new spacecraft or airplane isn't trivial, and isn't something some small company can just do, like you seem to believe. How is a new company going to start up that can actually do these things? The only reason SpaceX worked out is because a billionaire started it as a hobby project, and even there, it's been at it for well over a decade now. Same goes for Blue Origin: it wouldn't exist if it weren't for some billionaire who apparently had nothing better to do with his time. Even going back to the old days, this is what happened with TWA and Hughes aviation: Howard Hughes was a billionaire who inherited his fortune and got bored with oil drilling bits. You can't just award a hardware contract to some tiny company that has no ability to deliver on the contract in a reasonable timeframe.
Bell/AT&T had an effective telephone monopoly during the time of Bell Labs. They invented the transistor, MOSFETs, Unix, C, lasers, information theory, photovoltaic cells, Shor's algorithm, CCDs (basis for CMOS)...
It's not in the sweet spot of this forum, but aerospace research is incredibly capital intensive and difficult. There are economies of scale of combined R&D departments. Those labs don't last forever, but I hope my example of Bell Labs shows how productive and important they can be. You would not achieve that with Boeing divided into 10 parts.
This doesn't even go into just how impractical that is for commercial aerospace. The overhead for flying multiple types of aircraft is huge. Simulators are very expensive to produce. Pilots need large numbers of hours on different planes. It would be a maintenance nightmare since parts are tested more than anything outside of medicine and similarly stringent supply chains. The market won't support 10 commercial aircraft companies.
The problem with this argument is that Boeing hasn't produced any innovation lately that I can see, while their much smaller competitor, SpaceX, is the one that came up with reusable rockets that can land themselves.
Things were really different back in the days of Bell Labs when those inventions were made (another example is Xerox and PARC). Companies like those invested heavily in R&D; long-term stability and profitability was more important than next-quarter returns. This just isn't the case in America any more.
I don't think large R&D orgs have a monopoly on innovation. And they aren't guaranteed to produce good results. In context, though, the US put people on the moon 50 years ago at the cost in present dollars of $.5T. Easily that much or more has probably been invested in the defense side of rocket development. I think SpaceX was applying 50 years of incremental improvements to a technology that was relatively stagnant ("it just works" conservative approach/lower government funding/general risk aversion profile). This is not to slight them at all, just putting it in perspective.
But look at the payloads. Right now it's satellites/cargo. They still haven't done a manned mission. Commercial aircraft fly thousands of miles per day and carry hundreds of people. They do this while being repaired by different teams around the world.
It's not as glamorous, but incremental updates in adhesives, radar, rivets, and other safety and repairability are big deals and R&D into safety and reliability shouldn't be discounted. That's on top of the satellite division and Phantom Works (sustained and controlled hypersonic scramjet, wings that change shape). I am not a rocket scientist, so take it all with a grain of salt.
While capital certainly forms a barrier to entry benefits at scale has more of an effect on the equilibrium. Lets say that there is a rough quality/profitability function
vs size called EOS. More money can help skip to the far end but if it doesn't give enough of an edge there isn't much point. A given field's capital requirements and EOS aren't independent but neither are they necessarily always driven by capital requirements.
Essentially capital intensive makes it harder to enter and reduces the rate of attempts while EOS affects how well they can actually compete when they get there. Take Microsoft establishing the XBox Console ve Bing. Both took large ammounts of capital to launch but one is a success while the other is generally regarded as a joke. Once they got large enough as a platform the Xbox was "on the map" and a significant segment here to stay. Bing meanwhile remains a "second best".
Other examples of EOS in action - while there are larger law firms they haven't consolidated, and most artists take commissions independently instead of forming a megaconglomerate. They don't get much of anything from it due to specialization and lack of shared demand.
One example of EOS shifting is the decline of music producers with digital music and the rise of independent artists - the higher scale lost the relative edge.
They got fat and complacent, and they can no longer do the thing competitively at all. SpaceX has lapped them on new spacecraft development as well as cost. And I don't think they'll be the last, either. I'd sooner trust a new design out of Blue Origin over Boeing, as well.
The "IBM" of human spacecraft, that built the Apollo CSM and Space Shuttle was North American which indeed was acquired by Boeing. McDonnell built the earlier smaller Mercury and Gemini, and it also was acquired by Boeing. Space shuttle was started 50 years ago and Apollo 60 years ago. So not so much continuity. CCDev contracts were awarded 6 years ago in 2014.
Earlier, in 2011, Lockheed beat Northrop Grumman for the bigger Orion spacecraft.
There's an excellent book called Angle of Attack by Mike Gray that follows North American's development of their part of Apollo, as well as touching on the immense efforts of all the other contractors in the program. It's not too long a book and reads like an adventure story despite being nonfiction, which I think just speaks to the kind of cowboy mentality that went into building a moon rocket in such a short time. Also neat because the other big players are companies like Lockheed, Northrop, Martin, McDonnell, Grumman, and Douglass -- all before their eventual mergers!
The other thing is that for this area, Boeing the corporation is mostly a brand. The Apollo and Shuttle engineers are overwhelmingly retired or, err, passed. At this point, no company in America really has an edge with experienced employees with designing and qualifying a new human-rated space vehicle. SpaceX and Boeing were really pretty close to the same starting line.
And really SpaceX had a general experience edge in terms of coming off of designing a lot of wholly new and successfully working non-human rated space gear.
Boeing has experience in maintaining working space gear and process, but it's not the same scale of new product and process.
I think a major part of this, as applied to SpaceX and Boeing, is historically NASA micromanaged their contractors on cost+ based contracts. That is where companies like Boeing thrived, tell Boeing exactly what to do, they can do it and at the same time they can jack up the costs so cost+ is as lucrative as possible.
Now that contracting has changed and NASA no longer micromanages but just give the contractors top down goals within a given budget, we see how bad legacy contractors like Boeing are when not micromanaged, and given budgets, rather than cost+ contracts.
'“It is our belief we wouldn’t have found it if we hadn’t gone looking,” said Jim Chilton, senior vice president of the space and launch division at Boeing'
Boeing doesn’t know how to deliver on time and on budget any more. They have spent too long being able to do whatever they want that when faced with a nimble competitor they absolutely flopped.
And they finally found a circumstance that they couldn’t lie and bribe their way out of.
And remember everyone... NASA gave Boeing $2 billion more than they gave SpaceX.
They're a government contractor: they're not supposed to deliver a functional product right away. They're supposed to deliver a totally flawed product that's way behind schedule and that doesn't work, and then they're supposed to get more money from the government to fix it until it finally does work. That's how government contacts work.
>This seems much deeper than just not being nimble.
I think the glib cynicism underplays just how bad of a cf this was. This isn't just the normal 'good enough for government work' sort of shenanigans that defense contractors engage in.
You dont need to be nimble when you are a defense contractor, you just need to exist. There have been many times the government has paid defense company to stay afloat because its in the best interest of national security that they do. If you lose the ability to build big sticks you lose your power over the world, whether it's efficient or works every time.
It really seems like this might be the start of the collapse of an incompetent monopoly. Lots of new business opportunities here, this is a rare event.
"NASA gave Boeing $2 billion more than they gave SpaceX."
I saw this argument in a lot of places. Comparisons like this doesn't have a point without context. And with the context it's not something that you may want to suggest.
Edit: In a nutshell, the starting point is not the same. Crew Dragon is derived from the cargo Dragon created for COTS years ago. The Starliner was made from scratch just for the crew transportation program.
Except that Boeing's justification is that they are a "legacy" contractor needing more funding because of their bureaucracy. One which doesn't seem to add much value.
This is pretty much nonsense. Crew Dragon is a huge step from Cargo Dragon and its not like Boing has never before had money from the government to develop stuff like that.
The reason that Boeing gets more money is that companies could set their own price, and Boeing was very confident that they would be selected whatever the price.
And this came true, they set their price 1.5 billion higher then Sierra Nevada and got selected anyway.
Ada is still widely used and having decent Ada experience will get you a job pretty quickly, especially if you know how to get things done with most of the run-time disabled.
F-22 flight controls are all Ada for example. The new training tools are C#/C++ (Unity). Inventory tracking is Java.
People on HN and elsewhere frequently slam Ada (on the rare occasion it comes up) but I really liked it, in the one college course I took that used it. That may have to do with becoming fluent in Pascal before doing much with C, and considering the general appearance of similar languages to be more pleasant.
Boeing over the last 10+ years always got by far the most money for every program (or were the only one selected) and the excuse from NASA is always 'they have greater schedule certainty'. Something that never actually worked out in practice.
Starliner was selected over Dream Chaser for that reason. And Dream Chaser would have been a much better selection.
Boeing was also selected for Experimental Spaceplane (XSP) from DARPA and basically did nothing and dropped the program, while a company Masten Space Systems couldn't survive even while they had way more innovative technology.
This human space flight stuff has the potential to cost Boeing a huge amount of credibility, they got double the money compared to SpaceX and might end up catastrophically under-delivering. They will never again be able to argue that they should be preferred based on experience.
Where are you getting that crazy idea? Why wouldn't the US government trust them going forward? It doesn't matter if they don't deliver or their planes crash and kill hundreds: what's important is that they know all the right people, and they're "too big to fail". I see no indication that the government will stop preferring them.
Of course, foreign government and airlines might very well stop doing business with them, but that'll just cause the US government to prop them up even more.
> Because of the clock problem, Boeing engineers started searching to see if there were other flaws in the software. On the evening before landing, they found one.
I feel like, maybe, some baseline QA would be a good idea for a multi-million dollar rocket. If they were able to find this bug in the middle of the mission, then presumably with some more robust testing tools, it could have been caught in a relatively short time?
I wonder what the possible justification would be for skipping testing. As a cost-saving measure for getting an MVP or website change pushed out the door, I understand -- as the foundational control software for a difficult-to-replace, very costly hardware artifact, it seems like the very definition of penny-wise, pound-foolish.
I can't imagine that they skipped testing altogether. It seems much more likely that it was just very poor testing that managed to completely skirt the issue
There are plenty of QA-related requirements [1], [2]. To a certain extent, the incentives (cost, schedule) may align with contractors trying to skirt as many requirements as possible as long as NASA doesn't catch them:
>Mr. Loverro acknowledged that NASA failed to identify the weaknesses in Boeing’s work. “Our NASA oversight was insufficient,”
I might be alone in this sentiment, but is anyone else out there with a history working with Boeing or in the industry feel slightly irritated with the armchair hot takes seen in the comments sections?
IME it's just the opposite: my uncle is a retired mechanic who used to work at Boeing, and over Christmas dinner he opined that the press and armchair hot takes are if anything not going far enough, and that internally Boeing is even more of a disaster than we know.
People have been complaining about Boeing management for years, and if it seems like "years" to me, it's probably 20+. When people have been ignoring a problem for a very long time, and it finally gets to a tipping point that has dramatic public consequences, you can expect it to get worse for at least as long as people are coasting on the inertia from their previous assumptions everything was fine.
> An investigation team has found the cause of the incorrect time problem that emerged during the December flight; Starliner gets that information from the Atlas 5 rocket that propels it to orbit. Because of the programming error, the spacecraft queried the rocket too early, before the clock had been properly set.
Anyone who has seen a space movie or airplane pilot movie knows these guys go over checklists before doing anything major. Things are checked and double checked independently. But looks like this culture has has been lost. Clock was synced and not checked before launch. Mind boggling.
Boeing didn't provide any information on their ground displays for this particular clock, meaning that ground didn't have any way of verifying this with a checklist. It's a value that should have been properly telemetered and displayed, but was not. Don't knock JSC mission control, they're extremely professional and don't make silly mistakes like that.
But what's crazy is that once the capsule disconnects from the second stage of Atlas, the only remaining part of the whole system... is the Starliner. So what other clock could they be looking at? The Atlas V blows up in the ocean.
This also implies they never ran a simulation on the actual mission timeline (with the 11 hour delay between initial rocket startup and launch), or the error would have occurred during the sim.
There are a few points I'd like to make. First, no one writes perfect code on the first go. That's why we have testers. Second, the testers don't want to cut corners. Management pressures them to do so and/or hires people who don't know what to do. I hear there was one manager over flight software and avionics (fired a few months ago) that was directly responsible for most of these problems. However, it isn't all his fault. His managers wanted good news and he was happy to give it to them. Faster! Cheaper! Lay offs! Now all the money "saved" and more will be spent redoing what should have been done right the first time along with the cost of public humiliation.
But the flight software team had very smart people (those that were not driven off by the aforementioned manager) and they did what they were supposed to do. The rush to test and fly is a management problem, not a technical one.
150 comments
[ 3.2 ms ] story [ 230 ms ] threadSoftware is hard. Embedded software doubly so, but... Come on... This should not be happening to Boeing.
The thruster map error should have been detected in simulation. If it wasn't, the simulator is also faulty.
I'll assume the thruster map was only slightly wrong and that whatever simulator runs they did with it were very subtly off.
Still... What a mess...
[1]: https://www.youtube.com/watch?v=hxQ_ZV26E04
What in the world are you talking about? There’s been no reporting of anything like that. The MAX engineers failed all by themselves. No management told them to design MCAS in a brain dead fashion. Nobody “weakened” engineering leadership.
The problem is that Boeing management never said that. Moreover, the fired CEO was an engineer.
As an engineer, he was accountable to reality. As a CEO, he was accountable to the board and the financials.
Guess which won.
Moreover, the 737 Max program ran from 2011-2015. The CEO at the time was an MBA.
Traditional engineering companies don't respect software. You can tell by the fact that they pay well below market rates.
I suspect Boeing maybe treats software the same way as they've always done, just one of many boxes that need to be ticked in a long waterfall like process. They just throw more people at it hoping it gets done on whatever time line they imagine it needs to be done. They are likely to be very conservative with respect to tools, technologies, and practices (i.e. not much has changed there in decades).
The fact that SpaceX is able to come up with a new design for Starship and can then get it and the software working for several successful hovers & hops in around 9 months, tells me they are able to iterate quickly and adapt to quite major design changes. They launch a lot of rockets and they seem to be learning a lot every time they do this. I don't see how they'd be able to do this without very solid software development practices. I'd expect lots of CI & test harnasses, probably lots of emulators and other tools, and I have a hunch they probably use a lot of static code analysis etc. to prevent preventable bugs from happening.
Any engineer or tech at SpaceX is likely worth their weight in gold at any other company due to the company's perceived high bar. Additionally, I don't hear complaints about their pay either, which is likely extremely good.
No one is locking them into cages and telling them they can't leave. I would bet most everyone who is there strongly believes in the mission.
EDIT: Eh. They aren't paid fantastically but not slave wages either. Additionally this is spread out over three very different cities.
https://www.glassdoor.com/Salary/SpaceX-Engineering-Salaries...
That's true for a lot of other companies where people here have advocated the employees would benefit from unionization.
For the record, I am conflicted on engineer unionization, and my original statement was intended to be neutral.
As to this specific example, on one hand I understand the drive that comes from having a real mission (as opposed to an MBA-concocted "mission") and how a team who share that drive can execute with amazing speed; conversely, I have seen how poorly teams of clock-punchers putting in the bare minimum can perform.
On the other hand, I have also seen "the mission" used as a means to guilt trip people into overwork or accepting substandard compensation.
Of the two, I lean towards the SpaceX approach being the better one. In the absolute I don't particularly like either of them.
> EDIT: Eh. They aren't paid fantastically but not slave wages either. Additionally this is spread out over three very different cities.
> https://www.glassdoor.com/Salary/SpaceX-Engineering-Salaries....
Those salaries look to be in line with what equivalent engineers at Raytheon, Lockheed, and Boeing are making in those same locations, maybe a bit higher. Thus, if the WLB reports from SpaceX are to be believed, their per-hour pay is much lower. It's hard to tell, though, without a proper breakdown by years of experience. Also, there are certainly teams within those companies where the engineers put in just as much time and effort as SpaceX engineers. Boeing is the only one I am aware of that has unionized engineers, though.
Not that it's impossible, but you'll need tooling that prevents you from making the mistakes you'll unavoidably make when tired.
Einstein would certainly not have solved relativity by saying "Uh I guess from 9 to 5 I will work on this with a one hour break in between and then I am heading off to the fun stuff called leisure time".
No... Only if you think about these problems day in and day out you can work at in the top percentiles and this is the kind of people that get hired there.
People want to work on cool stuff (in the sense of rockets or in the sense of making the world better, or both). They will accept lower pay (per hour) to do that. Therefore, working on cool stuff demands longer hours and lower pay compared to less glamorous things. It's not an inherent requirement of the field; it's a side effect of human preferences.
I don't want a rock star anywhere near the software that operates a complex machine where a malfunction can kill people.
How can that even be "reviewed" in any way but superficially in a time frame less than 5 years?
https://en.wikipedia.org/wiki/Rogers_Commission_Report
I guess I’m spoiled working on a (mostly) Ruby project with ~100K-ish lines of code with a team of 25
We're about to start our next hiring spree in case you're looking for a new challenge ;)
>How can that even be "reviewed" in any way but superficially in a time frame less than 5 years?
If it can be written then it can be reviewed, you just need to pay for more engineers to do that.
You can do a pretty decent simulation of a space mission on a powerful cluster. You then plug in your simulated spacecraft and run the real flight software. Reentry is easily the most critical phase of flight for a crewed spacecraft, and they should have absolutely tested and caught this flaw in simulation. Ground testing could have also checked on the actual spacecraft that the correct valves were actuated when commanded to.
If I were to speculate, this suggests a pretty large disconnect between the software and hardware engineering teams. The software team should have been able to identify that the wrong valves were being actuated.
It can be quite common in the space industry to load non-flight or old versions late in the AIT phase in order to test on other modules, or even just to officially close old actions. But before delivering everything we would check all the versions, and make sure that everything is as expected and most importantly, as described. If that step was screwed up, it is bad news for the QA, but that would explain why the errors could be found and fixed quickly: it meant just going through all the non-conformances and change logs and understand what was different between the version used, and the version that should have been used.
SpaceX are not bug free [1] and their development culture is far from being perfect [2]
[1] https://news.ycombinator.com/item?id=20877142 [2] https://news.ycombinator.com/item?id=17835760
Bug-free is a ridiculous theoretical concept, where there are humans at work, there are bugs.
As for 'perfect development culture', same thing. Ridiculous concept. There is always something to improve on.
But would I personally, with my limited knowledge about Boeing and SpaceX, but with the catastrophic fails at Boeing regarding the 737 MAX in mind, judge SpaceX the more competent company in terms of management engineering skills, efficiency, to get a product as bug-free as they can without putting money before safety concerns? OF COURSE. There is no question.
Space is hard…
Boeing used to be that culture, but has abandoned its 100 years of wisdom. Now, instead of learning from a mistake, they use marketing, blame, and PR to steer the balance sheet, not the product. They try to influence regulators instead of physics.
And holy cow, this is a long list of incidents. https://en.wikipedia.org/wiki/List_of_accidents_and_incident...
By eyeballing the list the 737 has roughly 2-3 times the incident count. Both families have ~10k units produced according to wikipedia.
Which is why in any smart organization should ironically fire someone for "choosing IBM because nobody got fired for it". Not because it is the standard but because the butt covering over actual quality is a self fufilling prophecy of trouble. Essentially choosing IBM in itself is fine so long as you do at least a surface evaluation to be sure it is actually a better choice instead of cover which subverts the actual root function.
We've sacrificed meaningful constant progress at the altar of relatively speaking small economies of scale. That's a win... for the first few years; and a loss for the hundreds thereafter.
There's nothing necessarily wrong with a segment where there's essentially one one or two providers; some things may be natural monopolies. But if so - those suppliers are essentially governments of their fiefdoms, and deserve similar restrictions, including most particularly, no incentive-distorting profit motive.
Off the top of my head I have 100s of other questions and problems with your idea. I think picking a constant number to apply throughout the entire economy is going to be fraught with edge cases.
Yes, market size matters and the problem only really applies to companies who provide strategic infrastructure and services and are large enough to dictate terms to government, to customers, and to competitors, in ways that distort and limit their market freedoms.
None of this is mysterious or hard to understand, and it's been the foundation of antitrust legislation since Standard Oil.
Europeans got seriously spooked because they were afraid that if Italian Airlines flew Italian planes and such, fragmented markets would prevent any European company from being a viable player in the long term.
So they collaborated on the Concorde, and then the Airbus A300, consolidating the European aerospace industry quite successfully.
It costs a lot of money to train a pilot to fly a new aircraft, and on top of that you have mechanics, ground handlers, air traffic controllers and many other skilled people whose operations revolve it.
Thus there is some public and industry benefit to consolidation, but at the moment the competitive landscape looks very different for widebody (B787, A350) airliners where new technology is making it to to the public as opposed to narrowbody (B737, A320) airliners where the youngest design is 35 years old.
(Some of that parallels the lack of competition in the U.S. airline market which is dominated by narrowbody airliners as opposed to international flights on which you are likely to ride on a widebody. If you're going to compress people in all three axes to get them in a 737, why bother thanking them for flying with you?)
Essentially locality means Exxon and Mobil can merge but not the only two gas stations in town. Speaking of which I remember a few gas station chain mergers resulting in closing of gas stations at an intersection not because they were redundant but laws against the same owner having separate gas stations within a certain distance.
We have since realized that Anti-Trust in the Progressive period basically was applied with very little oversight and it was not applied consistantly. It was basically a way for politicans to go after the competition of their funders. This part of Anti-Trust history is often ignored in the glorifited 'Progessives fight evil capitalists'-story that is now so popular with the modern progressives.
Under the new rules, that the judges (specifically Richard Posner) came up with you actually have to show consumers were harmed. And since then all Anti-Trust cases are judged according to that criteria. And unsurprisingly the amount of Anti-Trust conviction has gone to basically nothing.
Standard Oil being case and point, during Standard Oil the oil price consistantly went down and the quality improved. The people that ran amoke against it were not in fact the poor consumers but rather local oil companies that were getting their ass handed to them by Standard Oil. They simply could not compete with the price or the quality.
This questions seems answered by the comment it's replying to. The previous poster said:
> There's nothing necessarily wrong with a segment where there's essentially one one or two providers; some things may be natural monopolies.
So, that seems like an answer to your question: no. There are some segments where it doesn't make sense.
For very mature industries, I believe studies have found that the optimal number of competitors is usually 3 large companies. For immature industries with a lot of change and innovation, that number is generally higher, but as the industry matures, players either die off or merge. A good example of this is the American auto industry.
I think that there will/can be competitors that spring out from the lower end. The trick is to prevent the oligopolists/monopolists from squashing them through manipulation of the legal system or other underhanded tactics.
https://en.wikipedia.org/wiki/CSeries_dumping_petition_by_Bo...
Bombardier is a foreign company that is heavily subsidized by the Quebec Pension Fund.
They probably were dumping.
I don't see how anti-trust applies here.
The CSeries is a fascinating story with so many twists and turns - almost unbelievable if it was a fiction novel!
The CSeries probably did not have a viable marketing and support story against Boeing.
If you want to get wound up, then the Boeing/Embraer Civil lashup is truly anti-competitive.
https://www.reuters.com/article/us-eu-usa-aircraft-wto/wto-s...
Your example shows exactly why this idea is flawed. Three competitors is what is called an oligopoly. There is nothing healthy about that.
Specially if there is also tons of international competition as well.
When there was a real need for a new company, because of EV shift, a new one happened. If it was a real oligopoly such could not be done. In fact the Big 3 attempt at using their market power against Tesla have mostly been marketing oppertunites for Tesla.
Specially if there is also tons of international competition as well.
And Fiat is European now I think.
If the government started bidding out smaller real hardware contracts to new companies, we would see a rise in new competitors and an accompanying rise in quality and roi. To do that though lobbying would have to be severely curtailed. Probably not going to happen.
Do you have any evidence that this would actually work?
Small companies generally don't have the resources to do large project. How is a small company going to build a state-of-the-art fighter jet? It's pretty much impossible. Look at other countries to see how they do it with small companies: they don't. The European nations basically have only a few large companies, the Chinese are the same, and so are the Russians.
The reasons America had a lot more aerospace companies in decades past are 1) planes were a lot simpler, 2) they were wiling to accept higher risk back then (some test planes crashing and killing the pilot was OK), and 3) the defense budget was enormous because there was a Cold War and the US's economy was gigantic thanks to being the only industrial power left standing after WWII. #3 is probably the biggest factor, and isn't going to happen again.
As for the stuff about golden parachutes, that's endemic to US companies in general, not just contractors. I don't see how it really affects competition much since everyone does it, except that it adds costs to American companies that, for instance, Chinese companies do not have, so it's probably bad in the long term for American competitiveness, but that really doesn't affect this argument for competitiveness between US government contractors (since they don't compete with foreign companies anyway). The bit about "a nice private sector job post government retirement" is pretty much the same: it's bad for American competitiveness, but doesn't affect things much internally, except that it probably makes upstart companies more competitive as far as costs go, but since acquiring contracts is very political it actually probably hurts upstarts since they're not connected the way the incumbents are.
Also, I should add they government actually does farm out smaller contracts to smaller companies. They even have a policy I think which causes them to do this more than they probably should, in an effort to help small companies. They don't farm out fighter jets to small companies, of course, but there are a bunch of processes that cause them to farm out small things to small companies.
The other thing that's important to understand is that most small companies do NOT want government contracts, if they're not already in that business. It's not like there's small companies that would be happy to bid on these things but are being prevented. In reality, government contracting is a giant pain in the ass, so the companies that do it are ones that are interested in it, and are good at it: their processes are set up to handle that kind of business model. Bidding on contracts is a pretty arduous thing and requires a lot of specialized skill and knowledge and special staff, so companies that aren't in that sector just don't bother.
First, the thing about cars doesn't work even for individuals, because it's not comparable. Buying an all-new spacecraft or aircraft isn't anything like buying a car. Cars are engineered and designed for a market, and manufactured in quantities of tens to hundreds of thousands or more. This is more like getting an extremely custom house built for yourself (think something like a Frank Lloyd Wright house, not a subdivision house): you have to find someone who can architect it, and someone to build it. Neither of these is easy because the house is entirely unique and not even like other houses. It's not something you can just go pay someone to do and not think much about, like you can with a car: you have to be actively involved in the design and construction process.
>If the government started bidding out smaller real hardware contracts to new companies
What companies have these abilities? Would you have a super-custom mansion designed and built by some random guy you met on a street corner? Of course not. You have to find someone who can actually do the work, and you have to be careful because a bunch of people will claim they can do it, but they really can't, so you can't accept their bids, and that means you have to do a lot of work in verifying they're capable of delivering.
Building a new spacecraft or airplane isn't trivial, and isn't something some small company can just do, like you seem to believe. How is a new company going to start up that can actually do these things? The only reason SpaceX worked out is because a billionaire started it as a hobby project, and even there, it's been at it for well over a decade now. Same goes for Blue Origin: it wouldn't exist if it weren't for some billionaire who apparently had nothing better to do with his time. Even going back to the old days, this is what happened with TWA and Hughes aviation: Howard Hughes was a billionaire who inherited his fortune and got bored with oil drilling bits. You can't just award a hardware contract to some tiny company that has no ability to deliver on the contract in a reasonable timeframe.
It's not in the sweet spot of this forum, but aerospace research is incredibly capital intensive and difficult. There are economies of scale of combined R&D departments. Those labs don't last forever, but I hope my example of Bell Labs shows how productive and important they can be. You would not achieve that with Boeing divided into 10 parts.
This doesn't even go into just how impractical that is for commercial aerospace. The overhead for flying multiple types of aircraft is huge. Simulators are very expensive to produce. Pilots need large numbers of hours on different planes. It would be a maintenance nightmare since parts are tested more than anything outside of medicine and similarly stringent supply chains. The market won't support 10 commercial aircraft companies.
Things were really different back in the days of Bell Labs when those inventions were made (another example is Xerox and PARC). Companies like those invested heavily in R&D; long-term stability and profitability was more important than next-quarter returns. This just isn't the case in America any more.
I don't think large R&D orgs have a monopoly on innovation. And they aren't guaranteed to produce good results. In context, though, the US put people on the moon 50 years ago at the cost in present dollars of $.5T. Easily that much or more has probably been invested in the defense side of rocket development. I think SpaceX was applying 50 years of incremental improvements to a technology that was relatively stagnant ("it just works" conservative approach/lower government funding/general risk aversion profile). This is not to slight them at all, just putting it in perspective.
But look at the payloads. Right now it's satellites/cargo. They still haven't done a manned mission. Commercial aircraft fly thousands of miles per day and carry hundreds of people. They do this while being repaired by different teams around the world.
It's not as glamorous, but incremental updates in adhesives, radar, rivets, and other safety and repairability are big deals and R&D into safety and reliability shouldn't be discounted. That's on top of the satellite division and Phantom Works (sustained and controlled hypersonic scramjet, wings that change shape). I am not a rocket scientist, so take it all with a grain of salt.
Good. What a waste of resources.
> Commercial aircraft fly thousands of miles per day and carry hundreds of people.
Jet engines are more reliable than rocket engines, and will always be.
In your opinion, what should be the end goal worthy of resource allocation in the space sector?
Essentially capital intensive makes it harder to enter and reduces the rate of attempts while EOS affects how well they can actually compete when they get there. Take Microsoft establishing the XBox Console ve Bing. Both took large ammounts of capital to launch but one is a success while the other is generally regarded as a joke. Once they got large enough as a platform the Xbox was "on the map" and a significant segment here to stay. Bing meanwhile remains a "second best".
Other examples of EOS in action - while there are larger law firms they haven't consolidated, and most artists take commissions independently instead of forming a megaconglomerate. They don't get much of anything from it due to specialization and lack of shared demand.
One example of EOS shifting is the decline of music producers with digital music and the rise of independent artists - the higher scale lost the relative edge.
You expect the 'safe, boring' choice to be slower, more expensive, more conservative in their approach.
On the other hand, you might expect them to have expertise, and you don't expect them to cut corners on good engineering to save a few dollars.
In my experience (from software) the big incumbents have the downsides you expect, but lack the upsides.
And NASA awarded them $2 billion more than SpaceX. Boeing is just a bloated, incompetent, and corrupt mess of a company.
Earlier, in 2011, Lockheed beat Northrop Grumman for the bigger Orion spacecraft.
Haven't read the book but that kinda sounds like SpaceX today.
And really SpaceX had a general experience edge in terms of coming off of designing a lot of wholly new and successfully working non-human rated space gear.
Boeing has experience in maintaining working space gear and process, but it's not the same scale of new product and process.
I think a major part of this, as applied to SpaceX and Boeing, is historically NASA micromanaged their contractors on cost+ based contracts. That is where companies like Boeing thrived, tell Boeing exactly what to do, they can do it and at the same time they can jack up the costs so cost+ is as lucrative as possible.
Now that contracting has changed and NASA no longer micromanages but just give the contractors top down goals within a given budget, we see how bad legacy contractors like Boeing are when not micromanaged, and given budgets, rather than cost+ contracts.
You don't say.
And they finally found a circumstance that they couldn’t lie and bribe their way out of.
And remember everyone... NASA gave Boeing $2 billion more than they gave SpaceX.
This seems much deeper than just not being nimble.
>This seems much deeper than just not being nimble.
Yep, it goes much deeper than just Boeing.
I saw this argument in a lot of places. Comparisons like this doesn't have a point without context. And with the context it's not something that you may want to suggest.
Edit: In a nutshell, the starting point is not the same. Crew Dragon is derived from the cargo Dragon created for COTS years ago. The Starliner was made from scratch just for the crew transportation program.
Boeing was awarded 50% more than SpaceX. If Boeing had to start from scratch why would NASA even bother?
The reason that Boeing gets more money is that companies could set their own price, and Boeing was very confident that they would be selected whatever the price.
And this came true, they set their price 1.5 billion higher then Sierra Nevada and got selected anyway.
Ada is still widely used and having decent Ada experience will get you a job pretty quickly, especially if you know how to get things done with most of the run-time disabled.
F-22 flight controls are all Ada for example. The new training tools are C#/C++ (Unity). Inventory tracking is Java.
Starliner was selected over Dream Chaser for that reason. And Dream Chaser would have been a much better selection.
Boeing was also selected for Experimental Spaceplane (XSP) from DARPA and basically did nothing and dropped the program, while a company Masten Space Systems couldn't survive even while they had way more innovative technology.
This human space flight stuff has the potential to cost Boeing a huge amount of credibility, they got double the money compared to SpaceX and might end up catastrophically under-delivering. They will never again be able to argue that they should be preferred based on experience.
Of course, foreign government and airlines might very well stop doing business with them, but that'll just cause the US government to prop them up even more.
Nothing is too big to fail, because as long as it's prevented from failing, the failure continues to scale up.
I suspect Boeing could use similar words to describe the year they have just experienced.
Was nice knowing ya!
I feel like, maybe, some baseline QA would be a good idea for a multi-million dollar rocket. If they were able to find this bug in the middle of the mission, then presumably with some more robust testing tools, it could have been caught in a relatively short time?
I wonder what the possible justification would be for skipping testing. As a cost-saving measure for getting an MVP or website change pushed out the door, I understand -- as the foundational control software for a difficult-to-replace, very costly hardware artifact, it seems like the very definition of penny-wise, pound-foolish.
>Mr. Loverro acknowledged that NASA failed to identify the weaknesses in Boeing’s work. “Our NASA oversight was insufficient,”
[1] https://snebulos.mit.edu/projects/reference/NASA-Generic/NAS...
[2] https://snebulos.mit.edu/projects/reference/NASA-Generic/NAS...
Anyone who has seen a space movie or airplane pilot movie knows these guys go over checklists before doing anything major. Things are checked and double checked independently. But looks like this culture has has been lost. Clock was synced and not checked before launch. Mind boggling.
But the flight software team had very smart people (those that were not driven off by the aforementioned manager) and they did what they were supposed to do. The rush to test and fly is a management problem, not a technical one.