82 comments

[ 2.7 ms ] story [ 153 ms ] thread
How many person years of salary is $84M for that manual from Boeing?
Assuming a cost of $200k per person, it's 420 man years, or 100 person team on a four year buildout like this. It does not sound unreasonable - or at least, about the right order of magnitude for such an undertaking.
(comment deleted)
What is the undertaking here and what does it involve? Can you please explain?
Collecting information from every single team that worked on the plane, describing how the plane is built and how to repair and maintain it.

The compiling all of that into a cohesive whole.

The article mentions the figure of 100k pages. Well, 100,000 pages of high quality and specific technical writing is quite an undertaking. If we take the size of the team in my previous post as a given, that's 1000 per writer over four years, or about 250 pages per year, or about a page per working day on average. It's quite a bit.
I hate stories like this. Yeah, excellent documentation for an immensely expensive one-off, bespoke, incredibly complex product costs a ton of money. Duh.
Totally agree, and hey, if you or anyone you know is looking for documentation, PM me!
Not $84M ,unless you're talking about a death star. And even then! I mean, back up a bit here, even the greediest tech company (oracle!) Has not thought of to charge for documentation. On what insane world would someone purchase a complex product like this without the docs? Why is it billed differently?
This works out to $840/page. How many hours does it take to write the page, by people who are making how much? And don't forget, they get revisited and revised as the design changes.

On the subject of why it's billed differently: well, first, transparency. What would you make you feel better? Getting a $4 billion lump quote for a giant project, or seeing that $4 billion itemized down so you can check the underlying data?

Edit: Oops, sorry, data entry failure. Thanks for the corrections. Fixed.

> This works out to $84/page.

You're off by an order of magnitude. It works out to $840/page.

Off by a factor of 10, $840 a page. I think you make a compelling argument but I’d rather see some data comparing costs of similar projects.
$840/page. 100k+ pages rather than 1M. Doesn’t change your other points, however.
(comment deleted)
No, things cost what buyers are willing to pay. Transparency would be saying the product costs the total sum so that everyone making purchasing decison knows the total cost. A manual in this case is as much part of the plane as is the wing! The product is a pile of metal without the docs. Is there a separate cost like this for R&D,labor,parts,paint, "service charge"? I mean come on,imagine the outrage here if microsoft made msdn cost even $10/year!

Salaried employees don't get paid by hour our output. If it takes a team a half a year to write the doc, they still get paid for the rest of the year.

Regardless, the cost nevet reflects the effort put into it. It is simply what they charge because they can and because the buyer pays and because the competition is not offering something better when bidding.

Salaried employees don't get paid by hour our output. If it takes a team a half a year to write the doc, they still get paid for the rest of the year.

One of the reasons I left aerospace is I found out that when a project ends, you and 2,000 of your closest friends all hit the street at the same time. :-)

Or as another friend put it when we were discussing the big companies: "If you work for Boeing, you'll always have a job. But you may have to move to Florida to keep it." And current events show the first part isn't even true any more...

> things cost what buyers are willing to pay

That's a ceiling but not a floor. Things never cost less than the expense of producing them. 840 per page or so does not sound excessive for this sort of thing--I'm surprise it's not 10x more.

(Yeah, loss leaders occasionally, but don't look for that in military contracts.)

At the MIL-SPEC level? $840/page sounds like a bargain. This entity is the polar opposite of "move fast and break things".

It's more like, we're going to bury this in a cavern, and when we dig it up in 100 years, it absolutely, positively has to work.

Also, it must not be forgotten that there is a lot more than writing the pages themselves.

On a document this big, you need to properly organize and index it so that the maintenance personnel would be able to find the relevant bits easily. that's a lot of editing job required, and a lot of coordination (to properly reference and label each subsystem for example).

Also, this document requires a lot of reviews and a lot of the maintenance procedures needs to actually be performed to actually check if they are clear enough to be executed reliably by operators/maintenance personnel.

All that is a lot of work, requiring a significant amount of man hours by skilled people and hardware and tooling available for validation.

> even the greediest tech company (oracle!) Has not thought of to charge for documentation

Oracle sells products/services that thousands or millions of organizations purchase. That's a completely different kind of business arrangement than something that a single entity purchases once. If you hire a software consultant to build you a custom system, they're definitely going to charge you for the time they spend writing documentation.

Not hard to imagine scenario after political wrangling where initiative is launched and the first thing the new entity does is buy Oracle for HR very special edition which eats up all the budget for the first years of its life. And nothing is left to the original reason for being for that entity.
Angry citizens want itemized lists of costs.
I wonder though if the expense is for producing the document, or because Boeing will no longer be the only company that can repair it.
I don't think Boeing does the repair, but rather Air Force personnel.
I think their point was that Boeing might be charging a premium as they will no longer service the plane.

However, I don't think this is the case, the cost probably has to do more with the specificity and overall data included.

Interesting to take that factor into account, at least in other areas of the program. I have it on good authority that a major producer of jet engines was essentially forced to provide their engines for free to a major airliner program, or have the engines excluded from the program entirely. They opted in, for those sweet operational revenues. Everybody seems to be focusing on milking the maintenance market.
My general rule in these cases: "1 is the worst number of times to do something" -- 84M for documentation for 2 airplanes seems like a lot. It looks a lot better if you have 84 airplanes or 8,400 airplanes. Alternatively, cost isn't so bad if those 84M manuals and the 2 airplanes are useful for 50 years.
"Why buy one when you can buy two for twice the price?" :-)

Yeah, not your point, but it was a great line in a great movie.

Meanwhile (ok, over a year ago) Angela Merkel took a commercial flight to G20 in Argentina....

https://m.dw.com/en/man-finds-himself-sitting-next-to-german...

Not by first choice, but still..

Respect!

I'm surprised they couldn't just call Lufthansa and get a plane with crew. But obviously Lufthansa doesn't just have that lying around, and needing to fly from Germany to Madrid first would also take a lot of time.

Meanwhile, Trump flew to Switzerland to the World Economic Forum (wow, that was actually this year, while Wuhan was in crisis) with 2 "Air Force One"'s...

I'm sure Lufthansa have aircraft and crew sitting idle now.

As for moving the president around - It's not just the vc-25a 'air force one's

They would have had at least one cargo plane to deliver his car.

And there was probably an E-4 Advance Airborne Command Post somewhere nearby too.

I don't recall exactly, but from what I remember, the choice to fly commercial was made to get to the destination as fast as possible. The time for readying a new plane and flying to the pickup airport was considered too long.
I have no knowledge of this particular situation, but...

Before I worked in aerospace as a structural engineer, I couldn't understand why airplanes are as expensive as they are.

Once I'd worked in aerospace as a structural engineer, I couldn't understand why airplanes aren't more expensive than they are.

I worked in stress analysis, where you'd analyze a part of an airplane to determine whether it fulfilled strength/fatigue life requirements. At this stage, the part's already been designed. So you take the part's material properties, geometry, surface preparation and coatings, machined features, and so on, and check it against anywhere from a few dozen to many thousands of load scenarios. You find the handful of critical load cases, and then you write up the analysis against those. The analysis is intended to be mostly self-contained, readable by any qualified engineer, and serve as a complete treatment of the part's structural sufficiency. It's not much of an exaggeration to say that every single structural component of an airplane is backed up by a stress analysis report running somewhere between 20 and 400 pages, written by one or more degreed engineers making what was, in the U.S. probably an average of $50-60/hr. Airliners are made up of many, many structural parts. The costs of proving the safety of those parts add up quickly.

Now, if I had to guess, instruction manuals are probably a bit cheaper to produce. Tech writers probably aren't as well compensated as engineers. But a lot of engineers' time will be eaten up nonetheless. (I once spent a memorable few weeks (!) proving that a dowel pin on a part that hadn't been machined correctly would not, during installation, shear off, so that existing installation documentation could still be used. Yay. After that I was known as the office dowel pin SME.) Good maintenance documents sweat the details and require input from designers, stress analysts, manufacturing engineers, supply chain representatives, and others. And then as the design evolves -- the design always evolves on the way to delivering a product -- these documents must be tweaked, added to, subtracted from, sometimes entirely rewritten, and those changes must be checked by a variety of folks.

So, $84 million for 100,000 pages might sound like a lot. But for what is in many ways a custom built aircraft, with divergences from the base aircraft that probably run into the millions, to me, it doesn't seem ridiculous.

Printing manuals, $1 million.

Research by thousands of highly trained engineers, compiled by experienced technical writers and illustrators: $83 million.

Paper: $1M

Research by highly trained engineers: $50M

Printer ink: $32M

Thanks for the detailed analysis, I was near ready to take my pitchfork out.
I can tell you in one sentence:

MATERIALS engineers are what the fuck is the problem.

We need superb materials engineers and materials engineering is HARD, and we need to drop off all the bdsm MIT cowards who get hard because they think they are going to be the bog swinging johnson after they leave, and are truly a bunch of cucks.

(This comment is deeper thAn you realize, so dont downvote the depth that you dont understand)

Edit: materials engineering is the first thing we should be throwing all things ML&AI at

That's the odd thing. To provide a custom aircraft with custom parts you have to do all the analysis and testing. The person installing each part then needs the manual for that part to installing it. The creation of manuals is organic to the creation of a new aircraft. To see manuals as a separate, very expensive, line item is odd.

But we must remember a famous line from a bad movie: "You don't actually think they spend $20,000 on a hammer, $30,000 on a toilet seat, do you?" Given what this aircraft is, what it does, I think it safe to assume a high degree of vagueness and miss-labeling in the non-classified version of the budget.

Couple of notes: You don't necessarily need the full maintenance manual to support initial installation. And the project may have been broken out differently to make the rest of the aircraft's sums seem more palatable -- bureaucratic budgetary games.
costs aside, I wonder how useful it is.

The average reading speed is about a page every 1-2 minutes (for non technical texts). 100.000 pages would then be a 800-1600 hour read, which is about half a year for a person spending 8 hours a day just flipping through pages.

Obviously such a manual is probably not meant to be read cover to cover, but even compared to a reference work, it's three times the volume of the entire Encyclopaedia Britannica - a giant amount of information to be easily used.

The cost of the information not being accessible when it is most urgently needed could easily be billions of dollars or many human lives.
You may never actually need all 100,000 pages, but you don't know in advance which pages you will need, and you'll definitely need that page when you need it.
Well it also covers all of the sub-systems on the aircraft as well, from countermeasures to communications down to how to evacuate the aircraft toilet waste.

The information in that manual is probably spread over the domain of 100's of people all performing different roles. I don't think their is any expectation that one single human would read the entirety.

Kinda like your car manual or university textbook. I've never read mine from cover to cover, but it is a useful reference sometimes. Particularly in the case of Air Force One where I doubt an internet search covers most troubleshooting requirements.

Comes out to 14 hours at $60/hour per page. Does it take about 2 days to produce a page of balls-on-dead-accurate guidance for one of the most complex flying machines on earth? I can believe that.
Well... yea, but there are some pictures in there and maybe it’s double spaced! /s
Former Boeing flight controls engineer here. Nice to see a stress guy!

I'm always amazed that for a few hundred bucks I can fly halfway around the world in a jetliner. Anyhow, you might be amused by this incident between me and the stress guy, when I was called over to visit them:

SG: We're concerned about the parts you're designing.

WB: Um, yes?

SG: Your parts are coming in at barely 1% stronger than the Ultimate Load.

WB: So they meet the requirement. What's the problem?

SG: How did you design them?

WB: I started with the load requirement, worked backwards to size the part, and rounded up the tolerances.

SG: We don't like that. We like to see 10% over.

WB: So, you want the parts to be overweight?

SG: No, we just feel better about stronger parts.

WB: But my parts meet the requirements at minimum weight. If you'd like stronger, heavier parts, you guys need to change the requirements.

SG: !@#$%^&*(

Hope you guys aren't still mad at me :-)

That's surprising from the Stress Guy. The ultimate load by definition should already have the safety factor + everything else factored in. Having a requirement go over the ultimate load, means its not really the ultimate load.
There's always some concern that the math models correctly reflect reality. That's why there's a 50% safety factor, and it's backed up by the engineers' experience that things "look right", bench testing, and flight testing. (In early flight test, they'll tape strain gauges all over the place to see if the math predictions of the stress matches reality.)
Did you ever experience any times when a part passed the math model, but didn't "look right"?
Sort of. I spent much time arguing with Stress over how to calculate the torsional rigidity of the elevator structure.
I don't mean to be too hard on Stress. They're good guys, we both had the best interests of the design at heart. Their manager paid me a great compliment by wanting me to join their team. But I liked design work too much.
BTW, this happens all the time when using structural optimization programs, where you tell the computer what material you're using, how it's constrained and loaded, and then it figures out what it thinks is the optimal shape for the part.

It means somebody left out a load or a constraint, or messed up the material properties, which is all another way of saying the math model is wrong.

So when a part doesn't look right but seems to pass the math, be very suspicious of the math! :-)

I believe that Boeing pushes for a little margin so they can revise the design without having to redesign a whole bunch of parts.
Geez Walter, you are full of surprises... When did you manage to do THAT?

(wikipedia does not have any other info, so I assumed you went to computer industry pretty much immediately after your aerospace eng degree...)

Haha, cool, I didn't know you did that! I enjoyed the back and forth between design and stress. I worked with a lot of really good people in both departments. On one program we were told to target 11% MS, only reason I could figure it wasn't written down was so that when revised loads inevitably knocked the margins down a bit, we wouldn't have to constantly be changing the fudge factor in the reports to avoid showing a negative margin that was effectively positive.

Humor me with a tangent: I think we should go back to design doing the stress work, for the most part, with some graybeards in stress and their anointed apprentices providing guidance and verification. AFAIK we really don't have the tech tools to do that efficiently, though, and nobody wants to invest the time. Design software needs access to all of the loads, temps, material properties from mil handbook, knockdowns, everything needs to be versioned and traceable, it needs to let you choose from a large catalog of analysis methods with good FEA as a fallback, show you the updated MS and critical load cases in realtime, always move your analysis nodes with the model as you edit it, export 3D models into the stress reports that remain live when the part/loads change so it changes in the next version of the report, etc. You mentioned git elsewhere. It would be so much better to validate stress report updates by viewing a diff. I could go on. Plenty to disrupt in this market, but the incentives aren't there for more than slow incremental change. I think only somebody who's willing to sink the money and open source the result has a chance at creating a revolution in how we design airplanes.

The 757 was the last bird drawn by hand on vellum, and the last with a wooden mockup. We mostly used calculators on formulae pulled from the Boeing Design Manual (a treasure), or derived formulae as needed. Some of the draftsmen did calculations by drawing geometric shapes and measuring them, which I'm sure is a lost art now.

We were mentored by greybeards who worked on the B-47 and 747, and sometimes would still use a slide rule. I received quite a wonderful education from those guys.

I was the first flight controls engineer, that I could tell, who stole time on a PDP-11 to much more efficiently do my work beyond what was practical with a calculator. (I was called on the carpet for that, the manager of the computer center wanted me censured for it, but my boss's boss's boss went to bat for me and told him to fork off.)

I understand why it costs this much. I don't understand why we think it is worth doing. Rather than fabricate a flying White House, why not deputize the vice president to carry out presidential duties while the president is in the air? There is a well-defined presidential line of succession that exists expressly to ensure continuity of government in case the president is incapacitated. Why reserve its use for emergencies? We can use it routinely.

Use the $5 billion to build housing for homeless people.

"Use the $5 billion to build housing for homeless people."

Homeless people are outside the circle of social distance and, therefore, social sympathy. ⁣Just sayin' the way to lower the cost is to produce more and sell more reprints of the manual at scale.

Finding housing for homeless people isn't a money problem. We just don't want to do it.

We find/print money for things we want to do.

> I understand why it costs this much. I don't understand why we think it is worth doing. Rather than fabricate a flying White House, why not deputize the vice president to carry out presidential duties while the president is in the air?

We have a national airborne command post, ultimately, in case the ground based command (the White House) is destroyed or under imminent threat requiring it to be (temporarily or permanently) abandoned. It's used for routine transport, and command and control while being used for routine transport, since it is desirable for it to be near the C-in-C at all times, and it doesn't make much sense not to use it.

After working in Aero, I'm still amazed airplanes fly at all.

Every single line of that document would have been double checked multiple times, and literally have 3x more documentation tracking why that piece of top level documentation exists and tracking any changes to it.

These days git handles the tracking changes thing very nicely and almost effortlessly.
Good for git? Gotcha comment is not useful when the industry (as far as I've seen) isn't using git to write manuals.
I can understand why a B737 can cost about a 100 million dollars, but I can't understand why a 4-seater Cessna costs as much as 300000 dollars... Can someone shed some light on it?
Very few units over which to amortize everything. I can't recall the exact numbers, but there were years in which (I believe it was) Continental shipped 80,000 small aircraft engines, and a decade or two later, they were down to shipping ~3,000/year.
I've always liked the detail that the space shuttle maximum thrust is rated at 104% because after some flights, they learned that the RS-25 could be run harder than the specs required, and it was simpler/less chance for error to just set max at 104% of rated thrust, rather than recalculating everything and changing all the documentation.
This is a prime example of why America will eventually fall: We pay taxes for the benefit of the few, instead of the good of the many. Meanwhile, the current president blames immigrants and liberals for all of the country’s problems, and tens of millions of Americans believe him. Fear and greed before science and compassion. I just wish all of the followers of this BS would fully reject science and go to war with only gut instincts and a bolt action rifle.

(I know i am going to be seriously downvoted for this comment, but so be it.)

America does not need its aircraft well documented as that only benefits a few??? 84 million for 100,000 pages is $840 a page and that's for a page with diagrams and one written by and reviewed by engineers.

Your imagination is leading you to think that they paid 84 million for a 10 page manual.

That's $840/page, right? So like if you have just 1 person responsible per page, and they cost $100/hr, that's 8.4 hours of work per page? I feel like I had to spend longer on my high school essays...
Yeah, that doesn't seem unreasonable at all, especially since producing a page on (made-up example, obviously) maintaining and replacing the starboard arglebargle frammistat is gonna require that the technical writer spend a fair amount of time talking with the engineers who designed the frammistat, as well as the techs who will perform the actual maintenance/replacement work. You've gotta know how to tell when the frammistat can be safely reused and when it has to be replaced, how much torque can be applied to it without messing it up, what lubricants and sealants can and cannot be used, maximum temperature ranges, all kinds of stuff, because if you mess up the docs, the plane falls out of the sky and kills the President of the United States, plus potentially many others on the ground. That would leave a significant resume stain.

Then, of course, the final document needs to be reviewed and signed off on by all those people, likely with multiple revisions.

8.4 hours per page seems like it'd be on the low side, if anything.

Well. It's $84 million to modify commercial 747-8 manual. Not re-write one from scratch.

"This contract modification is to modify commercial manuals, update with VC-25B-specific information and deliver integrated manuals for the VC-25B system"

I oppose 99.88% of taxes because it literally is all spent in this exact manner (YMMV in other countries, I'm just speaking of the USA). The money is best kept your pocket. Both Blue and Red need to oppose new taxes and advocate for keeping money in people's pockets.
Out of genuine curiosity, what's the 0.12% of taxes you support?
The stuff that ends up as a rounding error: fire, police, local guard.
I'm not going to comment on the cost of the manuals, but on a couple other concerns:

1) Mechanics and pilots are responsible for knowing the aircraft systems. Who has time to read and memorize 100,000 pages? How do you recruit new pilots in an efficient manner?

2) The cost could have been largely avoided by using an off-the-shelf airplane, instead of a billion dollar frankenstein. Possibly with the additional electronics in pods that have a limited interface with the airplane electrical system.

Source: commercially-rated airplane pilot.

I say this often: but not all dollars are created equally. I think the size of the recent stimulus packages: $2tr and ongoing trillions on top of it are case-in-point. These numbers are so astronomically large it is hard to fathom what they mean for the US and what it means for future generations.

My point about the headline number of $84 million... this is not a real number in that it reflects more than just the cost of manuals. It reflects the stimulus and backstop of strategically important private defense manufacturers like Boeing and others.

Well, Boeing does need money, overpaying for a bunch of crap already in the works is probably easier than a proper bailout.

Though I imagine both will occur in the fullness of time.

Bespoke professional services work costs bespoke professional service work rates, film at 11, but one factor not already discussed here:

A lot of contracting arrangements like this will ratably assign overhead to particular change orders or line items in ways which will cause the apparent cost of the line item to be higher than one might think necessary. This is, mostly, to avoid there being a "$250k for the manual; $5 billion for Miscellaneous Expenses" sort of bill.

(The question of whether government contracting is optimal is a long and involved one outside the scope of this comment.)

Back in the 90s, the documentation for a fighter jet, if printed out, would weigh more than the jet itself.