53 comments

[ 3.4 ms ] story [ 106 ms ] thread
It's been built but not delivered as its waiting for upgrades. One of the upgrades is the computer - which I had never thought about before. What platform does a cutting edge fighter even use??

Turns out it's a custom CPU/GPU developed by L3Harris [1] running the Green hills Integrity real time OS [2]. It's unsurprising that it's all super custom hardware and software for military use, but conceptually it's like alternative universe of computing. I wonder what it's like to work with.

1. https://www.l3harris.com/all-capabilities/high-performance-i...

2. https://www.ghs.com/products/rtos/integrity.html

>What platform does a cutting edge fighter even use??

Pretty sure it isn't Windows CE, but I'm not 100% certain

> What platform does a cutting edge fighter even use?

Generally speaking, something older than you might suspect. For instance, while the F-35 officially entered service with various branches of the US armed forces as early as 2015, its first flight was in 2006. The experimental plane first flew in 2000, and the earliest design work began in 1995.

Not that the compute/avionics hardware is ever truly locked into stone considering the successive F-35 production Blocks with both iterative software and hardware upgrades that have already been fielded, Service Life Extension Programs that the plane will receive in coming decades, and so on, but expect at least some of the silicon in there to share more with a Pentium 3 than a Ryzen 7.

Wow, 4.5 kW of power draw max for a server with 512 MB of memory and the equivalent computer power of a single x86 core from about 25 years ago!

I’m pretty sure an Arduino could run circles around that thing, using a battery.

Until you did a high G manuever or was cruising at 45k feet and get hit with a solar flare then your arduino would be rendered useless or at least introduce so many errors the watchdog hw would shutdown your newly acquired paperweight.
Ordinary consumer electronics can tolerate far higher G forces than the human pilot can. A standard technique is to simply liberally coat the electronics in epoxy resin, but that's only really needed for things like the guidance electronics in artillery shells.

E.g.: Here's a drone reaching nearly 13 Gs, which would incapacitate or even injure a human: https://imgur.com/SHbDK2l

One recent change is that modern electronics uses mostly surface-mount components, which are not easily dislodged by g forces. Older electronics used large discrete components with wire legs (capacitors, inductors, etc...), which had a tendency to bend back and forth and eventually break off.

However, tender requirements aren't changed just because technology has moved on. The paperwork must be completed as specified! The vendors must show that their new technology that doesn't have these problems includes solutions for problems only present in old technology! Etc...

This results in 20-30 year old technology being used in new warplanes rolling off the production line today.

But rest assured, it's being done properly, with all the correct paperwork filled out! Everything in order.

Meanwhile, for the price of one of the two cooling fans on that monstrosity, I could buy half a dozen embedded processors made with silicon-on-insulator technology, which is also rad-hard, and have a 5-way system with majority voting and a hot spare... still running off a battery and cooled passively.

It's not just about g-forces, it's about high temperatures, heating/cooling cycles, vibrations, humidity, EMP resistance, etc. etc.

It's cool that your drone can do 13 G's, but the more realistic scenario is doing 5-6 Gs over and over and over and over and over again, sitting out in the sun for weeks, sitting on the deck of a carrier in the tropics, getting launched off a carrier, getting shaken around and exposed to salty air, sitting a few feet from a high power radar - you get the point.

Also, god forbid, but these things are designed to drop nukes, so that has to be taken into consideration as well.

This is standard in a huge range of off-the-shelf consumer products, such as automotive electronics. They also are expected to operate under a huge temperature range, high humidity, etc...

Military hardware isn't magic unobtanium that exists in some parallel plane with unique physical conditions.

The choices being made for the F-35 have much more to do with government tender processes, military red tape, corruption, and general government incompetence than physics or engineering.

PS: It turns out I had misread the L3Harris info page. That spec isn't for the existing F-35 computers that are now about 20 years old, but for the TR-3 upgrade which will start rolling out in mid-2024!!! These new "flying supercomputers" will be 16x faster than before. This is equivalent to upgrading from a 1994 Intel Pentium to an Apple Watch Series 1 from 2015.

L3Harris is delivering just 1,000 units of these multi-kilowatt monstrosities for a low-low price of $300 million.

> The choices being made for the F-35 have much more to do with government tender processes, military red tape, corruption, and general government incompetence than physics or engineering.

Say who? what were the competing solutions and how did F-35 go with the selection?

The competing solutions were offered only by other bloated MIC corporations with similar habits of feeding from the government pork barrel.

I've been on both sides of this tendering process, and it's absurd how horrifically inefficient it makes things. Small, agile, and capable vendors are literally excluded from the process and can't even begin to meet the criteria for acceptance. But... that's the point. It's not about cost-efficiency or product quality, but about keeping a few well-connected companies happy.

Try to look at things from the perspective of someone like Elon Musk, or a leader you admire from any FAANG or successful modern company.

Now tell them with a straight face that you're going to keep using computer hardware from the 1990s for 20+ years without an upgrade because... the paperwork is too hard.

You'd be laughed at, and then booted from the organisation for gross incompetence.

In the military, that's normal way things are done.

Imagine running Facebook, Google, or Amazon on 20 year old hardware. Picture that for a second.

Or if you want space-rated flight hardware, imagine SpaceX using something from the 1990s for Falcon 9 rockets because of red-tape.

You're rationalising nonsense. You're trying to tell me that it's good and proper that America and its allies are going to war with planes that have flight hardware that is equivalent to what I wear on my wrist to tell me the time, but uses 100,000x the power, takes up 500x the volume, and costs 1,000x as much.

The needs of these systems are radically different.

We don't need ever more speed in processing, this isn't games.

Things like ultra reliable down to the nanosecond timing, etc. are important for these applications, not raw computing power.

You fundamentally don't understand some things about the application, therefore are presenting solutions that seem smart on the surface but are actually fundamentally flawed and non starters.

> The needs of these systems are radically different.

Different to what? Desktop PCs? Commercial avionics? Space-rated computers? Military fighter jet avionics from other planes?

The future upgrade to the F-35 is orders of magnitude worse at every metric than everything else in any of those categories.

For example, the latest F/18 has a computer that is several hundred times faster. It’s also a “networked” fighter jet with sensor integration, etc…

So what’s different about the F-35 avionics that justifies $300K per unit for the computer power of an outdated Apple Watch!?

From where I’m sitting it’s purely corruption and incompetence.

You’re saying it’s something else. Okay, what is it?

They just started delivering those block III super hornets in 2021. You are comparing hardware from different generations. Just because it's an older plane doesn't mean the avionics in it haven't been upgraded since. Don't worry, they'll get to upgrading the f35s and then they will have the newer hardware. It's the natural progression of these things.
I'm comparing the 2021 F/18 computers to the upcoming TR-3 upgrade of the F-35s which won't actually start until mid-2024!

They're upgrading the 1994 level of computer power to 2015 level of computer power... in 2024, when they already have directly comparable fighter jet avionics that has a "giga" where this thing has a "mega".

The F35 is a massive network of independent computer systems, you seem to be using “computer power” rather abstractly. Some of the computing components are state-of-the-art and others are rather old, based on requirements. Who cares if a computing component is slow if it doesn’t need to be faster than it already is. Other parts of the F35 exceed the capabilities of anything you can buy off-the-shelf.

The need to upgrade many of these computing systems is bottlenecked by the ability to upgrade the sensor systems that would necessitate upgrading the computing systems.

Also, you are talking about “mega” and “giga” when many modern US systems are “peta”. With all due respect, you might be working from an obsolete understanding of these systems.

Let me guess… do you work at Boeing?
Things like this are done for good reasons.

It's fair to say, the engineering minds at work here have reasons. Reasons that would survive this armchair whataboutism.

There is no evidence of these "good reasons".

In fact, there are many new articles bemoaning cost overruns, delays, poor performance, and more.

From what I've experienced first-hand with government tender processes, engineering considerations are a distant third on the priority list at best.

That's how you end up with the SLS using rocket engines from the Shuttle that cost more than an entire Falcon 9 rocket. Except that the the SLS uses 5 of these single-use engines, but the Falcon 9 is reusable dozens of times.

There really are situations that are this absurd, and the military industrial complex is notorious for them.

I would argue that the burden of proof is on you to demonstrate why F-35 jets are using hardware 100-1,000x worse in ever metric. Also please demonstrate why it uses bizarre non-standard CPUs instead of CotS hardware as seen in civilian avionics, SpaceX rockets, automotive ECUs, or pretty much anywhere else for that matter.

The evidence is 1000 operational planes
1,000 operational planes with cost overruns and persistent issues decades into the program — most of which are related to the avionics and their associated software!

The F-35 planes are a success despite the avionics, not because of them.

The upgrade to Apple Watch levels of computing has notably been delayed by a year into mid-2024!

How is that a successful demonstration of computer engineering?

Every single engineering effort of this level and complexity humankind has undertaken in the last 20 years have faced the same challenges.

Space shuttle, bridges, tunnels, skyscrapers, planes, trains, on and on.

Again, the burden of proof is on you to demonstrate why the F-35 has such over-priced but under-performing avionics when directly comparable engineering efforts have delivered much better electronics at much lower cost, size, power, etc...

I'm comparing the F-35 to projects such as the Falcon 9, commercial avionics, or even just automotive ECUs. All of those outperform the F-35 in every metric, including reliability.

Please list the specific engineering hurdles that explain this monstrosity, because I just can't of think of any.

However, I can think of dozens of political- and corruption- related reasons for why this came to be. There are news articles and everything...

>I'm comparing the F-35 to projects such as the Falcon 9, commercial avionics, or even just automotive ECUs. All of those outperform the F-35 in every metric, including reliability.

Apples and oranges.

Make any of those things you mention stealth, and armed, and the maintenance requirements will match that of the F-35.

Its the needs of the application that are driving many of the challenges you mention, not the implementation.

An example - the SR-71. Widely regarded as one of the great engineering feats of human history, a marvel of engineering for any time, but especially that time. It's by all accounts a maintenance nightmare. Not because it's not amazing, but because the specifics of the application demand so.

Its not easy to mass manufacture the most capable airframe in the world.

> Make any of those things you mention stealth, and armed, and the maintenance requirements will match that of the F-35.

You're conflating the attributes of the plane with the attributes of the components that make it up.

The buttons in cockpit are not "stealth". You don't shoot the pilot seat out of the cannon. Etc...

Just because the plane is stealth and armed, doesn't impose any special requirements on the computers! They're on the inside of the plane, whereas the military nature of fighter jets is mostly relevant to the outside: stealth shape, stealth coatings, armor, etc...

Similarly, the special maintenance requirements of fighter jets are mostly to do with the high performance engines and physical stresses imposed on their airframe by high-G manoeuvrers. They're nothing to do with the computers, which have no moving parts and generally are serviced infrequently.

This whole discussion reminds me of how many people don't realise that many things are fungible and are designed to be fungible. There's this tendency to "label" things so as to break this mental model on purpose. The most common example is OPEX versus CAPEX -- it's all just money!

A more directly relevant example is a customer that was grumbling about how their email was slow to sync to their phones. They were blaming the phones, because they knew in their gut that "phones have slow CPUs", right? Meanwhile, the primary mail server had about 1/3rd the computer power of the phone it was synchronising to! Literally a fraction of the CPU power, memory capacity, and even disk capacity throughput. It was hilarious.

What had happened is that they "invested" a lot of money ($50K) on a big and important rack mount server... 15 years ago. The purpose of the server is critical to the business. They mentally labelled it as "production", "enterprise", "server grade", "highly available", and so forth. That's the box they put the thing into in their brain because that's the purpose the thing served. Never mind that after more than a decade, the phone in their pocket outperformed the server in every metric! They had mentally labelled those devices as "consumer", "gadget", "end user", etc...

These are just human labels. They don't affect the metrics, the objective measurements of reliability. The "enterprise grade" server had been crashing weekly. It had disk reliability issues (old mechanical drives!). It had capacity issues. Etc... It was labelled as reliable, but it was basically a worthless anchor on the organisation. If they had replaced it with a $500 laptop, they would have been measurably and objectively better off as a business! They just couldn't bring themselves to this realisation. It just wasn't "clicking" in their brain that it doesn't matter how much money they spent, or how flashy the sales presentation was, or what the purpose of it all is.

To me, this whole thread is that story all over again. Everyone here keeps going on about how the F-35 has "this" or "that" attribute that consumer electronics don't, while completely ignoring the fact that its computers are crazy slow and out-dated even when compared to directly comparable fighter jet aircraft such as the F-18!

PS: After doing even more research, I'm starting to suspect that the cause of the suckiness of the F-35 computers is that the tender requirements probably included "must be entirely manufactured in the US by US citizens", which is a common clause in military contracts. That would dramatically shrink the pool of options, reducing the performance and quality that is available. Similar issues occur with Russian military procurement, because their internal chip fab capability is much worse than what the US can produce.

Have you ever designed or architected similar hardware?
Yes. I've worked with "non-stop" (lockstep dual-redundant) hardware and the like, and I'm reasonably familiar with what it takes to get hardware space-rated.

It's specialised and more expensive, sure, but not that much. Having a cost or some spec be worse by a factor of 5x or 10x as a consequence is within reason. 100,000x... not so much.

I've seen this play out many times with government tender processes. They write the requirements months or even years before the tender officially begins. They write about established technologies at the start of this authoring process. Then it takes a year or more for a vendor to be selected, years more for development, years for acceptance testing, years for production and rollout, etc...

That's how you end up with a 10-20 year delay between the current-gen technology and what is contractually required, by a legal document literally mandating that the technology be hilariously outdated by the time it is actually used.

The more exponential the growth curve of some technology, the worse this is. If the Army is putting out a tender for uniforms, tents, or MREs, this is no big deal. If the Air Force is building "high-tech flying supercomputers", it is crazy bad.

Government agencies, especially big ones, simply have no concept of aiming for where the puck will be, not where the puck was.

Combine that with requirements that tie the hand of the vendor behind their back, such as mandating 100% US-made parts, then costs can go into the stratosphere while the technical capability lags behind by decades.

You can keep trying to rationalise and defend this madness, but it's your taxpayer dollars being heaped into a giant pile and set on fire.

Dual redundant hardware, sounds like a server or IT equipment.

Airliners and the like are triple, sometimes quad.

Designing a Dell server for high uptime with dual power supplies is one thing. Designing stuff where it falls out of the sky and crashes into a high-school if the computer signal from one sensor to one actuator takes 2 milliseconds too long to traverse.

An Arduino has memory that maxes out at 96kB and the computing power of an 8 bit microchip from the early 2000s... because that's what it is.

I doubt to a high degree that an Arduino could handle anything that this computer can.

A lot of military silicon is, in microarchitectural terms, very old. People overestimate how much CPU is required. For example, reliable hypersonic kinetic intercept of hypersonic targets (i.e. the proverbial bullet shooting down another bullet, where both are hypersonic) can be, and has been, done all day with a MIPS R3000 CPU. The physical limits of materials restrict systems to much lower reaction times than even ancient CPUs are capable of.

The advantage of old CPUs is that you can harden the hell out of them and be made to have very predictable properties. Also, the CPUs tend to be for orchestration and control. If you need any kind of heavy exotic processing, there is a dedicated piece of silicon for that purpose.

Complex US weapon systems like the F35 are a mixture of silicon that was old before some people here were born and state-of-the-art beyond what is available in commodity systems. They generally use the oldest most reliable tech that won’t compromise requirements for any particular use case.

Not knowing much about these things, I wondered if 1,000 is a lot or a little. Wikipedia says that it is not very many [1]. But then again, we aren’t in the middle of a world war. Does anyone know what the final production number will be? As in, the vast majority that will ever be produced have already been ordered? Or is the future still wide open?

[1] https://en.m.wikipedia.org/wiki/List_of_most-produced_aircra...

For a military, non WW2 plane, 1000 aircraft is a lot. The US government plans on buying around ~1700 F35 and US allies have orders for another ~1000 F35. Overtime I would expect the number of orders for f35 to keep going up.
For a modern fighter jet, it's a lot. It's planned to remain in service till 2070.

One of the most important strategic innovations of the F-35 has been economies of scale.

I don’t have a citation for you at the moment, but I recall reading estimates of something like 4,500 F-35s anticipated to be produced over the program’s lifespan. I think I also recall that being something like a 50/50 split between the United States and the rest of the world, but I’m even hazier on that.

Taking a quick glance through the table you linked, the closest equivalent I saw was the F-4, which is a multirole fighter that was used by the US Navy and Air Force as well as broadly around the world. About 5,200 of those were ultimately produced, and that number is close to what I recall reading about anticipated F-35 production. For a variety of reasons, it’s a very different world now from when the F-4 was introduced but on balance a mid-four-figure production run for the F-35 feels about right.

For comparison, the F-16 had about 4,600 built over 50 years, while the F-15 has had maybe 2,000 built over 50 years. 1,000 for a relatively young fighter is a lot.
1000 aircraft for $1.7 trillion, so each plane costs $1.7 billion. That's nuts.
Umm each plane cost way less than 1.7 Billion. A F35A can be purchased for around $80 million. The 1.7 trillion figure is the cost of research & development, purchasing about 1700 aircraft, and sustaining them for 57 years. So that includes fuel, spare parts, upgrades, pilots, etc, for almost 60 years.
To further elaborate on this, it’s something like a few tenths of total GDP over that period.

Now, depending on how you look at it, you can consider that either a huge number or eminently affordable - I think both are true - but I think it’s a reasonable price for the roles the aircraft is designed to fill over that period.

The cost of each product must include a host of costs that fall outside of the simple manufacturing cost of each. These costs are myriad: R&D, engineering, tooling, maintenance program development, documentation, testing, re-designing, etc, etc. Thus the only fair assessment of the cost of each aircraft must be calculated as COST_OF_PROGRAM / NUMBER_OF_AIRCRAFT.

This phenomenon has been discussed ad nauseum when pricing the development cost of a new pharmaceutical drug. ALL of a pharma's annual corporate research and development costs must be divided only by the number of new drugs that year. Only THAT accurately accounts for the actual cost of developing each drug, given the number of drug candidates that fail during gestation.

Military hardware is no different. Many bids on new contracts fall through. That expense is part of the cost of doing business w/ the military. In fact, if all of Lockheed's many expenditures (productive and unproductive) were divided by the number of its shipped products (w/ cost proportionate to sale price), the actual cost of each shipped product would be substantially higher than is reported. This is an age-old game played by every government contractor to hide the true extent of how much each of their products finally cost the taxpayers. (Not to mention the practice of hiding the high fraction of cost-plus contract overruns in that business.)

From wikipedia:

  > By 2017, delays and cost overruns had pushed the F-35 program's expected acquisition costs to $406.5 billion, with total lifetime cost (i.e., to 2070) to $1.5 trillion in then-year dollars which also includes operations and maintenance
That big number is for the total of acquisition, operations and maintenance from the beginning until 2070.

F-35 has been a component of the US defense since at least 2020, so the taxpayers gets 50 years of F-35 for $1.5 BN, that's 30 billion per year. The US population is currently 332 million. That makes it $2.5 per American citizen per day.

Every morning I get a Grande coffee from Starbucks for $3.97.

I would rather not spend $912 per year for the rest of my life on fighter jets even though you choose to spend $3.97 on coffee everyday.
Then don't?
How do you suggest that I convince the US government not to tax me, or not to spend that money on the f-35 program?
You give up citizenship. You're acting like it's some impossible thing. Close to 10,000 people do so a year.

My money is on you actually enjoying the rest of the benefits America provides so you won't do it.

But if you cared enough, you would.

Sadly, my backup citizenship is Canadian, so I get to pay for F-35s no matter what I do.

On that note, giving up your US citizenship does not in any way prevent you from being taxed by the American government. There is a hefty fee to apply for expatriation, and then there are exit taxes that may have to be paid.

The hefty fee is only $2350, about two years of your F-35 payments.

The exit tax is just your unpaid taxes and only applies to high net worth individuals (over $2 million) or other special circumstances. If you already paid taxes on income, you won't be double taxed. It's basically the pro-rated "final tax bill" because you are leaving in the middle of the tax year.

could we spend that on schools and housing and infrastructure and welfare instead?
There are no lack of spending for schools and housing and infrastructure and welfare.

15% of federal spending is for defense, of which F-35 spending is part of.

To compare, 22% is for social security, 14% is for health, 10% is for medicare, 3% is for education. Note that this is federal, most of education spending is at state and local level.

https://fiscaldata.treasury.gov/americas-finance-guide/feder...

My city (NYC) alone spends $31.5 BN per year for preK-12 education [1]. NYC's population is 8.15 million, or less than 2.5% the population of the entire country.

For $30 BN/year the entire country gets 5th generation gets that are by far the best in the world, and for $31.5 BN a single city gets to fund its public education with results that are (maybe) average for the OECD countries.

[1] https://comptroller.nyc.gov/reports/spotlight-school-budget-...

Spending it on this is basically that, just one step removed
Nobody prices the cost of discrete products by the number of years of production. That'd be insane.
~500 of them have been to the US so far, more than 2000 still planned for the US alone.

The US has 4 of the top 7 largest air forces in the world.

1. US Airforce 2. US Army 4. US Navy 7. US Marines

That's interesting, I always thought the Navy was #2.
The US Army has thousands of helicopters, which count as aircraft. They have a limited inventory of fixed-wing aircraft. Only about a thousand of those helicopters are combat systems (almost entirely AH-64 Apache), there are a few thousand utility and transport helicopters that exist for quickly moving troops and equipment.

I always find the absolute numbers of US military systems surprising, particularly considering that they are largely state-of-the-art and continuously upgraded. US equipment inventories are vast, since they were originally trying to maintain parity with Soviet numbers. The situation in Ukraine suggests that the exchange rate is not anything like 1:1 but the equipment is a sunk cost at this point.