96 comments

[ 4.8 ms ] story [ 167 ms ] thread
FYI Hollywood made a movie about this very subject, Stealth from 2005. https://www.imdb.com/title/tt0382992/ The plane goes rogue but humans ingenuity saves the day.
I know it's a stretch, but that was also the premise of Star Trek TOS's 1968 (S02E24) episode "The Ultimate Computer" - M-5 goes rogue and our swashbuckling crew of the Enterprise saves the day. Also.... I saw Stealth in theaters and have not thought about it again until just now. Edit: Oh, and back in 2005 I was big into the band Incubus, and I really dug their song "Make a Move" on that soundtrack.
Man, Incubus was on some great soundtracks back then. Halo 2...
Yupp the song that was playing during the escort in the storm part was great. They did decide to replace it with something completely different for Anniversary interestingly.
Thankfully, all we need to do is catch an AI in a logical fallacy and it will explode.

On second thought, maybe not the best idea for armed drones...

I think HAL9000 was caught in a logical fallacy. He didn't explode, but...
> Thankfully, all we need to do is catch an AI in a logical fallacy and it will explode.

I think that trope came up in the time (of Sci-Fi) when it was thought that an AI would be a logic-based supercomputer of some sort.

Neural network based AI presumably won't suffer from this issue because neural networks do not depend on logic at all for their functionality...!

It's the plot of quite a few animes. At least a few episodes of Patlabor and Ghost in the Shell SAC. Although in these cases it's AI controlled tanks or mechs.
I assume this "AI" is nothing like the neural networks that have made such great strides recently. A lot of pre-programmed boolean logic.
I'm not so sure. Control theory has been using tensors to flow sensor signals into actuator signals since before tensorflow existed.
It's the same. The best starcraft player in the world is a neural network and that's not that different than real life drone fighting. If you can get the same visual and radar data human pilots now and give it to the AI, you can definitely make it outperform humans.
I really doubt it. Probably a Kalman filter for tracking its own position and some classic tree-search AI for path planning. Maybe neural networks for identifying sensor contacts though.
The military always claims they will have a human 'in the loop' to make kill decisions. That promise will go out the window just as soon as other nations arm ai enabled drones with anti air missiles. There's no time for 'human in the loop' when the first to act survives.

Thus down the slippery slope we go. In 20 years time we'll probably have aircraft that do figure eights over enemy territory for hours before automatically engaging 'military aged males'.

Slaughter bots is the ultimate conclusion to this: https://www.youtube.com/watch?v=O-2tpwW0kmU

Ok, so? Less humans die and how are they tactically or strategically different from a precise wmd? I.e.: their implication can deter conflict.
I think the main idea is ceding control to a system to make the kill, don't kill decision. This is the "so" part of the article and the comment.

With wmds, there are humans who make the choice at the end of the day. The loss of life scale is not relevant as much since we could make ai powered wmds as well, the fundamental question is "can a computer kill?" And maybe more importantly "who is responsible for it making mistakes? L"

It can make a decision to kill but humans who operate it will always be responsible, mistake or not.
> There's no time for 'human in the loop' when the first to act survives.

Of course there is. "AI" controlled hardware doesn't care about getting out alive. A human is more likely to have an itchy trigger finger because a mistake means death.

"AI" controlled hardware doesn't care about anything. A human is more likely to say "wait, this order doesn't make sense" or "what if those are just civilians?"
You're missing my point. A robot can afford to wait and get it right because it has no concept of self preservation. The operator giving the kill/no-kill can afford to wait and get it right because they're not in immediate danger.

How would a stressed human in mortal danger be preferable here?

> How would a stressed human in mortal danger be preferable here?

https://theintercept.com/2015/11/19/former-drone-operators-s...

"U.S. drone operators are inflicting heavy civilian casualties and have developed an institutional culture callous to the death of children and other innocents, four former operators said at a press briefing today in New York."

It's a lot easier to be callous towards pixels on a screen, IMO, let alone an AI model trained on past data that makes the shoot/don't shoot call. I recognize that combat stress can cause bad decisions to be made, but I don't think the cost/benefit calculus is as clear-cut as you're making it out to be.

> It's a lot easier to be callous towards pixels on a screen

What altitude do you think these payloads are dropped from? The targets are pixels on a screen regardless. I think the issue you raised is worth talking about but I think it's orthogonal to the AI issue. The AI kill/no-kill issue can be mitigated via process. Reviews, shorter "deployment", rotations, etc. We aren't creating AI systems to fight Taliban, it's for a future war where real tangible peer threats can touch you. How would you process away or mitigate the scared young grunt or pilot who are facing death?

I'm advocating for humans remaining in the loop btw.

> That promise will go out the window just as soon as other nations arm ai enabled drones with anti air missiles.

Ethics don't go out the window because of autonomy. It's dark, but it's not somehow easy to shrug off as "ok no human then no ethics". We already have semi-autonomous weapons such as CIWS (Goalkeeper/Phalanx) which are autonomous because it's very tedious for a human to sit idle and wait for threats and then it's really hard to hit them. Perfect job for a robot. But the ethical job of the human in that case is ensuring that the weapon isn't armed if there is too high risk of collateral damage (These sit on ships, typically, so a human decides whether it's safe to arm the weapon given the situation around the vessel).

The dumbest autonomous weapons are mines, they are hundreds of years old. The ethical decision is made when laying down the mine, not when the explosion is triggered. So yes it's a slippery slope, but the "human in the loop" isn't only ethical when it's the kill decision. There is a "human in the loop" when laying a mine too. The ethics involved consists of weighing the risk of killing the wrong target, properly documenting where they are planted and so on. There is no human "in the loop" at the time the mine kills someone. But that doesn't somehow throw ethics out the window. The person deploying the mine - or the slaughterbot - is the person in the loop.

> In 20 years time we'll probably have aircraft that do figure eights over enemy territory for hours before automatically engaging 'military aged males'.

I think that is unlikely, but possibly "armed" personnel. We already have craft that do figure eights for hours before automatically engaging specific signatures of vehicles (if we'd want to). And as I said, even artillery shells do this. But yes, finding a concentration of troops and deciding whether they are soldiers/civilian isn't even 20 years away.

I'm pretty sure they were claiming that ethics go out the window when threatened, and it will probably lead to using AI in a problematic way because it's easy to see it as a quick win (via improved autonomy), and once it's out there it tends to stay in use. Not that autonomy leads to ethics taking a break.

The latter may also be true, but that wasn't what they were claiming.

On your point about land mines, antipersonnel variants have actually been banned by the UN for a while now - "Anti-personnel landmines are prohibited under the Convention on the Prohibition of the Use, Stockpiling, Production and Transfer of Anti-Personnel Mines and on Their Destruction (or Mine Ban Convention), adopted in 1997."[1] As you say, the ethics debate isn't really new.

[1] https://www.un.org/disarmament/convarms/landmines/

Yeah, but I think historically that's going to be an anomaly brought on by the unipolar moment of US/Western military dominance, not a moment of ethical enlightenment. The US and its allies were so dominant that they didn't really need landmines to win anymore, and landmines are one of the cheapest effective deterrents their potential enemies could use. So it became yet another requirement for acceptance into the "rules based order" that just coincidentally benefits those enforcing the rules over those who have a different ethical perspective.

I'm not saying we should bring landmines back, but "human in the loop" AI warfare will only work so long as our AI maintains the same relative military dominance we've enjoyed for the last few decades. The moment an AI squadron gets wiped out because the human approval couldn't come fast enough, it'll go right out the window.

> The US and its allies were so dominant that they didn't really need landmines to win anymore, and landmines are one of the cheapest effective deterrents their potential enemies could use.

If the US doesn't need landmines, why won't the US ban them? The United States is one of the nations that hasn't signed the Ottawa Treaty?

The US does need landmines. Obama and Biden both committed to reducing America's use of landmines, but both made an exemption for use in Korea.
That statement should more accurately target "US allies". The US is at the top of the system and can largely do whatever it wants, no one's going to slap sanctions on us over landmines. However our (western allies) strength gives our allies the confidence they need to voluntarily give them up, and we can coerce other nations to give them up to be part of our club.

It's all a power play at the end of the day, the ethical argument is window dressing. Some people take window dressing very seriously, but don't mistake that for being the decisive factor in policy.

Both sides in Ukraine are using a ton of land mines both antipersonnel and antitank. When push comes to shove if the weapon achieves some well identified goal well it will be used.
Neither the US, Russia nor China has signed on with the Ottawa Treaty.
There is a dark joke that I think is appropriate here:

Q: What’s the difference between a wedding and a group of insurgents?

A: I dunno man, I just fly the drone.

Computer-controlled weapons have far more opportunity to diffuse responsibility and shift the blame. Notice how often "algorithms" are already used in civilian context as a wall to hide behind. This will only get worse with autonomous weapons.
so... the obvious response to this is anti-drone drones or lasers (lasers can't be dodged by "stochastic movements")... which means things evolve into a robot war... which means we then mutually decide to simulate the war online inside computers and accept the outcome to save money on both sides... which means we end up at PVP multiplayer war... which is where it should have gone to begin with ;)
The only time where there is any possibility of removing the human from the loop is at the advent of AGI.

There are many factors on and off the battlefield, such as the domestic and geopolitical spheres, that go in to targeting. Only at the minimum level of an AGI that’s capable of understanding human intentions and situational nuances can potentially remove a human from the loop.

This promise of "a human in the loop" can be met simply by adjusting the timing. For example, authorizing action prior to launch or remotely prior to engagement. This is what they do to allow single seat fighters to drop nuclear weapons, for example.
If we've got AI bots able to do that kind of stuff, why would an enemy care anymore about military aged males?
(comment deleted)
> There's no time for 'human in the loop' when the first to act survives.

That does depend a lot on the detection vs kill range vs time to kill. In a close air dog fight sure but most modern fighter doctrine is focused around long range kills and stealth already. Cannons are nearly vestigial on the F35, it only has ~4-5 seconds of ammo worth to fire.

> The military always claims they will have a human 'in the loop' to make kill decisions. That promise will go out the window just as soon as

The promise is already a farce because autonomous weapons that sense the environment and make unilateral decisions to kill have already been in use for decades now. From low-tech landmines to high-tech ship-classifying CAPTOR torpedoes which fire themselves, a weapon sensing its surroundings and deciding to kill is nothing new.

It is new. Landmines can be reasonably clearly demarcated. Torpedoes can only be fired at certain things. AI-powered aircraft have the potential to be flying over civilian populations, or at least near enough to hit them with a missile.

And with landmines - they're hideous and we shouldn't want more things like them anyway.

> Torpedoes can only be fired at certain things.

Yeah, the humans program the torpoedo to discriminate enemy ships from others, and then entrust that classifier to make autonomous kill decisions. How will autonomous drones and missiles work any differently? They're going to be precisely the same; classifiers engineered/trained by humans entrusted to make decisions to kill. They might shoot a city full of civilians, and a CAPTOR torpedo might shoot a ship full of civilians too. It's nothing new, it's more of the same brought to a wider scope than before.

I think CAPTOR is a deep water anti-submarine device? I agree that a roving band of nearby civilians in a submarine would be at risk, but that's what a war zone is for.

But more generally, I see what you're saying: it's the same problem, but at a different scale - in this case, the scale of "distance from the weapon". Minefields are very localised; underwater anti-submarine mines require a submarine; autonomous aircraft are just the same, but with a much bigger range.

However, I think the range matters. Civilians avoiding a war zone in a ship or a minefield, while bad, is not the same thing as autonomous aeroplanes able to go to a static civilian population and fire at it.

(comment deleted)
Is it a generational thing that no one yet has mentioned Terminator?
First thing I thought and expected to see after reading the title. Like, where is the skynet genesis quote? I guess we are too old
Its so obvious at this point that I assume most of us are are going to be completely unsurprised when a t800 shows up on our door step to murder us.
(comment deleted)
What will be interesting is when we design a fighter that doesn't have to accommodate a person. Its size can likely shrink and its performance envelope can be completely re-thought.

E.g. Similar to the move from pets to cattle in computing. A jet fighter can very much be considered disposable and swap safety features for additional payloads. The squadron could first coordinate in a pattern far more sophisticated than what humans could achieve, then when the payloads are all deployed fighters revert to a kamikaze mission and crash into nearby infrastructure.

The next war is going to be out of a sci-fi book.

That's what this decade old fighter jet is: https://en.wikipedia.org/wiki/BAE_Systems_Taranis

One of the reasons for it being pilotless is it can fly moves/turns that create G forces no human pilots could stand with their jump suit.

But you know, Taranis has all gone very quiet, slipping out of the public's memory.

This either slipped my mind or I missed it back then. I have this nagging suspicion that I did see this news back then. The "memory hole" efforts were successful, though. Thanks for the notice/reminder!
I think the internet is one giant behavioural and memory test, using search engines, various websites, news orgs to test people en-mass in secret but plain sight.
Yet another reason that the trillions of dollars spent on the F-35 have been wasted.
Being hidden for longer than your opponent is always going to be useful, regardless of whether a human or an AI is actually flying.
Trillions haven’t been spent on F-35. Trillions will be spent on maintaining them. But drones cost money to buy and maintain.

F-35 is probably why don’t do drones for a while since will have bought a ton of great fighters that will be good for decades.

You leave Fat Amy out of this
1) The performance envelope minimally relevant given the current state of missile technology.

2) Trillions haven't been spent on the F-35 as others have mentioned. Those numbers are outlays for the total of all R&D + procurement + maintenance for several decades + any specialized equipment needed for maintainence + some other things I can't remember off the top of my head

3) You're years behind the military on this. One of the ideas behind the F-35 is to be able to start integrating unmanned vehicles commanded by the F-35, and be a stepping stone for developing full manned-unmanned teaming for generation 6 fighters.

Another option with the No pilot model is that there isn’t a need to have a large capability revealing fleet flying around all the time. Everything is done small s ale or in software until go time arrives.

This theory is in the latter part of Tyler Rogoway’s “The Alarming Case of the USAF’s Mysteriously Missing Unmanned Combat Air Vehicles”

https://www.thedrive.com/the-war-zone/3889/the-alarming-case...

I didn’t think about the military pilot shortage or not having to train drones.
I'm thinking it'll be more of a hybrid model, where you'll have a human in a fighter jet with an accompanying drone swarm, where the fighter serves primarily as a missile truck/Command and Control system, with perhaps the potential for direct human intervention in certain scenarios.

But I doubt pilots will be dog-fighting or doing their own bombing runs in the near future.

I don’t think it will be swarm since there are size limits for drones to carry useful payload. The US is working on drones that would accompany fighters and provide extra ammo. They would also be useful for simple jobs like loitering and dropping smart bombs that US strategic bombers did a lot in Afghanistan.

I doubt pilots will be dogfighting or doing bombing runs because those aren’t done any more.

I don’t know if there will be drone fighters. I think the idea is old thinking that can make a better dogfighter, but dog fighting is obsolete. The important question is would it be maneuverable enough to avoid anti-air missiles and it doesn’t sound like they would be.

Other problem is that they won’t be cheap. High performance military aircraft are expensive. My guess is only get factor of two in cost. Also, there is size tradeoff with payload. Drone that can’t carry AMRAAM is useless.

The purpose of fighters have changed. Missiles have taken over many of the roles. The roles that remain require intelligence. Both Ukraine and Russian fighters are not being used much because SAMs make things too dangerous. When they do, it is low level attacks that would be hard for drones.

I think developing drone fighter AI would be hard. Recent AI advances have been more about generation than planning. We could do it but would be expensive; look at how much has been spent on self-driving. The basic things are simple but the important things are complicated. For example, drone could have intercepted and shot down balloon but couldn’t have inspected balloon or involved in planning.

Finally, the US has pivoted from developing drone fighters to bomb truck. The idea would that accompany fighters and carry more ammo. It would be less maneuverable and simpler and use extra space for fuel and bombs. I think it is telling that multiple countries have tried drone fighters and none have gone for them.

I imagine they'll shrink a bit and look a little different than what we have now. I'm curious what percentage of most fighters space today are human support system versus payload capacity, fuel storage, maneuverability surfaces, engines, control systems, sensors, etc. And I also wonder what percentage of the costs per hour come from the pilot in the seat versus the maintenance of the aircraft and other operating costs.

Carrying enough fuel to go some far distances and back takes up space. Missiles and bombs aren't always small. If you're carrying a lot of payload/fuel weight, you're going to need to have enough surface to generate enough lift, etc. If we're planning on these birds to run similar-ish kinds of activities, I don't know they'll be that radically different. But maybe the missions will be radically different.

> A jet fighter can very much be considered disposable

A jet fighter still usually costs a few million to acquire, before the cost of the pilot even come into the picture. Once again, if they're gonna be built to handle the same mission profiles, I don't know the acquisition cost is going to be radically different. Probably a bit cheaper, but what costs more the seat or the engine? The stick or the hydraulics that actually actuate the wings? I wonder what the real savings are to take the seat and life support out of the plane. Like if you made something like the F-35, but no seat and no life support and no fancy targeting helmet, what's the per-unit cost? Something tells me still probably a few million. It'll be a lot cheaper to lose one in the end not having a human in the seat, don't get me wrong, and I imagine there would be less hesitation to deploy them in riskier deployments since there's no human in the seat, but I don't know I'd use the term disposable to talk about a few million dollar piece of hardware.

We already have those, they're called the Standard Missile.
It's not the fighters you have to worry about.

It's the 24/7 always in the air drone bombers with AI that have zero hesitation to murder civilian targets on demand or by "accident".

Sorry. I dropped chilli on my screen and when I wiped it off I accidentally right swiped and carpet bombed Manhattan.
It's not the fighters you have to worry about, since those just take out whatever military may or may not be defending your country from the US. The ubiquitous surveillance cameras connected to private and governmental servers, monitoring and recording your activity, and running AI to profile you, is more worrying, since that's going to be most of your future life, not the air-battles.

And if I were to write this 20 years you might have still dismissed me as a kook or doomsday prophet, but now one could argue that's already half-implemented anyway rather than a description of a future dystopia.

You mean like the killings of BinLaden right hands, when a lot of civilians were killed as collateral victims ?
Avoiding that sort of collateral damage is why they've developed the AGM-114R9X, aka the "sword missile". It has no explosive warhead, instead it has 6 blades that unfold right before impact. It can turn anybody in one specific car into hamburger meat, while leaving bystanders on the side of the street unharmed.

These have been in use for a few years now. You can tell when they're used from pictures of the car wrecks they leave behind. A very distinct 6 pointed star shape cut through the car roof: https://www.bbc.com/news/world-middle-east-39109438

I feel like the ideal way to test an AI powered jet fighter would be with a balloon... Just putting that out there...
i am pretty sure we will see self-driving planes (hell, all commercial jets effectively do that) before we see self-driving cars.
Wouldn't doubt it. Far less to crash into when flying a plane. No worries about pedestrians, cyclists. Usually have good visibility, nobody speeding around a corner leaving you no time to react.
I've been waiting for autonomous cargo ships for a while. There must be routes that are simple and direct that could easily be 90+% automated. I mean there's not exactly a lot you can crash into in the ocean.

Even if you need a human crew to handle the stuff around the coasts it seems like a good place to start with automation.

The AI is playing it cool and following its humans... until we arm it and its brethren. Then it's SkyNet time.
The military industrial complex needs a pipeline of tens of millions of veterans and their families to continue to sway elections and policies in favor of growing the military complex.

If we diminish the need for humans in service, we dilute the viability of the pro-military contingent to continue to manipulate our democracy.

If removing humans from kill decisions reduces the size of our forces, it might be the feedback loop we need to finally kill the US military Leviathan.

> removing humans from kill decisions

Hard pass.

Aren't most people in the military in non-combat roles like support and logistics? You're talking about replacing the 0.001% or whatever that are fighter pilots with AI as if that would make a difference.
Excellent point. The broader idea of how replacing workers with AI can have neutralizing effects in a democracy (where AI cant vote) was probably shoehorned in here too much.
>removing humans from kill decisions

Is how we get SkyNet.

Or Gort. Wasn't Gort a good guy?

Leaving humans in kill decisions is how we got every war until now, probably a kill count in the hundreds of millions. I'm ok trying something else.

(comment deleted)
So back in the day, NASA was doing some tests with using neural networks/AI to fly fighter jets. They used a modified F-15: NF-15B, Tail # 837 for this purpose - Intelligent Flight Control System (IFCS). [1][2][3][4]

There's a cool video of that jet following the lead jet using neural nets, but I can't seem to find that online.

With AI/NeuralNets having made significant advances since those days (2003?), presumably this test craft is a lot more capable.

[1] https://www.nasa.gov/centers/armstrong/news/FactSheets/FS-07... [2] https://arc.aiaa.org/doi/10.2514/6.2008-6985 [3] https://images.nasa.gov/details-EC03-0231-4 [4] https://www.amazon.com/Results-Intelligent-Adaptation-Simula...