>The former Google CEO is on a mission to rewire the US military with cutting-edge artificial intelligence to take on China. Will it make the world safer?
No.
Hot take: making stuff that kills people doesn't make people safer.
What I fear are hypersonic weapons and other fast delivery systems that will require AI offensive and counter offensive weapons. At what point will humans need to be removed from the loop because a human response will mean that we would be destroyed. What happens when you have superpowers that have AI stats constantly checking on when the optimal nuke or emp time is, or even worse, faulty info or hacked info that tells it when are under attack and issues a counter offensive. Or it interprets human intent incorrectly.
What surprised me is how fast icbms travel. It's like 15 minutes from Russia to the US via ICBM. Other side of the planet in no time because it's traveling though outer space to get here
The US isn't on the other side of the planet to Russia. The missiles fly over the Arctic, and typically travel less than a quarter of the circumference of the planet.
There is actually a Stargate episode about this where two civilisations lived in bunkers and fought each other with drones without really knowing why anymore.
Actually what I recall was one side had real pilots, but the side SG1 came in on were flying drones. They were offered a chance to fly themselves then O'Neil was disturbed to see there was a person in the other aircraft.
There is the old Star Trek episode where two civilisations decided to forgo a real war for a simulated war where the computers would simulate attacks and then parlay the losses to each as a list of people who died in the attack who would then promptly commit suicide.
That reminds me of the Screamers movie about a war between the mining company (N.E.B.) and "The Alliance" (former mining and science personnel) on the planet Sirius 6B.
"Five years into the war, Alliance scientists created and deployed Autonomous Mobile Swords (AMS) — artificially intelligent self-replicating machines that hunt down and kill N.E.B. soldiers on their own. They are nicknamed "screamers" because of a high-pitched noise they emit as they attack. Screamers track targets by their heartbeats, so Alliance soldiers wear "tabs" which broadcast a signal canceling out the wearer's heartbeat and rendering them "invisible" to the machines.
A fragile stalemate is in effect between the two exhausted, poorly supplied, and undermanned armies. The Alliance recovers a message from a dead N.E.B. soldier, killed by screamers as he approached the Alliance compound, guaranteeing safe passage through N.E.B. territory to discuss a truce.
When Alliance commanding officer Joe Hendricksson reports this development to his Earth-based superiors, he is told that peace negotiations are already underway on Earth; but Private "Ace" Jefferson, newly arrived from Earth, says that is untrue. Hendricksson is not surprised; he has long suspected that both sides have simply written off Sirius 6B and abandoned their armies."
-- wikipedia
The script is loosely based on a story by Philip K. Dick:
"Set in a world where war between the Soviet Union and United Nations has reduced most of the world to a barren wasteland, the story concerns the discovery, by the few remaining soldiers left, that self-replicating robots originally built to assassinate Soviet agents have gained sentience and are now plotting against both sides."
-- wikipedia
Automated counter attack systems exist already, at least to a degree [0]. Hypersonic weapons may make a first strike more feasible, but not that much. SLBMs with a depressed trajectory already have very short flight times [1]. The defenses against this apply to hypersonic weapons as well. The defenses being maintaining a survivable second strike capability in the form of a very stealthy SSBN fleet and a launch on warning posture.
Having lethal automated weapons systems is a scary idea, but hopefully we've already learned how to avoid using them for strategic deterrence.
The title is complete click-bait. What they are actually doing:
> Ishtari uses machine learning to virtually assemble and test war machines from computer models of individual components, such as the chassis and engines, that are usually marooned on separate digital drawing boards
In all seriousness, working on projects like this should be a Mark of Death in the tech community. Programmers, VCs, anyone who actually gives a shit should outright refuse to work with people who make things like this.
Why? Surely you understand that your entire existence is predicated on being protected by a sovereign nation that prevents others from murdering you and stealing your stuff.
There's really only been 2 technological advances in weaponry that could possibly be an existential threat to humanity (nukes being the other, and existential threat to humanity is perhaps an exaggeration there). I don't think these require the same approach as non-world-destroying weapons where "speak softly and carry a big stick" can work.
Existential threats require real diplomacy. Not just, oh let's give the weapons to the "good guys".
The only way to stop this is for technologists to decide that we don't want this thing to exist. Virologists are trying the same approach with gain-of-function research, and it has mostly worked. Yes, it requires sanctions, and politics, and some people will attempt to circumvent laws and safeguards in thirst for power. Those people must be stopped.
This is not really the dunk you think it is. Light consensus among foreign policy experts is that sanctions are working[0][1]. Meanwhile our wars in the Middle East have not brought about world peace.
That's moving the goalposts. In a world with 'existential threat' (your words, not mine) weapons, making war inconvenient and take longer isn't enough, because one war ends everything. Sanctions and politics has to stop every war with 100% efficacy because the weapons in question cannot be matched. The fact that an invasion happened at all proves you wrong.
Severe sanctions should be applied to any country or individual attempting to develop autonomous weapons. Additionally, the "Mark of Death" should be applied to any technologist working on autonomous weapons. They should be outcast from society, given no support. Hardware manufacturers should not work with them, cloud computing companies shouldn't work with them. All it takes is for us as a profession to decide we don't want to destroy the world. Yes, there will be some people who try, but they won't be able to achieve this without the support of hundreds of thousands of people.
People in the tech community like to pretend like this is impossible, but very few people want to be social and professional outcasts. We have yet to see a lab-designed virus kill off humanity, despite it being theoretically possible to create.
And now you're just repeating yourself. All those measures were put in place against North Korea and they failed. They got nukes anyway. If anything, all your proposed solution will do is guarantee that the first group to get hold of this technology is largely comprised of violent sociopaths, and we won't have any means of stopping them.
I don't buy that North Korea would have nuclear weapons right now if the US and Germany didn't develop them first and lose international goodwill in the process.
Also, while NK having nukes is scary, their nuclear/ballistics capabilities are still not at existential threat level, even by their own exaggerated claims.
It doesn't matter whether it's nukes or Colonel Sanders' secret recipe, the fact of the matter is they tried exactly what you're proposing and it didn't work.
The reason nukes aren't used has more to do with mutually assured destruction (MAD) and less to do with diplomacy. What happens when a technological advance gives a nation a big enough advantage such that mutual destruction is no longer assured?
Why would we, as tech workers, build this system? Stuff like this takes tens of thousands of people deciding "sure, I'll contribute to the destruction of the earth for a few thousand bucks". Why? Why not go the gain-of-function route and just agree as a profession not to do it in large enough numbers to enable its development? Sure, this requires agreement among a lot of disagreeable people, but if we can't mostly agree on not destroying the world I think all hope is lost anyway.
> Sure, this requires agreement among a lot of disagreeable people
Disagreeable people implies they're unable to come to agreement, so you're kind of begging the question. Additionally, even if a vast majority of people favor peaceful solutions, there will always be those who are willing to exploit technological asymmetry. It's not a problem you can just wish away. It's best to accept that warfare is a reality we may need to face. It's not helpful to bury your head in the sand.
AI technologies like this require hundreds of thousands if not millions of people to develop. If chip manufacturers refuse to work with these people, cloud computing companies refuse to work with these people, then it won't be developed. Mark of Death.
Burying your head in the sand, IMO, looks like continuing to develop dangerous technologies knowing they will likely end in disaster because "technological progress can't be stopped". Somehow the medical profession has ethical guidelines that are followed enough to not result in existential crises, but we technologists could never I guess
We have lots of valid arguments for everything...but where do we think this is going to end? All living peacefully with AI as the ultimate deterrent? Nukes had to be used to become a deterrent. Now we have something no one can claim to be able to control...still let's keep on marching.
Or the opposite. War is already tech. The nuclear powers, and the ones controlling nukes control the world's fundamentals, with sideshows going on. When it comes to the ones who control the AI that will guide the nukes , who do we trust less, technologists or populist politicians? It is clear that in the future, weapons of manipulation will be just as powerful as thermonuclears, both superhuman and impossible to fight.
There’s a list of people I won’t work with alright- a list of people who throw down sweeping ultimatums and try to control others behavior and speech through fear. No thank you.
Well, fear can be applied through speech or through violence. I prefer speech.
People should be fearful about trying to develop autonomous weapons. There's nothing out there that says "I'm allowed to kill everyone in the world without ever feeling fear".
> a Mark of Death in the tech community... refuse to work with people
You called for individuals who work in this space to be ostracized, boycotted, blacklisted, etc.
You're hoping that people will read your comment and then panic at the thought of taking one of these jobs, for fear of being discriminated against further down the road the next time they look for employment.
This is you trying to control behavior through fear.
Cool, yes, you should be afraid when you go around making autonomous weapons. The rest of the world is. Not sure why anyone working on weapons of mass destruction would feel the right to be immune from fear.
I doubt you will have much luck with that. Every society is almost by definition built on using coercion and fear to keep the peace. Not saying that yunwals idea in particular is a good path to peace.
The lede is buried here in that the biggest problem is in defense procurement regulation and not really in technology or money. The other unspoken issue is labor and environmental regulation, which also influences the same problem, as does overall costs. If you think it's hard to build a condo, think about how hard it is to build all the foundries, chemical factories, and other similar heavy industries in the US. In the '70s-'80s, the US decided to "solve" its labor and environmental problems by sending heavy industry (with all of its attendant problems) elsewhere. This poses special problems for defense procurement because of the need to have trusted sources. There is no easy solution, so there will have to be catastrophe first before it starts to be addressed.
It seems like a lot of heavy industry could be environmentally friendly. It was just cheaper to send it overseas. Cars are still built in the US and that's pretty heavy. Excavation equipment is as well.
From the comments so far seems like nobody read the article. There is no "AI" or any involvement in the actual fighting. The start-up backed by Eric Schmidt is virtually testing component designs to search for faults, incompatibilities and such. It's a ML CI for CAD projects:
> Ishtari uses machine learning to virtually assemble and test war machines from computer models of individual components, such as the chassis and engines, that are usually marooned on separate digital drawing boards
> “Let's imagine we’re going to build a better war-fighting system,” Schmidt says, outlining what would amount to an enormous overhaul of the most powerful military operation on earth. “We would just create a tech company.” ... Independent studies and congressional hearings have found that it can take years for the DOD to select and buy software, which may be outdated by the time it is installed. Schmidt says this is a huge problem for the US, because computerization, software, and networking are poised to revolutionize warfare.
I'm not sure if "tech company" is the right model for weapons systems. Tech companies are often terrible at support and reliability. The military needs rugged and reliable systems with long service lives, and tech companies aren't know for that. They are known for taking shortcuts on QA, releasing with bugs and fixing them in a later release (if at all), not supporting products for more than a few years, etc.
Also, I'm not sure how military software can be "outdated," except compared to rival militaries. It's specialized stuff, and can be state of the art even it's written in Ada.
> Schmidt says, because of procurement rules that forbid the Pentagon from handing out money without going through careful but overly lengthy review processes.
The process can definitely be changed, but I feel like this is probably complaining about the tradeoffs. Isn't the current process designed to minimize corruption/maximize fairness (e.g. anyone has a fair shot if they can meet the requirements)? If you reduce the reviews, I bet there will be far more complaints of large contracts going to well-connected cronies.
73 comments
[ 2.7 ms ] story [ 138 ms ] thread>...some of that work can be done more effectively in the metaverse.
Wouldn't "simulation" be a more appropriate word? Buzzwords seem to devolve into meaninglessness.
1. Teach deathbots the value of bitcoin
2. Teach deathbots how to hash
3. Sit back and enjoy Pax Robotica
Or maybe the Screamers / The Second Variety.
What good will come of this?
To our chagrin, this increasingly describes the tech industry in general.
>The former Google CEO is on a mission to rewire the US military with cutting-edge artificial intelligence to take on China. Will it make the world safer?
No.
Hot take: making stuff that kills people doesn't make people safer.
I have a strong belief that the current AI weapons race could be the last one.
The world should look for a better consensus between all the species, not for a domination.
"Five years into the war, Alliance scientists created and deployed Autonomous Mobile Swords (AMS) — artificially intelligent self-replicating machines that hunt down and kill N.E.B. soldiers on their own. They are nicknamed "screamers" because of a high-pitched noise they emit as they attack. Screamers track targets by their heartbeats, so Alliance soldiers wear "tabs" which broadcast a signal canceling out the wearer's heartbeat and rendering them "invisible" to the machines.
A fragile stalemate is in effect between the two exhausted, poorly supplied, and undermanned armies. The Alliance recovers a message from a dead N.E.B. soldier, killed by screamers as he approached the Alliance compound, guaranteeing safe passage through N.E.B. territory to discuss a truce. When Alliance commanding officer Joe Hendricksson reports this development to his Earth-based superiors, he is told that peace negotiations are already underway on Earth; but Private "Ace" Jefferson, newly arrived from Earth, says that is untrue. Hendricksson is not surprised; he has long suspected that both sides have simply written off Sirius 6B and abandoned their armies." -- wikipedia
The script is loosely based on a story by Philip K. Dick: "Set in a world where war between the Soviet Union and United Nations has reduced most of the world to a barren wasteland, the story concerns the discovery, by the few remaining soldiers left, that self-replicating robots originally built to assassinate Soviet agents have gained sentience and are now plotting against both sides." -- wikipedia
Sometimes I wonder if TV was ever worth watching.
Having lethal automated weapons systems is a scary idea, but hopefully we've already learned how to avoid using them for strategic deterrence.
[0] https://en.wikipedia.org/wiki/Dead_Hand
[1] https://www.scienceandglobalsecurity.org/archive/sgs03gronlu...
> Ishtari uses machine learning to virtually assemble and test war machines from computer models of individual components, such as the chassis and engines, that are usually marooned on separate digital drawing boards
(before the latest "incident") https://www.politico.com/news/2022/07/05/u-s-militarys-newes...
I'd personally prefer we take the disarmament approach, but there's no money in that I guess.
1) Group of people gain a technological advantage
2) Said group of people use the technological advantage to kill or subjugate others.
The right strategy is probably to hope for the best and plan for the worst.
Existential threats require real diplomacy. Not just, oh let's give the weapons to the "good guys".
The only way to stop this is for technologists to decide that we don't want this thing to exist. Virologists are trying the same approach with gain-of-function research, and it has mostly worked. Yes, it requires sanctions, and politics, and some people will attempt to circumvent laws and safeguards in thirst for power. Those people must be stopped.
[0] - https://www.foreignaffairs.com/russian-federation/sanctions-...
[1] - https://www.npr.org/2022/12/27/1144226139/russia-sanctions-u...
People in the tech community like to pretend like this is impossible, but very few people want to be social and professional outcasts. We have yet to see a lab-designed virus kill off humanity, despite it being theoretically possible to create.
Also, while NK having nukes is scary, their nuclear/ballistics capabilities are still not at existential threat level, even by their own exaggerated claims.
Disagreeable people implies they're unable to come to agreement, so you're kind of begging the question. Additionally, even if a vast majority of people favor peaceful solutions, there will always be those who are willing to exploit technological asymmetry. It's not a problem you can just wish away. It's best to accept that warfare is a reality we may need to face. It's not helpful to bury your head in the sand.
AI technologies like this require hundreds of thousands if not millions of people to develop. If chip manufacturers refuse to work with these people, cloud computing companies refuse to work with these people, then it won't be developed. Mark of Death.
Burying your head in the sand, IMO, looks like continuing to develop dangerous technologies knowing they will likely end in disaster because "technological progress can't be stopped". Somehow the medical profession has ethical guidelines that are followed enough to not result in existential crises, but we technologists could never I guess
Neither. Luckily you only have to stop the technologists, since politicians are all technologically illiterate.
> try to control others behavior
Ah yes, speech is what controls other's behavior, not violence!
People should be fearful about trying to develop autonomous weapons. There's nothing out there that says "I'm allowed to kill everyone in the world without ever feeling fear".
You called for individuals who work in this space to be ostracized, boycotted, blacklisted, etc.
You're hoping that people will read your comment and then panic at the thought of taking one of these jobs, for fear of being discriminated against further down the road the next time they look for employment.
This is you trying to control behavior through fear.
> Ishtari uses machine learning to virtually assemble and test war machines from computer models of individual components, such as the chassis and engines, that are usually marooned on separate digital drawing boards
The current state of AI development seems to be on the same trajectory as the arpanet in 1975 with one exception, growth will be more exponential.
I'm not sure if "tech company" is the right model for weapons systems. Tech companies are often terrible at support and reliability. The military needs rugged and reliable systems with long service lives, and tech companies aren't know for that. They are known for taking shortcuts on QA, releasing with bugs and fixing them in a later release (if at all), not supporting products for more than a few years, etc.
Also, I'm not sure how military software can be "outdated," except compared to rival militaries. It's specialized stuff, and can be state of the art even it's written in Ada.
> Schmidt says, because of procurement rules that forbid the Pentagon from handing out money without going through careful but overly lengthy review processes.
The process can definitely be changed, but I feel like this is probably complaining about the tradeoffs. Isn't the current process designed to minimize corruption/maximize fairness (e.g. anyone has a fair shot if they can meet the requirements)? If you reduce the reviews, I bet there will be far more complaints of large contracts going to well-connected cronies.