A common retort to someone talking about extinction risk is "only people which don't work on AI research or understand transformers are scared. people that work on LLMs know better than to fear matrix multiplication".
That's like saying don't be scared of layoffs because you know better than to fear the spreadsheet. The technology is generally not a reason to fear, or not fear, something.
Just because a bit of harmless algebra is at the foundation of a technologie doesn’t make it so that all technology build with that is harmless too.
There are many valid reasons to fear the impacts that LLM’s have on society
It amazes me how pieces like this never stop to question whether these beliefs are genuine.
If I'm one of these researchers who's leaving their position, "I am so smart that I can make something world endingly dangerous, but also so principled that I'm willing to walk away from it" is a pretty slick bit of self promotion.
Even if I was making a career change for unrelated reasons, that's absolutely the story I'd be telling.
And no one wants to take action, they want others to. Anthropic CEO urges AI companies to slow model development amid fears over misuse [1]. If they truly believe that, they would announce "we are slowing model development."
The other half of it is that it is in the incumbents interest to have these stories because they want the government to regulate ai into an oligopoly of incumbents so they can maximize benefit to themselves and stop new entrants. Ai benefits will be competed away to the consumer benefit justlike every other benefit if the government doesnt ruin it with regulation, entrenching the current incumbents and causing deadweight loss to society forever
Sounds like you built a position that's immune to evidence?
If researchers say AI is dangerous but keep their position, well they're obviously hyping up their boss and doing it for the money, otherwise they'd quit. If they say AI is dangerous and quitting, they're self-promoting.
I have no idea what these researchers could do, if they were genuine, that you would accept as even weak evidence that they believed what they say.
"AI hardly needs to wipe out humanity in order to be harmful, though. Many experts predict that more powerful models will lead to a coming wave of AI-assisted cyberattacks. The technology is now widely used for disinformation campaigns, and military adoption of AI is accelerating rapidly."
I fear the vast (and growing) human incompetence in essential domains. Education and science are crucial to society, and reasoning is necessary to cooperation at every level.
The "zone" of public discourse is flooded with sewage. US news business has lost its mind. And government...
Humans being "in-the-loop" is not enough. Humans must know what they are actually saying and doing. Humans just being "in-the-loop" cannot fix incompetence in social policy and software design.
Humans are "in-the-loop" in corporations and that isn't working very well for society.
yeah, what's they're describing is essentially: look how morally unopposed and devoid corporations are and we're just letting them develop AI without any restraints on both capitalism and the power resultant from AI.
Given sufficient human incompetence, the technology doesn't even need to be powerful or efficient (or even work as advertised). It just needs to amplify/scale human incompetence.
Right now there's pretty limited set of actions that rogue AI can do with internet access. The most dangerous options are hacking into biolab facility, and maybe hacking into nuclear reactors or nuclear weapons.
These things should be as secure as possible, and I don't think rogue AIs, even super intelligent ones, have the capability to destroy humanity. Not to mention, super intelligent AIs would know that destroying humanity will also destroy themselves.
Maybe in 20 years once we have only self driving cars, robots in every household or maybe robots with weapons, then AI would pose a real extinction risk. But we're not there yet and it depends less on the development of AI and more on the development of physical real world robots.
As mentioned by another commenter, the real danger is that these tools act as amplifiers and enablers for human agents that want to do really bad things. Whereas some crazy group that wants to destroy X for some reason, but had no ability to do so before, with the help of advanced AI tools might have a chance.
The other is the human deploying these things don't even need to be bad (evil) actors. They can just be really dumb and/or lazy in how they use them and then inadvertently cause huge amounts of real damage.
As Margaret Atwood pointed out "Stupidity is the same as evil if you judge by the results"
Yep. Fire is actually a great example, its both useful and dangerous. And so we have probably 10s of thousands of fire regulations for products, buildings, vehicles. We have fire departments, we have regular fire drills, we think about fire safety constantly and for the very good reason that if we don't fire can kill us and destroy vast of amount of stuff.
So, hell yes. Let's treat AI with the same degree of care we treat fire.
AI is a problem only now where the people in power or their families are at risk of being assassinated. AI assisted drones were not a problem at all when they were sent for years to kill everybody in weddings at some obscure Asian country.
Is obvious that Trump fears an AI assisted killer droid attack. Change of hair tone, to make the automatic match more difficult, thick plastic screens seen behind him at rallies, Trump impersonators and limousines driving around some places trying to lure the assassins out...
He is reducing his Chiquito de la Calzada [1] impersonations, looks much less smug and was adopting a lower profile, maybe fearing the post-midterms. His social truth posts are filled with images of him with radioactive fire balls in his hands and escaping to somewhere in planes and vehicles. Asking for a ride out of US Trump?
The risk of some disruptive event in US in the next 60 days is greater than ever in US history, and it does not need to be assisted by AI. NS (natural stupidity) is more than enough.
I don't see how this could be stopped unless every country agrees to follow international laws of war. Sadly this can't happen until a few old men and war criminals are arrested, or they die of old age.
These men are the real problem. They will block or delay the climate actions until is too late. They will remove anti famine measures just by spite, killing thousands. AI can't kill everyone because this people will do it much before. To save themselves from jail, or to earn one million more.
Is it because they will get funding to stop the risk even if the risk never existed in the first place to any significant extent? Are most of these wild declarations not exclusively for market value and funding?
TL;DR: Researchers assumed it would be easier to align these models when they become smarter. In reality, agents are getting increasingly smarter but aligning them also becomes increasingly harder, or even impossible. They are losing their control over the agents. They don’t write their code anymore. They don’t know agents limits nor how to limit them.
They are like lab scientists in a Hollywood thriller, watching a mutant organism rapidly evolving in front of their eyes.
My bet is that humans will kill humans faster than AI. AI enables not only good actors, but also bad actors: Russians, Iranians, Houthis, other terrorist groups - they all now effectively use AI to speed-up development of their weapons as well as hybrid warfare.
100% agree. All of these stories seem off the mark...with the increased ability to create agentic hacking swarms, mass surveillance, killer AI drones, virus engineering, etc. it seems to me that we should all be most concerned with humans, enabled by AI, destroying everything. AI is the least of our concerns.
Nice soundbite. Do you have an actual suggestion too? I'd love to hear you suggestion to keep AI out of the hands of people who will certainly use it to kill.
Russia, for example, is already using AI targeting (Ukraine too but "with callback", ie. AI targeting asks permission to the commander and no permission? No targeting), China and Turkey are delivering AI targeting solutions to ... essentially anyone (the old Syrian government and Azerbeijan were both provided with them, and Azerbeijan probably used them against Armenian civilians in the blockade of Nagorno-Karabagh). Turkey provided them to Ukraine earlier in the war, then Russia flew a number of planes very close to Turkish borders and now they don't anymore.
The son-in-law of Turkey's prime minister, Özdemir Bayraktar, is among selling them.
As for diplomacy, the UN has already decided they will do nothing about it. And that means, btw, that the ICC will do nothing either.
So you want this? Excellent! So what are you going to do? Keep in mind that serious amounts of violence will be required to accomplish your ban on AI drones, because the main strength of AI drones used to be that only side has them, but that time is passed. Turkey has no doubt built up their production, Russia obviously has (and is producing 100+ per day), and China, well we don't really know, but it's China, so ...
When two sides have them (like in Ukraine), it seems to be a numbers game. So you want to make any headway, you'll have to produce large amounts of AI targeting drones to accomplish this.
37 comments
[ 0.26 ms ] story [ 3.3 ms ] threadA common retort to someone talking about extinction risk is "only people which don't work on AI research or understand transformers are scared. people that work on LLMs know better than to fear matrix multiplication".
If I'm one of these researchers who's leaving their position, "I am so smart that I can make something world endingly dangerous, but also so principled that I'm willing to walk away from it" is a pretty slick bit of self promotion.
Even if I was making a career change for unrelated reasons, that's absolutely the story I'd be telling.
I have mostly pity for you, that your life is so filled with dishonesty.
https://www.reuters.com/business/anthropic-ceo-urges-ai-comp...
If researchers say AI is dangerous but keep their position, well they're obviously hyping up their boss and doing it for the money, otherwise they'd quit. If they say AI is dangerous and quitting, they're self-promoting.
I have no idea what these researchers could do, if they were genuine, that you would accept as even weak evidence that they believed what they say.
The "zone" of public discourse is flooded with sewage. US news business has lost its mind. And government...
Humans being "in-the-loop" is not enough. Humans must know what they are actually saying and doing. Humans just being "in-the-loop" cannot fix incompetence in social policy and software design.
Humans are "in-the-loop" in corporations and that isn't working very well for society.
Given sufficient human incompetence, the technology doesn't even need to be powerful or efficient (or even work as advertised). It just needs to amplify/scale human incompetence.
These things should be as secure as possible, and I don't think rogue AIs, even super intelligent ones, have the capability to destroy humanity. Not to mention, super intelligent AIs would know that destroying humanity will also destroy themselves.
Maybe in 20 years once we have only self driving cars, robots in every household or maybe robots with weapons, then AI would pose a real extinction risk. But we're not there yet and it depends less on the development of AI and more on the development of physical real world robots.
The other is the human deploying these things don't even need to be bad (evil) actors. They can just be really dumb and/or lazy in how they use them and then inadvertently cause huge amounts of real damage.
As Margaret Atwood pointed out "Stupidity is the same as evil if you judge by the results"
So, hell yes. Let's treat AI with the same degree of care we treat fire.
Is obvious that Trump fears an AI assisted killer droid attack. Change of hair tone, to make the automatic match more difficult, thick plastic screens seen behind him at rallies, Trump impersonators and limousines driving around some places trying to lure the assassins out...
He is reducing his Chiquito de la Calzada [1] impersonations, looks much less smug and was adopting a lower profile, maybe fearing the post-midterms. His social truth posts are filled with images of him with radioactive fire balls in his hands and escaping to somewhere in planes and vehicles. Asking for a ride out of US Trump?
The risk of some disruptive event in US in the next 60 days is greater than ever in US history, and it does not need to be assisted by AI. NS (natural stupidity) is more than enough.
I don't see how this could be stopped unless every country agrees to follow international laws of war. Sadly this can't happen until a few old men and war criminals are arrested, or they die of old age.
These men are the real problem. They will block or delay the climate actions until is too late. They will remove anti famine measures just by spite, killing thousands. AI can't kill everyone because this people will do it much before. To save themselves from jail, or to earn one million more.
[1] https://en.wikipedia.org/wiki/Chiquito_de_la_Calzada
They are like lab scientists in a Hollywood thriller, watching a mutant organism rapidly evolving in front of their eyes.
They believe the sci-fi, rather than the reality. They've bought their own hype, and abandoned their critical thinking skills.
They're the guy saying to the computer "Say you're alive", and then being astonished when the computer says "I am alive."
Russia, for example, is already using AI targeting (Ukraine too but "with callback", ie. AI targeting asks permission to the commander and no permission? No targeting), China and Turkey are delivering AI targeting solutions to ... essentially anyone (the old Syrian government and Azerbeijan were both provided with them, and Azerbeijan probably used them against Armenian civilians in the blockade of Nagorno-Karabagh). Turkey provided them to Ukraine earlier in the war, then Russia flew a number of planes very close to Turkish borders and now they don't anymore.
The son-in-law of Turkey's prime minister, Özdemir Bayraktar, is among selling them.
As for diplomacy, the UN has already decided they will do nothing about it. And that means, btw, that the ICC will do nothing either.
https://news.un.org/en/story/2026/09/1168288
https://www.azernews.az/region/263412.html (Iran proudly declares they're doing this)
https://www.forbes.com/sites/pauliddon/2024/02/22/azerbaijan...
So you want this? Excellent! So what are you going to do? Keep in mind that serious amounts of violence will be required to accomplish your ban on AI drones, because the main strength of AI drones used to be that only side has them, but that time is passed. Turkey has no doubt built up their production, Russia obviously has (and is producing 100+ per day), and China, well we don't really know, but it's China, so ...
When two sides have them (like in Ukraine), it seems to be a numbers game. So you want to make any headway, you'll have to produce large amounts of AI targeting drones to accomplish this.
It may be the case that he was too idealistic, but those Three Laws seem foundational to discussion of what we do with AI.
[0] https://en.wikipedia.org/wiki/Three_Laws_of_Robotics