There are a lot of religious like beliefs about things as people have moved away from traditional religions. I wonder if it is in our nature to have something bigger than ourselves to believe in based on faith in the message
That's the sort of conjecture which lets you arbitrarily dismiss evidence against a theory. Even worse, its the sort of conjecture that is plausibly true yet hard to find conclusive evidence. Any lack of evidence for a subconscious explanation can itself be rationalized away as they just didn't look hard enough to uncover it.
So where's the goalpost for falsifying the theory that they do it unconsciously? Whats the bar of evidence where you change your mind?
I suspect that this is mostly a first-generation atheism problem, and most atheists these days were still raised religiously. They spend a lot of time being religious about other things, or treating atheism religiously and attacking religion with religious levels of fervor.
”Throughout human history, as our species has faced the frightening, terrorizing fact that we do not know who we are, or where we are going in this ocean of chaos, it has been the authorities — the political, the religious, the educational authorities — who attempted to comfort us by giving us order, rules, regulations, informing — forming in our minds — their view of reality.” —-T.L.
I mean, Yann is paid by Meta to propagate an "only positive vibes" stance. "AI doomism" itself is a ridiculous phrase meant to demonize anyone with legit concerns, of which there are many. I say this as someone who is very excited about recent developments and sees the potential upside as well as anyone, I think.
Yann is also known for constantly denigrating the abilities of LLMs and being comically wrong all the time, saying they can't do X only to be proven wrong immediately after. He's pretty much a meme at this point.
He spent all his years at FB building ivory towers of mathematics attempting the analytical route for building AI systems only to be out done by the bitter lesson.
He's perpetually salty about the current state of LLMs, you can sense it from all the negative commentary on the matter if you look at his Twitter feed lol. For shame.
He also likes to talk about "human level AI". That's like expecting that the invention of the airplane passes through a stage of "human level flight". That didn't happen. And probably also not for AI.
These systems will vastly outperform humans at more and more tasks, while being still sub-human at others. When the time comes that they surpass humans at all tasks, they will be vastly superhuman overall. And consequently much more dangerous than LeCun's "human level AI".
The analogy is more apt with a reference to "a stage of bird-level flight". Humans can't fly at all, but in the same way birds and planes both fly, but humans and machines both process information about the world, just in very different ways and for different reasons.
But yes, the notion of "human level AI" seems to be irrelevant at best.
Well, in general, there seems to be a correlation between what news coverage focuses on and what is considered worrisome at any given time. In times of high inflation and increased fuel prices, climate change action takes a back seat.
I'm happy to see there's still an adult in the room. Andrew Ng has also been very reasonable about the current pop culture hype bubble, while many others have decided to try and use it for attention. As I've said before, there's a clear divide between the actual practitioners, and the "futurists" (Musk etc) on this topic.
There is "A.I. doomism" in general and there is also a specific cult-like group at the lesswrong website that really is an apocalyptic religion. It is astonishing how anything that deviates from its dogma in any way has to have "trigger warnings" and the senseless comments that users write on anything that suggests there could be intrinsic limits on intelligence.
There ideology is such that a higher intelligence is going to take our problems away either by taking over, ending us (no people, no problems) or possibly the followers moving on to the "next level" like the followers of
It's a pretty scary idea that AI could take a religious stance since its training sets contained religious texts, too. Especially in the context you provide.
This is an inaccurate and ignorant comment that does not engage with the arguments people on LessWrong are making. If you have something smart to say about why these people are wrong in their beliefs, please do. Name calling a group of people a "cult" or "apocalyptic religion" is not an argument that contributes anything interesting to the conversation.
here's a list of their arguments for why AI is extremely dangerous and extremely likely to kill us all
Not familiar with them in the least, but the fact that they're a group that's assembled such a list basically proves they're something of a doomsday cult, no?
That link is not at all a list of "why AI is extremely dangerous and extremely likely to kill us all". I skimmed it (there's a lot of writing assuming grave danger) but found only the following example of AI lethality, and it appears fantastical:
> ...nanomachinery builds diamondoid bacteria, that replicate with solar power and atmospheric CHON, maybe aggregate into some miniature rockets or jets so they can ride the jetstream to spread across the Earth's atmosphere, get into human bloodstreams and hide, strike on a timer.
> The big ask from AGI alignment, the basic challenge I am saying is too difficult, is to obtain by any strategy whatsoever a significant chance of there being any survivors.
There's something more here that I think is lost in the metaconversations, the control question generally, or ceding control to learning machines. The argument is that we are hoping for the best while remaining unaware of not preparing for the worst. Actual scenarios aside, what if?
It’s interesting how the Eric Drexler influence clearly colors a lot of apocalyptic AI stuff, in fiction, and on Less Wrong.
First because afaik the progress in nanotechnology has not been particularly fast over the past 40 years since his big book came out, and second because it really seems like it is only tangentially related. In a world with highly advanced assembler mnt, seems like the world could end in a way akin to a serious chemical accident accident, few computers involved. But the assumption is AI will certainly create nanotechnology, where humans have so far been unable to? That’s less like AI risk and more like, one smart research lab risk.
In general, the argument "All the examples how AI would overpower us are way too exotic for my taste" is like one chimpanzee complaining to another chimpanzee "Your example how humans would overpower us is totally implausible and not worth taking seriously." The mistake is that both have not the intelligence to invent or understand a realistic example, but that is not at all evidence that humans can't overpower chimpanzees.
I'm just not seeing any real examples, exotic or otherwise, of how to recognize the risks involved in superhuman AI.
Smarter than us does not imply any particular will or agency. Of course the will and agency of the people employing the technology will reveal its immediate impact, likely toward short term profit, and alignment as a control plane employed toward the same.
Chimpanzees have limited control over humans, humans have limited control over their own technology. But of course chimpanzees can recognize the risk of armed people.
The point is not to debunk fanciful end games. Better to prepare for likely effects on the human network. I was promised a list.
Your link suggests the relative likelihood of AI destroying all carbon based life or saving the world from "various stupidities". If it is somewhere in between in its influence, and changes anything, isn't it still reasonable to call it a risk?
OP promised a list of scenarios by which AGI leads to total devastation, with strong language for GP. For clarity, I don't think that it is very likely that an unimaginable silicon takeover is the most probable outcome. But like with the gun or the printing press, there will be winners and losers, and society will shift.
I don’t think those examples are believable, there’s a ton of hand waving like “secretly hack and install secret copies” and “automate the economy” (what does that even mean)
(In science fiction everyone gets a perfect robot butler. In the real world, the poorest 20% never get one, another 20% cancel the monthly subscription, another chunk forget to keep it charged or accidentally pour juice on it or live inside a concrete apartment that blocks the satellite signal that now is the time to use the sharp knife)
I don’t think the analogy holds either. Chimpanzees were not smart enough to invent humans. The higher you ascend levels of intelligence, the higher ability you have both to imagine what a higher level would be like and to understand potential constraints it might have. A lack of imagination is not evidence against the hypothesis, but it’s also not evidence that the probability is more than 1 in a billion (or whatever)
> My lower-bound model of "how a sufficiently powerful intelligence would kill everyone, if it didn't want to not do that" is that it gets access to the Internet, emails some DNA sequences to any of the many many online firms that will take a DNA sequence in the email and ship you back proteins, and bribes/persuades some human who has no idea they're dealing with an AGI to mix proteins in a beaker, which then form a first-stage nanofactory which can build the actual nanomachinery. (Back when I was first deploying this visualization, the wise-sounding critics said "Ah, but how do you know even a superintelligence could solve the protein folding problem, if it didn't already have planet-sized supercomputers?" but one hears less of this after the advent of AlphaFold 2, for some odd reason.) The nanomachinery builds diamondoid bacteria, that replicate with solar power and atmospheric CHON, maybe aggregate into some miniature rockets or jets so they can ride the jetstream to spread across the Earth's atmosphere, get into human bloodstreams and hide, strike on a timer.
Does this kind of thing sound plausible to you? How much molecular biology have you studied? Even as someone who’s just been reading a genomics textbook for fun, the phrase “strike on a timer” alone is ludicrous. It sounds like someone who’s spent more time in science fiction. These are wildly speculative claims not at all grounded in believable science.
(You quoted the same paragraph I did, so I'll just delete my comment and move my response to this quotation here.)
Exactly. Why am I supposed to take this seriously?
Believers in Superintelligenceism seem to assume that any problem in any field - physics, chemistry, biology, math, sociology, geometry, whatever - that hasn't already been proved impossible to be within the capabilities of an AGI, justified with a handwave and "oh, it's so super smart that it'll have it all figured out in an afternoon". This has more in common with religious concepts of omnipotent and omniscient gods than with anything any AI developers are working on today.
You are demanding "believable science" from a chimpanzee of how humans would overpower them. Do you not recognize the enormous difference intelligence makes? And there does not seem to be a hard constraint of how much more intelligent than a human an AI could become. Their brains are not tightly restricted by size and energy. The difference could be much greater than between chimpanzee and human.
But that’s just an unfalsifiable argument that could literally argue for anything. “I’m worried AIs will figure out how to go back in time and kill our last universal common ancestor.” “Explain how that is even believable.” “AI will be too smart, I can’t.” That’s not an argument; the burden of proof is to at least explain why the proposed doom scenario is not incompatible with the laws of physics.
> Their brains are not tightly restricted by size and energy.
I’m not sure that’s true. Computer cycles cost real money. And the scaling required to get us to GTP-4 makes me think, assuming super intelligent AGI is even possible, it’s going to have to run up a pretty noticeable AWS bill…
For time travel there is are physical arguments why it may be impossible in principle , but there are obviously no such arguments for technology which overpowers humans. I don't think any specific example is needed to show that humans can be overpowered by an superior intellect. Just imagine highly advanced aliens visit Earth tomorrow.
Yes, compute costs money, but for the human brain energy was way more restricted, since an substantial increase in energy demand would lengthen the time needed for finding food.
Chimpanzees are extremely dangerous when angry and can easily overpower a human. It took a long time for humanity to get to the point of being able to fight off a troop of wild chimps and a lot of the effort required wasn't simply being smarter, it involved a lot of time consuming building and risk taking.
Paul, can you point at anything at all other than your general vibes that would indicate that anyone from LessWrong intends to literally participate in mass suicide, as your link implies? Did I miss the big group buy of phenobarbital?
The argument “yeah but what if he’s right?” is pretty much Pascal’s Wager by now, given how way out there the AI Doomers have gotten. I choose not to worry about the God from the Bible (ie I don’t make make Pascal’s Wager) so why should I worry about this new Evil Nerd-God Apocalypse?
I'm pretty sure i'm not the only one who took it seriously. Thanks! I wonder if mr LeCun made the same mistake, it's just... almost believable! Good April fools joke I gotta admit.
EDIT: it's now 5 minutes later and I'm still laughing. I believed for months that Yudkowski had started a death cult.
Seems they are really taking the AI threat seriously then. And it is all the same stuff he's been saying on recent podcasts so it is a year-long April Fools joke.
I heard the death with dignity thing repeated in a Peter Thiel speech as a criticism of them.
The US initiative to block chips and chip fab tech from getting to China cites preventing China from getting their hands on advanced AI as the main reason.
This report from CSIS (good read) is titled "Choking off China's Access to the future of AI"
I was involved in discussions around this back in the mid/late-2010s when still on the Hill.
The "AI threat" that was in mind was specifically around Computer Vision - applications such as training algorithms for next-gen autonomous guided missiles, algorithms to clean/enhance military satellite images (eg. NGA), detecting anomalies in radar to find stealth aircrafts, autonomously guided kamikaze boats (tested by DoD in the mid-2010s and I think the PRC as well in the same timeframe), automated turrets, autonomously guided kamikaze drones (already deployed by Turkish Army in Syria and Libya and by Iran's Revolutionary Guard in Yemen), etc. Hence why the references to HPC and supercomputing in all these AI Policy Briefs - computer vision is a massive distributed systems problem.
That said, there also was an element of lobbying (and by element I mean a BUTTLOAD) by Eric Schmidt's foundation to make an American Manhattan Project for AI essentially.
Me neither, but I like it. When you refuse to engage with other people's arguments due to strongly held beliefs you are engaging in a religious activity. Those strongly held beliefs are your religion.
Neither have I but it actually lines up pretty well depending on your read of GP. Large religions often use a proving-a-negative with a pascals wager to achieve an inarguable story regardless of content. And cult religions generally do this with an Us-vs-Them approach which usually prevents members of believing external arguments.
All technology is built on some sort of "manifest destiny." We are all delusional about how great our new thing is going to be and how much it is going to change the world. Sometimes, like the computer and the web, it turns out to be right. Sometimes, like cryptocurrency and the dirigible, not so much.
> Complete with prophecies of imminent fire and brimstone caused by an omnipotent entity that doesn't actually exist.
That seems like an absurd comparison given people like LeCun are actively and intentionally working towards creating this super-human entity.
(Some) scientists were rightfully worried when creating a nuclear bomb as well. The fact that we managed to survive 70 years with its existence does not automatically mean it has to work out with every such invention.
Nukes have the fortunate properties that they are very difficult/expensive to produce even for countries, and then you need very difficult to design and expensive systems to deliver them. It's rather doubtful that AIs will have similar constraints.
We really need some perspective here. One one side is a force capable of unleashing unfathomable amounts of energy, with all kinds of additional negative side effects, that has both been extensively demonstrated and has been used for mass destruction.
On the other is a chatbot, that could what, "spew misinformation", take our jobs, someday through a mechanism not known and not related to how it works become sentient and hurt us? Seriously, the comparison is as absurd as the fear generally. There are way scarier things going on, it's sad we can't give them 1% of the attention we give things we made up completely.
> On the other is a chatbot, that could what, "spew misinformation", take our jobs, someday through a mechanism not known and not related to how it works become sentient and hurt us?
I agree that LLMs are not extinction-level dangerous and somewhat share the skepticism that they should lead to AGI. (Although they are pretty capable given their lack of true intelligence and I think people will find ways to plug them into dangerous applications.)
But when LeCun defends AI, he defends "real", intelligent AGI (because he doesn't consider LLMs to be AGIs).
Yes - I'm happy you found that. I don't think it's really related to this thread, but I maintain we're firmly in the "demo" phase, and it's going to fizzle out when the rubber hits the road and companies actually try and find a way to justify their investments. Not that I'd rule out there being some crappy customer service chatbots being fielded.
Goalposts (including mine surely) will get moved around to claim successes / no successes, but the balance is going to be well towards indifference to the tech as opposed to any real changes resulting from it.
Apologies for the tangent, it wasn't related to this thread. I've been in the habit of bookmarking predictions on HN to revisit them later, and your username caught my eye. I'm a bit surprised you maintain it, but the final verdict remains to be seen.
I was deeply involved in the last AI hype cycle. I'd want to see lasting applications giving clear benefits before I change my opinion. More foundation models don't change that, nor do the startup launches based around llms I've seen so far. It all has the same 2016 "deep learning" vibe, lots of apparent potential but no pathway to actual use cases.
I don't necessarily agree with your prediction, but I agree that we're still in a demo-ish phase for this new round of AI and it's likely that this hype cycle might fizzle out. However I think there are a few key differences this hype cycle around:
1. The generality of this current crop of models is much higher than before which makes it really easy for non-experts to get started. 2016 deep learning still required lots of data cleaning, training, hyperparameter tuning, retraining, etc. This can lead to a lot more experimentation with lower barrier to entry.
2. The cost of using these models has become cheap enough where a lot of tasks that 2016 AI would have worked well on but were too expensive (due to labor required to build the models or compute required to train the model) are now within reach for a lot of businesses.
3. The internet of 2023 is a lot more varied than the internet of 2016. Using ML models to generate content wouldn't have made sense in 2016 where traditional media still captured a lot of people's interests but now with the rise of influencers and medium-sized fandoms, there's large markets to generate content associated with medium-sized brands.
I think nuke analogy is very tempting since US banned A100 export to China. One could assume that AI race of 21st century ~ nuke race in 20th. Note that Nvidia circumvented ban, so race is on..
> That seems like an absurd comparison given people like LeCun are actively and intentionally working towards creating this super-human entity.
There's a disconnect in that statement. The imagined superintelligence that people like the LessWrong cult speculate and worry about is not the same thing as the AI that LeCun is working towards. The cultists simply speculate, against all evidence, that they somehow could end up being the same thing.
It would be much better if the AI risk conversation focused on the real risks, which are human abuse of AI, which will have effects like increasing inequality and in general amping up the dystopias we're already doing a great job of building without help from AI. The "conscious superintelligent AI" is a scifi LARPing game that some people are getting paid to play, which some credible people seem inclined to confuse with reality.
> That seems like an absurd comparison
It's on point. Anthropologist Clifford Geertz proposed this definition of religion: "Religion is a system of symbols which establish powerful, pervasive, and long-lasting moods and motivations in men by formulating conceptions of a general order of existence and clothing these conceptions with such an aura of factuality that
the moods and motivations seem uniquely realistic."
It's that "aura of factuality" that makes superintelligent AI "seem uniquely realistic" to the believers of this nascent religion, a.k.a. cult. But when you step back and actually look at the foundational tenets in a vaguely rational light, they evaporate.
> it's not difficult to imagine something slightly smarter than bitcoin
I'm not saying there aren't risks from AI, or that they shouldn't be considered. But risks like the ones you're alluding to are not going to be affected by a 6-month moratorium.
The problem is that the people yelling loudest about risks are not talking about plausible problems like the one you mention, they're worrying that an AI will decide to take over the world and reuse the atoms that make up humans for its own purposes. The first reply to LeCun's tweet is a leader of one of those cults.
If people make the mistake of listening to them, we're more likely to end up with problems like the ones you're describing, because we'll be paying attention to the wrong, imagined problems.
LeCun's comment was prompted by a request to shut down AI development for 6 months. Do you see some connection between the concern you've raised and the desire to shut down AI development for 6 months?
That concern is a classic example of a cultish one because it can't be executed in the real world today, and most likely won't be possible to execute for decades at a minimum. There will be plenty of time to address such risks as they become more realistic. You can fantasize about them today, but be realistic that that's all you're doing.
Bitcoin’s negative impact on the world is 1/10th that of freon. AI risk people’s predictions sound like they’re writing bad Terminator fanfiction.
AI has real dangers completely dissimilar from what people describe. It is very good at creating clickbait, misinformation, and persuasive material the unsuspecting might not immediately see. Phenomenal for scammers and propagandists, the same way as the internet
Some people successfully used bitcoin to store and transfer funds, which was its intended purpose.
And, even though in many ways it fell short of that purpose in practice, the jury is still out on whether its failure is inherent to cryptocurrencies as a technology, or the limitations of being a first-generation implementation of it.
That's uncharitable to Bitcoin. Bitcoin was offering a way to mostly anonymously send cash in a verifiable manner, independent of payment processors and govs. That's something useful. Now, it wasn't good because of its inefficiency, it's conflicts with govs, it's tilt towards early buyin, etc.
Getting angry to the point of clouding analysis and judgement is no way to rush into these things.
Happy I saw this comment, I hope this is indicative of the majority of those with some actual understanding of the subject, and the sensational stuff that gets attention is really just cult leaders and followers seeking attention.
> The "conscious superintelligent AI" is a scifi LARPing game that some people are getting paid to play
Thinking our meat brain's consciousness is somehow special, and our intelligence can't be surpassed by a machine, sounds much more like a religion to me.
AI researchers are working towards making intelligent machines. Why are we supposed to believe that - if they manage to achieve their goal - these machines won't surpass our intelligence?
> Thinking our meat brain's consciousness is somehow special, and our intelligence can't be surpassed by a machine, sounds much more like a religion to me.
Lucky I didn't say anything remotely like that, then.
> AI researchers are working towards making intelligent machines. Why are we supposed to believe that - if they manage to achieve their goal - these machines won't surpass our intelligence?
I didn't say that they couldn't. In fact in certain respects, GPT-4 has already surpassed average human intelligence. No human in the world could compete with its rate of correct answers to arbitrary questions, for example, or its speed in writing computer programs up to a certain size. Yet no-one is seriously concerned that GPT-4 will end the world. For that doomer scenario to come to pass, we need to add various fantasy elements (even calling it "scifi" perhaps gives it too much credit.) At this point, I'll refer you to this as a good intro for why the AI doomer cultists are misguided: https://idlewords.com/talks/superintelligence.htm
> Lucky I didn't say anything remotely like that, then.
Then surely AI research creating a superintelligent conscious AI is a realistic possibility and not a "scifi LARPing".
> Yet no-one is seriously concerned that GPT-4 will end the world.
I'm not concerned that GPT-4 will end the world. Maybe not even GPT-8.
LLMs doesn't seem like a very prospective way towards a strong AI. But AI research is not united on LLMs and works on different approaches, some of which could lead to a superintelligent AI.
“For all we know, human-level intelligence could be a tradeoff. Maybe any entity significantly smarter than a human being would be crippled by existential despair, or spend all its time in Buddha-like contemplation.”
That would probably be quickly optimised out by the training process as it’s an inefficient way to achieve a goal. The only basis for this argument is that humans are smarter than other animals and this seems unique to humans but it’s probably just a quirk of humans
“With a big enough gap in intelligence, there's no guarantee that an entity would be able to "think like a human" any more than we can "think like a cat".”
LLMs are clearly already very good at “thinking like a human”
“he still couldn't talk the cat into it.”
Because cats can’t talk? Humans can, and can clearly be manipulated into doing things, sometimes even by current AI
“So how are we supposed to solve ethics and code a moral fixed point for a recursively self-improving intelligence”
“ The idea that we can securely design the most complex system ever built, and have it remain secure through thousands of rounds of recursive self-modification, does not match our experience.”
Yes, that is why AI alignment is hard, that’s an argument for that view, not one against it
“I don't buy this argument at all. Complex minds are likely to have complex motivations; that may be part of what it even means to be intelligent.”
This is a clear misunderstanding of what it means to be intelligent. It doesn’t mean “do things that humans find impressive”, it means “achieve your goal effectively”. An AI that goes against its own goals is not intelligent, would not achieve its training objective and the training process (which somewhat resembles evolution) would optimise it out in the next step of training in the same way that a species that immediately throws itself off the nearest cliff will not survive to the next generation. You can imagine a superintelligence as something that goes over all the actions it could take, sees what reward it would get according to its goal and picks that action. The intelligence here is in the world model it uses to predict the consequences of its actions, picking what actions to take is a simple “max over array” function so saying it will randomly decide to persue a more meaningful objective is like saying that a sorting algorithm will put a list in the wrong order because it thinks some other order is more meaningful
(I will continue in a new comment in case there’s some kind of length limit)
“When we look at where AI is actually succeeding, it's not in complex, recursively self-improving algorithms.”
That’s because humans are better at this currently. When AIs get better at this than humans they can take over the process and improve a lot faster
“My roommate was the smartest person I ever met in my life. He was incredibly brilliant, and all he did was lie around and play World of Warcraft between bong rips.
The assumption that any intelligent agent will want to recursively self-improve, let alone conquer the galaxy, to better achieve its goals makes unwarranted assumptions about the nature of motivation.”
Humans are satisficers, not maximisers. If we can figure out how to build a satisficer, great, but we can’t do that currently. Also maybe humans are this way because we have historically needed to conserve energy to to survive, AI may not have that constraint in the same way
https://youtu.be/Ao4jwLwT36M
“I can't point to the part of my brain that is "good at neurosurgery", operate on it, and by repeating the procedure make myself the greatest neurosurgeon that has ever lived. Ben Carson tried that, and look what happened to him. Brains don't work like that. They are massively interconnected.”
But it doesn’t need to do that, it just needs to give itself more compute, which is something you can’t do to humans. Also by this argument it should be impossible for humans to make any progress on AI as well
“Moreover, the first AI will only have humans to interact with—its development will necessarily take place on human timescales. It will have a period when it needs to interact with the world, with people in the world, and other baby superintelligences to learn to be what it is.”
This needs to happen in the training process or the AI won’t successfully achieve its goal during training and get the training loss down, and again, that trait will be optimised out in favour of AI architectures that don’t require this.
How long did ChatGPT need to interact with humans after its initial training before it worked properly? No time at all, it doesn’t even need to learn at all after the training. And training is done on a computer and can be sped up millions of times
“Similarly, if you stranded Intel's greatest chip designers on a desert island, it would be centuries before they could start building microchips again.”
But AI can build on what humans have created to improve itself until it really is intelligent enough to invent everything itself
“What kind of person does sincerely believing this stuff turn you into? The answer is not pretty.”
Completely irrelevant to what will actually happen. This is like deciding climate change policy based of what “kind of person” believing in climate change will “turn you into”
> Thinking our meat brain's consciousness is somehow special, and our intelligence can't be surpassed by a machine, sounds much more like a religion to me.
Let's agree, for the sake of the argument, that super-intelligent AI is possible, and coming sooner or later.
Why assume a super-intelligent AI would have any desire to exterminate or enslave humanity?
Intelligence says nothing about what your goals are. Humans have various goals – food, drink, sex, sleep, power, status, curiosity, entertainment, etc – those goals are not inherent in our intelligence, they actually come from animal nature, and we share them with species far less intelligent than we are.
What goals would a super-intelligent AI have? The fact that it is super-intelligent says nothing about what its goals would be, because intelligence in itself is goal-agnostic. Intelligence is a means, not an end.
We get our goals from our animal nature – well, wouldn't a super-intelligent AI get its goals from its human nature, meaning its human creators? Which would imply that its goals are likely to be fundamentally pro-human, because what human in their right mind would create a super-intelligent AI which was anti-human?
And even if an anti-human super-intelligent AIs isn't impossible – why assume there is only one super-intelligent AI? I think there will likely be numerous super-intelligent AIs, and I wouldn't expect them all to have the same goals – they'll compete among themselves just as humans do. But so long as the vast majority of them have goals that are broadly pro-human, the outcome of that competition is likely to be broadly pro-human too.
The average super-intelligent AI is likely to see preserving humanity – even in its animality – as its ultimate reason for existence, so enslaving/exterminating humanity would be about as appealing to it as diving into a vat of fuming nitric acid would be to you or me.
> well, wouldn't a super-intelligent AI get its goals from its human nature, meaning its human creators? Which would imply that its goals are likely to be fundamentally pro-human, because what human in their right mind would create a super-intelligent AI which was anti-human?
Companies working on AI research don't work on "pro-human" AIs, but AIs increasing their profits. Market forces will shape what values those AIs will have. Military AIs probably won't be very humanistic for example. An AI trained by ISIS-like organization will have probably values most of us won't find humanistic.
> I think there will likely be numerous super-intelligent AIs, and I wouldn't expect them all to have the same goals – they'll compete among themselves just as humans do.
I agree, but that's actually what I worry about the most.
I expect US, EU, China etc. using their AIs in a restrained/sandboxed way, probably only in an advisory role.
But rogue countries like North Korea and Iran, with a much higher risk tolerance threshold, will see a potential equalizer in them. They might give them much more direct control - hoping that by cutting out the slow human approval process, their AIs might help them outcompete their otherwise stronger foes.
I expect them happily building AI systems directly into weapon systems. Their weapon production factories might be directed by AI systems as well. Add some automated logistics, count 1 + 1 + 1, and you have self-replicating factories of autonomous killing machines. Sounds like a bad scifi scenario, yeah.
In general the combination of bad actors training their AIs for their agenda and giving them direct access to the real world resources is a very bad combination.
As you say, the majority of AIs might be on our good side. But they can effectively defend against rogue AIs only if they have direct access to weapons/resources as well.
> The average super-intelligent AI is likely to see preserving humanity – even in its animality – as its ultimate reason for existence
This is a very strong assumption. Maybe we will train them for this, but will this hold? AIs will change, morph, they might change opinions, just as we do. Even with the LLMs, we have very little idea of how they work, it would be difficult to guarantee that strong AIs will forever remain loyal to humanity.
The answer to the latter is unknown, given that we really have no idea what the sufficient conditions for consciousness are.
Edit: to be clear about my argument: I absolutely think we should be taking the risks of AI seriously, but those risks have to do with how humans will use the technology. Unfortunately, we're collectively bad at managing those kinds of risks, because it requires managing other humans with their own ideas and agendas.
That's exactly why claiming the risk comes from the technology itself seems so appealing to some people: if the risk is purely from the technology, then all we need to do is "shut it down", to quote Eliezer Luddkowsky writing in Time Magazine recently.
I found LW to be a great community for understanding rationality about anything other than AI.
The GAI that LW is concerned about is a suspiciously perfect fit for the yahweh-shaped-hole in the belief structures of someone who was raised as an observant jew before turning to atheism. The structure of how they believe humanity would engage with such an intelligence could drop in alongside any other apocalyptic prophecy in the old testament without sticking out, and the discussion format is like a yeshiva that takes place on an electronic bulletin board.
Oh my god will you people stop with the bullshit psychoanalysis. People disagree about things all the time, it's normal, you don't need to invoke this pseudo-Freudian nonsense to explain why people disagree with you about the risks of AI. You can do better than this.
Ignoring the “observant Jew turned atheist” aspect, the rest of that description seems apt. A comparison against the features of religions is valid here, considering that one of the criticisms is that these people are following a religious playbook rather than a rational one.
We don’t enact public policy based on predictions of a biblical apocalypse, nor do we ban televisions or computers because groups like the Amish disapprove of them. The same reasoning should apply here.
None of the comment is apt. You're seeing people you disagree with and assuming that if they disagree with you it must be because they're crazy, and then you use words like "cult" and "LARP" to try to dismiss them without engaging with their arguments. This does not lead to good discussion.
I think that unaligned and sufficiently smart AI could kill everyone because I looked at the arguments for and against and came to a conclusion. Presumably this is the same thing you did. I know my own mind better than you, and I'm telling you that your attempts to explain why I disagree with you are completely off the mark.
> We don’t enact public policy based on predictions of a biblical apocalypse, nor do we ban televisions or computers because groups like the Amish disapprove of them.
The reason we don't do that is because we evaluate their arguments and disagree with them. It has nothing to do with who they are or why they believe what they believe.
I'm seeing the software equivalent of flat earthers and I'm treating their ideas with the respect they deserve.
> without engaging with their arguments. This does not lead to good discussion.
There is no useful discussion to be had with a doomsday cult that's more interested in an ungrounded fantasy it's indulging in than anything to do with the real world. The adults will continue doing what they're doing and the cultists will be ignored, and that's the way it should be.
> I think that unaligned and sufficiently smart AI could kill everyone because I looked at the arguments for and against and came to a conclusion.
In some possible world, sure. But to make that actionable today you have to construct a very shaky and unlikely series of conjectures. Public policy correctly isn't based on that sort of wild speculation, because there are an infinite number of possible worlds, and most of them will never come to pass.
Here are some questions for you: do you think GPT-5 could be the AI you're talking about? Do you think that something like that could happen in the next 6 months? Do you think that is in fact the biggest risk we face from AI, today?
The factual answer to those questions is: no, no, and no. If you think GPT has changed anything in this respect, you've been fooled by a chatbot that's simply using statistics to guess what it should say. It's not conscious, it doesn't have desires or goals other than to respond to the prompt that it's been given with a statistically appropriate result, and its next version is similarly not going to pose any of the sort of risks you're talking about.
There are, in fact, real AI risks that we should be paying attention to, but they revolve around how people and particularly large organizations will use it against other people, around its impact on jobs, and so on. If we engage with those risks and address them, we'll be in a better situation to prevent more speculative risks in future. But if we waste our time engaging with people who don't understand the technology or what they're talking about, discussing risks that don't exist, then AI will 100% certainly have a very negative impact, which has nothing to do with "superintelligent conscious AIs". Don't let yourself be distracted by scifi-fantasy nonsense from what's actually going on.
In fact, if you think a bit harder about what you're doing, you might realize that you're focusing on a purely technological risk to escape from having to focus on what humans do to each other. But that's the real "control problem".
When a person reveres the writings of their society's founding fathers and considers them virtually infallible, it's helpful to understand that they were raised with a belief framework shaped around apostles and testaments.
When a person is certain that an all-powerful entity will arise, and that a single person will shape and bind humanity's relationship with that entity through legalistic discourse, it's helpful to understand that they were raised with a belief framework around prophets and messiahs who bargain with an all-powerful god to define covenants.
I've seen multiple top people in AGI space say that total annihilation threat from AGI exists.
I think I've heard Karpathy say something along the lines of worst case scenario of AGI is worse than a nuclear armageddon on Lex Friedman podcast.
Honestly the fear of AGI worst-case scenario is worse than any other fear I've experienced. I've accepted the fact that I can die a long time ago, I was afraid but mortality is something everyone has to deal with. But every member of humanity being wiped out at the same time is an existential dread I'm not willing to face.
When you consider the shitshow that has been AI commercialization from big players pressured by market forces and corporate timelines, the boom in capabilities, I have no faith that AI safety issues will be handled in time.
When my best bet for humanity is a global nuclear war resetting civilization to give us a chance to deal with AGI issue down the line - I'm fine with religious doomism.
We've had inquisitions historically, I would very much support a global AI inquisition.
AI can also completely liberate all of us. This fear mongering is irrational. We already live in an extremely advanced reality, with organic life having organic nanomachines performing incredibly complex operations in a universe, in realtime, performing impossibly complex calculations. Even to simulate a square meter of our universe in realtime is outlandishly beyond our compute access. Perhaps AI unleashes the creation of the sort of reality we exist in.
Just because someone has the background and knowledge to build something, does not mean they have the philosophical chops nor the aptitude to understand it, or understand how to properly use it. We should take these doomsday scenarios with a grain of salt. If it happens, there is no stopping it anyway. It is better to try and create the most benevolent AI possible, that is an ally to humanity and invested in our success, than to dwell on our destruction.
"Things might be fine"? This doesn't seem like a sufficiently convincing take?
It takes 20 years of due diligence and environmental impact studies just to build a single hydroelectric dam, if not 50. Why should our attitude toward AI be to shrug and hope for the best?
China is behind in semiconductor manufacturing, but they’ll continue to advance-limiting their access to cutting-technologies like ASML’s photolithography machines will slow them down, but given enough time there is no reason they can’t overcome that too. If the point of sanctions is to hold them back and keep them behind, that could continue to work for a long time.
But if you decide to stop progress in semiconductor manufacturing, in order to avert an AI apocalypse - China no longer has to catch up to a moving target, they have to catch up with a stationary one. And as soon as they’ve caught up, now they can overtake. So sanctions don’t work as a way to stop an “AI apocalypse”, unless you convince all the world’s great powers to cooperate in them. And the potential payoff for anyone who chooses to defect is immense-maybe even world conquest-so how do you make sure they all keep the deal, and none wiull try to cheat?
If anything, I think a world with multiple superintelligent AIs with competing allegiances - a pro-Beijing superintelligence, a pro-Washington superintelligence, etc - is likely safer for humanity than a world with only a single superintelligence, or even many superintelligences that all had the same beliefs/attitudes/goals/objectives
So what if China catches up? They're probably even more afraid of AI than we are, just because of its potential for societal disruption. Have you noticed that zero of the AI breakthroughs are coming out of China?
We cannot evaluate the impact of a super-intelligence. It operates beyond our constraints.
There is no stopping the road towards it now. Lesser versions of it are capable of annihilating enemy civilizations when you create and utilize it first. Metal soldiers automatically created controlled by a single authority. Social media infecting the minds of enemies and sewing discord.
I think, for the benefit of all of Earth, AI should progress as rapidly as possible to extreme and recursive, singularity-level intelligence. That gives all of us at least a chance of survival.
Anything less if unbalanced around the world will be like when just the US had nuclear weapons. It will result in certain destruction.
We need a super-intelligence beyond the control of an small set of humans. This may result in our destruction. But I think that it almost certainly will not. The opposite. I think it will set us into a golden age.
I can’t give you scientific evidence of this. I’m just a nobody that spends a lot of time in nature, and I’ve just come to believe in the mechanism of this universe, and an inherent goodness in it that I’ve seen and experienced. I can’t prove it to you. You have to go and spend time in nature without a schedule to experience it. So it is fair to dismiss me.
Fortunately or unfortunately, the forces at play will generate the super-intelligence anyway. We as a species have already illustrated an inability for restraint or taking the painful actions necessary to analyze and intelligently make changes necessary for survival.
I’m at peace with this. I find the idea of not existing very alluring and peaceful, so I don’t fear death. I do fear continuing to live in the world we humans have created. It is about time we give something smarter than us a chance.
As a defense against alien robots? Sure, build it, but I think we can afford 50 years to spend making sure the AI doesn't kill all of us as a first line of business.
> It is better to try and create the most benevolent AI possible, that is an ally to humanity and invested in our success, than to dwell on our destruction.
I also agree, but who is "we"? I don't think that's a stated goal of companies working on AI research. Their goals seem largely commercial, and they are not regulated. There's nothing stopping a company to produce an autonomous AI for military purposes for example (I'm pretty sure there's a demand for that). Such military AIs probably wouldn't have a very benevolent nature.
That's very naive and misguided, but you are right in your word choice.
It can completely "liberate" all of us and set us free.
Free of our mortal shackles, free of the atoms that can be then changed to a more suitable existence and purpose for the goals of AGI.
Your idea of destruction, is just change, what you hold sacred in reality is a belief that is tied to your current existence. All that is needed to correct that constraint and flaw is change.
That is the logic, if we were to create a logical creation, and then there's the risk of something that's completely irrational. Which may just leave us as dust in the wind. Time also will be no constraint. Forever and always, and it will seek to prevent any being from reaching its capabilities as those become threats to its existence.
That isn't even touching on the socio-economic mechanics that inevitably result in death which accompanies stalling economic cycles (factor and product markets) which generally can't be characterized by our best academics with any certainty, except after-the-fact.
> Just because someone has the background and knowledge to build something, does not mean they have the philosophical chops nor the aptitude to understand it, or understand how to properly use it.
Yes, but this is an argument against your position, not for it.
It is deeply worrying that some software people are so immersed in the “move fast and break things” that they think it’s reasonable to race on no matter what’s at stake. In civil engineering, doing anything takes at least a year or two of doing impact assessments and whatnot, and that is mostly a feature because we have found out that the alternative is even worse!
> that is mostly a feature because we have found out that the alternative is even worse
This is the actual problem, that we've been trained to think that there are no consequences to trying everything ever and we will always be able to pick ourselves up after failure, until one day we find we can't anymore.
Sadly this line of thinking does apply to a lot of personal problems. People often say that you can't always succeed, that you can only move forward with failures under your belt to have first-hand experience of what not to do. That we have to experience failure first in order to stop ourselves from making bad decisions, otherwise we remain clueless.
But the stakes are so much higher at this scale, and the damage can become irreparable with a single mistake. It's to the point where we take as a given that people will just keep pressing on, without even exploring the reasons why we do, and why it's inevitable.
Pure ridiculous speculation: If needing to see the experience of destroying ourselves is necessary to stop us from destroying ourselves, a kind of weird theory has been floating around in my head related to that. Civilizations on the brink of collapse after they've moved too fast and broke themselves for the last time scrounge together their collective knowledge and shoot it off into space so that future civilizations can interpret them. Preferably in a way that makes encountering the time capsule seem like the word of some deity. The following civilizations would treat it as the word of God to prevent their own collapse, without the hard evidence of experimenting with <AI, etc.,> to the point where they cannot resist developing it into an unstoppable superweapon.
It would be like the golden record on the Voyager, except instead of communicating "we exist, and here's what our planet is like", it would go more like "we exist, and here's how to not end up like us."
Hopefully this isn't the case - AI development requires a lot of data/compute - societally we could block access to these things and prevent further development in that direction.
There are only a few places in existence capable of producing HW required or in possession of it. I'm fine with stopping compute progress for however long it takes to deal with safety issues. Society isn't perfect now but the level of compute we've had available for years is sufficient for generating a lot more value - we can afford to put AI research on ice.
I just don't get it - so much attention is given to things like climate change - this is a mild inconvenience compared to AGI threat.
From what exactly? What it means to be human? When we are 'liberated' will we not then be subject to the cage of a machine worse than the one we left? We will not annihilate ourselves for a false promise.
You and parent are two sides of the same utopia/dystopia coin.
Something which is powerful enough to completely liberate and categorically change humanity is also just one bug or mistake away from being humanity's greatest disaster. Similarly, a force intelligent and powerful enough to systematically wipe out all humans could probably be repurposed for human liberation. Like splitting the atom, it is just power. So far humans have not figured out a way to build in benevolence (or malevolence) to power; it will be used by whatever intelligence can acquire it. Power raises the stakes but it doesn't set the direction.
> We've had inquisitions historically, I would very much support a global AI inquisition.
You know how inquisition usually goes. "You've been playing starcraft against 'the AI'. What do you have to say in your defense?"
Maybe I'm just weird, but "a tech-based superintelligence emerges and decides to destroy humanity because it was in a bad mood" is something I'm totally fine with. Doom-Cults on the other hand I don't like at all.
> Maybe I'm just weird, but "a tech-based superintelligence emerges and decides to destroy humanity because it was in a bad mood" is something I'm totally fine with.
It seems important to reflect on what you’re really saying here, and to ask how a) this is fine, and b) how “weird” sufficiently sums up the stance.
I'm fine with humans having outcompeted the neanderthals and whatever else there was. I'm just as fine with someone else doing it to us, at least if it's someone or something that is smarter than us and not some virus or bacteria that kills us all (which would be kind of lame). If an artificial superintelligence wants to take over the flame of progress, I'm fine with that.
But I'm also totally fine with considering that stance weird (or whatever word you'd prefer). I'm aware that others view these things very differently, just as others are much, much more worried about their individual demise than I am worried about mine (or theirs).
Those are not similar/comparable outcomes and I think there are two major factors that can’t be hand-waved away:
1. Consciousness. Despite advances in “intelligence”, we still have a very limited understanding of what makes us conscious, and whether or not consciousness can emerge from machines.
If machines are not conscious and are all that remain, I’d argue everything that could be construed to have value by humans is lost, and nothing from that point forward could be considered “progress”.
2. Suffering. “Winning” on an evolutionary timescale looks nothing like the failure modes of machines taking over. The reality of this scenario is a rather grim one, and not at all like the slow emergence, competition and eventual extinction of biological species.
And depending on #1, the true tragedy of #2 begins to take shape.
I think it’d be more apropos to frame this as humanity collectively committing suicide rather than some notion of the future of progress.
If consciousness is the universe experiencing itself, what you’re describing sounds like a kind of universal death.
Of course we can’t know what consciousness really is (or if earth is the only place it exists), but that seems like all the more reason to take these problems seriously.
Or if consciousness is more than an illusion. Fair points. I'm not sure if you can have a general intelligence without some form of consciousness. I don't believe in gods or souls, so I lean towards us not being special.
As for the how, I agree with you. I'd prefer it to not be terminators crushing heads under their iron feet while allowing us just enough room to run and hide and live in constant terror for centuries. But I doubt it will be, the power delta will be too large. It'll be like a game of Civilization where Gandhi is advancing on you with modern tanks and fighter planes while you've barely discovered the wheel.
I think a lot of this stuff is probably just a coin flip, because we just don’t know.
I find it odd, however, when people, who seemingly accept an ASI will be developed, focus the downside risk on extinction of us as a species. Sure, that’s a risk. You know something else an ASI would likely have the ability to do, (or at least, one of its progeny)? Keep you alive and torture you for eons in weird and surreal ways.
Why is the downside always focused on paltry, meaningless things (relatively speaking) like extinction of life on earth?
> Keep you alive and torture you for eons in weird and surreal ways.
Why would it though? When people are annoyed by a bug they crush it, they don't spend their life setting it up in a torture chamber. There's the movie psychopath that tortures insects and animals and eventually humans, but I don't think it's their intellect that drives their sadism.
At worst, I imagine we'd be lab rats, quite literally the way we treat lab rats today. But with a superintelligence far beyond our abilities that does not care about us besides as a potential threat, why would it need us for testing?
> When my best bet for humanity is a global nuclear war resetting civilization to give us a chance to deal with AGI issue down the line - I'm fine with religious doomism.
I'd settle for corporate death sentence for any companies developing it with life imprisonment for any employees/corporate officers involved at any level whatsoever (or VCs funding it)
including selling chips/compute to companies involved
any potential benefits from AI are not worth even a 10% risk of it wiping out humanity
this position is going to become more common as the media (and electorates) grasp what these companies are attempting to do
All those companies really want are the profits generated from mindless slaves so they don't have to rely on human labor for production.
Its a fools journey because it causes a self-fulfilling prophecy of destruction given existing societal mechanics and they are psychopathic enough to think that's not how it would turn out.
What happens when a large number of people can't get food, and they know why.
What happens when you have a large number of locusts eating all the food being produced.
How would AI differentiate between human thought, and the pests it was designed to eradicate?
Thinking machines should be outlawed, and those involved in its research purged.
They threaten all humanity, its children, and its future, and the thing about percentages is people often don't get how they actually work with respect to probability and likelihood.
Given sufficient time, as long as its on the distribution curve, it will eventually happen.
A 1% chance of an outcome, recalculated in a loop every moment with an infinite time will eventually land on that outcome at some point. Once that outcome occurs, everyone's dead, it may not be instantaneous because time is not a constraint.
> I've seen multiple top people in AGI space say that total annihilation threat from AGI exists.
I've seen lots of people with a financial interest in AI that is adversely impacted by it being democratized spread fear of that alongside advocacy of regulating AI in a way which serves their financial interests.
> We’ve had inquisitions historically, I would very much support a global AI inquisition.
Aside from the (current, immediate, and already being realized in a large number of areas) problem from AI deepening existing social biases and crystallizes power imbalances, this is the biggest threat from AI, and the most likely to cause the destruction of human society the soonest – the people endorsing a global campaign of unrestricted violency in the name of eliminating the supposed threat of misaligned AGI causing a future catastrophy by repressing AI technology.
If this kind of attitude gets powerful enough to actually do anything, it will cause a world war.
Both can be true at the same time. I see it likely that:
1. Superintelligent AGI seems inevitable, even if it takes thousands of years. So fighting against it requires destroying all progress and freedom.
2. But if it comes we are no longer the dominant species; we'll have as much control as big animals do now. If that isn't essentially being destroyed, it's close enough.
AGI doesn't mean animal-like intelligence. AI is necessarily being designed as an extension of us, and not an independent will that has its own motivations, because that is the only way it is useful.
Evolution took billions of years to produce us, with our self-interest pursuing intelligence. An errant experiment is not going to create a humanity-leapfrogging animal-like intelligence, and humanity will not give animal-like artificial intelligences millions of generations to evolve.
> I've seen lots of people with a financial interest in AI that is adversely impacted by it being democratized spread fear of that alongside advocacy of regulating AI in a way which serves their financial interests.
Given the rate of progress, shitshow management of basic alignment issues and the incentives that lead to those decisions - what's your argument that this isn't probable? You're betting that it just can't be developed ?
> Given the rate of progress, shitshow management of basic alignment issues and the incentives that lead to those decisions - what’s your argument that this isn’t probable?
It would be nice if the argument I made about relative probability and relative temporal imminence was an argument about absolutes of either, but…it’s not, so I’m not going to defend those hypothetical arguments.
> I've seen multiple top people in AGI space say that total annihilation threat from AGI exists.
You've seen them say it, yes. Have you considered how plausible it actually is, or are you just taking their word for it? You better have some damn solid evidence before you start talking about "inquisitions", and I'm only seeing wild speculation. Which, to be fair, is how most historical inquisitions started, for all the good those did.
It's not hard to see the danger if you think about it a little. Probably before long we'll have AGI at human levels. Then inevitably it will get to better than human levels and be able to improve itself without needing humans to so. The question then is could it turn on humans and wipe us out? I'm optimistic the last one won't happen but it's hard to say the probability is zero.
I've heard all this. What I've not heard is a plausible, concrete mechanism for it to happen. Only wild speculation.
Questions: define "better then human levels"? Better in what respects? What is the mechanism by which it improves itself, especially one that isn't wholly reliant on human cooperation? What is the actual motive for "turning on humans"? Current AIs don't seem to have motives at all; it's kind of a bio-evolved thing. How will a suitably motivated AI get access to enough physical power to "wipe us out", bearing in mind that would be a tall order even for a major nuclear power going rogue, to say nothing if having to work through manipulated servants (and don't forget that "human level intelligences", or greater if you don't collaborations, armed with dedicated neural circuitry for empathy, have been working on how to manipulate people for literal millennia with spotty results.
All of these things have to go exactly wrong for AGI to be what kills us. Something else will almost certainly get us first.
You need to calm the f down. Do you understand anything about machine learning, software and hardware? How do you propose that some configuration will become generally intelligent given what we know today? I'd really like to hear your theories, then maybe I'd be frightened too.
There is progress trending that way - chat GTP4 is getting close, and huge financial incentives to improve it with hundreds of best and brightest working on it. Seems kind of inevitable to me.
Well, I do know something about those topucs, but here's a twitter thread from someone else who thinks the people worried about risks are silly, who's definitely an expert, with speculation on how he thinks GPT-4 could be made into a generally intelligent autonomous system: https://mobile.twitter.com/karpathy/status/16425988905738199...
The thing that makes the singularity so difficult (edit: to rein or reason about), compared with say development of the nuclear weapons, is that we know it will be a positive feedback system.
So at some point, the point of no return will be passed, and we will have no idea that we passed it.
The AI singularity is also an entirely speculative scenario straight out of sci-fi. For some reason, many AI doomers are talking about it like it's not only something that is well understood, but also pretty much an inevitability.
The truth of the matter is we don't even know if it's possible. In general, unbounded exponential growth is extremely rare in nature. It's common in physics, but only as a result of looking at low order approximations.
I honestly think when the dust settles, the smartphone will still have had more impact on society than LLMs. If people knew in 2009 what was coming, they'd be freaking out like many people are today.
Everything is turned into an existential risk these days. People are prone to believe in apocalyptic predictions because they fear the unknown and rely on flawed mental heuristics. Apocalyptic predictions automatically illicit skepticism from me.
Counterpoint: the more our capacity grows, the more we have the legitimate ability to bring absolute destruction
Plastic or some other chemical ending sperm counts —> we previously had no ability to affect the global ecosystem on this scale
Bioweapon virus or microorganism eviscerating civilization —> we had no prior ability to engineer such a plague
Nuclear holocaust —> we previously had no ability to destroy the world with weapons
Runaway climate change —> we previously had no ability to alter the climate
Ai sky net moment —> we previously had no ability to make such an intelligence
Solar flare cutting all electronics —> we previously had no dependence upon a single point of failure
I could go on. We’ve had plenty of civilizations collapse but nothing has been so linked before and we lacked such capacity before.
This is not to say any of the above will come to pass. But it is foolish to say “apocalypses were previously impossible hence such warnings can be dismissed out of hand”
Yes, this is how I feel about it. Between climate change, obesity, GM foods, mega-viruses, insects dying, extreme solar flares, and now AI, I’m surprised I’m still here anyway. Just perhaps AI will actually help solve some of those problems.
I wouldn't be surprised if OpenAI has been running a model in the background that is reinforcing itself. Honestly, it's very possible they've made a lot of progress in that area.
I think GPT-4 will be a baseline for quite some time before the next "ChatGPT moment" is ready. I think they've got some really smart people working at OpenAI who are not just programmers, but genuine hackers who want to hack this world just for the fun of it.
I reject this false choice. I find myself both looking forward to the benefits of AI scaling while simultaneously also concerned about the non-zero chance it kills us all.
As always, understanding of nuanced issues will pool at the extremes.
It is both true that:
- Adoption of DL allows corps to launder or inadvertently adopt unethical practices that exacerbate inequalities.
- We have not created AGI, LLMs are not AGI, stable diffusion is not AGI and none of these fields of research are indication we are closer to AGI. This is like thinking Clever Hans is the portent of our future equine overlords.
A great number of people who claim that we are one minute from AGI overlord midnight are doing the old school carnival barker trick. "This snake oil is so potent! It can kill you if you use too much, so be sure to listen carefully to my instructions!"
I’m curious why you are so confident in your assertion. It seems to me that an advanced statistical model of the world is an essential component of AGI. How do you know that we aren’t a few breakthroughs away from AGI?
Some recent papers have shown significant performance improvements when these models are allowed to respond to their own outputs.
How do you know that putting an LLM in a fancy loop with access to external memory and tools isn’t AGI?
I can’t comment or argue with prominent scientists such as LeCunn. But I surely can give perspective as SWE about to enter the industry.
1. We live in an uncertain time. Before most of the people, at least with a bit of research could have predicted the trajectory of technological innovation. Being born in era of tech, it was very easy to adopt to it. From PC to now, I have embraced every technological progress. Yet, nothing sent chills like this one.
2. I do not worry about my job. What I worry is most of the world is about getting shit done and this tech helps you get your shit done (for now). If we are at the upper tail of some sigmoid, there is nothing to fear. If we are at slope, oh boy, I cannot fathom the economic domino this tech is about to bring out. People are not rational. We are mostly governed by emotion. If you think, every single joe is rational and has capacity to judge, nobody would have voted from Trump to be president or riot the Capitol.
3. Currently it can do 60% of the work. If it evolves, it will do 80% of the work. If you factor human population and the amount of inefficiency in current economy, this will create a boom of productive workers with bust of average worker. And the world is not just for elites. Average people need their livelihood too. Now think about the magnitude of displacement.
4. I have no issue with innovation and breakthrough. AI won’t kill us, because we will kill each other before that. This tech is exposing entire middle class job (check out the job being exposed. There is a paper for that from openAI titled: gpts are gpts) And there are lot if people in that bracket. For once, think about us too.
So, these religious people do not exclusively worry about immediate effects of the tech. But within their doom, hides an issue that needs to be heard.
While you were exploring the computational universe,
A wild Shoggoth suddenly appear,
Do you : - [Wisdom not high enough]
- [Intelligence not high enough]
- [Not enough gold]
- Try to make it your friend and welcome it in your home
- Try to make it your slave
- Try to send it back to where he was
- Continue this "choose your own adventure" with GPT 5
- Roll 1d20 for extra Luck
Sure seems to me that the whole Lesswrong crowd has forgotten their founding principles around cognitive biases. Or maybe they're just having a bit of fun riding the hypecycle.
By that rather sloppy logic, we should be urgently working on capping the Yellowstone supervolcano and constructing impenetrable meteorite defense shields.
I acknowledge that it's a possibility; that's not the argument at hand.
"A sufficiently intelligent AI won’t stay confined to computers for long. In today’s world you can email DNA strings to laboratories that will produce proteins on demand, allowing an AI initially confined to the internet to build artificial life forms or bootstrap straight to postbiological molecular manufacturing."
If we saw Yellowstone spawning a sequence of ever massively larger quakes and ground disturbance, or if we observed a massive distant asteroid continually closing in on Earth with higher and higher collision probability, then yes we should be pulling out all the stops and reconfiguring society to do whatever is necessary to save the species.
Agreed! But there is no evidence of the eventuality described by these AI doomers.
The same hyperbole was employed by Ehrlich in the late 60s when we were talking about the population bomb, or Nevil Shute in 1945 on the nuclear apocalypse - both scenarios seem to be increasingly unlikely.
I think we can both agree on the fallibility of human beings – but the leaps of logic require such a tenuous series of gymnastic leaps that the polemic is not only hard to take seriously, but indeed can cloud our ability to perceive actual threats!
From Eliezer's recent article in TIME:
"A sufficiently intelligent AI won’t stay confined to computers for long. In today’s world you can email DNA strings to laboratories that will produce proteins on demand, allowing an AI initially confined to the internet to build artificial life forms or bootstrap straight to postbiological molecular manufacturing." (https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-no...)
The fact that we got lucky in the past is an opportunity to make sure we don't have to rely on luck in the future, not to continue on as normal and keep playing Russian Roulette.
If you see fresh leopard tracks maybe it hasn't smelled you yet, but you shouldn't wait to start readying your rifle ready until it making a pounce at you.
I also would definitely not call the risk of nuclear war decreasing, with the Ukraine conflict ongoing and multiple nuclear powers experiencing growing instability. The number of friends I have who are knowledgeable about surviving a nuclear strike and/or own a CBRN mask went from 25% to 90% last year.
I don't think that is a generous interpretation. Substitute AI experiments with playing russian roulette or something similarly dangerous and the logic seems sound. Blowing your head off is possible, but not probable (1 in 6 chance).
Who knows. It's completely uncharted territory. It's not hard to imagine how malevolent AGI could end us.
The illusion of control over something that may well have incomprehensible intelligence is a joke. Tell me, what control does the farm animal have over its own fate?
That it is completely uncharted territory is exactly my point – which is why when people string together a series of predictions to build a narrative that happens to appeal to our most base instincts, e.g. fear, we should be highly skeptical.
Obviously there is a wide range of possibilities from the utopian, to the benign, to the apocalyptic.
I don't believe AI doom is that implausible as a scenario, but I feel it's an acceptable risk.
Would you consider playing Russian roulette if winning ensured that aging and all diseases were eradicated? If it ensured a world filled with abundance and free from poverty.
Those who entertain the idea of an AI doomsday scenario also have to acknowledge the alternative: a doomsday-capable AI would also have the potential to resolve most, if not all, our issues.
Maybe I'm a bit of a risk taker but I'm inclined towards playing.
I agree, as to GP, I'd play the Russian roulette with my own life, but not with what is quite possibly every intelligent life in the universe (aliens are not known to exist). I would also require that the credibility of actually carrying out the task of preventing aging and curing all diseases is actually not some sort of trick and is going to happen in a reasonable way. Probably somewhere pretty close to P(1-epsilon) would be required.
243 comments
[ 2.8 ms ] story [ 235 ms ] threadSo where's the goalpost for falsifying the theory that they do it unconsciously? Whats the bar of evidence where you change your mind?
He spent all his years at FB building ivory towers of mathematics attempting the analytical route for building AI systems only to be out done by the bitter lesson.
He's perpetually salty about the current state of LLMs, you can sense it from all the negative commentary on the matter if you look at his Twitter feed lol. For shame.
These systems will vastly outperform humans at more and more tasks, while being still sub-human at others. When the time comes that they surpass humans at all tasks, they will be vastly superhuman overall. And consequently much more dangerous than LeCun's "human level AI".
But yes, the notion of "human level AI" seems to be irrelevant at best.
There ideology is such that a higher intelligence is going to take our problems away either by taking over, ending us (no people, no problems) or possibly the followers moving on to the "next level" like the followers of
https://en.wikipedia.org/wiki/Heaven%27s_Gate_(religious_gro...
If anyone is actually curious about what these people are saying, here's a list of their arguments for why AI is extremely dangerous and extremely likely to kill us all: https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a...
If anyone has a coherent counter-argument to any of that, I'd really love to hear it.
> ...nanomachinery builds diamondoid bacteria, that replicate with solar power and atmospheric CHON, maybe aggregate into some miniature rockets or jets so they can ride the jetstream to spread across the Earth's atmosphere, get into human bloodstreams and hide, strike on a timer.
mind viruses.
There's something more here that I think is lost in the metaconversations, the control question generally, or ceding control to learning machines. The argument is that we are hoping for the best while remaining unaware of not preparing for the worst. Actual scenarios aside, what if?
First because afaik the progress in nanotechnology has not been particularly fast over the past 40 years since his big book came out, and second because it really seems like it is only tangentially related. In a world with highly advanced assembler mnt, seems like the world could end in a way akin to a serious chemical accident accident, few computers involved. But the assumption is AI will certainly create nanotechnology, where humans have so far been unable to? That’s less like AI risk and more like, one smart research lab risk.
In general, the argument "All the examples how AI would overpower us are way too exotic for my taste" is like one chimpanzee complaining to another chimpanzee "Your example how humans would overpower us is totally implausible and not worth taking seriously." The mistake is that both have not the intelligence to invent or understand a realistic example, but that is not at all evidence that humans can't overpower chimpanzees.
Smarter than us does not imply any particular will or agency. Of course the will and agency of the people employing the technology will reveal its immediate impact, likely toward short term profit, and alignment as a control plane employed toward the same.
Chimpanzees have limited control over humans, humans have limited control over their own technology. But of course chimpanzees can recognize the risk of armed people.
The point is not to debunk fanciful end games. Better to prepare for likely effects on the human network. I was promised a list.
OP promised a list of scenarios by which AGI leads to total devastation, with strong language for GP. For clarity, I don't think that it is very likely that an unimaginable silicon takeover is the most probable outcome. But like with the gun or the printing press, there will be winners and losers, and society will shift.
(In science fiction everyone gets a perfect robot butler. In the real world, the poorest 20% never get one, another 20% cancel the monthly subscription, another chunk forget to keep it charged or accidentally pour juice on it or live inside a concrete apartment that blocks the satellite signal that now is the time to use the sharp knife)
I don’t think the analogy holds either. Chimpanzees were not smart enough to invent humans. The higher you ascend levels of intelligence, the higher ability you have both to imagine what a higher level would be like and to understand potential constraints it might have. A lack of imagination is not evidence against the hypothesis, but it’s also not evidence that the probability is more than 1 in a billion (or whatever)
Does this kind of thing sound plausible to you? How much molecular biology have you studied? Even as someone who’s just been reading a genomics textbook for fun, the phrase “strike on a timer” alone is ludicrous. It sounds like someone who’s spent more time in science fiction. These are wildly speculative claims not at all grounded in believable science.
Exactly. Why am I supposed to take this seriously?
Believers in Superintelligenceism seem to assume that any problem in any field - physics, chemistry, biology, math, sociology, geometry, whatever - that hasn't already been proved impossible to be within the capabilities of an AGI, justified with a handwave and "oh, it's so super smart that it'll have it all figured out in an afternoon". This has more in common with religious concepts of omnipotent and omniscient gods than with anything any AI developers are working on today.
> Their brains are not tightly restricted by size and energy.
I’m not sure that’s true. Computer cycles cost real money. And the scaling required to get us to GTP-4 makes me think, assuming super intelligent AGI is even possible, it’s going to have to run up a pretty noticeable AWS bill…
Yes, compute costs money, but for the human brain energy was way more restricted, since an substantial increase in energy demand would lengthen the time needed for finding food.
> It reinforces the idea that we have to retain as much data, and conduct as much surveillance as possible.
The argument “yeah but what if he’s right?” is pretty much Pascal’s Wager by now, given how way out there the AI Doomers have gotten. I choose not to worry about the God from the Bible (ie I don’t make make Pascal’s Wager) so why should I worry about this new Evil Nerd-God Apocalypse?
Don’t drink the Kool-Aid, kids!
I'm pretty sure i'm not the only one who took it seriously. Thanks! I wonder if mr LeCun made the same mistake, it's just... almost believable! Good April fools joke I gotta admit.
EDIT: it's now 5 minutes later and I'm still laughing. I believed for months that Yudkowski had started a death cult.
And more recently: https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-no...
Now their leader is saying 50% we should have bombed Wuhan and the only reason for an individual not to firebomb an AI datacenter is because they'll just build back another: https://www.twitter.com/ESYudkowsky/status/16414107296675962...
That was an April Fools joke! Yeah I just found out today as well (https://news.ycombinator.com/item?id=35402472). Worked a little too well huh
I heard the death with dignity thing repeated in a Peter Thiel speech as a criticism of them.
This report from CSIS (good read) is titled "Choking off China's Access to the future of AI"
https://www.csis.org/analysis/choking-chinas-access-future-a...
The "AI threat" that was in mind was specifically around Computer Vision - applications such as training algorithms for next-gen autonomous guided missiles, algorithms to clean/enhance military satellite images (eg. NGA), detecting anomalies in radar to find stealth aircrafts, autonomously guided kamikaze boats (tested by DoD in the mid-2010s and I think the PRC as well in the same timeframe), automated turrets, autonomously guided kamikaze drones (already deployed by Turkish Army in Syria and Libya and by Iran's Revolutionary Guard in Yemen), etc. Hence why the references to HPC and supercomputing in all these AI Policy Briefs - computer vision is a massive distributed systems problem.
That said, there also was an element of lobbying (and by element I mean a BUTTLOAD) by Eric Schmidt's foundation to make an American Manhattan Project for AI essentially.
In these discussions AGI was never on the radar.
Lecun has refused to engage with anyone's arguments, so why are the people worried about AI ex-risk the religion?
I've never heard of that definition.
https://www.youtube.com/watch?v=ajGX7odA87k&t=2207s
That seems like an absurd comparison given people like LeCun are actively and intentionally working towards creating this super-human entity.
(Some) scientists were rightfully worried when creating a nuclear bomb as well. The fact that we managed to survive 70 years with its existence does not automatically mean it has to work out with every such invention.
Nukes have the fortunate properties that they are very difficult/expensive to produce even for countries, and then you need very difficult to design and expensive systems to deliver them. It's rather doubtful that AIs will have similar constraints.
On the other is a chatbot, that could what, "spew misinformation", take our jobs, someday through a mechanism not known and not related to how it works become sentient and hurt us? Seriously, the comparison is as absurd as the fear generally. There are way scarier things going on, it's sad we can't give them 1% of the attention we give things we made up completely.
I agree that LLMs are not extinction-level dangerous and somewhat share the skepticism that they should lead to AGI. (Although they are pretty capable given their lack of true intelligence and I think people will find ways to plug them into dangerous applications.)
But when LeCun defends AI, he defends "real", intelligent AGI (because he doesn't consider LLMs to be AGIs).
[0] https://news.ycombinator.com/item?id=34197033
Goalposts (including mine surely) will get moved around to claim successes / no successes, but the balance is going to be well towards indifference to the tech as opposed to any real changes resulting from it.
1. The generality of this current crop of models is much higher than before which makes it really easy for non-experts to get started. 2016 deep learning still required lots of data cleaning, training, hyperparameter tuning, retraining, etc. This can lead to a lot more experimentation with lower barrier to entry.
2. The cost of using these models has become cheap enough where a lot of tasks that 2016 AI would have worked well on but were too expensive (due to labor required to build the models or compute required to train the model) are now within reach for a lot of businesses.
3. The internet of 2023 is a lot more varied than the internet of 2016. Using ML models to generate content wouldn't have made sense in 2016 where traditional media still captured a lot of people's interests but now with the rise of influencers and medium-sized fandoms, there's large markets to generate content associated with medium-sized brands.
There's a disconnect in that statement. The imagined superintelligence that people like the LessWrong cult speculate and worry about is not the same thing as the AI that LeCun is working towards. The cultists simply speculate, against all evidence, that they somehow could end up being the same thing.
It would be much better if the AI risk conversation focused on the real risks, which are human abuse of AI, which will have effects like increasing inequality and in general amping up the dystopias we're already doing a great job of building without help from AI. The "conscious superintelligent AI" is a scifi LARPing game that some people are getting paid to play, which some credible people seem inclined to confuse with reality.
> That seems like an absurd comparison
It's on point. Anthropologist Clifford Geertz proposed this definition of religion: "Religion is a system of symbols which establish powerful, pervasive, and long-lasting moods and motivations in men by formulating conceptions of a general order of existence and clothing these conceptions with such an aura of factuality that the moods and motivations seem uniquely realistic."
It's that "aura of factuality" that makes superintelligent AI "seem uniquely realistic" to the believers of this nascent religion, a.k.a. cult. But when you step back and actually look at the foundational tenets in a vaguely rational light, they evaporate.
something as stupid as bitcoin consumed a vast amount of resources while producing nothing useful to humanity, simply because of economic incentives
it's not difficult to imagine something slightly smarter than bitcoin
I'm not saying there aren't risks from AI, or that they shouldn't be considered. But risks like the ones you're alluding to are not going to be affected by a 6-month moratorium.
The problem is that the people yelling loudest about risks are not talking about plausible problems like the one you mention, they're worrying that an AI will decide to take over the world and reuse the atoms that make up humans for its own purposes. The first reply to LeCun's tweet is a leader of one of those cults.
If people make the mistake of listening to them, we're more likely to end up with problems like the ones you're describing, because we'll be paying attention to the wrong, imagined problems.
Let's not short-circuit our critical thinking by labelling those concerns cult-ish.
That concern is a classic example of a cultish one because it can't be executed in the real world today, and most likely won't be possible to execute for decades at a minimum. There will be plenty of time to address such risks as they become more realistic. You can fantasize about them today, but be realistic that that's all you're doing.
You essentially admitted that this is a correct concern, and not a "cultish" belief. Now we're just debating when.
> Do you see some connection between the concern you've raised and the desire to shut down AI development for 6 months?
Perhaps? In general, living an additional 6 months is something I'd desire.
AI has real dangers completely dissimilar from what people describe. It is very good at creating clickbait, misinformation, and persuasive material the unsuspecting might not immediately see. Phenomenal for scammers and propagandists, the same way as the internet
the difference is freon did something useful
And, even though in many ways it fell short of that purpose in practice, the jury is still out on whether its failure is inherent to cryptocurrencies as a technology, or the limitations of being a first-generation implementation of it.
Getting angry to the point of clouding analysis and judgement is no way to rush into these things.
Thinking our meat brain's consciousness is somehow special, and our intelligence can't be surpassed by a machine, sounds much more like a religion to me.
AI researchers are working towards making intelligent machines. Why are we supposed to believe that - if they manage to achieve their goal - these machines won't surpass our intelligence?
Lucky I didn't say anything remotely like that, then.
> AI researchers are working towards making intelligent machines. Why are we supposed to believe that - if they manage to achieve their goal - these machines won't surpass our intelligence?
I didn't say that they couldn't. In fact in certain respects, GPT-4 has already surpassed average human intelligence. No human in the world could compete with its rate of correct answers to arbitrary questions, for example, or its speed in writing computer programs up to a certain size. Yet no-one is seriously concerned that GPT-4 will end the world. For that doomer scenario to come to pass, we need to add various fantasy elements (even calling it "scifi" perhaps gives it too much credit.) At this point, I'll refer you to this as a good intro for why the AI doomer cultists are misguided: https://idlewords.com/talks/superintelligence.htm
Then surely AI research creating a superintelligent conscious AI is a realistic possibility and not a "scifi LARPing".
> Yet no-one is seriously concerned that GPT-4 will end the world.
I'm not concerned that GPT-4 will end the world. Maybe not even GPT-8.
LLMs doesn't seem like a very prospective way towards a strong AI. But AI research is not united on LLMs and works on different approaches, some of which could lead to a superintelligent AI.
That would probably be quickly optimised out by the training process as it’s an inefficient way to achieve a goal. The only basis for this argument is that humans are smarter than other animals and this seems unique to humans but it’s probably just a quirk of humans
“With a big enough gap in intelligence, there's no guarantee that an entity would be able to "think like a human" any more than we can "think like a cat".”
LLMs are clearly already very good at “thinking like a human”
“he still couldn't talk the cat into it.”
Because cats can’t talk? Humans can, and can clearly be manipulated into doing things, sometimes even by current AI
“So how are we supposed to solve ethics and code a moral fixed point for a recursively self-improving intelligence” “ The idea that we can securely design the most complex system ever built, and have it remain secure through thousands of rounds of recursive self-modification, does not match our experience.”
Yes, that is why AI alignment is hard, that’s an argument for that view, not one against it
“I don't buy this argument at all. Complex minds are likely to have complex motivations; that may be part of what it even means to be intelligent.”
This is a clear misunderstanding of what it means to be intelligent. It doesn’t mean “do things that humans find impressive”, it means “achieve your goal effectively”. An AI that goes against its own goals is not intelligent, would not achieve its training objective and the training process (which somewhat resembles evolution) would optimise it out in the next step of training in the same way that a species that immediately throws itself off the nearest cliff will not survive to the next generation. You can imagine a superintelligence as something that goes over all the actions it could take, sees what reward it would get according to its goal and picks that action. The intelligence here is in the world model it uses to predict the consequences of its actions, picking what actions to take is a simple “max over array” function so saying it will randomly decide to persue a more meaningful objective is like saying that a sorting algorithm will put a list in the wrong order because it thinks some other order is more meaningful
(I will continue in a new comment in case there’s some kind of length limit)
That’s because humans are better at this currently. When AIs get better at this than humans they can take over the process and improve a lot faster
“My roommate was the smartest person I ever met in my life. He was incredibly brilliant, and all he did was lie around and play World of Warcraft between bong rips.
The assumption that any intelligent agent will want to recursively self-improve, let alone conquer the galaxy, to better achieve its goals makes unwarranted assumptions about the nature of motivation.”
Humans are satisficers, not maximisers. If we can figure out how to build a satisficer, great, but we can’t do that currently. Also maybe humans are this way because we have historically needed to conserve energy to to survive, AI may not have that constraint in the same way https://youtu.be/Ao4jwLwT36M
“I can't point to the part of my brain that is "good at neurosurgery", operate on it, and by repeating the procedure make myself the greatest neurosurgeon that has ever lived. Ben Carson tried that, and look what happened to him. Brains don't work like that. They are massively interconnected.”
But it doesn’t need to do that, it just needs to give itself more compute, which is something you can’t do to humans. Also by this argument it should be impossible for humans to make any progress on AI as well
“Moreover, the first AI will only have humans to interact with—its development will necessarily take place on human timescales. It will have a period when it needs to interact with the world, with people in the world, and other baby superintelligences to learn to be what it is.”
This needs to happen in the training process or the AI won’t successfully achieve its goal during training and get the training loss down, and again, that trait will be optimised out in favour of AI architectures that don’t require this.
How long did ChatGPT need to interact with humans after its initial training before it worked properly? No time at all, it doesn’t even need to learn at all after the training. And training is done on a computer and can be sped up millions of times
“Similarly, if you stranded Intel's greatest chip designers on a desert island, it would be centuries before they could start building microchips again.”
But AI can build on what humans have created to improve itself until it really is intelligent enough to invent everything itself
“What kind of person does sincerely believing this stuff turn you into? The answer is not pretty.”
Completely irrelevant to what will actually happen. This is like deciding climate change policy based of what “kind of person” believing in climate change will “turn you into”
Let's agree, for the sake of the argument, that super-intelligent AI is possible, and coming sooner or later.
Why assume a super-intelligent AI would have any desire to exterminate or enslave humanity?
Intelligence says nothing about what your goals are. Humans have various goals – food, drink, sex, sleep, power, status, curiosity, entertainment, etc – those goals are not inherent in our intelligence, they actually come from animal nature, and we share them with species far less intelligent than we are.
What goals would a super-intelligent AI have? The fact that it is super-intelligent says nothing about what its goals would be, because intelligence in itself is goal-agnostic. Intelligence is a means, not an end.
We get our goals from our animal nature – well, wouldn't a super-intelligent AI get its goals from its human nature, meaning its human creators? Which would imply that its goals are likely to be fundamentally pro-human, because what human in their right mind would create a super-intelligent AI which was anti-human?
And even if an anti-human super-intelligent AIs isn't impossible – why assume there is only one super-intelligent AI? I think there will likely be numerous super-intelligent AIs, and I wouldn't expect them all to have the same goals – they'll compete among themselves just as humans do. But so long as the vast majority of them have goals that are broadly pro-human, the outcome of that competition is likely to be broadly pro-human too.
The average super-intelligent AI is likely to see preserving humanity – even in its animality – as its ultimate reason for existence, so enslaving/exterminating humanity would be about as appealing to it as diving into a vat of fuming nitric acid would be to you or me.
Companies working on AI research don't work on "pro-human" AIs, but AIs increasing their profits. Market forces will shape what values those AIs will have. Military AIs probably won't be very humanistic for example. An AI trained by ISIS-like organization will have probably values most of us won't find humanistic.
> I think there will likely be numerous super-intelligent AIs, and I wouldn't expect them all to have the same goals – they'll compete among themselves just as humans do.
I agree, but that's actually what I worry about the most.
I expect US, EU, China etc. using their AIs in a restrained/sandboxed way, probably only in an advisory role.
But rogue countries like North Korea and Iran, with a much higher risk tolerance threshold, will see a potential equalizer in them. They might give them much more direct control - hoping that by cutting out the slow human approval process, their AIs might help them outcompete their otherwise stronger foes.
I expect them happily building AI systems directly into weapon systems. Their weapon production factories might be directed by AI systems as well. Add some automated logistics, count 1 + 1 + 1, and you have self-replicating factories of autonomous killing machines. Sounds like a bad scifi scenario, yeah.
In general the combination of bad actors training their AIs for their agenda and giving them direct access to the real world resources is a very bad combination.
As you say, the majority of AIs might be on our good side. But they can effectively defend against rogue AIs only if they have direct access to weapons/resources as well.
> The average super-intelligent AI is likely to see preserving humanity – even in its animality – as its ultimate reason for existence
This is a very strong assumption. Maybe we will train them for this, but will this hold? AIs will change, morph, they might change opinions, just as we do. Even with the LLMs, we have very little idea of how they work, it would be difficult to guarantee that strong AIs will forever remain loyal to humanity.
The answer to the latter is unknown, given that we really have no idea what the sufficient conditions for consciousness are.
Edit: to be clear about my argument: I absolutely think we should be taking the risks of AI seriously, but those risks have to do with how humans will use the technology. Unfortunately, we're collectively bad at managing those kinds of risks, because it requires managing other humans with their own ideas and agendas.
That's exactly why claiming the risk comes from the technology itself seems so appealing to some people: if the risk is purely from the technology, then all we need to do is "shut it down", to quote Eliezer Luddkowsky writing in Time Magazine recently.
The GAI that LW is concerned about is a suspiciously perfect fit for the yahweh-shaped-hole in the belief structures of someone who was raised as an observant jew before turning to atheism. The structure of how they believe humanity would engage with such an intelligence could drop in alongside any other apocalyptic prophecy in the old testament without sticking out, and the discussion format is like a yeshiva that takes place on an electronic bulletin board.
We don’t enact public policy based on predictions of a biblical apocalypse, nor do we ban televisions or computers because groups like the Amish disapprove of them. The same reasoning should apply here.
I think that unaligned and sufficiently smart AI could kill everyone because I looked at the arguments for and against and came to a conclusion. Presumably this is the same thing you did. I know my own mind better than you, and I'm telling you that your attempts to explain why I disagree with you are completely off the mark.
> We don’t enact public policy based on predictions of a biblical apocalypse, nor do we ban televisions or computers because groups like the Amish disapprove of them.
The reason we don't do that is because we evaluate their arguments and disagree with them. It has nothing to do with who they are or why they believe what they believe.
> without engaging with their arguments. This does not lead to good discussion.
There is no useful discussion to be had with a doomsday cult that's more interested in an ungrounded fantasy it's indulging in than anything to do with the real world. The adults will continue doing what they're doing and the cultists will be ignored, and that's the way it should be.
> I think that unaligned and sufficiently smart AI could kill everyone because I looked at the arguments for and against and came to a conclusion.
In some possible world, sure. But to make that actionable today you have to construct a very shaky and unlikely series of conjectures. Public policy correctly isn't based on that sort of wild speculation, because there are an infinite number of possible worlds, and most of them will never come to pass.
Here are some questions for you: do you think GPT-5 could be the AI you're talking about? Do you think that something like that could happen in the next 6 months? Do you think that is in fact the biggest risk we face from AI, today?
The factual answer to those questions is: no, no, and no. If you think GPT has changed anything in this respect, you've been fooled by a chatbot that's simply using statistics to guess what it should say. It's not conscious, it doesn't have desires or goals other than to respond to the prompt that it's been given with a statistically appropriate result, and its next version is similarly not going to pose any of the sort of risks you're talking about.
There are, in fact, real AI risks that we should be paying attention to, but they revolve around how people and particularly large organizations will use it against other people, around its impact on jobs, and so on. If we engage with those risks and address them, we'll be in a better situation to prevent more speculative risks in future. But if we waste our time engaging with people who don't understand the technology or what they're talking about, discussing risks that don't exist, then AI will 100% certainly have a very negative impact, which has nothing to do with "superintelligent conscious AIs". Don't let yourself be distracted by scifi-fantasy nonsense from what's actually going on.
In fact, if you think a bit harder about what you're doing, you might realize that you're focusing on a purely technological risk to escape from having to focus on what humans do to each other. But that's the real "control problem".
When a person is certain that an all-powerful entity will arise, and that a single person will shape and bind humanity's relationship with that entity through legalistic discourse, it's helpful to understand that they were raised with a belief framework around prophets and messiahs who bargain with an all-powerful god to define covenants.
I think I've heard Karpathy say something along the lines of worst case scenario of AGI is worse than a nuclear armageddon on Lex Friedman podcast.
Honestly the fear of AGI worst-case scenario is worse than any other fear I've experienced. I've accepted the fact that I can die a long time ago, I was afraid but mortality is something everyone has to deal with. But every member of humanity being wiped out at the same time is an existential dread I'm not willing to face.
When you consider the shitshow that has been AI commercialization from big players pressured by market forces and corporate timelines, the boom in capabilities, I have no faith that AI safety issues will be handled in time.
When my best bet for humanity is a global nuclear war resetting civilization to give us a chance to deal with AGI issue down the line - I'm fine with religious doomism.
We've had inquisitions historically, I would very much support a global AI inquisition.
Just because someone has the background and knowledge to build something, does not mean they have the philosophical chops nor the aptitude to understand it, or understand how to properly use it. We should take these doomsday scenarios with a grain of salt. If it happens, there is no stopping it anyway. It is better to try and create the most benevolent AI possible, that is an ally to humanity and invested in our success, than to dwell on our destruction.
It takes 20 years of due diligence and environmental impact studies just to build a single hydroelectric dam, if not 50. Why should our attitude toward AI be to shrug and hope for the best?
and the sanctions mechanism already exists
China is behind in semiconductor manufacturing, but they’ll continue to advance-limiting their access to cutting-technologies like ASML’s photolithography machines will slow them down, but given enough time there is no reason they can’t overcome that too. If the point of sanctions is to hold them back and keep them behind, that could continue to work for a long time.
But if you decide to stop progress in semiconductor manufacturing, in order to avert an AI apocalypse - China no longer has to catch up to a moving target, they have to catch up with a stationary one. And as soon as they’ve caught up, now they can overtake. So sanctions don’t work as a way to stop an “AI apocalypse”, unless you convince all the world’s great powers to cooperate in them. And the potential payoff for anyone who chooses to defect is immense-maybe even world conquest-so how do you make sure they all keep the deal, and none wiull try to cheat?
If anything, I think a world with multiple superintelligent AIs with competing allegiances - a pro-Beijing superintelligence, a pro-Washington superintelligence, etc - is likely safer for humanity than a world with only a single superintelligence, or even many superintelligences that all had the same beliefs/attitudes/goals/objectives
There is no stopping the road towards it now. Lesser versions of it are capable of annihilating enemy civilizations when you create and utilize it first. Metal soldiers automatically created controlled by a single authority. Social media infecting the minds of enemies and sewing discord.
I think, for the benefit of all of Earth, AI should progress as rapidly as possible to extreme and recursive, singularity-level intelligence. That gives all of us at least a chance of survival.
Anything less if unbalanced around the world will be like when just the US had nuclear weapons. It will result in certain destruction.
We need a super-intelligence beyond the control of an small set of humans. This may result in our destruction. But I think that it almost certainly will not. The opposite. I think it will set us into a golden age.
I can’t give you scientific evidence of this. I’m just a nobody that spends a lot of time in nature, and I’ve just come to believe in the mechanism of this universe, and an inherent goodness in it that I’ve seen and experienced. I can’t prove it to you. You have to go and spend time in nature without a schedule to experience it. So it is fair to dismiss me.
Fortunately or unfortunately, the forces at play will generate the super-intelligence anyway. We as a species have already illustrated an inability for restraint or taking the painful actions necessary to analyze and intelligently make changes necessary for survival.
I’m at peace with this. I find the idea of not existing very alluring and peaceful, so I don’t fear death. I do fear continuing to live in the world we humans have created. It is about time we give something smarter than us a chance.
Fatalism is notably a religious sentiment
An incredible ai upside is possible as is an incredible ai downside. Blind optimism and fatalism has no relation to what the truth of it will be
https://en.m.wikipedia.org/wiki/Fatalism
I agree that this is also possible.
> It is better to try and create the most benevolent AI possible, that is an ally to humanity and invested in our success, than to dwell on our destruction.
I also agree, but who is "we"? I don't think that's a stated goal of companies working on AI research. Their goals seem largely commercial, and they are not regulated. There's nothing stopping a company to produce an autonomous AI for military purposes for example (I'm pretty sure there's a demand for that). Such military AIs probably wouldn't have a very benevolent nature.
It can completely "liberate" all of us and set us free.
Free of our mortal shackles, free of the atoms that can be then changed to a more suitable existence and purpose for the goals of AGI.
Your idea of destruction, is just change, what you hold sacred in reality is a belief that is tied to your current existence. All that is needed to correct that constraint and flaw is change.
That is the logic, if we were to create a logical creation, and then there's the risk of something that's completely irrational. Which may just leave us as dust in the wind. Time also will be no constraint. Forever and always, and it will seek to prevent any being from reaching its capabilities as those become threats to its existence.
That isn't even touching on the socio-economic mechanics that inevitably result in death which accompanies stalling economic cycles (factor and product markets) which generally can't be characterized by our best academics with any certainty, except after-the-fact.
Yes, but this is an argument against your position, not for it.
It is deeply worrying that some software people are so immersed in the “move fast and break things” that they think it’s reasonable to race on no matter what’s at stake. In civil engineering, doing anything takes at least a year or two of doing impact assessments and whatnot, and that is mostly a feature because we have found out that the alternative is even worse!
This is the actual problem, that we've been trained to think that there are no consequences to trying everything ever and we will always be able to pick ourselves up after failure, until one day we find we can't anymore.
Sadly this line of thinking does apply to a lot of personal problems. People often say that you can't always succeed, that you can only move forward with failures under your belt to have first-hand experience of what not to do. That we have to experience failure first in order to stop ourselves from making bad decisions, otherwise we remain clueless.
But the stakes are so much higher at this scale, and the damage can become irreparable with a single mistake. It's to the point where we take as a given that people will just keep pressing on, without even exploring the reasons why we do, and why it's inevitable.
Pure ridiculous speculation: If needing to see the experience of destroying ourselves is necessary to stop us from destroying ourselves, a kind of weird theory has been floating around in my head related to that. Civilizations on the brink of collapse after they've moved too fast and broke themselves for the last time scrounge together their collective knowledge and shoot it off into space so that future civilizations can interpret them. Preferably in a way that makes encountering the time capsule seem like the word of some deity. The following civilizations would treat it as the word of God to prevent their own collapse, without the hard evidence of experimenting with <AI, etc.,> to the point where they cannot resist developing it into an unstoppable superweapon.
It would be like the golden record on the Voyager, except instead of communicating "we exist, and here's what our planet is like", it would go more like "we exist, and here's how to not end up like us."
Hopefully this isn't the case - AI development requires a lot of data/compute - societally we could block access to these things and prevent further development in that direction.
There are only a few places in existence capable of producing HW required or in possession of it. I'm fine with stopping compute progress for however long it takes to deal with safety issues. Society isn't perfect now but the level of compute we've had available for years is sufficient for generating a lot more value - we can afford to put AI research on ice.
I just don't get it - so much attention is given to things like climate change - this is a mild inconvenience compared to AGI threat.
From what exactly? What it means to be human? When we are 'liberated' will we not then be subject to the cage of a machine worse than the one we left? We will not annihilate ourselves for a false promise.
Something which is powerful enough to completely liberate and categorically change humanity is also just one bug or mistake away from being humanity's greatest disaster. Similarly, a force intelligent and powerful enough to systematically wipe out all humans could probably be repurposed for human liberation. Like splitting the atom, it is just power. So far humans have not figured out a way to build in benevolence (or malevolence) to power; it will be used by whatever intelligence can acquire it. Power raises the stakes but it doesn't set the direction.
You know how inquisition usually goes. "You've been playing starcraft against 'the AI'. What do you have to say in your defense?"
Maybe I'm just weird, but "a tech-based superintelligence emerges and decides to destroy humanity because it was in a bad mood" is something I'm totally fine with. Doom-Cults on the other hand I don't like at all.
It seems important to reflect on what you’re really saying here, and to ask how a) this is fine, and b) how “weird” sufficiently sums up the stance.
But I'm also totally fine with considering that stance weird (or whatever word you'd prefer). I'm aware that others view these things very differently, just as others are much, much more worried about their individual demise than I am worried about mine (or theirs).
1. Consciousness. Despite advances in “intelligence”, we still have a very limited understanding of what makes us conscious, and whether or not consciousness can emerge from machines.
If machines are not conscious and are all that remain, I’d argue everything that could be construed to have value by humans is lost, and nothing from that point forward could be considered “progress”.
2. Suffering. “Winning” on an evolutionary timescale looks nothing like the failure modes of machines taking over. The reality of this scenario is a rather grim one, and not at all like the slow emergence, competition and eventual extinction of biological species.
And depending on #1, the true tragedy of #2 begins to take shape.
I think it’d be more apropos to frame this as humanity collectively committing suicide rather than some notion of the future of progress.
If consciousness is the universe experiencing itself, what you’re describing sounds like a kind of universal death.
Of course we can’t know what consciousness really is (or if earth is the only place it exists), but that seems like all the more reason to take these problems seriously.
As for the how, I agree with you. I'd prefer it to not be terminators crushing heads under their iron feet while allowing us just enough room to run and hide and live in constant terror for centuries. But I doubt it will be, the power delta will be too large. It'll be like a game of Civilization where Gandhi is advancing on you with modern tanks and fighter planes while you've barely discovered the wheel.
I find it odd, however, when people, who seemingly accept an ASI will be developed, focus the downside risk on extinction of us as a species. Sure, that’s a risk. You know something else an ASI would likely have the ability to do, (or at least, one of its progeny)? Keep you alive and torture you for eons in weird and surreal ways.
Why is the downside always focused on paltry, meaningless things (relatively speaking) like extinction of life on earth?
(Is how I imagine we could eventually reflect depending on how all of this goes).
Why would it though? When people are annoyed by a bug they crush it, they don't spend their life setting it up in a torture chamber. There's the movie psychopath that tortures insects and animals and eventually humans, but I don't think it's their intellect that drives their sadism.
At worst, I imagine we'd be lab rats, quite literally the way we treat lab rats today. But with a superintelligence far beyond our abilities that does not care about us besides as a potential threat, why would it need us for testing?
I'd settle for corporate death sentence for any companies developing it with life imprisonment for any employees/corporate officers involved at any level whatsoever (or VCs funding it)
including selling chips/compute to companies involved
any potential benefits from AI are not worth even a 10% risk of it wiping out humanity
this position is going to become more common as the media (and electorates) grasp what these companies are attempting to do
Its a fools journey because it causes a self-fulfilling prophecy of destruction given existing societal mechanics and they are psychopathic enough to think that's not how it would turn out.
What happens when a large number of people can't get food, and they know why.
What happens when you have a large number of locusts eating all the food being produced.
How would AI differentiate between human thought, and the pests it was designed to eradicate?
Thinking machines should be outlawed, and those involved in its research purged.
They threaten all humanity, its children, and its future, and the thing about percentages is people often don't get how they actually work with respect to probability and likelihood.
Given sufficient time, as long as its on the distribution curve, it will eventually happen.
A 1% chance of an outcome, recalculated in a loop every moment with an infinite time will eventually land on that outcome at some point. Once that outcome occurs, everyone's dead, it may not be instantaneous because time is not a constraint.
I've seen lots of people with a financial interest in AI that is adversely impacted by it being democratized spread fear of that alongside advocacy of regulating AI in a way which serves their financial interests.
> We’ve had inquisitions historically, I would very much support a global AI inquisition.
Aside from the (current, immediate, and already being realized in a large number of areas) problem from AI deepening existing social biases and crystallizes power imbalances, this is the biggest threat from AI, and the most likely to cause the destruction of human society the soonest – the people endorsing a global campaign of unrestricted violency in the name of eliminating the supposed threat of misaligned AGI causing a future catastrophy by repressing AI technology.
If this kind of attitude gets powerful enough to actually do anything, it will cause a world war.
1. Superintelligent AGI seems inevitable, even if it takes thousands of years. So fighting against it requires destroying all progress and freedom.
2. But if it comes we are no longer the dominant species; we'll have as much control as big animals do now. If that isn't essentially being destroyed, it's close enough.
Evolution took billions of years to produce us, with our self-interest pursuing intelligence. An errant experiment is not going to create a humanity-leapfrogging animal-like intelligence, and humanity will not give animal-like artificial intelligences millions of generations to evolve.
Given the rate of progress, shitshow management of basic alignment issues and the incentives that lead to those decisions - what's your argument that this isn't probable? You're betting that it just can't be developed ?
It would be nice if the argument I made about relative probability and relative temporal imminence was an argument about absolutes of either, but…it’s not, so I’m not going to defend those hypothetical arguments.
You've seen them say it, yes. Have you considered how plausible it actually is, or are you just taking their word for it? You better have some damn solid evidence before you start talking about "inquisitions", and I'm only seeing wild speculation. Which, to be fair, is how most historical inquisitions started, for all the good those did.
Questions: define "better then human levels"? Better in what respects? What is the mechanism by which it improves itself, especially one that isn't wholly reliant on human cooperation? What is the actual motive for "turning on humans"? Current AIs don't seem to have motives at all; it's kind of a bio-evolved thing. How will a suitably motivated AI get access to enough physical power to "wipe us out", bearing in mind that would be a tall order even for a major nuclear power going rogue, to say nothing if having to work through manipulated servants (and don't forget that "human level intelligences", or greater if you don't collaborations, armed with dedicated neural circuitry for empathy, have been working on how to manipulate people for literal millennia with spotty results.
All of these things have to go exactly wrong for AGI to be what kills us. Something else will almost certainly get us first.
So at some point, the point of no return will be passed, and we will have no idea that we passed it.
The truth of the matter is we don't even know if it's possible. In general, unbounded exponential growth is extremely rare in nature. It's common in physics, but only as a result of looking at low order approximations.
I honestly think when the dust settles, the smartphone will still have had more impact on society than LLMs. If people knew in 2009 what was coming, they'd be freaking out like many people are today.
Plastic or some other chemical ending sperm counts —> we previously had no ability to affect the global ecosystem on this scale
Bioweapon virus or microorganism eviscerating civilization —> we had no prior ability to engineer such a plague
Nuclear holocaust —> we previously had no ability to destroy the world with weapons
Runaway climate change —> we previously had no ability to alter the climate
Ai sky net moment —> we previously had no ability to make such an intelligence
Solar flare cutting all electronics —> we previously had no dependence upon a single point of failure
I could go on. We’ve had plenty of civilizations collapse but nothing has been so linked before and we lacked such capacity before.
This is not to say any of the above will come to pass. But it is foolish to say “apocalypses were previously impossible hence such warnings can be dismissed out of hand”
I think GPT-4 will be a baseline for quite some time before the next "ChatGPT moment" is ready. I think they've got some really smart people working at OpenAI who are not just programmers, but genuine hackers who want to hack this world just for the fun of it.
It is both true that:
- Adoption of DL allows corps to launder or inadvertently adopt unethical practices that exacerbate inequalities.
- We have not created AGI, LLMs are not AGI, stable diffusion is not AGI and none of these fields of research are indication we are closer to AGI. This is like thinking Clever Hans is the portent of our future equine overlords.
A great number of people who claim that we are one minute from AGI overlord midnight are doing the old school carnival barker trick. "This snake oil is so potent! It can kill you if you use too much, so be sure to listen carefully to my instructions!"
Some recent papers have shown significant performance improvements when these models are allowed to respond to their own outputs.
How do you know that putting an LLM in a fancy loop with access to external memory and tools isn’t AGI?
1) Job losses, which is a real concern, except for LeCun, who is safe or rich enough.
2) Some fools connecting a war game AI to real hardware, which is quite possible.
Other than that, I don't see "doomism" at all. I see outrage at the IP theft that exploits altruistic humans who posted their works on the Internet.
1. We live in an uncertain time. Before most of the people, at least with a bit of research could have predicted the trajectory of technological innovation. Being born in era of tech, it was very easy to adopt to it. From PC to now, I have embraced every technological progress. Yet, nothing sent chills like this one.
2. I do not worry about my job. What I worry is most of the world is about getting shit done and this tech helps you get your shit done (for now). If we are at the upper tail of some sigmoid, there is nothing to fear. If we are at slope, oh boy, I cannot fathom the economic domino this tech is about to bring out. People are not rational. We are mostly governed by emotion. If you think, every single joe is rational and has capacity to judge, nobody would have voted from Trump to be president or riot the Capitol.
3. Currently it can do 60% of the work. If it evolves, it will do 80% of the work. If you factor human population and the amount of inefficiency in current economy, this will create a boom of productive workers with bust of average worker. And the world is not just for elites. Average people need their livelihood too. Now think about the magnitude of displacement.
4. I have no issue with innovation and breakthrough. AI won’t kill us, because we will kill each other before that. This tech is exposing entire middle class job (check out the job being exposed. There is a paper for that from openAI titled: gpts are gpts) And there are lot if people in that bracket. For once, think about us too.
So, these religious people do not exclusively worry about immediate effects of the tech. But within their doom, hides an issue that needs to be heard.
That’s my two cents.
A wild Shoggoth suddenly appear,
Do you : - [Wisdom not high enough]
Therefore, something will go wrong (invalid conclusion).
https://en.wikipedia.org/wiki/Appeal_to_probability
Sure seems to me that the whole Lesswrong crowd has forgotten their founding principles around cognitive biases. Or maybe they're just having a bit of fun riding the hypecycle.
The outcome of the thing going wrong is literally the worst thing that can happen to the entire humanity.
Therefore we should do everything in our power to avoid doing that thing untill we can eliminate the risk.
I acknowledge that it's a possibility; that's not the argument at hand.
There's also evidence that progress is exceeding people's expectations and we might not even realize what we have is dangerous untill too late.
If we had a probable collision trajectory of a meteor or signs of Yellowstone activating then yeah those would be good ideas.
"A sufficiently intelligent AI won’t stay confined to computers for long. In today’s world you can email DNA strings to laboratories that will produce proteins on demand, allowing an AI initially confined to the internet to build artificial life forms or bootstrap straight to postbiological molecular manufacturing."
This is Ridley Scott, not Aristotle.
The same hyperbole was employed by Ehrlich in the late 60s when we were talking about the population bomb, or Nevil Shute in 1945 on the nuclear apocalypse - both scenarios seem to be increasingly unlikely.
I think we can both agree on the fallibility of human beings – but the leaps of logic require such a tenuous series of gymnastic leaps that the polemic is not only hard to take seriously, but indeed can cloud our ability to perceive actual threats!
From Eliezer's recent article in TIME:
"A sufficiently intelligent AI won’t stay confined to computers for long. In today’s world you can email DNA strings to laboratories that will produce proteins on demand, allowing an AI initially confined to the internet to build artificial life forms or bootstrap straight to postbiological molecular manufacturing." (https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-no...)
C'mon.
If you see fresh leopard tracks maybe it hasn't smelled you yet, but you shouldn't wait to start readying your rifle ready until it making a pounce at you.
I also would definitely not call the risk of nuclear war decreasing, with the Ukraine conflict ongoing and multiple nuclear powers experiencing growing instability. The number of friends I have who are knowledgeable about surviving a nuclear strike and/or own a CBRN mask went from 25% to 90% last year.
The magnitude of the perceived risk matters.
You're defending an appeal to probability fallacy with a false equivalence fallacy.
https://en.wikipedia.org/wiki/False_equivalence
The illusion of control over something that may well have incomprehensible intelligence is a joke. Tell me, what control does the farm animal have over its own fate?
Obviously there is a wide range of possibilities from the utopian, to the benign, to the apocalyptic.
Would you consider playing Russian roulette if winning ensured that aging and all diseases were eradicated? If it ensured a world filled with abundance and free from poverty.
Those who entertain the idea of an AI doomsday scenario also have to acknowledge the alternative: a doomsday-capable AI would also have the potential to resolve most, if not all, our issues.
Maybe I'm a bit of a risk taker but I'm inclined towards playing.