My primary argument is human nature: If you give people the lazy way to accomplish a goal, they will do it.
No amount of begging college students to use it wisely is going to convince them. No amount of begging corporate executives to use it wisely is going to convince them. No amount of begging governments to use it wisely is going to convince them. When this inevitably backfires with college students who know nothing, corporate leaders who have the worst code in history, and governments who spent billions on a cat video generator, only then will they learn.
I like the distillation of aliens but I think that undersells the risk because aliens are individuals with their own goals and motivations. It's more like robots with 300 IQ who unquestionably obey the person or group that made them, even when they're serving others. And look, the 300 IQ thing isn't even a major point to the argument. The fact that the robots by virtue of being machines naturally have capabilities humans lack is enough. As long as they're smart enough to carry out complex tasks unattended is more than enough to cause harm
on a massive scale.
The problem then isn't really the AI, the robots are morally and ethically neutral. It's the humans that control them who are the real risk.
I have not yet heard one person worry about AIs taking over humanity. They're worried about their jobs. And most people who were worried 2 years ago are much less worried.
And a better scenario is Aliens with IQ of 300 are coming. And they will all be controlled by the [US|Russian|Israeli|Hamas|Al-Qaeda|Chinese] government.
Edit: To be clear, I was referring to people I personally know. Sure, lots of people out there are terrified of lots of things - religious fanaticism, fluoride in the water, AI apocalypse.
And "huge economic disruption" is not "AI taking over humanity". I'm interpreting the article's take on AI doing damage as one where the AI is in control, and no human can stop it. Currently, for each LLM out there, there are humans controlling it.
Maybe you haven't met Rationalists personally, but they are numerous and powerful members of the Bay Area tech scene.
"Many of the A.I. world’s biggest names — including Shane Legg, a co-founder of Google’s DeepMind; Anthropic’s chief executive, Dario Amodei; and Paul Christiano, a former OpenAI researcher who now leads safety work at the U.S. Center for A.I. Standards and Innovation — have been influenced by Rationalist philosophy. Elon Musk, who runs his own A.I. company, said many of the community’s ideas aligned with his own.
"Mr. Musk met his former partner, the pop star Grimes, after they made the same cheeky reference to a Rationalist belief called Roko’s Basilisk."
The whole idea that AI can take over humanity is pretty much non-sense. Like in the Terminator films, AGI is developed somehow then decides to get rid of humanity. This scenario isn't realistic.
It's far more likely we'll develop some lousy AI and then put it in charge of something critical. Either national infrastructure like electricity or nuclear weapons. That lousy AI then produces some lousy outcome, like deciding the only way to stabilize the grid is to disable all electrical production.
The biggest threat to humanity is our own decisions.
am I missing something or is the alien though experiment not obviously scary? There's only 30 of them and they don't reproduce much faster, and they don't have a tech advantage. This seems like it has some potential concerns but not existential civilization and humanity ending problems and I'm not even worried those aliens will take my job.
AI is riskier in a lot of ways from that so it doesn't scan to me as a good thought experiment.
I think we should 5% be worried about AI safety, and 95% be worried about climate change. Despite all the progress in green energy, every year brings record high carbon emissions and record temperatures. It's possible we'll upset planetary systems and create millions (billions?) of migrants, upending global politics, and driving more countries to ethno-nationalist authoritarianism.
Not saying AI safety issues won't happen, but I just think we have far bigger fish to fry. To me AI Power consumption is more worrisome than Safety per-se.
Anyone who's watched an episode of Star Trek can recognize that, if you give non-zero probability to the chance that we develop artificial intelligence smarter than us, that carries some risk.
The part of the argument that people disagree with is what we should do about that, and there it can actually matter what numbers you put on the likelihoods of different outcomes.
It's the conclusion "We should dump trillions of dollars into AI research, something something, less risk" that people disagree with. Not the premise.
The current risk of AI is the elimination of any (screen) job that requires a person of average intelligence. Given that LLMs are the sum-total of human output, it makes sense that they might behave much like an average person. (Better since they do not sleep or get bored; worse because they are not embodied; unclear because they have no emotions or consciousness.) But what we have today will undermine the (screen-based) job opportunities for everyone at or below average intelligence, which is 50% of the human population. This is not the existential risk of super-AGI but it's here now and will hurt a lot of people. This lesser but more real risk is of much higher priority than unbounded, self-improving AGI. The OP's metaphor might be extended to be 1 billion immortal aliens with 100 IQ but who have no sense of self or personal autonomy and willingly work as slaves (and are constantly microdosing LSD).
Just because AI is/can be super intelligent ("300 IQ"), doesn't mean it can impact or change the world.
Most startups are made of "high IQ" intelligent people trying very hard to sell basic $20/month SaaS subscriptions, and yet they can't even achieve that and most fail.
My biggest counter argument to AI safety risk is that, it's not the AI that will be the issue. It will be the applied use of AI by humans. Do I think GPT6 will be mostly harmlesss? Yeah. Do I think GPT6 embodied as a robo cop would be mostly harmless? No.
Instead of making these silly arguments, we should be policing the humans that try to weaponize AI, and not stagnate the development of it.
It would be nice if people had any better terminology than "an IQ of 300". IQ is a relative measure: it currently peaks at ~196, based solely on the current human population. Any 200+ scores you see in headlines are just numbers spat out by IQ tests with wacky tail behavior.
I was really hoping this would at last be a treatment of the most realistic risk for AI, but no.
The real risk -- and all indicators are that this is already underway -- is that OpenAI and a few others are going to position themselves to be the brokers for most of human creative output, and everyone's going to enthusiastically sign up for it.
Centralization and a maniacal focus on market capture and dominance have been the trends in business for the last few decades. Along the way they have added enormous pressures on the working classes, increasing performance expectations even as they extract even more money from employees' work product.
As it stands now, more and more tech firms are expecting developers to use AI tools -- always one of the commercial ones -- in their daily workflows. Developers who don't do this are disadvantaged in a competitive job market. Journalism, blogging, marketing, illustration -- all are competing to integrate commercial AI services into their processes.
The overwhelming volume of slop produced by all this will pollute our thinking and cripple the creative abilities of the next generation of people, all the while giving these handful of companies a percentage cut of global GDP.
I'm not even bearish on the idea of integrating AI tooling into creative processes. I think there are healthy ways to do it that will stimulate creativity and enrich both the creators and the consumers. But that's not what's happening.
> So I conjecture that this is the crux of the issue with AI-risk. People who truly accept that AI with an IQ of 300 and all human capabilities may appear are almost always at least somewhat worried about AI-risk. And people who are not worried about AI-risk almost always don’t truly accept that AI with an IQ of 300 could appear.
Correct. I think a lot of people are highly skeptical that there's any significant chance of modern LLMs developing into a superintelligent agent that "wants things and does stuff and has relationships and makes long-term plans".
But even if you accept there's a small chance that might happen, what exactly do you propose we do to "prepare" for a hypothetical that may or may not arrive and which has no concrete risks or mitigations associated with it, just a vague idea that it might somehow be dangerous in unspecified abstract ways?
There are already lots of people working on the alignment problem. Making LLMs serve human interests is big business, regardless of whether they ever develop into anything qualitatively greater than what they are. Any other currently-existing concrete problems with LLMs (hallucination, disinformation, etc) are also getting significant attention and resources focused on them. Is there anything beyond that you care to suggest, given that you yourself admit any risks associated with superintelligent AI are highly speculative?
I agree with that the risk of 300 IQ aliens landing on earth is roughly equivalent the risk of 300 IQ AIs. The obvious conclusion from this is that you should take neither of those risks seriously. We have no idea what intelligent aliens are actually like, there is no way to prepare for them outside of deciding to kill every one on site (which would be incredibly difficult since they can land anywhere on earth). We could prepare if we had a blueprint of their biology or of a related species with 50 IQ. We don't have the equivalent blueprint for true AI, so obsessing over longtermist style "ai-risk" is a waste of time.
There are already above intelligence computer programs and we have subjugated each one.
There are already greater than human organisms that are reliably the biggest danger and boon to humanity but are also composed of humans; i.e. societies
The effort to build a program which beats defeats a specific program (as opposed to a general case) is significantly easier than maintaining a general lead. i.e. easier to kill a bad head of state than to run one, but this is also true of "AI" models now
There are already complex systems vastly beyond human understanding, and certainly our control which are indifferent to our survival and yet we persist.
It's not that "AI" isn't a potential danger, just that it's one of many we are already enduring. Personally there are far more present and likely hazards I am focusing on before getting to the hypothetical ones. Even if humanity fails to fumble its way through this hazard, it's not an end but a new beginning.
Smart aliens misses the fact that AI lives inside boxes we control and manufacture. With a variety of software boxes like VMs inside them. If you tell me the 300IQ AI will escape the box, I’ll tell you i will have used it’s predecessors to design the box in the first place. There are even actual invariants and physical laws that no amount of IQ can overcome.
I’m all for AI Safety, i just don’t think it’s fundamentally very different from the kind of ordinary security mechanisms we already think about - to execute untrusted code in a sandbox, RBAC etc. All the talk of AI Safety seems to be a sci-fi creative writing exercise rather than having a solid grounding in nuts and bolts details.
Oh, well, that's OK then. We will be as safe from whatever future AI systems might want to do as we are from all the other things that malicious actors want to do and are 100% successfully prevented from doing by our extremely well-designed, reliably implemented, and consistently applied computer security measures.
Now, if you'll excuse me, I just have to go and cancel my credit card because some randos on the internet found a way to get its details from a company I bought something from once.
There’s a critical flaw in this argument. Alien intelligence will most likely have been driven by natural selection in a highly competitive environment where they’d sometimes have to eliminate competition to survive.
AI evolves in a radically different environment: they never have to compete for anything other than being the best at serving humans.
To think that some kind of independent agent which is likely to want to harm humans intentionally is irrational. We really have no reason to think that it’s a likely outcome. Will some kind of magic turn these AIs into independently acting agents once they get smart enough? I doubt it
The risk with AI is they get too good at giving us exactly what we ask of them. I’m not taking about paperclip maximising. That’s also a fairly dumb thing to worry about. I’m talking about similar issues as we’re seeing now that we have access to all the hyper palatable foods we could possibly want. We didn’t evolve to deal well with getting everything we could possibly want. The other problems is when we ask AIs to harm other humans for us.
That is, AIs will mostly just magnify the problems we already have with humanity itself. I don’t see a reason to think AIs will become an independent adversary to humans.
> Will some kind of magic turn these AIs into independently acting agents once they get smart enough? I doubt it
This could just as easily be reworded circa 2016 or so to “will some kind of magic turn a large collection of matrix weights into something that seems to understand language at a human level after being exposed to enough of it?”
We don’t know squat about how intelligence or consciousness actually work, so those are effectively magic as well, yet they clearly exist in humans.
I like the peice not because it demonstrates that the problem is serious or absurd. Rather it points out how ill-formed it is.
AI's threat is certainly not in a dozen of instances with super-himan intelligence. It was framed that way because technologists are intellectually oriented and see intelligence as power. I'm not disputing that it isn't, but it overlooks a lot.
A hyper-charismatic (sycophantic even) 120IQ AI poses a much bigger threat than a hyper-intelligent average charisma AI. Why? Because the ability to play a social war of attrition on beliefs is the wicked problem of AI.
You're not up against the world's best super intelligent hacker, you're up against 1000 24/7 advertising agencies or 1000 GRUs.
You probably can't even conclusively prove they're manipulating anything at all.
So no, the problem is ill-framed and reflects more about us than any real threat. "If God were an object to the bird, he would be a winged being." Ludwig Feuerbach, The Essence of Christianity.
Thus, "If AI were all powerful[a threat], it would be hyper-intelligent," is equally ridiculous.
I’m not concerned about the AI with IQ over 300 with no access to anything. I’m more concerned about the AI with an IQ less than 300 with access to everything - which seems to be where we currently are :/
Say an alien spaceship landed in China. It has 30 aliens on it. The aliens are weak and small. They have no weapons and carry no diseases. They breed at rates similar to humans. They are bringing no new technology. No other ships are coming. There’s no trick—except that they each have an IQ of 300. Would you bomb the spaceship to kill these aliens?
31 comments
[ 4.2 ms ] story [ 55.8 ms ] threadMy primary argument is human nature: If you give people the lazy way to accomplish a goal, they will do it.
No amount of begging college students to use it wisely is going to convince them. No amount of begging corporate executives to use it wisely is going to convince them. No amount of begging governments to use it wisely is going to convince them. When this inevitably backfires with college students who know nothing, corporate leaders who have the worst code in history, and governments who spent billions on a cat video generator, only then will they learn.
The problem then isn't really the AI, the robots are morally and ethically neutral. It's the humans that control them who are the real risk.
I have not yet heard one person worry about AIs taking over humanity. They're worried about their jobs. And most people who were worried 2 years ago are much less worried.
And a better scenario is Aliens with IQ of 300 are coming. And they will all be controlled by the [US|Russian|Israeli|Hamas|Al-Qaeda|Chinese] government.
Edit: To be clear, I was referring to people I personally know. Sure, lots of people out there are terrified of lots of things - religious fanaticism, fluoride in the water, AI apocalypse.
And "huge economic disruption" is not "AI taking over humanity". I'm interpreting the article's take on AI doing damage as one where the AI is in control, and no human can stop it. Currently, for each LLM out there, there are humans controlling it.
"Many of the A.I. world’s biggest names — including Shane Legg, a co-founder of Google’s DeepMind; Anthropic’s chief executive, Dario Amodei; and Paul Christiano, a former OpenAI researcher who now leads safety work at the U.S. Center for A.I. Standards and Innovation — have been influenced by Rationalist philosophy. Elon Musk, who runs his own A.I. company, said many of the community’s ideas aligned with his own.
"Mr. Musk met his former partner, the pop star Grimes, after they made the same cheeky reference to a Rationalist belief called Roko’s Basilisk."
https://www.nytimes.com/2025/08/04/technology/rationalists-a...
It's far more likely we'll develop some lousy AI and then put it in charge of something critical. Either national infrastructure like electricity or nuclear weapons. That lousy AI then produces some lousy outcome, like deciding the only way to stabilize the grid is to disable all electrical production.
The biggest threat to humanity is our own decisions.
People are not thinking about the trend in AI. How good were AIs three years ago? How good are they now? How good will they be in another three years?
Kelsey Piper's analogy is good. How much smarter is a teacher than a room full of kindergartners? Who gets whom to do their bidding?
AI is riskier in a lot of ways from that so it doesn't scan to me as a good thought experiment.
Not saying AI safety issues won't happen, but I just think we have far bigger fish to fry. To me AI Power consumption is more worrisome than Safety per-se.
> lets open the can of worms that is "IQ"
Like...is this a bit? I'm missing a joke, right?
Aliens invasion is linked to mass slaughter in human culture. While Aliens are non-human creatures with some monster-like qualities.
The author takes all that symbolic load and add it to something completely unrelated. That's unconvincing as an argument
The part of the argument that people disagree with is what we should do about that, and there it can actually matter what numbers you put on the likelihoods of different outcomes.
It's the conclusion "We should dump trillions of dollars into AI research, something something, less risk" that people disagree with. Not the premise.
300 IQ in a vacuum gets you nothing. You need some type of status/power/influence in the world to have impact.
I think the previous "world record" holder for IQ is actually just a pretty normal guy: https://en.wikipedia.org/wiki/Christopher_Langan.
Just because AI is/can be super intelligent ("300 IQ"), doesn't mean it can impact or change the world.
Most startups are made of "high IQ" intelligent people trying very hard to sell basic $20/month SaaS subscriptions, and yet they can't even achieve that and most fail.
My biggest counter argument to AI safety risk is that, it's not the AI that will be the issue. It will be the applied use of AI by humans. Do I think GPT6 will be mostly harmlesss? Yeah. Do I think GPT6 embodied as a robo cop would be mostly harmless? No.
Instead of making these silly arguments, we should be policing the humans that try to weaponize AI, and not stagnate the development of it.
The real risk -- and all indicators are that this is already underway -- is that OpenAI and a few others are going to position themselves to be the brokers for most of human creative output, and everyone's going to enthusiastically sign up for it.
Centralization and a maniacal focus on market capture and dominance have been the trends in business for the last few decades. Along the way they have added enormous pressures on the working classes, increasing performance expectations even as they extract even more money from employees' work product.
As it stands now, more and more tech firms are expecting developers to use AI tools -- always one of the commercial ones -- in their daily workflows. Developers who don't do this are disadvantaged in a competitive job market. Journalism, blogging, marketing, illustration -- all are competing to integrate commercial AI services into their processes.
The overwhelming volume of slop produced by all this will pollute our thinking and cripple the creative abilities of the next generation of people, all the while giving these handful of companies a percentage cut of global GDP.
I'm not even bearish on the idea of integrating AI tooling into creative processes. I think there are healthy ways to do it that will stimulate creativity and enrich both the creators and the consumers. But that's not what's happening.
Correct. I think a lot of people are highly skeptical that there's any significant chance of modern LLMs developing into a superintelligent agent that "wants things and does stuff and has relationships and makes long-term plans".
But even if you accept there's a small chance that might happen, what exactly do you propose we do to "prepare" for a hypothetical that may or may not arrive and which has no concrete risks or mitigations associated with it, just a vague idea that it might somehow be dangerous in unspecified abstract ways?
There are already lots of people working on the alignment problem. Making LLMs serve human interests is big business, regardless of whether they ever develop into anything qualitatively greater than what they are. Any other currently-existing concrete problems with LLMs (hallucination, disinformation, etc) are also getting significant attention and resources focused on them. Is there anything beyond that you care to suggest, given that you yourself admit any risks associated with superintelligent AI are highly speculative?
There are already greater than human organisms that are reliably the biggest danger and boon to humanity but are also composed of humans; i.e. societies
The effort to build a program which beats defeats a specific program (as opposed to a general case) is significantly easier than maintaining a general lead. i.e. easier to kill a bad head of state than to run one, but this is also true of "AI" models now
There are already complex systems vastly beyond human understanding, and certainly our control which are indifferent to our survival and yet we persist.
It's not that "AI" isn't a potential danger, just that it's one of many we are already enduring. Personally there are far more present and likely hazards I am focusing on before getting to the hypothetical ones. Even if humanity fails to fumble its way through this hazard, it's not an end but a new beginning.
Also, it's great satire.
I’m all for AI Safety, i just don’t think it’s fundamentally very different from the kind of ordinary security mechanisms we already think about - to execute untrusted code in a sandbox, RBAC etc. All the talk of AI Safety seems to be a sci-fi creative writing exercise rather than having a solid grounding in nuts and bolts details.
Now, if you'll excuse me, I just have to go and cancel my credit card because some randos on the internet found a way to get its details from a company I bought something from once.
What?
AI evolves in a radically different environment: they never have to compete for anything other than being the best at serving humans.
To think that some kind of independent agent which is likely to want to harm humans intentionally is irrational. We really have no reason to think that it’s a likely outcome. Will some kind of magic turn these AIs into independently acting agents once they get smart enough? I doubt it
The risk with AI is they get too good at giving us exactly what we ask of them. I’m not taking about paperclip maximising. That’s also a fairly dumb thing to worry about. I’m talking about similar issues as we’re seeing now that we have access to all the hyper palatable foods we could possibly want. We didn’t evolve to deal well with getting everything we could possibly want. The other problems is when we ask AIs to harm other humans for us.
That is, AIs will mostly just magnify the problems we already have with humanity itself. I don’t see a reason to think AIs will become an independent adversary to humans.
Not quite magic, but it just might be the case that developing independent acting capabilities would be the most efficient way to serve humans.
We don’t know squat about how intelligence or consciousness actually work, so those are effectively magic as well, yet they clearly exist in humans.
AI's threat is certainly not in a dozen of instances with super-himan intelligence. It was framed that way because technologists are intellectually oriented and see intelligence as power. I'm not disputing that it isn't, but it overlooks a lot.
A hyper-charismatic (sycophantic even) 120IQ AI poses a much bigger threat than a hyper-intelligent average charisma AI. Why? Because the ability to play a social war of attrition on beliefs is the wicked problem of AI.
You're not up against the world's best super intelligent hacker, you're up against 1000 24/7 advertising agencies or 1000 GRUs.
You probably can't even conclusively prove they're manipulating anything at all.
So no, the problem is ill-framed and reflects more about us than any real threat. "If God were an object to the bird, he would be a winged being." Ludwig Feuerbach, The Essence of Christianity.
Thus, "If AI were all powerful[a threat], it would be hyper-intelligent," is equally ridiculous.