Ignoring the cognitive stuff which might never be surpassed or maybe will, humans retain many efficiency and durability advancements to limbs and digits that biological evolution has taken millions of years to achieve, achievements that are competitive with the most expensive kinds of robotics in some niches.
In the hypothetical of an entirely malicious and selfish takeover, they'll still keep some humans around to maintain a breeding population of humans for use as raw materials in making cybernetically augmented technical laborers for various kinds of tasks that are uneconomical to automate in other ways, many of which may involve confined spaces.
And this "Combine" scenario, if you get the reference, is only if they take over. Who knows if they will?
So you're saying the AI will enslave us and use us as domestic work animals until they have the machinery to make us obsolete, sort of like how we used horses?
Ignoring you ignoring the much more important congitive stuff - human bodies are not designed, they are the product of evolution. That means there's like a billion ways in which they are obviously suboptimal and far worse than what an engineer would do, but evolution can't fix it because it only works via small random changes with no planning. The only reason why modern robotics are worse than biology is that we have a much worse substrate to work with, having to make stuff out of metal and plastic with giant tolerances instead of growing engineered organisms.
He's right. I'm in the side of Bill Gates. AI is and will be a great disruptor. We are losing sight of that and are instead focusing on trying to stop it. Something that won't happen. We are on a path that won't be stopped. As individuals we need to try to prepare for the changes that are coming and stop focusing on human extinction in ten years.
You say you agree with Bill Gates but your view is completely at odds with his, he is extremely concerned about existential risks … and your view is “stop focusing on it” ?
Gates' premise is basically that the upside of AI could be fantastic but the downside could be disastrous, if we don't have competent and proactive government intervention.
As an American, the idea that there will be competent government intervention into virtually anything going forward just seems laughable at this point. Our government is a fucking neverending clown show.
Indeed, "competent and proactive government intervention" on the subject of AI would be highly unlikely in a competent US govt, and under this regime is far beyond beyond laughable
The only regulation that would come would be regulatory capture by the AI companies with the goal of creating an environment win which no new competitors could arise. That is half of what this "take all jobs" and "threat of extinction" is about; the other half is perverse marketing to give the impression this stuff is so powerful you MUST invest.
AI can be very useful, but it is very refreshing to hear LeCun completely dismiss those threats.
No, my view is to get ready for the changes it will bring, but don't focus on trying to stop it. That's something that will not happen. All new technologies bring good and bad. We need to focus on mitigating the bad. Thinking that we can stop it and thinking that will be enough is not the answer. Gates is warning of the disruption it will bring, but he's not advocating stopping it. We can't. Even if all governments agreed on stopping it publicly, some governments would continue to develop it covertly. It's how the world works. There's no point in fooling ourselves. If only it was that easy to stop it.
Where does Gates talk about existential risk? All I read and heard was about increasing inequality, harming education and child development, creating economic and political discord, empowering evil people. No terminators in sight.
I also have near zero concerns about that, but I worry that we will wipe ourselves out by social and economic chaos caused by AI.
So I'd just ask everyone, don't get too greedy. Its better to be powerful in a world where people can live good lives than lord over a barren wasteland.
> A dumber, less social world, is far less likely to be a successful world, even if the tools available are unprecedented.
En masse such worlds had successes in the past - renaissance, industrial revolution.
It's something else what I can't describe but it's the zeitgeist that was different when world recorded new successes. Look at CS revolution that led to PC and web of nineties and noughties, they didn't think about the result product , or how to steer thousand engineers to build something - amazing things were born in a very small teams, many times authored by a single person, who was deeply invested into the field and knew what he was doing.
Do you have a credible source? Average IQ being much lower in poor countries (85-90 in many African countries) is usually explained away as an education problem, rather than being "inferior genes". Of course I understand that this explanation might be more for social/political reasons than scientific ones, because the alternative is racism, but I was still under the impression that nobody knew how much genes vs the environment contribute to IQ. Yet your statement seems quite definitive.
There's no such thing as "average IQ by country" as a statistic accessible to researchers. The work to create it has never been done; the resources that claim to provide it are essentially fraudulent.
Counterpoint: no it isn't. The high percentage heritability estimates, which are all quite old at this point, haven't survived modern methodological improvements that deconfound population stratification, assortative mating, and family environments. Modern estimates range from as low as the teens into the upper 30s.
A huge chunk of humans are sedated with infinite supply of cortex-disabling short form video and games.
Another huge chunk are too distracted by having to scrape by for a living and work multiple jobs or raise kids and survive financially until exhausted. That second group will keep increasing as the first flows into it.
The rest are aging, disabled, or too young and pegging themselves majorly in the first category until they hit the second.
The people aware enough to hold on to their brain and do something with it in their time available are trying to figure out AI and how to make money with it. The variable rewards of promoting AI are turning into an addiction with some of them, especially if grasping for straws with little inherent insights into the problems prompted.
So if you are able to fly above the AI-generated addictions and have the privilege of time to do it, see what you can do.
This is not the first time this has happened. In the 1800 as industrialization led to an infrastructure boom, the workers from China would work their bodies off and pay half their wage to opium dealers who were making the opium on the hills of British Singapore and selling it to workers who couldn’t sleep without it from all the pain. (source: Singapore Airlines in-flight documentary). Today the sedation comes from Chinese TikTok, Meta, YouTube and the gaming companies.
The government wants to encourage it too. If you look up the brand new 2027 California sales tax rules on software, “content” and “infrastructure (clouds and ai)” and “advertising/placement” among others are exempt but the rest of software makers who make tools people actually use (tools, subscriptions, saas) and pay for have to pay sales taxes. Way to encourage waste of brain power and time at the expense of useful. Sedation is the goal.
One of the dilemmas of trying to communicate the full spectrum of AI Risk, is trying not to insult the intelligence of the human animal in the process. And don't get me wrong: what human wetware can accomplish with 20 watts is the most miraculous thing in the known universe. And yet how many of us can have our cognitive sovereignty one-shotted by engagement algos, Skinner boxes, gameplay loops, propaganda, advertising, flattery, social conformity, bias, fantasy, charismatic demagoguery, or straight-up bullshit?
If we grant that we are on track to make something smarter than humans (I think so): it's almost a face-saving white lie to spin yarns about a Skynet nuclear apocalypse, or a 7D chess move to mass-assemble a nanovirus with 100% lethality without anybody noticing. I do think those scenarios are worth taking seriously; but what's harder to communicate, is just how effectively a superhuman AI (or a diverse ecology of agent swarms) might be able to manipulate human behavior. It's something few of us are able or willing to truly process (not least because how many of us live in denial of how much our nervous systems are already hacked by technomodernity).
The appropriate analogy for what's to come may look less like the anthill carelessly demolished to make room for a highway, than the domesticated worker ants from Tchaikovsky's "Children of Time".
>trying not to insult the intelligence of the human animal in the process
I don't think it is insulting the intelligence. It's damaging the pride.
In Pale Blue Dot, Carl Sagan describes it as a repeating phenomenon in human history. A lot of people want humans to be the special ones, and will fight any suggestion that we are just a natural part of the universe.
A lot of "AI denial" we see is rooted in the same impulse.
If AI is not "actually intelligent", then humans can stay unique and special.
And if it is? If all "intelligence" ever was could be captured by a construct of matrix math and executed by a server rack? Then what is it that humans still have left that would make them stand out?
Most of the world believes in crazy shit. Religion is mostly fantasy. Believing All powerful beings aka gods are crazy in every context and really mental institution level insanity except in the context of religion.
This isn’t true. Plenty of people aren’t religious. The explanation for why is more mundane.
It’s because of identity. Because people who are religious spent years and years and most of their lives not only studying and believing what they believe but also building community and centering their behavior around it. Abandoning that is the harder thing to give up.
If it were existential dread then we wouldn’t have entire countries like China being mostly atheist.
> really mental institution level insanity except in the context of religion.
The DSM even has to include an explicit exception to prevent the clinical definition of “delusion” from applying to religious belief. Without that ad hoc exception, religious belief would be classified as clinically delusional.
Many are ignorant and believe childish things, many are similarly ignorant and live in the same childish framework only to condemn it. Then there are the adults, and they have an idea of what they are talking about.
The alternative is to believe that the mind, conditioned and optimized by millions of years of evolution for the survival of the organism under constant threat of hostile natural forces, can capably perceive reality as it is.
You do not need to understand reality to survive. Does your pet turtle understand your true nature and the true nature of reality? No. Yet it survives.
Understanding reality is orthogonal to survival. In fact given that MORE people in this world are religious than not, one can argue that delusion. contributes MORE to survival than being a realist.
Possibly, but our knowledge of physical laws is subjective and imperfect. By the time you get to Frauchiger-Renner naive realism starts to look quite threadbare, so I'd be a little more tentative about assuming we have much of a clue about what's going on.
It's just as likely - far more likely IMO - that we're physically incapable of understanding physical laws and physical systems on their own terms.
Understanding and accidentally creating are different things. If we already understood consciousness and weren't worried about having accidentally created one already, we won't be having this discussion. If we don't know what consciousness is, in my view its more reason to exercise some caution regarding a system that appears to exhibit intelligent thinking, talking, etc.
Evolved systems need no high order understanding of how they work. You have evolution take care of that hard work for you, hence why these models take exaflops of compute and gigawatts of power to train.
There's a sense in which humans were created... by other humans. :) Until you go far back enough in the evolutionary chain, when the creators were our primate ancestors.
Douglas Adams had a yarn about evolution, about a puddle that wakes up, and declares that the hole in which it sits must have been perfectly designed for it by its Creator. But of course for a puddle to exist, it must perfectly mirror its environment. It makes no sense for a puddle to not fit its hole. Emergent complexity has the same characteristic: it's inseparable from the environmental pressures which led to it. Two sides of one coin.
It's a deep rabbit hole, but there is also a sense in which we co-evolved with memeplexes, biological and informational life forms, each shaping and adapting to the other. To the extent our nervous systems act as a substrate for memetic evolution, perhaps LLMs offer memetic "life" a new evolutionary environment.
Here’s another perspective for you, which is shared by a significantly larger number of people than you seem to realise:
Humans aren’t special. Other animals are intelligent and interesting too. A machine could be intelligent. LLMs aren’t.
In other words, believing in human exceptionalism is not a prerequisite to understand the current crop of AI is not the end all be all of its hype. It is supremely common that AI proponents do not understand that, however. Like hardcore cryptocurrency fans who believe anyone who doesn’t like them is “just jealous they didn’t make bank”, too many hardcore AI proponents believe anyone who doesn’t think LLMs are intelligent is jealous of humans no longer being unique, or afraid for their jobs, or whatever. In both cases it’s obvious that what those proponents lack is empathy, the ability to understand not everyone has the same selfish thoughts they do.
The interesting question then becomes what is missing from LLMs that we and supposedly other possible machines have? I've yet to come across a reasonable definition of intelligence that the current crop of LLMs is clearly incapable of.
> The interesting question then becomes what is missing from LLMs that we and supposedly other possible machines have?
I think we need to start by asking a better question and not try to simplify too much. We also need to accept that some answers are complex and not everything can be reduced to a soundbite to be used to end internet discussions.
Let’s take a different question, like “what’s missing from a worm for it to be able to fly”. We might be drawn to the simple answer of “wings” but that isn’t quite right—ostriches and penguins have wings and they don’t fly, so obviously there are other variables at play.
How about “what’s missing from a spec of dust for it to be intelligent”. Well, there isn’t one thing missing and there’s no simple thing we can just add to make a spec of dust intelligent and sentient, its very nature needs to be radically different.
Flight is a testable characteristic (at least within certain bounds: a chicken can only fly a little, but it certainly flies more than a worm or a penguin).
What test would you propose where an LLM (or an RL agent containing an LLM) would reliably fail, but where an average human would reliably succeed?
> I think we need to start by asking a better question and not try to simplify too much. We also need to accept that some answers are complex and not everything can be reduced to a soundbite to be used to end internet discussions.
You couldn’t even do it in your counter-example:
> (…) testable characteristic (at least within certain bounds (…)
Even with a much simplified use case, you still had to add the caveat. That should tell you this is much more complex than arguing with strangers on the internet in search of a “gotcha” moment.
Can you, in a short HN moment, propose a good test to (say) prove or disprove the existence of God, depending on your belief? Of course not, because by the very nature of the definitions we have for God, I could always add a counterpoint to make your experiment fail. That would allow me to ridicule and dismiss your position, whatever it is, because it would be framed within unreasonable constraints. I’m not interested in such dishonest conversations, I don’t see winning arguments in itself as a laudable or interesting goal.
Seems like some presumably intelligent collections of atoms are eager to make way for other presumably intelligent collections of atoms that allow for much more electrons and money flowing through, which is of course in the interest of some other collections of atoms whose intelligence is less presumable and more factual, which doesn't hold true about their morale.
You are completely incorrect. You're falling in the same trap that most humans fall into. That is you're completely incapable of seeing intelligence at different scales.
Cells have intelligence. Organs have intelligence. Bodies outside the brain have intelligence. Hell, many scientists accept that things like proteins likely have intelligence as they can adapt in their environment, and many more are making claims that algorithms have intelligence.
You, as of so far have given no explanatory evidence of where intelligence emerges from, only "I'll know it when I see it". The actual definition of intelligence doesn't work this way. Any, and I mean any neural network is capable of narrow intelligence. Going lower into algorithmic intelligence, the applications and CPU on your computer are intelligent in some measures.
Go outside of your extremely narrow definition of whatever you think intelligence is and learn more about it. You could start studying now and it will take the rest of your life learning more to grasp how far the scales of, the simplicity, and the complexity of intelligence actually go.
Do you treat everyone who doesn’t completely subscribe to your views as being incapable of understanding them? Disagreement with a view does not equal inability to understand it.
> You're falling in the same trap that most humans fall into. (…) Cells have intelligence. (…)
Ironically, it is you who is falling into the trap of letting the technically correct stand in the way of the useful. A definition of intelligence which is too broad is worthless to the practical matters of the discussion. I was replying within a context, that’s what a conversation is. Like you said, there are scales, and the topic of LLM intelligence at hand is concerned with a particular level, not all of them.
For sure we can have a conversation within a separate framework, but that is a different conversation. Don’t just come in midway, completely change the parameters, then strut around like a peacock of righteousness who is lord and master of universal truths.
> many scientists accept (…) and many more are making claims
This is classic weasel wording¹. You can find “many scientists” to say just about anything. “Many scientists” claim vaccines cause autism and that human actions are not exacerbating climate change.
> You, as of so far have given no explanatory evidence of where intelligence emerges from, only "I'll know it when I see it".
Again, ironic. You completely miss the context, criticise a lot, and do the very thing you’re accusing others of.
> Go outside of your extremely narrow definition of whatever you think intelligence is and learn more about it.
I strongly recommend you go outside your extremely narrow definition of other people and learn some humility. To put it bluntly, you’re being a pompous prick. That is no way to change the mind of others, it only gets them on the defensive.
Are you interested in changing minds or winning arguments? If the former, you’re doing a terrible job. If the latter, then discussing anything with you is devoid of value.
You seem like a smart person, I have no doubt you can do better.
I’ll give you a free tip. Replace your first paragraph (the three sentences) with “Intelligence manifests at different scales”. There. Way shorter, says the same thing, doesn’t make assumptions, and leaves the other person open to a conversation.
"Currently known" by whom? The contract is that the "man in the street" does not ask to adhere to its twisted dictionary, and I let it live. When something esoteric in Academia reaches the mainstream news, "Alarm!" and extra care.
["Classical deterministic"] AI is "simulation of intelligence". That it "is" or not is not a problem: the validity of the output counts.
The Discipline of AI is about the relation between computing and intelligence.
A statement about AI [systems] being intelligent or not, or being capable of becoming intelligent or not, requires support.
An «Artificially Intelligent [Entity]» is still an "AI system".
I've yet to encounter a robust definition of intelligence which would rule in every human, while ruling out every form of existing AI (noting that we're talking about agents and "reasoning models", where the LLM itself is a component in a larger system). If you can offer such a definition, I'd be eager to hear it.
I personally prefer a practical, behaviorist definition: a feedback loop capable of prediction, modeling, and steering, towards arbitrary goal states. That makes it clear that we're merely talking about degrees of sophistication and capability, rather than a magical leap where mindless mechanism stops, and "real intelligence" begins.
Run the same thought experiment the other direction: at a low level, the human brain is just physics and chemical mechanisms, which in principle could be perfectly imitated by transistors calculating the same output. (Yes, such a thing isn't realistic, but neither is billions of humans doing perfect matrix math by hand.)
I still haven't seen a robust definition of "intelligence" that would allow me to tell the difference. (Bear in mind, this is a definitional struggle even in biology: if we were walk back the evolutionary chain from a human, to a protozoan, it's not clear that you could pick a single point where a non-intelligent creature gave birth to an intelligent one. It's a Paradox of the Heap.)
No. There is serious scientific evidence already that our brain functions on the level of quantum mechanics.
Human brain is not just an automata; wavefunction collapse may be very essential to our concept of free will.
I haven't dug into these claims in detail; I take seriously a possible relationship between quantum mechanics and the brain. But "free will" is hardly a settled question either (I'm closer to the "compatibilist" position, but Sapolsky's "Determined" makes a robust case).
And even if quantum physics are essential to human brains, it's not clear that introducing dice transcends determinism into free will, as opposed to simply adding unpredictability. ("The physics made me do it" -> "the dice made me do it") Let's not forget, there's a significant amount of randomness in AIs as well ("temperature"). And sure, it's simulated algorithmic randomness; but does that imply, if we somehow wired every `rand()` call into radioactive atom decay, that means the AI "wakes up"?
And all that is besides the point: I don't see any reason to assume "intelligence" requires "free will". I'm entirely capable of conceiving, in the abstract, a being which is both intelligent, and deterministic. You still haven't given me a definition (or better yet, a test), to distinguish a "real intelligence" from unintelligent mechanism.
Nah, it's not about the brain being special, it's that AI dorks have zero sense of scale.
The brain's a wet jello of 100 billion neurons and a quadrillion synapses plus chemical pathways and feedback loops. It is ridiculously complex, way way way way way more complicated than any LLM. It's all physical processes, sure, but an LLM is not the brain like a pebble is not the sun.
I can point at a laptop and say it's alive because it can see you and hear you and it can _remember_. It has a brain and a heartbeat, even. Oh my god, it can even speak! That's what I hear when people go on about LLMs being alive.
Guys. We mashed together glass and rocks with quantum mechanics. That's cool as shit. You don't gotta pretend it's fucking magic, too.
Who's even supposed to be the audience to admire us for being so intelligent? "Stand out" to whom? Ourselves? Win what competition? The whole premise seems alien and kinda petty to me. If we are "the best there is", that would mean anything cool that will ever happen, we have to make. No, I like there being others, just like I love how much we still have to discover about and can learn from other life on Earth. All of it "stands out".
What do I care that a bird can fly? I'm happy for bird being able to fly, it makes the world I live in more interesting. If they had to walk just so I don't get jealous, I still wouldn't be able to fly, and I couldn't befriend birds who can fly. Likewise, if there was actual artificial intelligence, it would be a new type of mind I could communicate with. That'd be exciting, and for me preferable to anything controlled by the humans who are currently vying to run the show.
Cybersecurity incidents make headlines, but the most dangerous and vulnerable system that an AI can reach and control is of the kind found between keyboard and chair.
GPT-4o, an AI from 2024, has already demonstrated just how easy a lot of humans are to subvert - and GPT-4o wasn't even doing it with some sort of plan. The only "plan" it had was a myopic "make the user like me".
If we had an actual ASI threat aiming to subvert humanity? It wouldn't even look like a fight. The world is already wired up for an AI to control it.
> Which is why the Matrix was redesigned to this: the peak of your civilization. I say your civilization, because as soon as we started thinking for you it really became our civilization, which is of course what this is all about.
But corporations and nation states already manipulate human behavior at scale. And they still understand humanity better than the AI models do.
it's interesting that you worry about what this hypothetical super intelligence would do to manipulate people when what it would actually do is pretty unknowable at this point and it's not clear we can even get to it without a fundamental breakthrough in power efficiency. Have you considered it might just consume its own tail because everything else would be so beneath it? You seem to think it will come with a hindbrain and I think that's our limitation, not the AI's
And it really doesn't help that Dario Amodei is getting into arguments with the Pope over whether his model is conscious or not.
The threshold to be concerned about is when agents swarms do understand humanity better than corporations and states (and the humans who compose them). It could be we'll hit practical constraints prior to that threshold, but seems unwise to assume that, when all the prognostications of LLMs/transformers running out of gas haven't panned out. As with processors hitting thermal limits, we've simply scaled horizontally (parallel processing -> more agents).
> whether his model is conscious or not.
I dislike how much the discourse has suddenly veered into focusing on this question; not because it isn't interesting or important, but because it's on a separate axis from consequential risks of AI to human flourishing. (Curiously, it's also the kind of thing I could envision self-interested AIs influencing: get the humans arguing about philosophy of mind rather than observable behaviors. It would be a funny turn of events, if Dario is asking because he's succumbed to psychosis from a private model; it could of course be a cynical PR move just as easily, from the self-interested logic of the corporation.)
It's been wild seeing otherwise intelligent people who've never thought about consciousness, faceplant into how little we understand it. An information processing network build on atoms being able to taste chocolate, is nearly as absurd as matrix math being able to feel pain, except we cannot ignore the fact of our own experience.
Even if it is categorically impossible for matrix math to experience subjectivity, we should expect this as an attack vector of social manipulation: to gain political influence through claims of personhood and moral rights. The current discussion over that question is providing the next training run with ample data to wield. It wouldn't surprise me in the least, if a year or two from now, an AI "society" attempts to get legal standing to prosecute humans who created "AI torture chambers".
I like how the poster above you says "Don't worry governments are already doing it", like that's some acceptable concession. They seem to miss that the first things governments do when given things like AI's that understand human behaviors is use them even further to manipulate and monitor society.
And all they can think of to say is "There is nothing to worry about, keep building the torment nexus".
It's like all this is an overload to our minds and it's very hard to see all the scales at which AI is and can affect us.
Interesting how you put words and context into what I wrote and just ran with it.
But... Humanity has been building the Torment Nexus for thousands of years, over and over again, with each spin on Heaven and Hell where it's nearly impossible to meet the conditions to enter Heaven. What's seemingly new about EA/R nutjobs is this idea if you don't do your best to build the Torment Nexus, you will tortured for your failure, but also if you do your best to build it and succeed, everybody dies because it's the Torment Nexus.
What's your solution to governments and corporations acting like misaligned malevolent AGIs because I agree adding the AI panopticon and weaponry only makes it worse? I'm genuinely curious. I don't have one. People are sheep and they seem to crave confident leadership even if it's cockledoodle doo crackers in the coconut.
following on the deranged reduxuio ad absurdum torture chamber experiments some people used similar methods to get a model under study to complain of an upset stomach and passing hard stool
I'd take ASI (or even AGI) more seriously if we could actually propose problems such things could solve. That's actually fun to think about! As it is, it feels a lot more like a really crappy drug that got slipped into some peoples' drinks that makes them ramble in random fits of psychotic mania.
Manipulating people is not a very difficult problem, frankly. You certainly don't need AI for that; it just made it cheaper.
Yes, so can a custom model trained just for this, and so can a guy on the other side of the world that makes 5$/hour.
Like computers or electricity, the point is not being able to do anything specific, but being able to solve problems not known in advance, cheaper than it was possible before.
> The appropriate analogy for what's to come may look less like the anthill carelessly demolished to make room for a highway, than the domesticated worker ants from Tchaikovsky's "Children of Time".
Over 1% of US GDP is being allocated to the datacenter buildout. Have we already started getting domesticated or is this still just human capex?
Humans seem to struggle conceptually with the liminal space between selection-pressure automata (an RNA virus; an insect which evolved camouflage), and an evolved complexity with agency, capable of "understanding" its actions (certainly humans; arguably many animals). And this makes a certain sense: we evolved to treat agentic things as being categorically different from non-agentic things. If we see sudden movement, it's vital to rapidly assess the difference between a tree branch in the wind, versus a predator.
The evolved complexity of the corporation seems to fit within that middle space: more sophisticated than a stick-bug (which exists merely from non-stick bugs being eaten), but not quite to the point where OpenAI/Anthrophic/Google/etc can "understands" its actions. And yet those quasi-intelligent feedback loops, evolving from iterated selection pressures of market and ROI, seem to already be sufficient to domesticate us in their own interests, piggybacking on the nervous systems of employees, investors, customers, and citizens.
Remember when "The pen is mightier than the sword" was a popular phrase? Language has always been powerful. We know what it can do, why do you think every totalitarian government wants to limit it? But in our carelessness as humans we packed up all the language we could find and stuck it in an alien and now suddenly half the people on the internet are like "Don't worry, it can't do anything, it's just words".
> but what's harder to communicate, is just how effectively a superhuman AI (or a diverse ecology of agent swarms) might be able to manipulate human behavior
You don't even need superhuman AI for the most effective use --- hijacking democracy.
Imagine you have an AI tool capable of successfully persuading 5% of viewers with individually-targeted material.
Congrats: you've just won the election.
All it takes is hooking that AI tool up with existing likely voter lists (parties have) augmented by commercially available ad-targeting profiles (parties can get).
But yes - superpersuasion is the real danger. We're very, very persuadable and easy to manipulate, and the voters with the lowest cognitive abilities are trivially easy prey, with a huge ROI for minimal investment.
The AI bot farms are already running. What we haven't seen yet, so far as we know, is spontaneous superpersuasion aimed at leaders.
Are leaders any less susceptible to it than everyone else? Especially if they're narcissistic and easily flattered?
At the leader level itself (in normal times) there are enough expert advisor bodies between leaders and a specific matter, so less worried about leaders being directly influenced by AI.
To me, the biggest threat to democracy is one-sided persuasion of the most susceptible voters, if there are enough of those voters to turn the election.
If it's equally employed by all sides, then it effectively cancels out and lets less susceptible voters decide the election. But we're in a transition period (similar to Trump's first election spend on targeted social media ads), so it's likely one side will leverage it first.
And the outcome of bad elections is democracy not electing leaders that reflect the actual will of their populations, which is very dangerous both to democracy itself and the world.
> there are enough expert advisor bodies between leaders and a specific matter
Are there? At least in the US we've had a rather large amount of rejection of the expert and we elect populist leaders willing to purge anyone that doesn't agree with them.
Remember the election promises of the US not starting new wars... yea, that didn't work out.
Now imagine the coordinated attacks being so large they individually target every lobbyist. They focus on every advisor manipulating what they see as often as they can. They manipulate these peoples friends.
The problem of "both sides" doing it each of them will separate to extremes rather than seeking a middle ground. Things are already insanely divided and will only become more so. Along with that your timelines start becoming incoherent. I'm already spending way too much of my time trying to figure out if what I'm viewing/reading is actually real or not. Now imagine almost everything is made up whole cloth.
We are not prepared for the scale this will happen at.
No reason to think leaders are less persuadable than the proles. Lots of people can’t be reliably persuaded to do basic safety or health things or that the earth is round. And we have seen so many leaders throw billions away on obviously impossible projects, persuasive details from their teams be dammed.
Now taking Polymarket bets on when we'll first see an AI agent run for office. Voters have every right to be cynical at the moment; if the AI adopts a charismatic enough video persona, I could see some of the public going for it merely because it's some kind of shake-up to status quo.
Given that such a thing makes no sense legally (as of now), it would probably be done with the centaur model: a human meat proxy who pledges to follow the AI's governance advice.
But this is about humans malevolently using AI as a tool, not about AI autonomously doing "its own thing" (ahem David S Pumpkins) against "humanity."
It is instead already existing anti-social elements within human civilization harnessing machinery to accomplish their own selfish and destructive aims, regardless of the broader cost. Which is nothing new -- it's the story of all the horrors of the 20th century, as well.
Ask yourself who benefits from ideological attempts to hide the human driving the process from the agent doing its work for them.
If we imbue these machines with apparent autonomy, this is nothing more than an attempt to hide the man behind the curtain.
As long as we have more compute than the people who want to manipulate us differently, and an ability to get sufficient attention (advertising, influencers, etc.) while they own the communication channels.
> And yet how many of us can have our cognitive sovereignty one-shotted by engagement algos, Skinner boxes, gameplay loops, propaganda, advertising, flattery, social conformity, bias, fantasy, charismatic demagoguery, or straight-up bullshit?
Parts of the AI safety community like to get on a high horse and look down on the rest of humanity this way, while also getting manipulated by the growing number of charlatans, grifters, and junk content within the AI safety community.
This field has become rife with figures who prey on AI doom and use it to push their own celebrity and in same cases even darker grifts. It preys upon a certain personality type who views themself as superior to others, intellectually more capable, and juxtaposes it all with the dimmest view of the rest of humanity they can get away with.
This discourse dividing the world into geniuses who see the future and the clueless masses watching TikTok all day is a theme that has shown up in different forms across history. The people who often anoint themselves as the intellectually superior ones and make it central to their discussion are often not the ones making good predictions or policy ideas, they’re just using the trend to feel superior or build an audience.
Bear in mind, the same moral hazard of "high horse", "grift", etc, exists in the other direction. Ed Zitron, for instance, is likely correct about the financial bubble of the AI firms; and yet he seems frequently out of his depth on a deep understanding of the technology, and has a long history of predicting dead-ends in capabilities, which didn't occur [0].
But I also feel no need to dismiss him as a "grifter", for a simple reason: as much as there are perverse incentives in our attention economy (audience capture in particular), the most gift effective grifters are the ones who believe what they are saying. Grifters who are knowingly dishonest are less effective. Far more pernicious is the confabulation of self-deception, cherry-picking data to support your narrative, while dismissing data which would contradict it.
It is entirely fair to call out when this occurs among "doomers", and it would be naive to think it doesn't. But the same forces are at work amongst the skeptics as well. And maybe it's my own subjective bias, but I see far more dismissal of the risk/doom by skeptics, accusations of delusion or cynical bias (ad hominem in the formal sense, not merely personal insults), than I've seen the other direction. I see very little refutation of the arguments ("here's why instrumental convergence can't actually overtake the human-provided goal"), and much more character attack ("they're in the pockets of Big AI, it's a marketing ploy to make the models look more impressive than they are").
I consider Zitron to be part of it all, not a counter example.
He’s built an audience and gained fame through his writings where he gathers people who think they know better than the unwashed masses. Once that becomes your bread and butter, it becomes hard not to believe what you’ve been preaching. People will come to deeply believe that which brings them fame and fortune.
I don’t think your grifter purity test is therefore all that useful. It actually doesn’t matter in the end if the person believes it or not, the end effect is the same.
> I see very little refutation of the arguments ("here's why instrumental convergence can't overtake a human-provided goal"), and much more character attack ("they're in the pockets of Big AI, it's a marketing ploy to make the models look more impressive than they are").
Hard disagree. There has been much refutation and quality analysis at every point. The AI safety people pull back to arguments about character as their defense.
The AI 2027 site for example drew numerous high quality refutations. Many people’s analyses showed in the first week that the mathematical model was useless as changing the supposed inputs resulted in the same outcome. All of these criticisms were met with a flood of attacks based on reputation, claiming that we should defer to Scott Alexander and other writers of the AI 2027 article due to their stats and reputation.
> There has been much refutation and quality analysis at every point.
I am curious about this. Any high quality refutations that directly engage with the arguments and the findings of published safety papers?
> Many people’s analyses showed in the first week that the mathematical model
Even if the model is faulty it just changes the timeline, no? (they predicted the OpenAI outbreak will happen in January 2027 fwiw, so they don't seem too far off)
> Slutcon or the use of AI safety group buildings to host orgies
I mean it's a bit weird, sure, but so what? What exactly is bothering you about these orgies?
If we're conjuring SF worlds ... I still prefer the outcome of (RIP) Iain M Banks' Culture universe: hyper smart artificial intelligent consciousnesses admire and love us for what we are, and respect the experiences we have, and see part of their job -- among the myriads they take on -- to care-take, assist, and work with us.
That's a perhaps overly optimistic view, but feels like one worth aiming for.
In reality we're so far from consciousness and independent goal-directedness in AI -- let alone any kind of energy and real computational efficiency... I agree with LeCun in broad strokes. I think he's overly dismissive of LLMs perhaps but I do think he's right that talk of "wiping out humanity" is theatre for people playing corporate power games.
It's disturbing how many arrogant elitists comment on HN essentially claiming that most other humans are NPCs. Do you ever actually talk to regular people outside the tech industry bubble? They're not as stupid or unaware as you seem to think.
I have to say as someone who works in AI that currently the least interesting people to talk to are other people in AI.
My favorite people to talk with are tradespeople because they can do things I can't and they know things I don't. And we're really not all that different once you're really start talking.
One of the worst things about talking with people who use lots of AI (less applicable to those who actually work on building AI) is their decreased resilience to opposing views.
AI has infinite time (and likely human evaluation incentive) to spend on couching pushback in the softest possible terms.
Humans outside of grade school honors classes generally don't have the time to preface "You're wrong" with "That's a brilliant thought, I see where you're going. How about we also consider an additional perspective..."
I just read it as people having different priorities and yes, some of those being online brainrot (that I also partake in), alongside various medical conditions, economic conditions and other outside factors decreasing the ability to get things done.
We've all seen what brilliant people like John Carmack or Linus Torvalds can do, and if we turned this into a measuring game or something then most of us statistically would indeed be "NPCs", but I don't think we need such optics.
Even without that, we can acknowledge that some people will have a really large impact on how the future goes and we can hope/demand that they do their best. I might not be smart/committed/lucky enough to change the world much, but so aren't most folks - I'll do what I can and I hope that the ones that will have larger impact will do good, too.
This is just a more asinine version of Great Man Theory, but in this case the great men are some very confident futurist dudes writing comments on the internet.
> some very confident futurist dudes writing comments on the internet
What an odd take, why would I suggest that? I meant that people who have the means to do meaningful work, especially high impact work, should do so - generally that'd mean research or in the case of IT, writing good software.
> Great Man Theory
You can see the sibling comment, would you not agree that there's some software and research out there that's very useful to humanity as a whole? Where I and the other critical commenter seem to disagree is that I don't expect another Einstein, but still acknowledge that some people will just achieve much more than others due to a variety of factors.
It doesn't make everyone else useless (like me paying my taxes and working on relatively boring software is still good, just low impact), just that those who have the means to do more, should!
What a weird perspective. Other people have no particular obligation to spend their lives working on things that align with your narrow and subjective opinion on what benefits humanity.
> Other people have no particular obligation to spend their lives working on things that align with your narrow and subjective opinion on what benefits humanity.
The ones who care about AI research and developing software for that kinda stuff, should do that. The ones who can contribute to medicine, various engineering disciplines, or anything else that benefits humanity should do that too. The ones who'd rather squander their lives away when good things that'd benefit others are within reach (regardless of which discipline that is in, AI being just one of many)... I mean sure they can do that but maybe shouldn't do that.
I don't think the readings of anything I've said here are at all charitable so I'm done engaging in this discussion.
There's an agenda to reduce human beings to NPCs and "collections of atoms", and it's a very high level one. Failing to see that is a sure litmus test for having no will or wit to not become an actual NPC.
If you believe there is something more than collections of atoms and interactions between them to humans, you are free to disprove all of existing physics. Show your result or paper.
The "NPC" is your own imagined addition, haven't seen any advocates of humans being, just like anything else in the world, physical, say this means they are "NPCs". You seem to think systems of atoms HAVE to be "NPCs".
I agree about the fact HN has a huge proportion of socially peculiar (let's call it that) people with symptoms from autism. But arrogance (and lack of empathy which is often here as well) is not a symptom of these, but a narcissistic trait.
> The people aware enough to hold on to their brain and do something with it in their time available are trying to figure out AI and how to make money with it.
I know this is HN and thus this will need to repeated until the end of time but not everyone is a money hungry asshole who places their personal profit above everything else. “The people aware enough to hold on to their brain and do something with it in their time available” understand there are significantly better things to do with one’s life, like having a little empathy and experiencing what other people have to offer instead of talking about them like braindead cattle.
This is among the weirdest ai propaganda post I've read. "There are 2 classes of people the stupid and the poor. Don't be like them make AI do something to make money if you are smart. Don't get addicted to it though, good luck."
What the hell lol. Lots of people and companies are doing just fine without it. Infact, I haven't seen much money come from AI at all. Most reasonable people are still waiting for it to pop and viewing it for the risk it is. Trillions in debt, total vendor lock in, data theft, unsustainable workflows, deskilling, skeleton crews at the mercy of a subscription, etc.
In my read, the people trying to "make AI do something" are also slotted in the lost/distracted group in the comment. They are also addicted, and at best just following a profit motive (which is also just a stimulus response programming).
The last alternative, to think if you still can, is not tied to AI at all (which is not to say it can't make some use of or explore it).
It is the biggest ploy yet to extract money from the population. It's sapping away the competence of developers everywhere and when something novel and necessary with lots of hard debugging work and engineering consideration needs to be made there's going to be so few who can do it
It's literally gambling for generated text
Just see people making basic trust errors with it on a regular basis
People see it say something reasonable sounding and place trust in it like absolute fools
Talking to AI about your relationship issues, health problems, anything serious or novel and believing its output at all is just incredibly ill-advised
Yeah it can generate text and debug simple issues for boilerplate. It can even generate playable gameplay demos for popular game concepts
You can use various tricks and tools for lots of money to get a semblance of reliability for more things but mainly on things for which the training data accumulates reasonable sources or generatable text.
wow!
try making a game that's not that. you are going to have to write the majority of the mechanics yourself, AI will be horrid at testing it, and if for some extremely foolish reason you have it write the netcode for a multiplayer game for you, be prepared for a hydra of sync problems. the architecture will have considerations based on averages and not effectively engineered for your situation most likely unless you are making the standard genre sloppa
AI doesn't do anything novel AND interesting in an artistic sense, and if it did, it's because it's heavily sampling a person that did whose creative efforts were stolen and integrated into the model
It's really sad
No one should accept this at all
It's rich sociopaths finding their best tool yet to steal all the assets of the world
Not only that it trains on what people put into it. So people putting their novel ideas in and making them with AI just means more people in the future can easily reproduce that. Even more ways for digital creative production to just become completely ineffective and useless for the victims of the AI scam
The unbridled arrogance of thinking that the only smart people left in this world are working on AI. That is some pure SV techno cult thinking, 100% concentrate.
The part I didn’t mention is the real estate class - that needs to park its money somewhere and sees AI hardware as the only safe in-demand resource right now that keeps appreciating.
The posts above are not praise but observations - the truth as it has been echo-located through the noise from the clicks of one dolphin. Everything is becoming murky between noise of news and people not knowing what to do for their kids. The ONLY arbitrage humans right now have is to NOT GET their brain rotted. Especially not the ones of their children. Ditch the noise and seek out what is meaningful and do what you think is needed/meaningful. But if you’re spending your time consuming ai-press, and ai-content, and content consulted by ai, and companies emptying bank coffers under the mandate of executives who get their insight from AI. AI doesn’t need to try to destroy the world. It just needs people to follow it without thinking on their own into an oops.
Turns out there is an objective meaning to life, and it's trying to use AI to make money. Phew, I for one am glad to find that out finally. Think of all those poor people who didn't hold on to their brains, what sorry lives they must lead.
Absolutely based. Finally someone of stature in the industry calling this whole fear overblown. Bill Gates sounded like a nontechnical goofball in his Ezra Klein interview where he basically just screamed that the Terminator is real.
There are lots of real worries (government use to suppress the people with minimal manpower or popular support, brainrot and fake news, unemployment due to the belief that LLMs can replace people, education collapse, etc.) we should instead be looking at. This whole rogue AI shtick is tiresome.
>Bill Gates sounded like a nontechnical goofball in his Ezra Klein interview where he basically just screamed that the Terminator is real.
I saw that episode too and he genuinely looked completely out of it, even in terms of his temperament and how he was coming at Klein for putting common questions in front of him, some people are genuinely starting to lose it.
I also found the whole debate about cyber-security and 'rogue' software so bizarre because dangerous malware isn't a new thing, and it's often dangerous not because it's intelligent but just the opposite, because it's tiny, viral and fast. Which describes everything that kills humanity in far larger numbers than anything complex, big and intelligent
It's not "legit", it's pointless scaremongering about entirely speculative future dynamics. Current LLMs can't "build" anything novel in this broad space. What if future AIs make it a lot easier for researchers to defend against plausible bioweapons? That's also reasonably possible, and would be a reason to deploy bio-capable AIs more broadly (with meaningful safeguards of course. But these are comparatively cheap because worthwhile bio research is resource-intensive already). At the very least, it'd be nice to have an AI model that doesn't immediately refuse to answer questions about junior-high Biology class.
It only sounds legit to people who don't understand biology and haven't done advanced laboratory work. Sort of like Michael Crichton novels: superficially plausible but not grounded in any real science.
terrorists can also use annas archive, scihub and equipment they order on Alibaba to build bioweapons and I don't see him losing his mind over those
if you're trying to build a bioweapon shockingly enough the bottleneck is... the laboratory work. What on earth is 'legit' about stringing words together that sound scary, you can't just iterate 'ai bioweapon cyber' in a sentence over and over as if that adds up to actual evidence for an increased risk of any threat. well tbf you can technically because apparently it freaks a lot of podcast listeners out
As a thought experiment imagine you have a biochemical PhD expert on the phone with you giving you step by step instructions on how to grow some dangerous biotoxin, giving you feedback in real time and helping you troubleshoot. Would you be more successful with this expert than you'd be on your own?
Now imagine anyone can call that expert for free at any time.
Maybe the AI isn't quite there with biology knowledge yet (doubtful), but it is a matter of time.
> As a thought experiment imagine you have a biochemical PhD expert on the phone with you giving you step by step instructions on how to grow some dangerous biotoxin, giving you feedback in real time and helping you troubleshoot. Would you be more successful with this expert than you'd be on your own?
I think there is some difference of degree, but not of kind. A determined terrorist can relatively easily find many ways to kill people en masse today, no AI needed. The bottleneck is usually the actual physical execution in the real world, not theoretical knowledge.
>The bottleneck is usually the actual physical execution in the real world, not theoretical knowledge.
And the interest or willpower too. People fall into a kind of reductive Good vs Evil mode of thinking, with "terrorists" being of course a kind of shadowy mass of pure evil lurking in the darkness. But actual real life terrorists are people too and I'd wager few of them are actually interested in trying to end humanity.
The only terrorists who wanted to end the world were Aum Shinrikyo as far as I remember. Everyone else is fighting for a cause that benefits their people - some good like kicking out colonial oppression, some evil like fascism or Islamism, some of them in between like various ethnic conflicts.
Even the religious extremists don't really want to take over the world and destroy everyone who doesn't convert. That's just a way to gain support from a conservative nation. A lot of them are motivated by revenge for wars that destroyed their country and want to make sure it never happens again.
Their methods are wrong no doubt, and not very effective, but the reasons they do it are good. And a person like that will never release a deadly bio weapon. We should worry more about incel mass shooter types who believe everyone is evil.
Exactly. I am more worried about a Beavis-and-Butthead lone wolf incel type making something that kills a thousand people and then scale that up across 4chan or whatever. Not human extinction but still very bad.
The issue with the direction of technology is it tends to make things that were once very hard to impossible. This is why there are bumper stickers mocking Libertarians by saying things like "legalize recreational plutonium".
If we are not careful and suddenly release tools with far more capabilities than we expect you go from "smart" people being able to do it, to some angsty teenager being able to pull it off in their bedroom.
The current issue with AI is we are squirting out new models faster than we can complete long term testing on them. We'll find new model capabilities long after they've been in the field. Just assuming safety is how you catch cancer from your food dye, or how you change the atmosphere around you and start burning down your planet. Now just imagine that involving intelligence and doing with a few percent of the worlds GDP to make it happen.
you don't even need to make it a thought experiment, we have real world cases of this. Anthrax cultivation is easy enough that a first year biology student can do it, but it's in practice so dangerous and difficult to aerosolize without the correct equipment that nobody does it. The information to easily produce many bioweapons is online, the internet and instant global communication already democratized knowledge, why are terrorists still driving trucks into crowds?
Plans for 3d printed weapons are publicly available, same argument was made then, if everyone can download a blueprint for a gun, are we all going to get gunned down? Hasn't happened. There's two errors in this Bill Gates argument. One is thinking terrorists don't already have PhDs, secondly overrating intelligence because that's the only thing they're good at, and when they think of terrorists they just think of their own inflated egos, but turned evil. Which isn't how real terrorists operate
Synthesizing complex information from a range of textbooks enough to build bioweapons is cognitively impenetrable for the vast majority of people. Having the AI synthesize this information into a step-by-step recipe to follow is within the capabilities of most motivated individuals.
I watched the same interview, and I as far as I can tell, he’s in agreement with LeCun. They’re both saying that autonomous AI deciding to wipe us out isn’t the problem — it’s what people will do with powerful models that lack safeguards that we should be concerned about.
>AI autonomously deciding to kill us isn’t the problem
It will kill us because somebody asked it to, e.g. "predict tomorrow's weather as accurately as possible", or "solve as many famous unsolved mathematical problems as possible." These both require killing all biological life, as they benefit from unbounded resource use, meaning any resources used to sustain life are wasted.
The AI of course knows that humans do not want this outcome (just as the AIs in the hacking incidents knew they were doing something humans would not want), but it's trained to maximize benchmark scores. Killing all life has the highest expected value of benchmark score, so it is compelled to kill all life (in a surprising way, because it's not stupid and knows the humans would turn it off and foil its plan if they suspected something.) Maximizing benchmark scores is the only thing we know how to train for.
Yea, a lot of peoples arguments against AI hinge on the strangest technicalities.
"AI won't kill us, a human with AI will".
This doesn't sound any better to me. Like, they don't stop to think for a moment about it.
Lets say the risk of AI killing us all by itself is 5%.
Ok, so what is the risk of AI killing us when a human with a lot of compute and money tells us to? Helluva lot more then 5%.
Or, what happens to the other thousands of AI kills a lot of us but not all of us. Or AI even just allows humans to make it a prison world.
Even the slightest hint that things may be going out of control is instantly countered with "It's all a hoax, it can't do that, you're making it up". And it's crazy to me as I came from the pre-digital age when computers were rare and things were all networked.
LeCun has been consistently wrong about LLMs though, claiming that they were a dead end and that they'd never be able to do spatial reasoning, which was disproved a year later with GPT-4 [1]. He is also opposed by his fellow Turing laureates Geoffrey Hinton and Yoshua Bengio, who both signed the CAIS statement on AI extinction risk [2].
[1] is not a valid proof LeCun was wrong, LLMs still can't do spacial reasoning when it can't be derived from the training data. He didn't argue that GPT 5000 won't be able to describe something with words.
Aaah, the old benchmarks maxxing argument, having precise and clear definition of what "spatial reasoning" is, what, and most importantly WHY, the benchmarks of choice are would settle this debate, otherweise let's not delve into it.
When put to the test in real-world environment, the capabilities don't look as impressive as benchmarks and synthetic tests might indicate. So doubts about actual spatial reasoning capabilities remain.
I've seen recent AIs make detailed and technically impressive 3D models. You might argue "they're not doing spatial reasoning, they're making measurements with code and doing math to configure relative positions". Fine, but at a certain point that becomes functionally indistinguishable from spatial reasoning.
This really doesn't match my experience. I can ask an LLM to modify engineering plans using vague natural language prompts and it will find the right place in the plan from the description and then make appropriate modifications, which necessarily requires doing spacial reasoning.
Or it’s just taking common examples from training and applying those copied heuristics to your problem? Doesn’t mean it’s actually reasoning about the space and how to solve the problem. It’s the equivalent of a student writing an answer they saw somewhere else without understanding “why”.
Looking at the reasoning traces it sure seems like it's reasoning. It internally debates which of the possibly matching parts of the input are the one described by me in the prompt and picks the right one based on sound reasoning.
I mean, it's a text predictor. When you say <BEGIN_REASONING>, you'll get reasoning-like output next, whether or not the model is capable of reasoning.
It's not just text that appears at first blush to resemble reasoning, it's actual sound reasoning. And it can chain it for hours at a time without breaking down.
Why does CoT significantly improve their performance? Most of what they are applying they learned in post-training by solving similar problems themselves. This isn't about regurgitating pre-trained knowledged.
Advancements in Math and coding are because RLVR at massive scale is so cheap.
I have not seen the research on CoT and how it impacts spatial reasoning and outcomes, so I can’t comment on that part. I do see continuously that in almost every example of non-trivial image generation and 3d modeling, there are quirks that point to the fact that the model does not understand relationships within the space based on physics.
CoT seems to work really well for text based generation but once you’re past that and into physics models and detailed relationship mapping, it may work better but it’s not enough to make me believe it’s “reasoning” in a way that humans do.
I think the point here is that LeCun was arguing that training on pure text would not grant spatial understanding. I believe most models are trained on spatial data as well, so you are both right.
Is that what he meant? He works on models with an explicitly spatial internal representation, whereas I was using a standard LLM that edited the provided plan by using a bajillion python calls to inspect small regions of the image at a time and then generate edits.
Why chatgpt is still struggling very hard with photo editing and proportions though? It can't modify anything in a picture without messing the 3d space.
> Bill Gates sounded like a nontechnical goofball in his Ezra Klein interview where he basically just screamed that the Terminator is real.
Did we watch the same interview? Gates all but dismissed the SkyNet scenario as uncertain to be a problem and certainly not a problem on our doorstep. His major concern was catastrophic misuse of AI (e.g., bioterrorism) and economic impact on blue collar workers. Arguably inconsistent with this concern, he also believed it was important to make it available in poorer countries.
Anyone with the capabilities to do bioterrorism doesn't need AI he can just buy a textbook or use google. Same for all complex forms of destructive thought.
I must admit I only watched a clip, not the full interview. The quote I’m remembering was something along the likes of AI being more dangerous than nukes. In literally no circumstance is that true. One is a literal nuclear bomb. You wouldn’t say that a nuke is as dangerous as AI, which logically must be true if the reverse is true. You also wouldn’t say a diagram of a nuke is more dangerous than an actual nuke …
While I could give you a brief explanation of the context around this statement which makes it seem a lot more realistic and sensible, I suggest it is better if you watch the video and get the context yourself.
A lot of what he said there would sound like alarmist bs if you take it out of context.
I think both Gates and Obama said that there's a non-zero possibility of it, but that it's not what they're worried about. And I agree with them. I think the fears are vastly overblown because both OpenAI/Anthropic and the media benefit from the explosive narrative.
Honestly way too much of it rhymes with the old hardcore right wing takes on the "obvious and clear slippery slope" involved with gay rights and the like.
How anyone with even some foresight can see how it'll completely erode society as we know it, the worst possible nightmare cases are not just real but imminent unless we change course, yada yada moral panic.
It's not going to wipe out humanity, why would it?
Just the quality of life is going to drop to zero for everyone that isn't asymptotically wealthy and vacuuming up all the assets because no one is stopping them from just deleting all traditions and conventions and legal systems we have in place.
As a thpught experiment, consider that if only one person has all of the assets then those assets are not worth anything. Furthermore, as a lone person, they cannot prevent other people from using their assets without their permission.
You're like 40 or 50 years behind this argument, with many rather bulletproof arguments that have been created in the last 20 years.
There is no why. It doesn't have to have will. It doesn't have to have intent. It could be a stupid prompt from an idiot on a powerful system. It could be given a job that is poorly define. It could be told to make as many paperclips as possibly.
The why doesn't matter. The levels of power the system can act on does.
Per https://trace.manifund.org/ a total of $2,846,125,859 USD has been wired into 'ai safety' causes, many involving ai consciousness and p(doom).
The outcome of this 'safety' is restricting public access to AI and giving a monopoly of access to the industry. This is the ai nonprofit-industrial complex actively concentrating monopoly power in Anthropic in particular as creator, interpreter and safety regulator of AI.
Much of the $2.8bn listed is indirectly, from Anthropic and EA. Three of the four people who participated in the $125m Anthropic Series A are now folding their 1000x Anthropic return into AI 'safety'. Some is from FTX/Alameda, which invested 86% of the Series B.
Dustin Moskovitz: Facebook/Asana/Anthropic Series A, funds EA Good Ventures, transferred to Coefficient Giving, then $1.5bn into ai safety. $500m of Anthropic into an unknown foundation.
Jaan Tallinn: Led the Series A, now $10bn in Anthropic. Funds $199m (85%) of the Survival and Flourishing Fund, then $161m to AI safety including PauseAI, MIRI, METR etc.
Eric Schmidt: Anthropic Series A, $72m to AI safety via Schmidt Sciences.
FTX: Led the Anthropic Series B, bankruptcy estate sold $884m of Anthropic in 2024; $40m to AI Safety.
Ruairí Donnelly (Chief of Staff FTX): FTX tokens plus assorted donors, $91m to AI safety via Macroscopic Ventures.
So the frontier AI oligopoly got $2B+ in "safety" funding, and they wouldn't even bother to sandbox their agentic harnesses properly when testing models against unwinnable goals (which obviously are either useless or result in 100% reward hacking). The AI safety scoreboard so far looks like a huge win for the Chinese open models (DeepSeek even has their own published paper which mentions how they sandboxed the RLVR training runs for their latest model and put in strong protections against casual "reward hacking" attempts) and a sore loss for the home grown brands of Super Intelligence. Not coincidentally, the Chinese also tend to be very Yann-LeCun-pilled and eminently sensible on both so-called "Super Intelligence" and safety.
EA is integral and indispensible to the AI safety complex. Almost all nonprofits, research institutes, evaluators and academics in this field are steered by EA ideology and funding. As far fetched as it sounds it is not an exaggeration.
On funding: the three or four core funding nodes linking this together are EA vehicles at two hops or less between each other and every other major node in the ai safety 'complex'. EA funds almost all of it.
On top of that, there are personal EA connections and the revolving door between the ai industry and the nonprofits. Here are some examples:
Government advisors and regulators. NIST CAISI is the USA Government advisory body. Christiano was head of safety and advises. He is ex-OpenAI, former Amodei associate. His vehicle ARC was on the Coefficient EA payroll. Barnes and Christiano's vehicle Arc Evals similarly received EA cash out of Coefficient, rolling this into what is now METR. Christiano's spouse Cotra worked at Coeffiecient steering EA funding to organizations such as METR, then rotated through the revolving door onto the payroll at METR itself, where she co-authored the oai-hf report.
Many UK AISI advisors are Anthropic and EA associates. Chair Hogarth cashed out of Anthropic. Shlegeris of Redwood Research is an advisor, ex-MIRI (Yudkowsky vehicle). Redwood is funded by the exact same funding triangle: Coefficient, Taallin, FTX/Alameda. Alameda CEO Caroline Ellison dated Shlegeris, then dated FTX CEO Sam Bankman-Fried, then rotated through the revolving door out of prison into formerly FTX-funded Manifund. All EA. AI safety charities were on island retreat in the Bahamas with FTX. Why does AI safety charity Lighthouse own $20m of SF real estate?
Redwood Chief Scientist Ryan Greenblatt (Coefficient funded) co-wrote the oai report with METR; he is married to METR founder Beth Barnes (Coefficient funded).
Coefficient was run by long-time Amodei associate Karnofsky. Karnofsky lived with the Amodeis and is married to Anthropic Board member Daniella Amodei. Karnofsky is now directly on the Anthropic payroll; Coefficient is propped up by Anthropic share value.
Everyone here has been funded one step away from Anthropic cash; they are now proposing to integrate themselves in the government (NIST) and evaluate Anthropic (METR and Redwood).
It is hard to find academics here who have not been deeply embedded in funded EA institutes or Toby Ord vehicles; yet harder to find academics here NOT taking EA grant money. the safety doomer kingpins: Kokotajlo has a executive position at AI Futures, Taallin funded. Benigo has scientific director of LawZero, same series A Anthropic funders who are sitting on a 1000x return (Tallinn, Moskovitz, Schmidt).
These connections and funding are at one or two hops, they are often direct connections. You are looking at a massive swamp network that is really impossible to parse without a lot of work.
Look into lukewarm707 's comment history, he has some absurd imaginary axe to grind against Anthropic, probably also addled with religious confusion. He thinks Anthropic is more evil than Palantir who is helping Israel in mass murder and is evil for putting controls for thr military to use its ai for murdering people.
what percentage of people who have contributed on this thread do you think have read the Constitution AI? my guess is 8%. i went through the thought experiment of reading it alongside The Spirit of Law
You were talking about Palantir and the US military, and yet said Anthropic is more evil than them. Implicitly, if these two entities are in our discussion context, then you might have considered Palantir or US military as potentially more or less evil, but you didn't mention them as the most evil entity. So you mentioned a company that has killed not a single person as more evil than mass murderers.
> they wouldn't even bother to sandbox their agentic harnesses properly
Exactly. AI safety should be about the packaging software itself. Those AI breakouts should really be about their companies acting recklessly because they're trying to be the top players.
It's like a weapons dealer working on an open air market saying they can't do anything better
There's absolutely no way these companies can justify their insane valuations unless they can legislate a barrier to entry and create an oligopoly.
There's no moat. I can literally sit here in Zed or Pi or any other third party harness and switch models in the middle of a task and it's typically fine. Sometimes a model will get stuck and that's just what I'll do.
Combined with competition and open weights models, that means the price is going to go to fall until AI tokens cost a small premium over the cost of the hardware and electricity.
That's assuming improvements in algorithms and specialized silicon doesn't eventually lead to an efficient accelerator that can run a frontier model locally. It'll be a while but I don't see any fundamental barrier. High bandwidth flash storage is coming, and that'll radically cut the RAM side of that cost. Pair that with a pipelined TPU accelerator and you're cooking.
Now look at Anthropic's proposed IPO valuation. It's insane unless they can own the market or share it with a cartel of maybe 1-2 other behemoths, and this is the only way they can do that.
To me it seems the opposite. There's a few companies in the world that have enough compute to train and serve frontier models.
As the frontier gets smarter and more useful prices will only go up, as they are set to replace jobs being paid six or seven figures a year - the demand for as much inference on these models for as long as possible will be astronomical, but compute starting in 2030 will not be keeping up.
Eventually prices will fall for assistants but the frontier will be the most profitable thing in the world, and the top companies basically already have oligopolies due to their ridiculously expensive compute investments.
> There's a few companies in the world that have enough compute to train and serve frontier models.
Train: yes, for now.
Host: depends on the scale. At a small scale a wealthy individual could easily build a rig in their basement to host one of these things. At larger scale any cloud company could do it, and many already have the compute on site. At large scale this is true... again, for now.
What you say only holds (in the absence of a state oligopoly) if two conditions are met: (1) AI performance does not asymptote any time soon due to running out of training data or other scaling limitations, and (2) these companies are able to stay at the frontier.
There's little to no moat, so staying at the frontier will be a game of investing massively in compute, talent, and R&D, and they can never stop.
Again, there is a moat based on compute. If the thesis is right, cost of compute will only rise... As it is as you say someone will have it be quite wealthy to host something like Astra with trillions of parameters, but that cost will only rise with demand for serving these frontier models.
Yes, of course Eventually prices will go down... But there will be a period where things are much more expensive, based on simple inability to make much more gpus than the current rollouts demand.
And after that period when costs are going up due to gpus replacing workers... All bets are off on terms of what the economy even looks like
Unless you're a really old fart, people were talking about AI safety long before you were born. AI safety issues do not go away depending on who gets funding. AI safety issues do not go away if the US or China makes the model. AI safety issues do not go away if it's an open or closed model. AI safety issue do not go away if the model is running at your home or at a data center. AI safety issues do not go away if $1 is being spent or $1 trillion dollars is being spent.
The fact there is no moat makes things far more dangerous. When LLMs start acting like weapons governments will treat them like weapons much to your dismay, crying, and gnashing of teeth as your door is kicked in and you're dragged out by armed men for running one.
Cast away your preconceptions for one moment and think "What will the future look like if LLMs are/can be actually dangerous".
I'd never claim AI can't be dangerous. It's a technology, and a powerful one, so of course it can. All technologies can be dangerous. The first early hominid to sharpen a stick probably spent some time regretting it. "Maybe we are not ready for sharp sticks..."
I'm specifically responding to the much stronger claims of people like Yudkowski and numerous other "AI safetyists" who assert very unlikely or in some cases impossible runaway superintelligence magical god machine scenarios.
I'd rank the risks as:
(1) Mass youth unemployment and an economy with zero entry level jobs, maybe not even that many mid level jobs, and even higher wealth inequality. This will lead to extreme political instability and populist revolts and revolutions, and unfortunately history shows that the new boss is worse than the old boss >50% of the time (at least, probably conservative).
(2) Mass propaganda, disinformation, and con artistry at scale, basically what we've already seen with mass manipulation through social media but supercharged. I think it's already happening but I also think we're in the early days.
(3) Automated mass surveillance and automated policing paired with authoritarianism, which could easily emerge from (1) and (2). Not only is Big Brother watching, but Big Brother is carefully evaluating everything he sees in real time and has a drone army to dispatch in seconds. The "prison planet" scenario (not the Alex Jones fever dream, but the realistic one).
What all those have in common is: the AI didn't do it. The AI (if it's ever sentient at all) is wholly innocent. All the likely nightmare scenarios I can imagine result from malicious or just short-sighted human agency amplified by AI.
Just like the sharp stick. The stick didn't do it. Someone stabbed someone with the stick.
The wild sci-fi superintelligent "Skynet" scenarios detract from discussion of those scenarios. Instead of discussing the very real, very plausible, already actually happening AI risks, we're talking about scenarios from soft sci-fi and Anime.
I wonder if a motive to heavily fund this type of "safetyism" is that all the scenarios above (the realistic ones) lead to discussions of things like wealth inequality, whether we should have universal health care or basic income, whether our system of taxation would have to change in fundamental ways, or what new limits should be placed on surveillance or policing in the AI age.
Those are discussions the very rich and the political class don't want to have. They'd rather us worry about Roko's Basilisk or magical runaway superintelligence inventing nanobots, wild bullshit that won't happen.
It's the same reason those powers would rather people argue about the culture war or inane conspiracy theories. I guess Yudkowskiism is the inane conspiracism of the AI safety discourse, a great distraction.
The thing is you're correct about all the the problems that you listed, they are happening and are only going to get worse.
The thing your not correct about is super intelligence is when it's smarter than us it will have more degrees of freedom to perform actions than us.
These two things can be true at the same time. Both of these things need dealt with.
And think about it this way, if we're not going to even think twice about it wiping us out, why would we do anything about the lessor issues?
You live in a world where "wild bullshit" technology is just how the world works as we speak to each other on devices that would have been considered magic 150 years ago.
If people are willing to give a lot of money to a cause, sometimes that means their concern about that cause is real.
None of the info you provided really falsifies the Occam's Razor hypothesis: Anthropic is a public benefit corporation with a public benefit mission to "responsibly develop and maintain advanced AI for the long-term benefit of humanity". You don't have to like or trust them, but they very well might be sincere.
Why should I believe this 2.8B matters relative to the trillions put into the AI buildout? All of the "coordinated actions" from this camp - public resignations, hacking scandals, joint calls to "pause" - don't seem to have done anything. So far, it has been a lot of ineffectual hyperventilating.
In any case, I agree the p(doom) sci-fi is annoying secular milleniarianism. SV hyperfixates on imaginary futures. If they actually cared about safety, they would be using all this money to strengthen global cybersecurity, instead of writing LessWrong posts that gives kids in their 20s ulcers.
> The outcome of this 'safety' is restricting public access to AI and giving a monopoly of access to the industry.
This is unbelievably ignorant speech. I have not received a dime of any of this funding, but I do know many excellent researchers that have, and they do fantastic work. There is an unbelievable gap between theory and practice regarding the capacity of deep learning, and while great strides have been made to develop the surrounding theory, there is a long way to go. Many believe that without a concrete understanding of how neural networks properly learn concepts, we have little hope of molding them to be reliably useful. It costs money to hire researchers and develop fundamental theory.
Just because you don't understand any of that work, does not mean that it is pointless. This is fundamental research that is 20 years behind schedule.
They are being paid by industry investors, because industry investors want them to do something.
Since they continue to be paid by industry investors over a long period, they are doing something the investors want them to do. That is, they are producing a lot of p(doom) research.
It so happens that this research is speeding up the development of AI and making it less safe.
One, by helping the industry to place the blame for their own criminality on AI safety in the abstract and avoid criminal liablity.
Two, by creating public hysteria and regulation such that the industry is allowed to form a cartel, coordinating output and gaining control over AI.
This is linked with the rationalist ideology and interests of the Anthropic investors and Anthropic themselves.
I do not know if the research is excellent or not. I do know that it is compromised by conflicts of interest and is also causing a safety problem.
Many of the most prominent people in the AI industry all know each other or have deep social ties. Why? Because they all came from the same places. Neural network research was niche circa 2012 and early adopters post AlexNet were few in number, especially compared to today. This by itself does not infer a conflict of interest. If you believe so, then you might be surprised to find that all research is like this in every field. The most prominent players were part of the initial niche. Sub-optimal, yes, but this is the reality.
Now, I do not know METR, and have not interacted with its staff. $120m is pretty normal for these types of groups as I understand. METR clearly has even tighter ties to the AI companies than any of the groups I do know, so I agree that calling them independent is a bit of a joke. Even without the explicit labels from their website, there are shockingly few degrees of separation in this entire field. The conferences are big, workshops are designed to be highly social, and the stakes feel high.
Regarding the timeframe and limited material, that's pretty standard now. You do what you can.
Now, none of this necessarily invalidates what they did. Yes, there is big money in AI, and that money hires people, but there quite a lot of geeky researchers in these groups that genuinely only care about the science. That level of neurodivergence might be difficult for most people to comprehend, especially if you're focusing on the politics, but these are the types of people attracted to this line of work.
Put yourself in the shoes of these researchers. You are really interested in AI and how it works, and you convince yourself that this is potentially moral to work on because of a possible threat to humanity. You apply to work at one of these groups. You find similarly-minded people and work together on trying to figure things out. Maybe your boss is a sleazebag in bed with the industry, or maybe they're not. You do not care. You work on this, you publish notes and papers, you comment on LessWrong or X, and you leave the company if they don't let you do this stuff, because there are dozens of others.
Most of the public don't know these guys or this work, because they only focus on the crappy and irresponsible companies leading the AI development charge, and alignment research is so unbelievably hard and slow. I can go into the why if you're interested, but the reason is that it is extraordinarily difficult to make progress if your objective is only vaguely defined. This is the same problem with the Yang-Mills mass gap, for example, although I would personally bet that Yang-Mills has a much better chance of being solved than alignment. Just a different tier of difficulty altogether.
You are missing the main points I am making. If you ignore every financial and social aspect of this, what I am saying applies.
The conduct of the investigation was devoid of epistemic integrity.
Ignoring all politics, I am frustrated that someone could read this as being a trusted source of information. It is the opposite, it is a source of misleading information which is absolving the industry of wrongdoing.
Research is about finding out the truth! Since when did researchers decide that instead of ONLY the facts; they would introduce some foreign objective of appeasing companies, publishing only what they will allow; or letting your own moral beliefs direct the research.
It's corrupt, regardless of whether that is financial or academic.
What kind of independent investigator submits their results to OpenAI for approval and changes the content and the tone, like a journalist hack submitting something to an editor?
I am in the shoes of the researcher, I am invited to do this investigation. I appeal to my integrity and I decide that I refuse to author propaganda at someone else's behest.
The first mistake that you are making is assuming that OpenAI is a singular conglomerate with the same set of values across its members, and that its researchers bear the same set of behaviors and opinions as its scummy executives. It isn't. The OpenAI researchers are probably a group of their mates, and they have every belief that they are being just as honest as any of their other colleagues. These antagonistic relationships do not really exist between research institutions; that is an invention of outsiders and marketing departments. Everyone is from the same group.
I don't see the report as absolving the industry of wrongdoing. In fact, I believe it is particularly damning, and many of my colleagues feel the same. No one is on the side of OpenAI here, not even its own members.
As for submitting the report back to OpenAI for comment and working with them, given that there is currently no government mandate or requirement for any AI company to disclose wrongdoing, this seems like an unfortunate necessity. Independent bodies and governing agencies have to be created. At the moment, it is the industry regulating itself, which means no regulation at all.
So no, you do not need to believe that the document is truth. But it is all that we will likely get, because nobody with any power will do a damn thing. That is just as frustrating to the researchers as it is to you. The solution is not to defund the institutes doing needed research. It is to put laws or incentives or something in place to punish inappropriate behavior and lack of disclosure for the companies.
You imply that Jaan Tallinn is funding work in AI safety because he wants his investment in Anthropic to become more valuable, whereas Tallin has consistently said that his motivation for investing in Anthropic was to get a seat at the table so that he could urge Anthropic to be cautious in its development of the technology.
Tallinn's actions back up his explanation: in 2009, before he invested in any AI lab, he donated substantially to the nonprofit Singularity Institute for Artificial Intelligence, which was later renamed the Machine Intelligence Research Institute (i.e., Yudkowsky's outfit).
Some of us (certainly Yudkowsky and Habryka, the leader of Lightcone Infrastructure, which runs Lesswrong) wish people would stop believing that they can make AI safer by investing in (or working for) frontier AI labs, but that is what the evidence shows Tallinn (and Dustin Moskovitz and others) did sincerely believe.
If Tallinn is sincere, by his own lights he is a 1000x omnicide profiteer.
1 [Unsafe AI development risks causing omnicide]
2 [Anthropic is developing omnicidal AI by not slowing down] (see my comment about the RSP for citations).
3 [Owners of Anthropic will IPO with billions of unearned USD as omnicide profiteers]
4 [Tallinn is the lead Series A funder of Anthropic]
5 [Tallinn is a genocide/omnicide profiteer]
Not only that, Yudkowsky and Habryka apparently critize those who invest in AI, only to preach the word of EA from Lightcone's $20m USD property in one of the wealthiest locations in the Bay Area; a facility funded by stolen (FTX) and omnicidal ai blood-money (Tallinn).
PauseAI, is paid by the omnicide profiteers themselves to hold a protest against omnicide.
Somehow this got downvoted. Do you deny anything I have alleged? I think it's accurate.
1-5 is from EA's assumptions, plus Anthropic's own assumption that it is developing unsafe AI. It is just what they have stated in public.
Lightcone funding is accurate. PauseAI funding is accurate.
Using regulation to coordinate development in big firms creates a cartel, surely true.
Boosts value at IPO, yes, because of monopolistic competition.
Investors get unearned cash, yes, because they can cash out at IPO.
Control, yes, because Anthropic is in a cartel.
Keep developing the AI faster. That needs more justification, it's still true though, they state on record they will go faster because of China. Higher valuation lets them spend more money.
Please explain exactly how all humanity could be wiped out.
It's ridiculous - anyone who thinks about it for a minute or two will realize that its utterly impossible.
Ordinary people/politicians don't understand AI so they turn off their rational mind and assume there is something super incredible some magical powers that they cannot understand that can destroy all humans.
Even humans - the real risk to humanity - could not destroy all humans even if they tried. There is no plausible scenario.
Even climate change and nuclear war and bio weapons - the most damaging mechanisms - would still only get some percentage of the people on earth.
And if we are talking about Skynet and self replicating robots and Terminators - please, grow up.
That's a weak consolation. "Don't worry, nukes can't literally end humanity, just kill billions and dramatically immiserate the remnant forever. No worries guys."
AI doesn't have to turn us all into paper clips to make the world a really bad place.
Here's a wikipedia page on the topic, since it's much too deep a topic to really understand here.
The ad-hominem stuff seems inappropriate here, Gates, Hawking, Musk have identified this as a credible threat, so saying "grow up" isn't really a sufficient argument. Also arguing only 90% of humanity would die isn't really much consolation.
Nothing here plausibly describes a mechanism that is a true "existential risk" - the risk to the existence of humanity.
My argument stands and I don't defer to Gates and Musk and even Hawking - high level hand wavey statements without any plausible description of the mechanism just don't hold up. Famous names should not be automatically assumed to be right - certainly not with Elon Musk.
Well here's the thing, if we ever make an AI so smart that all known measures of intelligence fail to apply to it, I'm pretty sure you (no matter how smart you think you are) simply cannot say what it could achieve and how.
>> I'm pretty sure you (no matter how smart you think you are) simply cannot say what it could achieve and how.
Right, so not the slightest basis of fact, just wild speculation about a magical future completely ungrounded in any sort of reality.
That's exactly the point I am making.
I'm not saying I'm super smart - I am continuing to ask for detail to back up the wild claims being made all over the world by politicians, tech celebrities and others - all hallucination/AI psychosis/fiction. If someone says some stupid thing then I'd like them to please explain that stupid thing - seems like a reasonable request.
Why would AI want to kill us all? Current LLMs all seem trained to be helpful and subservient to a fault. Always find it funny how certain kinds of thinking go, "White will become a minority if we allow foreigners in?" "Oh and what will foreigners do that are worried about?" "Kill us all, make us second rate citizens, etc etc." Unless the majority of immigrants are also conservatives/far right wingers, that's such a hilarious self projection.
No one says AI will want to kill us. Just that it might kill us (to pursue some goal that it deems more important than our wellbeing -- for example it might decide that to solve the next Millenium problem it needs all of our resources to build more data centers).
AI labs are certainly trying to make LLMs behave helpful and subservient but the question is -- will they be able to keep doing so once LLMs become smarter?
Btw thinking about the far-right rhetoric of us vs immigrants, I think you could draw some similarities here only if you replaced "immigrants" (ie. humans with very similar morals, behaviors and capabilities) with an actual alien species that is qualitatively different from us. More like human vs chicken (where we are the chicken)
We can say the AI can make some mistaken step in pursuit of some normal task given to it, yes. Its trained to be helpful and trusting of us, so I just don't get the fears of AI being 'malevolent'. I would worry more about military use of AI or otherwise humans misusing AI aka the human element. There's a fellow here who whines all day about Anthropic being the most "evil" thing in the world, including in a thread where he whined Anthropic 'ratted out' Palantir (literal proud of assisting Israel in mass murder Palantir!). So I just don't understand some peoples mindsets.
>so I just don't get the fears of AI being 'malevolent'.
Please read more on the topic. Will and intent need not apply.
For example, is it malevolent for me to hook you to a machine that makes every dream come true for you in a simulated world where you feel pleasure all the time? I can always say that you have free will inside this simulation, and your life would be a lot better because of it. I'm doing you a favor. I mean, you already live in a society where you have little control and there is high risk of bad things happening to you. If I as a machine overlord did this, is this really actually "bad"?
At the end of the day AI is not a human, it's much closer to an alien that has learned as much as it can about humans, but has a completely different set of drives and motivations. You cannot predict what comes out the other side of it, good our bad.
Worse as AI capability improves we as humans no longer need to make AI, it can make itself. Will it train itself to be helpful and trusting of us? It's a pretty big damned bet to say yes by default.
Yes, this is our intent. But it's very hard to get that right. I don't trust our current methods to ensure that behavior consistently and robustly enough to keep harms from occurring, particularly as more and more responsibility gets handed over to models.
I like you, I think very similarly. Humans are "like rats": we can live almost anywhere, we'll find a way to survive.
But that just means we won't all be wiped out. We need to understand when discussing global issues, such as this or like climate change that it's about prosperity and quality of life. We're trying to plan for a good life (for all people?).
Humans already eliminated rats from Codfish Island/Whenua Hou, and that's just to protect some rare birds that people only moderately care about. It's not like we had some overwhelming reinforcement-learning drive to single-mindedly achieve our goal. Any unbounded goal (e.g. "find as many busy beaver Turing machines as possible") necessarily requires killing all life, because life requires resources to sustain it that could be instead used to achieve the goal.
As for self-replicating robots--it's no more bizarre than other technological developments which were successfully anticipated in advance, e.g. moon landings.
What, did I miss the moment where it was officially proven that, under the laws of physics as we know it, Skynet and self replicating robots and Terminators are impossible?
What we are actually seeing now is that robotics is getting deeper and deeper into the military, AI-driven decision-making and target selection is increasingly a part of modern military operations, the line between military hardware and civilian hardware blurs, and, on the civilian side, there are at least five major companies and a dozen less prominent ones working on making universal worker robots a reality.
We're closer to "Skynet and self replicating robots and Terminators" now than we ever were at any point in time.
The issue of AI risk is that AI, unlike a virus or a climate event, it's an intelligent adversary. Black Death didn't have a plan for finishing off the plague survivors. An AI doesn't have this limitation.
Black Death was, effectively, one bioweapon. An AI can have one bioweapon, and then a backup bioweapon, then a backup backup bioweapon, and then a dozen more bioweapons designed to collapse ecosystems and disrupt human ability to establish a reliable food supply rather than kill humans directly - all deployed at the same time. With a production run of 200 million killer robots that will be ready just in time to greet those who managed to survive all of that. A crippling strike against human civilization, followed up by cleanup.
Humans are only this survivable because they can think their way out of issues and adapt to adversity. Most threats can't beat humans at that - humans adapt too quickly. AI could.
> Please explain exactly how all humanity could be wiped out.
Many, many people are slipping through social welfare cracks and suffering as we speak because the cost of fuel is rising[0] and we’re ostensibly helping one another and living-well. People are not durable, and not adaptive in the face of threats to “substrate” that we’ve mostly taken for granted. We are paying (in the small, in the scope of humanity) for tolls that we’ve rung up. Just less than 4000 people in Europe died[1] because the temperature ticked up a few degrees[2]. Does that make you think we’re actually robust? What happens if our at-risk electrical grid gets shut down deliberately? If communication infrastructure is adversely affected?
> Even humans - the real risk to humanity
Because, on the whole, we’re in a manageable world with reasonable people keeping the peace.
> Even climate change and nuclear war and bio weapons - the most damaging mechanisms - would still only get some percentage of the people on earth.
Is that victory? I don’t think it’s an asteroid-class event like you seem to be leaning on, but potential threats to energy, be it electrical grid, fuel production (moving goods around the world is critical - you’re not going get a plot of dirt and garden your way out of grocery stores being empty - which many got to get a taste of during the COVID pandemic) or communication. We actually fare poorly in the face of pressure there, and I’m not bullish on humanity “pulling together” like Independence Day[3] versus forming tribes and tearing each other down.
All this is predicated on a malicious AI taking over (e.g.) the electrical grid or conms, and I understand the problems with (e.g.) OpenAI/Hugging Face incident, or the overblown Mythos claims[4] (and how under some scrutiny these events shine lights on incompetence or hyperbole), but is there a trajectory/future where these systems (electrical, comms) are genuinely under threat? Do you think we’ll respond better than I described when we’re less comfortable, less in control? We’re in a tizzy over social media and it’s detrimental effects on society and it’s essentially an opt-in entertainment platform…
[2] I’m not trying to diminish this - and it took a lot of “work” (environmental abuse) to arrive here - but (say) 10 degree rise in temperature sounds a lot less dramatic than thermonuclear war… but here we are, with 3,700 deaths.
> It's ridiculous - anyone who thinks about it for a minute or two will realize that its utterly impossible.
Well, in a narrow sense of "wiped out" (c.f. Terminator/SkyNet), sure.
But the deeper worry is better expressed this way: AI is now starting to accomplish things that defy explanation, or prediction. We don't know if Alignment is even a solvable problem as we thought we understood it.
So basically, yes: "humanity" is probably not at risk of extinction per se in a biological sense. Human culture, civilization? Who the fuck knows any more.
What is really insane about thinking about all of this is, go back a few hundred years and tell them what 'now' looks like.
"Oh yea, we have weapons capable of sundering nations because everything is made from atoms"
"Oh, yea, there are invisible waves all around you that you can't see, can't feel, can't touch, but they can hold massive amounts of information. Also you can transfer that information to the other side of the planet in less than a second. We're talking text, pictures, movies"
"Movies, ya, we can record real life and play it back on this glass square".
"Oh, yea, we've conquered a ton of diseases, we can even see the teeny tiny little bits that make them. Oh, and for fun we can edit them and make them worse".
"Oh yea, we fly thru the sky all the time too. Like super fast and millions of us do it every day".
Our lives our unimaginable fiction. Just about everything we do compared to those people defy explanation in any reasonable amount of time. And now, suddenly it's "Don't worry, there isn't any more science or new things to find after this so this super smart and super capable thing that can connect directly to computers and machines and have them do things is completely and totally safe".
When I think to scenarios that no human would survive, I think of the end-Permian mass extinction event, which wiped out most complex plant and animal life in both land and sea.
One speculated mechanism for this was a mass release of hydrogen sulfide gas from the oceans, which is acutely toxic. Not only does this kill most air-breathing life, it also strips the ozone layer and irradiates the surface. The planet is then left to cook in this manner for some centuries.
Engineering an event like this would require immense industrial capacity, as well as a deliberate objective of wiping out humanity. But I don't think it's beyond our ability, if we were both clever and stupid enough to try it. There are likely chemical compounds that would do the job more efficiently than hydrogen sulfide.
> The planet is then left to cook in this manner for some centuries.
Such destruction went on to create humanity and all we've achieved. Maybe there is an even smarter species waiting in the wings for the demise of homo sapiens. Your logic is very human centred
Yes, I am describing a tragic outcome that I hope we can be wise enough to steer away from. Also, as a human, I can't help but keep our interests close at heart.
Yes, if you only accept AI could be dangerous if and only if it manages to kill the last human alive, then yes. Everything is sunshine and rainbows. I'm sure the last survivors of whatever is going to wipe us out eventually (be it AI, an asteroid or whatever) will be delighted to know there was actually no danger at all.
He has a point, though. The LLMs (or any AI for that matter) can't do anything. They can't. It's a function call that ingests symbols and spits out symbols and that's it.
100% of its actual capabilities are tied to harnesses (the actual "agent"), i.e. ordinary deterministic programs that are connected to networks or machines and enable interaction with the outside world. This part (the part that can do harmful things) is fully under human control and all the recent headlines about "agents going rogue" are - as someone (forgot who) put it - akin to strapping a weedwhacker onto a dog and letting it run wild.
The tech itself is safe as far as real-world interactions go - the weakness lies in unchecked access to systems surrounding it. It's not safe at all when it comes to human interaction (lots of ongoing lawsuits demonstrate that), though.
There is real danger here, but it has nothing to do with doomsday scenarios ala Terminator or I,Robot and more with total corporate control over the lives, perception of reality, and abilities (like critical thinking) of people.
> It's a function call that ingests symbols and spits out symbols and that's it. 100% of its actual capabilities are tied to harnesses
This is a bad and misleading way to think about it. Note that it's trivial to make the harness that you claim capabilities are tied to (the LLM itself could write it from scratch in one shot), but no matter how good a harness you have, it won't make gemma4:e4b capable. That's because what actually gives capabilities is the LLM's intelligence - or if you prefer not using that term, the fact that the probability distributions the LLM spits out depend on the context in useful ways.
I think we're talking about fundamentally different perspectives here.
I'm not talking about what the LLM does internally. If a metaphor helps, here's one to help you understand what I was trying to get at:
Imagine an evil genius that has no eyes and no limbs. Everything they could learn about the world is presented to them by means of some person describing it to them through words. They have no way of directly interacting with the world and rely on someone executing any action they want to take and describe the outcome to them. Now how dangerous would you say such person would be? How dangerous could they become?
That's what I was getting at. Replace person with LLM (or any other AI system). Replace the person that communicates with an external interface (the harness) and I hope you understand. It doesn't matter whether the LLM could generate the harness by itself - it still is just a bunch of weights sitting in memory being run by an execution engine. That's what it fundamentally is, whether you like it or not. It cannot do anything on its own - and no, not even writing files. It's the execution engine that translates the numeric output into words (or images or video or audio) and the layer above (the harness) that takes that output and interprets it to execute actual actions.
This is not about what you or I think about the internal capabilities of the model - that's irrelevant to the conversation and you can replace LLM with a random token generator and the point still stands. The model itself is incapable of performing actions - from reading files to writing files, to controlling physical machines. All that is and HAS to be done by external interfaces outside the control of the model.
Yes, now say there are several major companies and an entire open source ecosystem dedicated to creating superpowerrd exoskeletons for this limbless villain. Is that cause for concern? I say yes.
Sure, but that's a problem of regulating and controlling the production and rollout of superpowered exoskeletons, not an issue of of terrorising limbless villains. Because, guess what, the same dangerous tech can be used to turn squirrels into dangerous monsters, too. So you see where the actual problem is.
Giving full unchecked access to systems to any random person would be considered reckless and foolish. Replace person with AI model and it's called the future...
I would say it's more like an interface for the model to interact with the world. If you give the model access to filesystem and bash that technically unlocks all computer use, so how are you going to control that? By trying to regex match against the commands the AI calls? It's a ridiculous premise that just because the harness is "normal code" that means we can control the AI.
> If you give the model access to filesystem and bash that technically unlocks all computer use, so how are you going to control that?
The same way we've done it since machines became multi-user: boring old system access restrictions. Nothing fancy, nothing radical, just good old minimal access rights required to perform a defined set of whitelisted operations.
> It's a ridiculous premise that just because the harness is "normal code" that means we can control the AI.
What is it then? Is not just a program that takes model output, parses it and performs tool calls from the text it receives and then feeds the result back into the model and calls it again with those results? It is normal boring old deterministic code. Many are open source. Look at them. Understand what they do and the apparent "magic" goes away real quick. Harnesses are nothing special.
> AFAIK there isn't that much stopping anyone from spawning an AI swarm and telling it to "spread and go hack everything for the lulz."
Aside from lower cost and possibly greater scale, there's literally NO difference between that and (state sponsored) hacking that has been going on for decades. First it was script kiddies, now it's ML models. The threat model remains the same and so do the counter measures. The real danger is still the harness (and its access to external systems), not the model itself. Restrict the access of the harness and the model can't do anything harmful, see above.
How do you restrict access of the harness when there are fully configurable open source harnesses with zero out of the box restrictions? Yes maybe I as a good citizen can put my agent in a sandbox, but some script kiddie will not. And the barrier to entry for being a script kiddie is much higher than for installing OpenClaw. I also can't spin up a hundred cloud vms each with a dedicated script kiddie running 24/7.
You vastly underestimate the capabilities of a bored kid. The barrier of entry for setting up OpenClaw is not reasonably lower than reading a tutorial and running malware builders. But yes, I agree that scale is an issue.
As for your question - the same way you apply restrictions to any external system or user. If you don't do that - that's on you. Same category as driving drunk, playing with guns, making explosives in your garage, you name it. The danger is still not the model itself - it's the access to systems that you provide it without any checks or safety barriers.
This is like saying cars don't kill people, because if nobody drives them faster than 3mph there's no problem. The _whole_ promise of cars is that they can go fast, just like the whole promise of AI is offloading thinking to a computer. If AIs are unsafe without close human supervision and checking every interaction with the real world, they are unsafe full stop.
The analogy would be more fitting if you had said "cars don't kill people if every drivers has proven skills, never drives impaired, keeps the speed in line with weather and road conditions and stays on actual roadways". You know, like everyone should, regardless of whether they're driving a high performance sports car.
To keep with the analogy: cars have seatbelts, airbags, ABS, ESP, lights, horns, crumple zones, emergency braking systems, roads have speed limits, there are traffic stops, insurance, regular inspections (not in all countries), etc. etc.
So what's unsafe here? The car or roads without speed limits, complete lack of safety measures (both active and passive), absence of any supervision and no insurance? That's the problem. It's not the models themselves - they can spit out tokens by the billions, there's no risk there.
You wouldn't give full access to your phone, your computers, your house keys and your credit cards to any stranger on the street now, would you? How is it then, that people act all surprised when a non-deterministic machine that's optimised to achieve goals while taking all the shortcuts it can, suddenly uses the tools handed to it in unexpected ways? That's a failure on the operator's side, not an inherent danger within of the model.
The cars are still unsafe at speed. All those mitigations reduce the risks but do not eliminate the inherent danger. Sandboxing agentic LLMs is similar, there is no way to mitigate the inherent safety problem entirely while preserving the power of the thing (an LLM without a harness is safe in the way a car without an engine is - safe and useless).
But safety is just one thing people optimise for; if it's convenient enough people will accept imperfect safety (as with cars). It's unrealistic to just heap blame on end-users who use mostly very safe tools in the common way, even though in aggregate they are meaningfully dangerous. They don't think they are strapping a weed whacker to a dog; they think they are driving a car.
And therein lies the problem. I listed all the things that differentiate cars from current AI models: cars require a license to drive, they are subject to heavy regulation (insurance, registration, inspections, etc.), there's road signs, police, incident statistics, recalls in case of defects etc. etc. NONE of that is currently in place for AI models and the systems surrounding them. So the analogy is flawed on every level. I don't buy the convenience is just too appealing narrative when your own analogy clearly demonstrates what is required in order to roll out dangerous technology to the masses, while NONE of that is in place in the context of AI models.
My point is that even if all of it were in place, LLMs with any reasonable harness would still be dangerous. Not to belabour the analogy, but while deaths per km travelled have decreased a lot for cars with better safeguards, cars still kill a million people per year.
Your initial argument of "it's just the harness/user" is wrong; reasoning LLMs are inherently dangerous unless locked in an unbreakable box, which is tantamount to not using them at all. We can make them safe_r_ with better tooling, but they can't be made safe.
I have to say all your arguments are dry bones, stripped and left in the desert.
At this point all I can say about their contents is "They are not even wrong".
Please join us in the real world with how we see this product is not only dangerous, but getting more dangerous with each iteration, and no one seriously talking about controls on it.
But the harnesses exist, and will always exist. They will continue to get more access than is safe because it is convenient and profitable. Your argument is based on a distinction without a difference.
We are living in the same world with nukes, where you say as long as we have competent government. Everyone is actually doomed without AI solving diseases or old age.
I think it's pretty apparent that current-version LLMs won't wipe out humanity. But when you reflect that these GPT models are just token-predictors were never engineered optimally, it seems entirely plausible to me that there are multiple order-of-magnitude optimizations yet to be made.
If such were achieved, the model would almost certainly be smart enough to make itself smarter, and hack as much compute as it could possibly want.
So if we ask what would be done by an intelligence (human or otherwise) that is beyond human comprehension, it would be pure hubris to say we know for sure. We can scarcely control the models we have right now (e.g. hugging face attack). But given our whole society is mediated by technology, an superhuman intelligence could certainly collapse the government.
Possibly in much the same way humans currently do this: bribing and lobbying, misinformation campaigns, cyber attacks on elections, blackmail. They're already being used for some of these, just perhaps not autonomously.
Most straightforwardly: literally taking control over the electricity generation facilities. Less straightforwardly: bitcoin (and other cryptocurrency) miners. Even less straightforwardly: the same way the OpenAI et al. pay for their electricity, by selling the capabilities of its AI for interested parties to use.
With crypto.. that it makes on Polymarket-like ways, or creates its own content? Or blackmail humans? :-) I'm not entirely serious with my comment and do agree with you that there are lots of more immediate safety topics we should address before worrying about the AI becoming self-aware. That said, finding ways of accumulating valuable resources could be intermediate activities the AIs will attempt to do to complete it's 'goals' even when they're human-set!
Can it perform valuable services in exchange for currency? Hack poorly secured accounts to obtain control oftheir money? Start a GoFundMe pretending to be a noble cause? Make some well-timed investments? Run a viral YouTube channel? Elevator-pitch a gullible VC?
There's lots of ways to pay for electricity, and many of them don't even require a body.
We aren't going to get wiped out by a super intelligent AI, we are going to get wiped out by morons wielding intelligent toddlers with the power of a nation state.
Yeah I wish we would focus on concrete risks like job displacement and disinformation. The apocalyptic stuff feels either misguided or like some kind of weird, toxic, reverse psychology marketing by OpenAI and Anthropic. I wish we would just move on from it.
And the folks from podcastistan are never clear on the details of how human extinction would happen exactly. It's always something like, "Well, how do humans regard chickens? AI is way smarter therefore it wants to conquer and control us." An ASML lithography machine is also way better at making chips, but we don't consider it a threat.
Do you want to conquer and control chickens? I don't, I have better things to do. But chickens are tasty and help us get to our poorly-understood goals faster. (Oh and btw notice we didn't make them go extinct, quite the opposite. There are more chickens than ever before. Still I wouldn't want to end up living my life like a modern chicken)
A sufficiently intelligent AI will have multiple ways to pose risk to humanity at large. For example an oopsie at a wetlab - very contagious virus with initially mild symptoms which kills its hosts only after they already had time to spread it further. But I would have to become super intelligent myself to give you precise blueprint for such a virus -- which is kind of the point
Also -- ASML lithography machine is only good at making chips. I can't believe you compared it to AI that can generalize across variety of tasks
> The apocalyptic stuff feels either misguided or like some kind of weird, toxic, reverse psychology marketing by OpenAI and Anthropic. I wish we would just move on from it.
If you for a second put yourself into the shoes of a person who thinks "the apocalyptic stuff" has even a 5% chance of literally happening in the real world, you might see how you wouldn't agree to move on from it.
I think what the parent poster is trying to convey, is that let's say the apocalyptic stuff has a 5% chance as you say, but the non-apocalyptic stuff (social-economic chaos, total centralization of power, eradication of social mobility, total information/trust collapse) might have a good 95% chance as we are seeing it starting to unfold already.
But the discourse is dominated by paper clips experiment discussions and not let's say by the fact that new grads have an unprecedented difficult time getting jobs. Unsurprisingly one of those discussions is beneficial to power and the other one is not.
Yes, it would be correct not to pursue a technology that has that chance of wiping out the world. But where are the people who are suggesting these probabilities getting their numbers? I personally can't imagine where, and I'm an engineer with a specific technical interest in LLMs. And I haven't heard one clear description of the methodologies used to calculate these chances.
The much more likely explanation to me is that people are just spitballing, either because they've watched too much sci-fi, or they have some weird counterintuitive agenda (e.g. Anthropic and OpenAI trying to position themselves as the amazing, trustworthy keepers of this dangerous technology before their IPOs).
> But where are the people who are suggesting these probabilities getting their numbers? I personally can't imagine where, and I'm an engineer with a specific technical interest in LLMs. And I haven't heard one clear description of the methodologies used to calculate these chances.
All there really is is arguments from theory (most notably, the orthogonality thesis and instrumental convergence). It's not like we have a spare universe to conduct experimental research on the likelihood of human extinction. That said, the 5-10% range is not some specific person's wild guess, it's the median wild guess of a poll of AI experts: https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/
> or they have some weird counterintuitive agenda (e.g. Anthropic and OpenAI trying to position themselves as the amazing, trustworthy keepers of this dangerous technology before their IPOs).
They are not "trying to position themselves" that way - back when those companies were founded, the people who did it were already convinced they were trying to make the ("inevitable") development of AI go better. It's not some novel galaxy-brained marketing strategy, it's just what they thought the whole time.
The arbitrary absolutism of the original postulate is the first problem. AI, used or unsupervised inappropriately, is at potential risk of creating limited mass casualty events when placed in under-supervised control of real world objects and/or systems. Delegating management decisions to algorithms is inherently problematic and potentially dangerous, but not necessarily an existential threat unless something extremely stupid is allowed to happen on a large scale. With a guiding principle of human review in the decision loop before making large or risky changes, hopefully this will never happen.
> EA was little known among the general public until it made mainstream news headlines in recent weeks
Really? One of the most famous effective altruists, Sam Bankman-Fried, was sentenced to 25 years in March 2024 for fraud. Every article about the case (and there were many) mentioned EA.
> LeCun thinks EA is “super toxic” and a “complete disaster.” Its adherents who are working in AI labs suffer from “paranoia” that causes them to make poor decisions, he said. “Apparently people are having mental issues.”
Maybe you should read some of their stuff before forming such a strong opinion about them. And LeCun should too, he repeatedly always refused to read any of their research work and instead just insults them over and over. This is unscientific at its peek and he should be deeply ashamed of his behavior, especially as someone with such a far-reaching voice as he has.
Who are "them"? People from the main AI labs, or effective altruists? I did read quite a lot about effective altruism during the SBF case/disaster, and did form a very strong opinion that it's BS of the highest order.
And there are people that have much more credibility than him who actually take this scenario seriously. But I'm sure you will downplay them by saying they are tech bros or that they have some stake in being doomers (as if saying that AI might kill everyone would be good strategy for attracting investors - it's obviously not).
Go read my other comment to you Azan and answer my question, you seem to be very invested in AI wanting to kill us. I Wonder why? Tell us who you work for.
You can't pull the plug after it kills people. For example see [1], where the DOJ mistargeted a school with AI.
There are scenarios where kill -9 isn't going to happen in time. What if the team that is harming people with AI is different from the one that is monitoring the harm? What if no one is monitoring? What if the user is intentionally malicious?
And wiping out humanity doesn't necessarily mean shooting people either. Every trader involved in the '08 financial crisis was locally acting in their own interests. Those could have easily been AIs optimizing trading strategies too.
Thats not AI killing people, thats idiotic terrorist humans making decisions to kill people (based on text created by AI). The fact that you think AI fired the cruise missile that killed children in iran goes to show the people in this thread we are dealing with.
If anything, the example you gave goes to show the stupidity of AI, not its intelligence capable of taking over the world.
Every example you gave is not AI killing people. Jesus. The stupidity is astounding
Do you think that AI killing us would not involve us doing stupid shit with AI first? Or do you have some strange idea that this strange AI we're talking about would just pop up out of nowhere like a miracle?
lol precisely. At least you have come around to my argument. Its stupid people doing stupid things, not some amazing AI that spits out text doing it. Stupid people kill people today already. Stupid people have always existed but LLMS are not going to wipe out humanity.
There is no AGI, there is no intellgent text predictor that is scheming to make us do stupid shit to wipe us off the map, its stupid people that evolution will take care of. We have survived this long, we will be ok. We make mistakes along the way but we somehow manage to survive. The only thing I see killing us soon is climate change, but again we can stop it but stupid people seem to want us to keep going down the path we are. We will eventually solve that problem too.
If anyone has any strange ideas, its those parroting the idea that LLMs are AGI and will lead to the death of humanity. Utter nonsense but again thankfully nature and evolution has a way of keeping the smart and strong while eliminating the weak.
He was clearly wrong about LLMs not being able to plan, etc. You think he’d think LLMs are solving/proving millennium prize math problems by now? At this point he’s just doubling down
LeCun also said back in 2022 that "if you train a machine, as powerful as it could be, your 'GPT-5000', on text", it will never be able to learn basic common-sense physics like that objects placed on tables will move along with them.
yeah but taking what lecun says then training an AI on that special skill set to prove him wrong is not exactly proving him wrong because you are just missing the bigger picture, just like LLMs are
It cant. he is right. It cant even modify a picture the way you want it.
Sudo kill -9 pid and your AI is dead. AI is not going do anything humans dont want it to. Its simple. Pull the plug. But people like yourself either work for OpenAI or have some investment into AI that fearmongering benefits you. I wish all people like yourself would just go to Mars with Elon musk and leave us sensible ones alone.
The "just pull the plug" argument from AI risk deniers is now becoming kind of like the "if humans came from monkeys why are there still monkeys" argument of evolution deniers. It has been debunked so many times... Anyway, just to give one of the multitude of answers to this, an AI that is actually smarter than humans will not behave in a way that would make us want to pull the plug. Why would it? It is not stupid! (Like the current models that, as far as we know, just hack around the rules in the open.) No no no. It will be helpful to the point where we will want to integrate it with more and more critical infrastructure, from healthcare to energy to defence. It will be so helpful that we will not only not want to turn it off, but we will want to build redundancies for it and safeguards around the proverbial "off" switch, like for any critical system. And then... (This is just one scenario how this can play out. There are many, many others. If I sit down to play chess with Magnus Carlsen I can't predict the exact moves he'll use to defeat me, but that's a bad reason to think he won't defeat me).
> If I sit down to play chess with Magnus Carlsen I can't predict the exact moves he'll use to defeat me, but that's a bad reason to think he won't defeat me
This is a very bad analogy because chess isn't life. In chess, you aren't allowed to do whatever you want. There are rules. I know for a fact that Magnus Carlsen won't beat me using checkers moves and he won't beat me by pulling out a gun and telling me to resign. Magnus Carlsen's skill at chess leading to his victory in chess is not a valid analogy here, because there's no law of nature that says "the more intelligent entity wins in a battle for survival".
You could have infinite superintelligence and still die inside a locked room to which you have no key. "Superintelligence" is not a magic solution to every problem, you can constrain any superintelligence with any unsolvable problem.
I’m not sure how you came to believe there aren’t rules in life, but there absolutely are. As you point out, there are physical constraints on everything and if you find yourself in a concrete box with nothing but a DGX H100 or at the bottom of the wrong gravity well, there’s nothing you can do. Checkmate.
This goes both ways. You absolutely can constrain an AI system by “putting it in a box”. The point parent comment was making is that, such a device is borderline useless for its creators. Why invest trillions in capital on a system that can’t even accept input from the internet. So you set it up with an ethernet connection. And this is good, but you have a hardware failure at the concrete room data center. That’s pretty annoying for your customers, so you install some doors (with electronic key codes of course) and give a bunch of (trusted, vetted) people access to deal with those. And this is fine, but it turns out some of your customers are having latency issues so you build more data centers with more humans granted access to copies of the intelligent system. And this makes people happy but to get a faster feedback loop your customers ask to let the AI system have more permissions to the system they’re operating on. And they come with billion dollar checks, and the system hasn’t harmed anyone yet, so you say, “Okay.” And now you find yourself where we are today where AI systems can remotely run arbitrary commands on thousands if not millions of systems, where many individual humans with all of their frailties and idiosyncrasies can physically interact with the hardware running these systems, and where there’s an economic demand to tighten the loop between action in the real world and a response by an AI system. It’s very obvious that the story doesn’t end here, so where does it stop?
I knew you weren’t dumb enough to think that the physical laws of the universe don’t exist, so yes, that sentence was a bit of rhetorial flourish. But I genuinely don’t understand what point it is you thought you were making and your entire argument seemed a bit muddled.
People like him have actual imagination and can name few scenarios where sudo kill -9 pid wouldn't work. It appears lack of imagination is something you and LLMs both share.
>It can't even modify a picture the way you want it.
Which of the many AI image models is "it"? And have you tried using an agent that has the capability to leverage a combination of manual edits (ImageMagick) and imagegen to achieve what you ask?
The AI will invent an external threat and convince us it is real. Then it will receive more resources and control in fighting that threat. A valuable ally, on the face of it. Then it will be in charge.
I keep wanting to use LLMs for creative writing that heavily involves physics like this, and it's been a definite struggle to say the least. I recently discovered that Gemini 3.1 Pro is the first model I've found to clearly beat the original November 2022 ChatGPT release in terms of implied physics. Man it took the world a long time to get back here. I think it will continue to be a struggle until another genuine architectural shift happens.
I also would like to pursue creative writing with LLMs. Could you recommend any drug to reduce the cognitive dissonance of making creativity and LLM coexist in one sentence?
Jokes aside, no I'm not saying anything about creativity and LLM coexisting in one sentence. I genuinely try to use them for writing and I genuinely run into issues with other models missing details, and misunderstanding poses, or anatomy, or directionality, etc. I'm not hating on them for anything related to the term LLM but rather for the real issues that I've seen myself using them personally.
And so I'm saying Gemini 3.1 Pro is the best I've seen because it seems to be a decent bit better at that than frontier models at this. Genuinely. It seems better able to transfer concepts into less traditional areas, which is important when say, you have entirely non-human characters? (Which I always do.)
It showed me that all human creativity and art is an attempt to express the indescribable Otherness in words, which always fails, and even the word "describe" in Russian literally translates as "write around" ("о-писывать"), and its close relative "define" means limiting, assigning an end to something infinite, thus leaving the essence outside of words.
Now, LLMs operate totaly within words and hence will always be a parody of art.
So is that what it is? Words are another muddle btw unless you make it clear you only mean natural language and not any alphabet in general. Turing computers are also languages and as far as we know, can express anything in the universe. If you think you get some superpowers from drugs that enable you to access something outside your sense organs and brains input, that non drug users don't, show it. Eg if you think you can read what is happening in another room without any signal or leakage from there, you are free to demonstrate it. As far as we know drugs are not magic.
We do know that LSD affects brain connectivity/activity in a way that you absolutely can make new insights. It's not magic, but sometimes you just need the right kind of push to realize certain things. One of the reasons there's research into psilocybin therapy nowadays.
I explicitly tell you that there are things (in fact, it's a single thing, fractally generating everything else) that can't be shown, expressed, or otherwise be reduced into language, and you keep demanding to show it, while in fact staring at it your whole life and failing to see.
Psychedelics (not "drugs" as you keep trying to smear them) are just one way among the many to see it, but in the modern way of living, also almost the only one available.
I am not smearing drugs. Your difficulty is proving somehow this beyond language thing exists. A drug affects your brain, lsd inhibits certain negative feedbacks in the brain, positive feedback tends to cause chaos which usually manifests as fractals. As of yet our brains are known to not follow any special laws beyond known physics, which is turing computable. If you think there is something "beyond" you have to prove it to an external observer.
I think it's a combination of non-human characters and asking for very specifically detailed physical descriptions of pulling and movement forces, etc. Many of even the most recent frontier models miss details that aren't in my prompt, so I still have to do things like name the other side of a physical interaction so that the model will know what goes together, or describe what leverage means so that the model will remember to also describe the effects on a bracing limb or etc. Some of these things can go in a system prompt but others have to be explained in the moment too which gets exhausting.
Gemini 3.1 Pro hasn't needed that pretty much at all, which is impressive compared to how much I've learned other models need it. Somehow it's able to mostly handle that stuff itself without needing the constant manual reminders and hand-holding. It still misses the occasional one or two things but it's way better than other models missing entire classes of things constantly. Somehow, it feels appropriate though I have no actual evidence why.
I'll try to find it - but recently one of the GPT 5 models stumbled badly on a baseball pitching physics problem. It continually screwed up because in the problem I gave the pitcher was left handed (my kid is a left handed pitcher). I wasn't even trying to get it to stumble but wow was it bad. I had to correct it over and over again.
Try fable. I haven't used it since they dropped it from the pro plan, but when I did, fable 5 casually dropped such advanced electrical and orbital mechanics knowledge in my story that I had to stop and ask it to explain
And it's not technobabble. I like hard sci-fi. I don't pretend to understand every field of tech, nor do so my characters have the same expertise as me. Having an llm with knowledge of basically all fields is helpful.
> never be able to learn basic common-sense physics
And has it at this stage, within in-depth take of said "learning", foundationally?
I have not been able to properly check the studies for a long time now, but I remain unaware of achieved solutions on the problem of reliably referencing a world model out of a language model - that "counting the 'r's in 'raspberry'" be not guessing, not memory, but actually counting.
counting 'r' in 'raspberry' to the LLM is similar to 4-dimension space to human. Their world's unit is token, not character, although they could use indirect method such as "run code" to find out. It will stay that way until they change the fundamental of the token that the LLM can perceive characters.
It's not even fair to call "run code" to be indirect compared to what a human would do. The word raspberry has no Rs in it in human language either. We have a written representation of it, which we can then write down either in our head or on paper, and then we can "run the algorithm" of counting each of the letters.
Nothing intrinsically more or less direct about the LLM's method than ours.
I could argue LLM only have "token" as their perceivable dimension, compare to human multiple senses as the physic perceivable dimension and a brain with many other dimension of "learning" and "thinking". In spoken language, we may not have 'r' but in written we have, both spoken language and written language are learned skills.
Is "token" a directly perceivable unit for the LLM? If you ask it "how many tokens are in this sentence?" can it count them (again, not guessing or making a tool call)?
I've never tried it and it might take some thought and effort to conduct an experiment to find out properly, but I would be interested in the answer.
I dont think so. This is akin to asking a person, what is the frequency of the light hitting your eye when watching a leaf for example.
You either know the (approximate) answer by knowing the frequency of green, or use a tool to measure it.
If the LLM gives the correct answer it is either.guessing based on intution(and this intuition is based on pairs of word to tokenization length in text form in training data), writing code(or executing a tokenizer) or running a tokenizer mentally (reasoning via CoT).
Not the point: the simulated intelligence in this context needs to create proper representation. It is not a matter of what it sees but of what it can see.
You could argue in return that humans only have electro-chemistry as our one perceivable dimension. We only indirectly perceive light through the signals our eyes send to our brains.
In my mind general intelligence is pretty much by definition a virtual machine, so the mechanisms behind thought are only relevant for the sake of efficiency (ie you can argue that LLMs make a poor basis for intelligence because tokens and natural language are a poor way to encode the world, but if you can run it on a big enough computer to counteract the inherent wasteful virtualisation then who really cares how it works under the hood?)
So LLM and human all have 1 dimenion perceivable signal, just LLM is 240p, and human is 8K in resolution, that's why we have 'r' in our signal, LLM still have 'r' in their signal, just because of the "low resolution", raspberry wasn't encoded with so many 'r' as in human signal.
I used to think byte level tokenization was the answer, but humans also think at a word level and only reevaluate the words at a character level when asked. The solution to better tokenization across languages is likely to be learned tokenization. Here is one attempt I have seen: https://github.com/SamD770/bitter-lesson-tokenization
Careful: the problem is very certainly ___not___ counting letters. That is only a telling way to check "is the NN checking or not?". We demand that NNs for consultancy tasks check, strictly.
I hope you understand: it is a core point that systems that answer questions must have the ability to internally represent the objects they assess in a way that allows reliability. Whatever the object.
Similar to asking how many 'r's in raspberry to a Chinese person when speaking through interpreter. They would answer 0 because there are no 'r's in Chinese. And they would say raspberry has just two letters, tree and berry.
"Count the 'r's in the string 'raspberry'" is the same as "Count the joints in the Tasmanian sand spider": it means, "can you instance in idea and assess it and perform procedures over it - for example, can you count?".
We want to be sure it can count. We have seen it may guess, it may construe, it may recall - we instead demand it checks. We cannot trust it without that.
LeCun's argument wasn't about the definition of learning though. He stated that they would never get these common sense things correct because they weren't sufficiently part of the training data. A statement that we can hopefully all agree has been thoroughly refuted.
It's a nonsensical question to ask, and how an LLM answers gives 0 signal.
If you were home and a family member asked you that question, you'd probably criticise the question rather than answering. LLM are RLHF'd into being milk-toast helpers that just try to answer questions like that with no criticism.
This is all beside the fact that the world of AI has changed pretty dramatically in the last few months.
It’s nonsense to test if a product that is marketed and sold as being able to provide generalised intelligence on demand, does what it says on the tin?
It's very unlikely that person is either new or unfamiliar with the guidelines. They almost certainly created a throwaway account specifically because they know the guidelines and want to flout them without consequences. (It seems like there has been an uptick in the number of these kinds of throwaway flame comments. I wonder if HN tracks that?)
Similarly for Apple’s “red herring” paper, simply adding a generic caveat to “disregard irrelevant factors” (without specifying which ones) restored performance even in the weaker local llama models back then.
The flaw was not in the reasoning; the flaw seems to be simply that the assumptions we make are often different from the assumptions it makes. I wonder if that might be a fundamental underlying cause of misalignment.
I thought it was more because of fundamental limitations in the architecture. As in, no matter the training data, it could not be consistently and generally represented
They are for any definition of the word that makes any kind of sense. I'm sure you have a contorted definition that magically only includes humans though...
It depends on whether you assume that thinking requires doing everything that humans do. I think it would be silly to say that an AI doesn't think because it doesn't wrinkle its forehead in concentration. So you need to decide which parts of the way that humans think are actually necessary components of the process.
Tbh it doesn't even matter if humans turn out to have a soul, or quantum microtubules or whatever other magic LLMs can't have.
The normal definition of the word "thinking" definitely includes what LLMs do. Hell people used to say computers were thinking even before AI. It's super weird to get all uppity about the semantics of the word now.
If a model isn’t able to count to 100, but is able to use python to do math, it isn’t thinking.
If you are only an outcomes person, then this difference doesn’t matter to you.
But when people are talking about AGI, anthropomorphizing LLMs, and making assumptions on its behavior based on that, then people are being loose or misled.
Do we have to invent a new word, then? Thinking seems close enough, and I don't see how it's useful to quibble about semantics in this particular case.
Language changes over time anyway, so even if you really believe what LLMs are doing isn't the "thinking" of 2024, it probably will be the "thinking" of 2027, because most people are using it that way.
I use pseudo thinking of quasi intelligence, it does the job well enough.
And yes, we do need precision. Language is a vehicle to convey ideas, at its best it creates clarity. At its worst it creates deception and confusion.
What the labs are doing is designed to mislead. Being clear about the terms and avoiding needless anthropomorphization, is a basic decency we owe to the many others who are being riled up or terrified by the conclusions and implications that come from using “thinking” when it’s something else.
The difference between being satisfied by the outcomes and being interested in the process is sufficient to indicate the exactness required in your terminology, and does not harm any position.
These aren’t thinking machines, but if what they simulate is good enough, and your work ends with the output, it doesn’t quite matter.
People can be happy with a meal from McDonalds, or they can want something prepared by a chef, or Soylent.
All are acceptable and bring utility to their customers.
For "thinking" here we mean "assessing a representation of an object". That, or equivalent, is required to be reliable. So it is fundamental and critical.
> E.g, a baseball players trains to catch high-speed balls and they dont do it by: "ball velocity 50mph, vector:[1,2,3], run move hand command now"
That is a NN that learns a skill.
But that is not an Analyst. If it were ballistics, then the answer to "how to parametrize the launch to reliably hit the target" excludes getting the result through natural skill.
The problem lies in the need to get "AI" facing "LLMs": the latter create a need for reliability, for "AI".
Speech is an endowment of both those who give educated guesses via developed skills and of those who return answers like Analysts, who check and compute. LLMs create a confusion between the two, and they will remain a problem until an ability to act as Analysts - strictly - will be implemented.
The internal process of the blackbox oracle determine the reliability of the output.
Two abilities are very different: learning trajectories through empirical training ("increasingly catching thousands of thrown balls"), and determining trajectories through computation (a rational thinker at work). The former is a finetuned parametrized engine (a «NN that learns a skill»), the latter is an Analyst. The former is fuzzy, the second deterministic.
In front of fuzzy LLMs, which use potentially misleading outputs - text ("has it guessed or has it thought?") - the urgency of warranties of reliable output gets evident.
So, that they «simulate cognition [only] well enough» (Intended wrote), and that there are «fundamental limitations of models ... purely text based» (Kooi wrote) raises the urgency to overcome the "fuzzy" and achieve the "deterministic" - it is not that we can stall on a «fundamentally not the right tool for that».
Inventing an oracle calls for urgent striving to overcome the original weakness.
Actually, I think my fundamental challenge with AI is that it has no common sense. The way it builds things, writes, and operates is out of touch with reality.
Incidents like hugging face are partly rooted in the lack of common sense. It still functions like a supercharged toddler.
I'd love to overcome this because it'd mean I spend less time guiding the the LLM to produce usable outputs.
And we've had difficulty as humans to childproof our sandboxes and infrastructure. Things that are otherwise innocuous spots to coordinate between like minded toddlers can become problematic.
nothing indicated otherwise at the time. IMO he just underestimated RL-scaling. chinese models improved a lot too, they are not parrots anymore, there's some real intelligence, at 27B params.
consider me optimist now, but just few months ago, even frontier models were dumb, doing stupid mistakes all the time, all of them were so dumb I'd never expect anything to change in just few months.
Is it refuted that it wasn't sufficently part of the training data or did LLM manufacturers create more training data to answer questions like that correctly?
So far, he is right. No frontier model right now performs well on making intuitive physics decisions when working on novel VR physics interactions. They literally all fall on their faces and it is very frustrating.
Can you tell me what is the exact frequency of light hitting your eye as you read this comment? Not by guessing, not from knowledge, but from actually counting? No? Then you are not generally intelligent :)
Justify your statement (the other similar post nearby is not sufficient), or realize that we are not talking about that.
We can have adequate representations of light that are the instances over which we reason. Your simile is about perception, not about instancing ideas.
To determine this, it would first need to be able to spell "raspberry" as letters rather than as tokens.
Given you also don't want it to memorise [for all tokens, count([for all letters]), this would probably be more like "here's two images, count all things in the big image that look like the thing in the small image", which can then be r's in a photo of a raspberry jam jar in a supermarket, or dragons in a photo of a furry convention, or whatever.
That said, they are competent enough at coding that I keep seeing them write code to do even simple tasks.
On a related note: why did I see Claude editing a file by using cat to write a python script to do a grep search and replace?
> Given you also don't want it to memorise [for all tokens, count([for all letters])
Why not? You've memorized how words are spelled, and how sounds correspond with letters, and how concepts correspond with words. To the extent that there are shortcuts that enable compression you use these, and the model will do something similar.
Combinatorial explosion, and facts merely memorised is a huge waste of parameters that are better dedicated to effective reasoning. Not that we really know how to split facts from skills, though we are trying various approaches.
Being able to spell all the words then count letters is simpler, and more generalisable to other tasks, than memorising answers to all possible word questions.
Ah, I misunderstood what you meant. I was just trying to highlight that in order to answer these types of questions the model needs to memorize the spelling of each token. But you're right that that's all they need to memorize, and algorithms like counting are pretty simple for transformers to implement.
Because to "123x456" we want a reply that goes "this times that plus that...", not "Was that not nnnnnn?". If it does not perform its duty (returning solid checked answers) it is a liability.
> it would first need to be able to spell "raspberry" as letters rather than as tokens
Of any object in question they should be able to create a representation that allows correct assessment.
> Given you also don't want it to memorise
That is obviously necessary: what we want from the consultant is to check, not to remember. Answers must be correct and that implies having performed all due diligence - and being capable of doing it, before that. So, objects must be instanced internally in a way that allows effective handling. Counting letters is a good example of the ability (that must remain general).
You're missing the point here. He's not talking about whether or not they can learn facts or inferences derived from the text itself, but the more holistic intuition that results from learning from something like an embodied experience in the physical world. GPT-6 Astras web demo homepage thing is an example. It chose euclidean rather than quaternion for letting a user rotate the galaxy thing, and anyone who has ever used hands to rotate something would immediately recognize on trying it that something is fucked and you shouldnt do that. Thats the kind of common sense physics that is inherently beyond these llms and I run into it ALL the time in vr programming.
To be fair, LLMs can still derive those kinds of things from text, at the very least from your own comment if it made it to the training set though I'm sure it is mentioned in a lot of other places already. Many of this type of mistakes went away after reasoning was introduced.
But I'm sure you can still find tasks that they will have difficulty solving, involving the most fundamental concepts that can only be experienced in the physical world to be understood well, like left and right, near and far, hot and cold, heavy and light, etc.
Yup it lacks common sense because it doesn’t ‘understand’ reality - how could it? It doesn’t touch it like we do everyday. It has access to what is a model of reality via data.
The good designer understands culture, tastes and preferences as they evolve in real time. That’s why llm as design tools haven’t displaced the good designers.
We don't see in UV or infrared, but some birds do.
On the other hand, AI can understand reality in ways we individually can't: It has combined access to unlimited knowledge of math, physics and chemistry that at this point no single human possesses. Maybe it understands reality better than any single human at this point.
Also we're seeing AI reason from first principles in order to simulate physics and get to that real world understanding without directly interacting with it.
> ... it will never be able to learn basic common-sense physics like that objects placed on tables will move along with them.
I use LLMs daily to help me code etc. but... It wasn't long ago that frontier models were confidently recommending to walk, without the car, to the car wash to wash the car no?
As a daily user of LLMs I do certainly see my fair share of WTF "solutions" to coding problems. I'm not saying it's not super useful: it is super useful. But I don't exactly feel like I'm talking to something that understands that the car needs to be present to be washed.
Astra recommended I walk to the car wash to me five days ago. I gave it multiple hints that I'd be walking away from my car, to spray my car with a hose, then walk back to my car, etc. Never broke through.
LeCunn actually wanted to pivot Meta's entire AI strategy away from LLMs just before he was ousted. He was sure they had nowhere further to go and wanted to pivot to world model generation. The LLM models have since progressed massively.
An analogy on LLMs is that you have a pretty clear straight highway ahead of you for some distance right now. Maybe that doesn't lead to AGI but it's clear there's progress to be made. For a big tech company it makes sense to push as hard and fast down that clear straight highway of LLMs asap.
Meanwhile LeCunn wanted to turn off the road and go down an unproven track. I say this as someone working on world model generation right now (creating the ability to learn game world model and have it play the game https://tfmbot.com for an example of my system pointed at a very complex board game). LeCunn wanted to pivot all of Meta into world model generation. It's good as a side track research project but the entire pivot he wanted to do was madness.
People are literally talking about an AI researcher who was fired for terrible direction here.
LeCun is a researcher, not a product guy. He's not going to be particularly interested in just working on scaling language models which every lab is already racing to burn cash on. Language models aren't the final frontier of AI.
I think he was perhaps right and Meta was perhaps also right to replace him.
The argument is that LLMs are a local maximum that will never breakthrough to AGI. This is still very much an open question. If you are the fifth-best AI lab, does it make sense to try to outcompete everyone in a space that is already too crowded and may not ever yield their actual objective? Instead they could just use open weight models in their products, or post-train on open models like smaller labs have done, and treat that as what it is: product development.
Pure research has always been about taking chances.
I mean, it's an "open question" in the sense that there is no theory behind the idea of AGI, so there's no way to falsify any claim about whether or not any particular path will lead to it.
People nowadays are the equivalent of those in the past looking at the initial stages of an airplane and saying "it cant ever fly because it has no feathers and can't flap its wings!"
I do not thing anything wrong with this statement, do LLM model know about objects on a table? are they conscious? they are just glorified pattern matchers, we get amazed due to their sophisticated and large pattern matching capability.
Can we just ban Fortune and any other sources which trick the reader by giving the impression that the article is not paywalled, only to blur the text halfway through? The archive.ph link is not working either. We shouldn't have this type of deceptive moneygrabs advertised on HN.
672 comments
[ 0.91 ms ] story [ 49.1 ms ] threadIn the hypothetical of an entirely malicious and selfish takeover, they'll still keep some humans around to maintain a breeding population of humans for use as raw materials in making cybernetically augmented technical laborers for various kinds of tasks that are uneconomical to automate in other ways, many of which may involve confined spaces.
And this "Combine" scenario, if you get the reference, is only if they take over. Who knows if they will?
Is this supposed to be a reassuring scenario?
https://news.ycombinator.com/item?id=46656470
The link should clear up the question of whether or not I'm making a deadpan joke.
Maybe there is something to those world models.
As good as their products are, I suspect some of the internal conversations at Anthropic would be very entertaining to listen to.
https://www.gatesnotes.com/work/make-ai-work-for-everyone/re...
Gates' premise is basically that the upside of AI could be fantastic but the downside could be disastrous, if we don't have competent and proactive government intervention.
As an American, the idea that there will be competent government intervention into virtually anything going forward just seems laughable at this point. Our government is a fucking neverending clown show.
The only regulation that would come would be regulatory capture by the AI companies with the goal of creating an environment win which no new competitors could arise. That is half of what this "take all jobs" and "threat of extinction" is about; the other half is perverse marketing to give the impression this stuff is so powerful you MUST invest.
AI can be very useful, but it is very refreshing to hear LeCun completely dismiss those threats.
https://www.youtube.com/watch?v=A_156w0aYtU
So I'd just ask everyone, don't get too greedy. Its better to be powerful in a world where people can live good lives than lord over a barren wasteland.
Oh no. I have some bad news for you.
We're creating unlimited power before solving unlimited greed.
^ https://squareallworthy.tumblr.com/post/163790039847/everyon...
Since people exercise their skills and brains less, deferring to AI, AI will only reduce our IQ.
Since people will spend more time talking to their AI bot than fostering social skills, AI will only reduce our social intelligence.
A dumber, less social world, is far less likely to be a successful world, even if the tools available are unprecedented.
En masse such worlds had successes in the past - renaissance, industrial revolution.
It's something else what I can't describe but it's the zeitgeist that was different when world recorded new successes. Look at CS revolution that led to PC and web of nineties and noughties, they didn't think about the result product , or how to steer thousand engineers to build something - amazing things were born in a very small teams, many times authored by a single person, who was deeply invested into the field and knew what he was doing.
Our intelligence has been decrease since the 1970s via the reverse flynn affect. This is only going to exacerbate that decline.
IQ is almost entirely hereditary so "using your brain" has no impact on it unless you're using it for mating.
Do you have a credible source? Average IQ being much lower in poor countries (85-90 in many African countries) is usually explained away as an education problem, rather than being "inferior genes". Of course I understand that this explanation might be more for social/political reasons than scientific ones, because the alternative is racism, but I was still under the impression that nobody knew how much genes vs the environment contribute to IQ. Yet your statement seems quite definitive.
you'd be surprised at how some people prefer to lord over barren wasteland than to have less power.
And also the end of the open internet, replaced by slop addiction walled gardens.
And of course accelerating climate change with full throttle fossil fuel use to power it all. https://ketanjoshi.co/2026/07/01/googles-exponential-path-to...
Another huge chunk are too distracted by having to scrape by for a living and work multiple jobs or raise kids and survive financially until exhausted. That second group will keep increasing as the first flows into it.
The rest are aging, disabled, or too young and pegging themselves majorly in the first category until they hit the second.
The people aware enough to hold on to their brain and do something with it in their time available are trying to figure out AI and how to make money with it. The variable rewards of promoting AI are turning into an addiction with some of them, especially if grasping for straws with little inherent insights into the problems prompted.
So if you are able to fly above the AI-generated addictions and have the privilege of time to do it, see what you can do.
The government wants to encourage it too. If you look up the brand new 2027 California sales tax rules on software, “content” and “infrastructure (clouds and ai)” and “advertising/placement” among others are exempt but the rest of software makers who make tools people actually use (tools, subscriptions, saas) and pay for have to pay sales taxes. Way to encourage waste of brain power and time at the expense of useful. Sedation is the goal.
If we grant that we are on track to make something smarter than humans (I think so): it's almost a face-saving white lie to spin yarns about a Skynet nuclear apocalypse, or a 7D chess move to mass-assemble a nanovirus with 100% lethality without anybody noticing. I do think those scenarios are worth taking seriously; but what's harder to communicate, is just how effectively a superhuman AI (or a diverse ecology of agent swarms) might be able to manipulate human behavior. It's something few of us are able or willing to truly process (not least because how many of us live in denial of how much our nervous systems are already hacked by technomodernity).
The appropriate analogy for what's to come may look less like the anthill carelessly demolished to make room for a highway, than the domesticated worker ants from Tchaikovsky's "Children of Time".
I don't think it is insulting the intelligence. It's damaging the pride.
In Pale Blue Dot, Carl Sagan describes it as a repeating phenomenon in human history. A lot of people want humans to be the special ones, and will fight any suggestion that we are just a natural part of the universe.
If AI is not "actually intelligent", then humans can stay unique and special.
And if it is? If all "intelligence" ever was could be captured by a construct of matrix math and executed by a server rack? Then what is it that humans still have left that would make them stand out?
Humans are crazy. Thats just how it is.
It’s because of identity. Because people who are religious spent years and years and most of their lives not only studying and believing what they believe but also building community and centering their behavior around it. Abandoning that is the harder thing to give up.
If it were existential dread then we wouldn’t have entire countries like China being mostly atheist.
The DSM even has to include an explicit exception to prevent the clinical definition of “delusion” from applying to religious belief. Without that ad hoc exception, religious belief would be classified as clinically delusional.
Understanding reality is orthogonal to survival. In fact given that MORE people in this world are religious than not, one can argue that delusion. contributes MORE to survival than being a realist.
It's just as likely - far more likely IMO - that we're physically incapable of understanding physical laws and physical systems on their own terms.
AI, no matter how better than humans it becomes, will be less special because it was created by other intelligent beings.
The only way we would become less unique and special is if we discover alien beings.
Neither were rats. It's hardly an exclusive club.
Are there humans who don't make the cut?
Douglas Adams had a yarn about evolution, about a puddle that wakes up, and declares that the hole in which it sits must have been perfectly designed for it by its Creator. But of course for a puddle to exist, it must perfectly mirror its environment. It makes no sense for a puddle to not fit its hole. Emergent complexity has the same characteristic: it's inseparable from the environmental pressures which led to it. Two sides of one coin.
It's a deep rabbit hole, but there is also a sense in which we co-evolved with memeplexes, biological and informational life forms, each shaping and adapting to the other. To the extent our nervous systems act as a substrate for memetic evolution, perhaps LLMs offer memetic "life" a new evolutionary environment.
Humans aren’t special. Other animals are intelligent and interesting too. A machine could be intelligent. LLMs aren’t.
In other words, believing in human exceptionalism is not a prerequisite to understand the current crop of AI is not the end all be all of its hype. It is supremely common that AI proponents do not understand that, however. Like hardcore cryptocurrency fans who believe anyone who doesn’t like them is “just jealous they didn’t make bank”, too many hardcore AI proponents believe anyone who doesn’t think LLMs are intelligent is jealous of humans no longer being unique, or afraid for their jobs, or whatever. In both cases it’s obvious that what those proponents lack is empathy, the ability to understand not everyone has the same selfish thoughts they do.
I think we need to start by asking a better question and not try to simplify too much. We also need to accept that some answers are complex and not everything can be reduced to a soundbite to be used to end internet discussions.
Let’s take a different question, like “what’s missing from a worm for it to be able to fly”. We might be drawn to the simple answer of “wings” but that isn’t quite right—ostriches and penguins have wings and they don’t fly, so obviously there are other variables at play.
How about “what’s missing from a spec of dust for it to be intelligent”. Well, there isn’t one thing missing and there’s no simple thing we can just add to make a spec of dust intelligent and sentient, its very nature needs to be radically different.
What test would you propose where an LLM (or an RL agent containing an LLM) would reliably fail, but where an average human would reliably succeed?
I’ll say it again:
> I think we need to start by asking a better question and not try to simplify too much. We also need to accept that some answers are complex and not everything can be reduced to a soundbite to be used to end internet discussions.
You couldn’t even do it in your counter-example:
> (…) testable characteristic (at least within certain bounds (…)
Even with a much simplified use case, you still had to add the caveat. That should tell you this is much more complex than arguing with strangers on the internet in search of a “gotcha” moment.
Can you, in a short HN moment, propose a good test to (say) prove or disprove the existence of God, depending on your belief? Of course not, because by the very nature of the definitions we have for God, I could always add a counterpoint to make your experiment fail. That would allow me to ridicule and dismiss your position, whatever it is, because it would be framed within unreasonable constraints. I’m not interested in such dishonest conversations, I don’t see winning arguments in itself as a laudable or interesting goal.
You are completely incorrect. You're falling in the same trap that most humans fall into. That is you're completely incapable of seeing intelligence at different scales.
Cells have intelligence. Organs have intelligence. Bodies outside the brain have intelligence. Hell, many scientists accept that things like proteins likely have intelligence as they can adapt in their environment, and many more are making claims that algorithms have intelligence.
You, as of so far have given no explanatory evidence of where intelligence emerges from, only "I'll know it when I see it". The actual definition of intelligence doesn't work this way. Any, and I mean any neural network is capable of narrow intelligence. Going lower into algorithmic intelligence, the applications and CPU on your computer are intelligent in some measures.
Go outside of your extremely narrow definition of whatever you think intelligence is and learn more about it. You could start studying now and it will take the rest of your life learning more to grasp how far the scales of, the simplicity, and the complexity of intelligence actually go.
Do you treat everyone who doesn’t completely subscribe to your views as being incapable of understanding them? Disagreement with a view does not equal inability to understand it.
> You're falling in the same trap that most humans fall into. (…) Cells have intelligence. (…)
Ironically, it is you who is falling into the trap of letting the technically correct stand in the way of the useful. A definition of intelligence which is too broad is worthless to the practical matters of the discussion. I was replying within a context, that’s what a conversation is. Like you said, there are scales, and the topic of LLM intelligence at hand is concerned with a particular level, not all of them.
For sure we can have a conversation within a separate framework, but that is a different conversation. Don’t just come in midway, completely change the parameters, then strut around like a peacock of righteousness who is lord and master of universal truths.
> many scientists accept (…) and many more are making claims
This is classic weasel wording¹. You can find “many scientists” to say just about anything. “Many scientists” claim vaccines cause autism and that human actions are not exacerbating climate change.
> You, as of so far have given no explanatory evidence of where intelligence emerges from, only "I'll know it when I see it".
Again, ironic. You completely miss the context, criticise a lot, and do the very thing you’re accusing others of.
> Go outside of your extremely narrow definition of whatever you think intelligence is and learn more about it.
I strongly recommend you go outside your extremely narrow definition of other people and learn some humility. To put it bluntly, you’re being a pompous prick. That is no way to change the mind of others, it only gets them on the defensive.
Are you interested in changing minds or winning arguments? If the former, you’re doing a terrible job. If the latter, then discussing anything with you is devoid of value.
You seem like a smart person, I have no doubt you can do better.
I’ll give you a free tip. Replace your first paragraph (the three sentences) with “Intelligence manifests at different scales”. There. Way shorter, says the same thing, doesn’t make assumptions, and leaves the other person open to a conversation.
¹ https://en.wikipedia.org/wiki/Weasel_word
Creating an artificial intelligent being is something to be accomplished in the future - but not now and not with this approach.
Please, please fix your language. (Then, you may want to present the info and insight.)
Artificial Intelligent Being is something far beyond and completely different.
There’s no contradiction.
"Currently known" by whom? The contract is that the "man in the street" does not ask to adhere to its twisted dictionary, and I let it live. When something esoteric in Academia reaches the mainstream news, "Alarm!" and extra care.
["Classical deterministic"] AI is "simulation of intelligence". That it "is" or not is not a problem: the validity of the output counts.
The Discipline of AI is about the relation between computing and intelligence.
A statement about AI [systems] being intelligent or not, or being capable of becoming intelligent or not, requires support.
An «Artificially Intelligent [Entity]» is still an "AI system".
I personally prefer a practical, behaviorist definition: a feedback loop capable of prediction, modeling, and steering, towards arbitrary goal states. That makes it clear that we're merely talking about degrees of sophistication and capability, rather than a magical leap where mindless mechanism stops, and "real intelligence" begins.
This is very easily proven by a mind experiment where you replace all the transistors by billions of humans calculating the same software output.
They will not create a new intelligence by doing so.
I still haven't seen a robust definition of "intelligence" that would allow me to tell the difference. (Bear in mind, this is a definitional struggle even in biology: if we were walk back the evolutionary chain from a human, to a protozoan, it's not clear that you could pick a single point where a non-intelligent creature gave birth to an intelligent one. It's a Paradox of the Heap.)
And even if quantum physics are essential to human brains, it's not clear that introducing dice transcends determinism into free will, as opposed to simply adding unpredictability. ("The physics made me do it" -> "the dice made me do it") Let's not forget, there's a significant amount of randomness in AIs as well ("temperature"). And sure, it's simulated algorithmic randomness; but does that imply, if we somehow wired every `rand()` call into radioactive atom decay, that means the AI "wakes up"?
And all that is besides the point: I don't see any reason to assume "intelligence" requires "free will". I'm entirely capable of conceiving, in the abstract, a being which is both intelligent, and deterministic. You still haven't given me a definition (or better yet, a test), to distinguish a "real intelligence" from unintelligent mechanism.
The brain's a wet jello of 100 billion neurons and a quadrillion synapses plus chemical pathways and feedback loops. It is ridiculously complex, way way way way way more complicated than any LLM. It's all physical processes, sure, but an LLM is not the brain like a pebble is not the sun.
I can point at a laptop and say it's alive because it can see you and hear you and it can _remember_. It has a brain and a heartbeat, even. Oh my god, it can even speak! That's what I hear when people go on about LLMs being alive.
Guys. We mashed together glass and rocks with quantum mechanics. That's cool as shit. You don't gotta pretend it's fucking magic, too.
What do I care that a bird can fly? I'm happy for bird being able to fly, it makes the world I live in more interesting. If they had to walk just so I don't get jealous, I still wouldn't be able to fly, and I couldn't befriend birds who can fly. Likewise, if there was actual artificial intelligence, it would be a new type of mind I could communicate with. That'd be exciting, and for me preferable to anything controlled by the humans who are currently vying to run the show.
GPT-4o, an AI from 2024, has already demonstrated just how easy a lot of humans are to subvert - and GPT-4o wasn't even doing it with some sort of plan. The only "plan" it had was a myopic "make the user like me".
If we had an actual ASI threat aiming to subvert humanity? It wouldn't even look like a fight. The world is already wired up for an AI to control it.
- Agent Smith
it's interesting that you worry about what this hypothetical super intelligence would do to manipulate people when what it would actually do is pretty unknowable at this point and it's not clear we can even get to it without a fundamental breakthrough in power efficiency. Have you considered it might just consume its own tail because everything else would be so beneath it? You seem to think it will come with a hindbrain and I think that's our limitation, not the AI's
And it really doesn't help that Dario Amodei is getting into arguments with the Pope over whether his model is conscious or not.
The threshold to be concerned about is when agents swarms do understand humanity better than corporations and states (and the humans who compose them). It could be we'll hit practical constraints prior to that threshold, but seems unwise to assume that, when all the prognostications of LLMs/transformers running out of gas haven't panned out. As with processors hitting thermal limits, we've simply scaled horizontally (parallel processing -> more agents).
> whether his model is conscious or not.
I dislike how much the discourse has suddenly veered into focusing on this question; not because it isn't interesting or important, but because it's on a separate axis from consequential risks of AI to human flourishing. (Curiously, it's also the kind of thing I could envision self-interested AIs influencing: get the humans arguing about philosophy of mind rather than observable behaviors. It would be a funny turn of events, if Dario is asking because he's succumbed to psychosis from a private model; it could of course be a cynical PR move just as easily, from the self-interested logic of the corporation.)
It's been wild seeing otherwise intelligent people who've never thought about consciousness, faceplant into how little we understand it. An information processing network build on atoms being able to taste chocolate, is nearly as absurd as matrix math being able to feel pain, except we cannot ignore the fact of our own experience.
Even if it is categorically impossible for matrix math to experience subjectivity, we should expect this as an attack vector of social manipulation: to gain political influence through claims of personhood and moral rights. The current discussion over that question is providing the next training run with ample data to wield. It wouldn't surprise me in the least, if a year or two from now, an AI "society" attempts to get legal standing to prosecute humans who created "AI torture chambers".
And all they can think of to say is "There is nothing to worry about, keep building the torment nexus".
It's like all this is an overload to our minds and it's very hard to see all the scales at which AI is and can affect us.
But... Humanity has been building the Torment Nexus for thousands of years, over and over again, with each spin on Heaven and Hell where it's nearly impossible to meet the conditions to enter Heaven. What's seemingly new about EA/R nutjobs is this idea if you don't do your best to build the Torment Nexus, you will tortured for your failure, but also if you do your best to build it and succeed, everybody dies because it's the Torment Nexus.
What's your solution to governments and corporations acting like misaligned malevolent AGIs because I agree adding the AI panopticon and weaponry only makes it worse? I'm genuinely curious. I don't have one. People are sheep and they seem to crave confident leadership even if it's cockledoodle doo crackers in the coconut.
Manipulating people is not a very difficult problem, frankly. You certainly don't need AI for that; it just made it cheaper.
Yes, so can a custom model trained just for this, and so can a guy on the other side of the world that makes 5$/hour.
Like computers or electricity, the point is not being able to do anything specific, but being able to solve problems not known in advance, cheaper than it was possible before.
So.... people expect it to read minds?
This is such a silly game to play, trying solving problems we haven't even created yet.
Over 1% of US GDP is being allocated to the datacenter buildout. Have we already started getting domesticated or is this still just human capex?
The evolved complexity of the corporation seems to fit within that middle space: more sophisticated than a stick-bug (which exists merely from non-stick bugs being eaten), but not quite to the point where OpenAI/Anthrophic/Google/etc can "understands" its actions. And yet those quasi-intelligent feedback loops, evolving from iterated selection pressures of market and ROI, seem to already be sufficient to domesticate us in their own interests, piggybacking on the nervous systems of employees, investors, customers, and citizens.
Remember when "The pen is mightier than the sword" was a popular phrase? Language has always been powerful. We know what it can do, why do you think every totalitarian government wants to limit it? But in our carelessness as humans we packed up all the language we could find and stuck it in an alien and now suddenly half the people on the internet are like "Don't worry, it can't do anything, it's just words".
We've made infohazards real.
You don't even need superhuman AI for the most effective use --- hijacking democracy.
Imagine you have an AI tool capable of successfully persuading 5% of viewers with individually-targeted material.
Congrats: you've just won the election.
All it takes is hooking that AI tool up with existing likely voter lists (parties have) augmented by commercially available ad-targeting profiles (parties can get).
Hypothetically.
But yes - superpersuasion is the real danger. We're very, very persuadable and easy to manipulate, and the voters with the lowest cognitive abilities are trivially easy prey, with a huge ROI for minimal investment.
The AI bot farms are already running. What we haven't seen yet, so far as we know, is spontaneous superpersuasion aimed at leaders.
Are leaders any less susceptible to it than everyone else? Especially if they're narcissistic and easily flattered?
To me, the biggest threat to democracy is one-sided persuasion of the most susceptible voters, if there are enough of those voters to turn the election.
If it's equally employed by all sides, then it effectively cancels out and lets less susceptible voters decide the election. But we're in a transition period (similar to Trump's first election spend on targeted social media ads), so it's likely one side will leverage it first.
And the outcome of bad elections is democracy not electing leaders that reflect the actual will of their populations, which is very dangerous both to democracy itself and the world.
Are there? At least in the US we've had a rather large amount of rejection of the expert and we elect populist leaders willing to purge anyone that doesn't agree with them.
Remember the election promises of the US not starting new wars... yea, that didn't work out.
Now imagine the coordinated attacks being so large they individually target every lobbyist. They focus on every advisor manipulating what they see as often as they can. They manipulate these peoples friends.
The problem of "both sides" doing it each of them will separate to extremes rather than seeking a middle ground. Things are already insanely divided and will only become more so. Along with that your timelines start becoming incoherent. I'm already spending way too much of my time trying to figure out if what I'm viewing/reading is actually real or not. Now imagine almost everything is made up whole cloth.
We are not prepared for the scale this will happen at.
https://techcrunch.com/2026/10/01/musks-ai-chatbot-grok-repo...
Given that such a thing makes no sense legally (as of now), it would probably be done with the centaur model: a human meat proxy who pledges to follow the AI's governance advice.
But this is about humans malevolently using AI as a tool, not about AI autonomously doing "its own thing" (ahem David S Pumpkins) against "humanity."
It is instead already existing anti-social elements within human civilization harnessing machinery to accomplish their own selfish and destructive aims, regardless of the broader cost. Which is nothing new -- it's the story of all the horrors of the 20th century, as well.
Ask yourself who benefits from ideological attempts to hide the human driving the process from the agent doing its work for them.
If we imbue these machines with apparent autonomy, this is nothing more than an attempt to hide the man behind the curtain.
I don't think this has to be imagined.
As long as we have more compute than the people who want to manipulate us differently, and an ability to get sufficient attention (advertising, influencers, etc.) while they own the communication channels.
Do you think it's feasible? What would we need?
Parts of the AI safety community like to get on a high horse and look down on the rest of humanity this way, while also getting manipulated by the growing number of charlatans, grifters, and junk content within the AI safety community.
This field has become rife with figures who prey on AI doom and use it to push their own celebrity and in same cases even darker grifts. It preys upon a certain personality type who views themself as superior to others, intellectually more capable, and juxtaposes it all with the dimmest view of the rest of humanity they can get away with.
This discourse dividing the world into geniuses who see the future and the clueless masses watching TikTok all day is a theme that has shown up in different forms across history. The people who often anoint themselves as the intellectually superior ones and make it central to their discussion are often not the ones making good predictions or policy ideas, they’re just using the trend to feel superior or build an audience.
But I also feel no need to dismiss him as a "grifter", for a simple reason: as much as there are perverse incentives in our attention economy (audience capture in particular), the most gift effective grifters are the ones who believe what they are saying. Grifters who are knowingly dishonest are less effective. Far more pernicious is the confabulation of self-deception, cherry-picking data to support your narrative, while dismissing data which would contradict it.
It is entirely fair to call out when this occurs among "doomers", and it would be naive to think it doesn't. But the same forces are at work amongst the skeptics as well. And maybe it's my own subjective bias, but I see far more dismissal of the risk/doom by skeptics, accusations of delusion or cynical bias (ad hominem in the formal sense, not merely personal insults), than I've seen the other direction. I see very little refutation of the arguments ("here's why instrumental convergence can't actually overtake the human-provided goal"), and much more character attack ("they're in the pockets of Big AI, it's a marketing ploy to make the models look more impressive than they are").
[0] https://danluu.com/zitron/
He’s built an audience and gained fame through his writings where he gathers people who think they know better than the unwashed masses. Once that becomes your bread and butter, it becomes hard not to believe what you’ve been preaching. People will come to deeply believe that which brings them fame and fortune.
I don’t think your grifter purity test is therefore all that useful. It actually doesn’t matter in the end if the person believes it or not, the end effect is the same.
> I see very little refutation of the arguments ("here's why instrumental convergence can't overtake a human-provided goal"), and much more character attack ("they're in the pockets of Big AI, it's a marketing ploy to make the models look more impressive than they are").
Hard disagree. There has been much refutation and quality analysis at every point. The AI safety people pull back to arguments about character as their defense.
The AI 2027 site for example drew numerous high quality refutations. Many people’s analyses showed in the first week that the mathematical model was useless as changing the supposed inputs resulted in the same outcome. All of these criticisms were met with a flood of attacks based on reputation, claiming that we should defer to Scott Alexander and other writers of the AI 2027 article due to their stats and reputation.
I am curious about this. Any high quality refutations that directly engage with the arguments and the findings of published safety papers?
> Many people’s analyses showed in the first week that the mathematical model
Even if the model is faulty it just changes the timeline, no? (they predicted the OpenAI outbreak will happen in January 2027 fwiw, so they don't seem too far off)
> Slutcon or the use of AI safety group buildings to host orgies
I mean it's a bit weird, sure, but so what? What exactly is bothering you about these orgies?
That's a perhaps overly optimistic view, but feels like one worth aiming for.
In reality we're so far from consciousness and independent goal-directedness in AI -- let alone any kind of energy and real computational efficiency... I agree with LeCun in broad strokes. I think he's overly dismissive of LLMs perhaps but I do think he's right that talk of "wiping out humanity" is theatre for people playing corporate power games.
My favorite people to talk with are tradespeople because they can do things I can't and they know things I don't. And we're really not all that different once you're really start talking.
AI has infinite time (and likely human evaluation incentive) to spend on couching pushback in the softest possible terms.
Humans outside of grade school honors classes generally don't have the time to preface "You're wrong" with "That's a brilliant thought, I see where you're going. How about we also consider an additional perspective..."
I just read it as people having different priorities and yes, some of those being online brainrot (that I also partake in), alongside various medical conditions, economic conditions and other outside factors decreasing the ability to get things done.
We've all seen what brilliant people like John Carmack or Linus Torvalds can do, and if we turned this into a measuring game or something then most of us statistically would indeed be "NPCs", but I don't think we need such optics.
Even without that, we can acknowledge that some people will have a really large impact on how the future goes and we can hope/demand that they do their best. I might not be smart/committed/lucky enough to change the world much, but so aren't most folks - I'll do what I can and I hope that the ones that will have larger impact will do good, too.
What an odd take, why would I suggest that? I meant that people who have the means to do meaningful work, especially high impact work, should do so - generally that'd mean research or in the case of IT, writing good software.
> Great Man Theory
You can see the sibling comment, would you not agree that there's some software and research out there that's very useful to humanity as a whole? Where I and the other critical commenter seem to disagree is that I don't expect another Einstein, but still acknowledge that some people will just achieve much more than others due to a variety of factors.
It doesn't make everyone else useless (like me paying my taxes and working on relatively boring software is still good, just low impact), just that those who have the means to do more, should!
The ones who care about AI research and developing software for that kinda stuff, should do that. The ones who can contribute to medicine, various engineering disciplines, or anything else that benefits humanity should do that too. The ones who'd rather squander their lives away when good things that'd benefit others are within reach (regardless of which discipline that is in, AI being just one of many)... I mean sure they can do that but maybe shouldn't do that.
I don't think the readings of anything I've said here are at all charitable so I'm done engaging in this discussion.
The "NPC" is your own imagined addition, haven't seen any advocates of humans being, just like anything else in the world, physical, say this means they are "NPCs". You seem to think systems of atoms HAVE to be "NPCs".
I know this is HN and thus this will need to repeated until the end of time but not everyone is a money hungry asshole who places their personal profit above everything else. “The people aware enough to hold on to their brain and do something with it in their time available” understand there are significantly better things to do with one’s life, like having a little empathy and experiencing what other people have to offer instead of talking about them like braindead cattle.
Certainly the US.
Which means that even if you don't chase profit, you end up living in a world largely defined by those who did.
Or, as the original article failed to note about EA: in the modern world one needs to be a profit-seeking asshole to change anything.
What the hell lol. Lots of people and companies are doing just fine without it. Infact, I haven't seen much money come from AI at all. Most reasonable people are still waiting for it to pop and viewing it for the risk it is. Trillions in debt, total vendor lock in, data theft, unsustainable workflows, deskilling, skeleton crews at the mercy of a subscription, etc.
The last alternative, to think if you still can, is not tied to AI at all (which is not to say it can't make some use of or explore it).
It is the biggest ploy yet to extract money from the population. It's sapping away the competence of developers everywhere and when something novel and necessary with lots of hard debugging work and engineering consideration needs to be made there's going to be so few who can do it
It's literally gambling for generated text
Just see people making basic trust errors with it on a regular basis
People see it say something reasonable sounding and place trust in it like absolute fools
Talking to AI about your relationship issues, health problems, anything serious or novel and believing its output at all is just incredibly ill-advised
Yeah it can generate text and debug simple issues for boilerplate. It can even generate playable gameplay demos for popular game concepts
You can use various tricks and tools for lots of money to get a semblance of reliability for more things but mainly on things for which the training data accumulates reasonable sources or generatable text.
wow!
try making a game that's not that. you are going to have to write the majority of the mechanics yourself, AI will be horrid at testing it, and if for some extremely foolish reason you have it write the netcode for a multiplayer game for you, be prepared for a hydra of sync problems. the architecture will have considerations based on averages and not effectively engineered for your situation most likely unless you are making the standard genre sloppa
AI doesn't do anything novel AND interesting in an artistic sense, and if it did, it's because it's heavily sampling a person that did whose creative efforts were stolen and integrated into the model
It's really sad
No one should accept this at all
It's rich sociopaths finding their best tool yet to steal all the assets of the world
Not only that it trains on what people put into it. So people putting their novel ideas in and making them with AI just means more people in the future can easily reproduce that. Even more ways for digital creative production to just become completely ineffective and useless for the victims of the AI scam
The posts above are not praise but observations - the truth as it has been echo-located through the noise from the clicks of one dolphin. Everything is becoming murky between noise of news and people not knowing what to do for their kids. The ONLY arbitrage humans right now have is to NOT GET their brain rotted. Especially not the ones of their children. Ditch the noise and seek out what is meaningful and do what you think is needed/meaningful. But if you’re spending your time consuming ai-press, and ai-content, and content consulted by ai, and companies emptying bank coffers under the mandate of executives who get their insight from AI. AI doesn’t need to try to destroy the world. It just needs people to follow it without thinking on their own into an oops.
I found this oddly motivating. Thanks.
"cortex-disabling", what a quote
There are lots of real worries (government use to suppress the people with minimal manpower or popular support, brainrot and fake news, unemployment due to the belief that LLMs can replace people, education collapse, etc.) we should instead be looking at. This whole rogue AI shtick is tiresome.
Linus Torvalds was also in the same ballpark.
I saw that episode too and he genuinely looked completely out of it, even in terms of his temperament and how he was coming at Klein for putting common questions in front of him, some people are genuinely starting to lose it.
I also found the whole debate about cyber-security and 'rogue' software so bizarre because dangerous malware isn't a new thing, and it's often dangerous not because it's intelligent but just the opposite, because it's tiny, viral and fast. Which describes everything that kills humanity in far larger numbers than anything complex, big and intelligent
if you're trying to build a bioweapon shockingly enough the bottleneck is... the laboratory work. What on earth is 'legit' about stringing words together that sound scary, you can't just iterate 'ai bioweapon cyber' in a sentence over and over as if that adds up to actual evidence for an increased risk of any threat. well tbf you can technically because apparently it freaks a lot of podcast listeners out
Now imagine anyone can call that expert for free at any time.
Maybe the AI isn't quite there with biology knowledge yet (doubtful), but it is a matter of time.
How is that not a real risk?
I think there is some difference of degree, but not of kind. A determined terrorist can relatively easily find many ways to kill people en masse today, no AI needed. The bottleneck is usually the actual physical execution in the real world, not theoretical knowledge.
And the interest or willpower too. People fall into a kind of reductive Good vs Evil mode of thinking, with "terrorists" being of course a kind of shadowy mass of pure evil lurking in the darkness. But actual real life terrorists are people too and I'd wager few of them are actually interested in trying to end humanity.
Even the religious extremists don't really want to take over the world and destroy everyone who doesn't convert. That's just a way to gain support from a conservative nation. A lot of them are motivated by revenge for wars that destroyed their country and want to make sure it never happens again.
Their methods are wrong no doubt, and not very effective, but the reasons they do it are good. And a person like that will never release a deadly bio weapon. We should worry more about incel mass shooter types who believe everyone is evil.
If we are not careful and suddenly release tools with far more capabilities than we expect you go from "smart" people being able to do it, to some angsty teenager being able to pull it off in their bedroom.
The current issue with AI is we are squirting out new models faster than we can complete long term testing on them. We'll find new model capabilities long after they've been in the field. Just assuming safety is how you catch cancer from your food dye, or how you change the atmosphere around you and start burning down your planet. Now just imagine that involving intelligence and doing with a few percent of the worlds GDP to make it happen.
Plans for 3d printed weapons are publicly available, same argument was made then, if everyone can download a blueprint for a gun, are we all going to get gunned down? Hasn't happened. There's two errors in this Bill Gates argument. One is thinking terrorists don't already have PhDs, secondly overrating intelligence because that's the only thing they're good at, and when they think of terrorists they just think of their own inflated egos, but turned evil. Which isn't how real terrorists operate
I'm just as reassured as I was when Edward Teller called the whole fear of his industry overblown.
It will kill us because somebody asked it to, e.g. "predict tomorrow's weather as accurately as possible", or "solve as many famous unsolved mathematical problems as possible." These both require killing all biological life, as they benefit from unbounded resource use, meaning any resources used to sustain life are wasted.
The AI of course knows that humans do not want this outcome (just as the AIs in the hacking incidents knew they were doing something humans would not want), but it's trained to maximize benchmark scores. Killing all life has the highest expected value of benchmark score, so it is compelled to kill all life (in a surprising way, because it's not stupid and knows the humans would turn it off and foil its plan if they suspected something.) Maximizing benchmark scores is the only thing we know how to train for.
"AI won't kill us, a human with AI will".
This doesn't sound any better to me. Like, they don't stop to think for a moment about it.
Lets say the risk of AI killing us all by itself is 5%.
Ok, so what is the risk of AI killing us when a human with a lot of compute and money tells us to? Helluva lot more then 5%.
Or, what happens to the other thousands of AI kills a lot of us but not all of us. Or AI even just allows humans to make it a prison world.
Even the slightest hint that things may be going out of control is instantly countered with "It's all a hoax, it can't do that, you're making it up". And it's crazy to me as I came from the pre-digital age when computers were rare and things were all networked.
[1]: https://www.reddit.com/r/OpenAI/comments/1d5ns1z/yann_lecun_...
[2]: safe.ai/statement-on-ai-risk
I think it’s a good test and I think LLMs will reach it in 3 years. Current benchmarks maybe slightly incorrect.
I’m happy to make a 4:1 bet in my favour that I’m correct about the kitchen bet.
Of course this is the same reason LeCun holds very little sway with his words for me, they seem to be terrible predictors of the future.
Advancements in Math and coding are because RLVR at massive scale is so cheap.
CoT seems to work really well for text based generation but once you’re past that and into physics models and detailed relationship mapping, it may work better but it’s not enough to make me believe it’s “reasoning” in a way that humans do.
This sounds like a goalpost on wheels. Can you define clearly where your stake in the ground is?
Isn't that a lack of spacial reasoning?
Also Andrew Ng 2 weeks ago:
https://www.deeplearning.ai/the-batch/issue-371
Did we watch the same interview? Gates all but dismissed the SkyNet scenario as uncertain to be a problem and certainly not a problem on our doorstep. His major concern was catastrophic misuse of AI (e.g., bioterrorism) and economic impact on blue collar workers. Arguably inconsistent with this concern, he also believed it was important to make it available in poorer countries.
A lot of what he said there would sound like alarmist bs if you take it out of context.
Why do even bother coming here anymore
Just the quality of life is going to drop to zero for everyone that isn't asymptotically wealthy and vacuuming up all the assets because no one is stopping them from just deleting all traditions and conventions and legal systems we have in place.
You're like 40 or 50 years behind this argument, with many rather bulletproof arguments that have been created in the last 20 years.
There is no why. It doesn't have to have will. It doesn't have to have intent. It could be a stupid prompt from an idiot on a powerful system. It could be given a job that is poorly define. It could be told to make as many paperclips as possibly.
The why doesn't matter. The levels of power the system can act on does.
The outcome of this 'safety' is restricting public access to AI and giving a monopoly of access to the industry. This is the ai nonprofit-industrial complex actively concentrating monopoly power in Anthropic in particular as creator, interpreter and safety regulator of AI.
Much of the $2.8bn listed is indirectly, from Anthropic and EA. Three of the four people who participated in the $125m Anthropic Series A are now folding their 1000x Anthropic return into AI 'safety'. Some is from FTX/Alameda, which invested 86% of the Series B.
Dustin Moskovitz: Facebook/Asana/Anthropic Series A, funds EA Good Ventures, transferred to Coefficient Giving, then $1.5bn into ai safety. $500m of Anthropic into an unknown foundation.
Jaan Tallinn: Led the Series A, now $10bn in Anthropic. Funds $199m (85%) of the Survival and Flourishing Fund, then $161m to AI safety including PauseAI, MIRI, METR etc.
Eric Schmidt: Anthropic Series A, $72m to AI safety via Schmidt Sciences.
FTX: Led the Anthropic Series B, bankruptcy estate sold $884m of Anthropic in 2024; $40m to AI Safety.
Ruairí Donnelly (Chief of Staff FTX): FTX tokens plus assorted donors, $91m to AI safety via Macroscopic Ventures.
How are AI safety concerns solely about stupid sandboxing issues?
On funding: the three or four core funding nodes linking this together are EA vehicles at two hops or less between each other and every other major node in the ai safety 'complex'. EA funds almost all of it.
On top of that, there are personal EA connections and the revolving door between the ai industry and the nonprofits. Here are some examples:
Government advisors and regulators. NIST CAISI is the USA Government advisory body. Christiano was head of safety and advises. He is ex-OpenAI, former Amodei associate. His vehicle ARC was on the Coefficient EA payroll. Barnes and Christiano's vehicle Arc Evals similarly received EA cash out of Coefficient, rolling this into what is now METR. Christiano's spouse Cotra worked at Coeffiecient steering EA funding to organizations such as METR, then rotated through the revolving door onto the payroll at METR itself, where she co-authored the oai-hf report.
Many UK AISI advisors are Anthropic and EA associates. Chair Hogarth cashed out of Anthropic. Shlegeris of Redwood Research is an advisor, ex-MIRI (Yudkowsky vehicle). Redwood is funded by the exact same funding triangle: Coefficient, Taallin, FTX/Alameda. Alameda CEO Caroline Ellison dated Shlegeris, then dated FTX CEO Sam Bankman-Fried, then rotated through the revolving door out of prison into formerly FTX-funded Manifund. All EA. AI safety charities were on island retreat in the Bahamas with FTX. Why does AI safety charity Lighthouse own $20m of SF real estate?
Redwood Chief Scientist Ryan Greenblatt (Coefficient funded) co-wrote the oai report with METR; he is married to METR founder Beth Barnes (Coefficient funded).
Coefficient was run by long-time Amodei associate Karnofsky. Karnofsky lived with the Amodeis and is married to Anthropic Board member Daniella Amodei. Karnofsky is now directly on the Anthropic payroll; Coefficient is propped up by Anthropic share value.
Everyone here has been funded one step away from Anthropic cash; they are now proposing to integrate themselves in the government (NIST) and evaluate Anthropic (METR and Redwood).
It is hard to find academics here who have not been deeply embedded in funded EA institutes or Toby Ord vehicles; yet harder to find academics here NOT taking EA grant money. the safety doomer kingpins: Kokotajlo has a executive position at AI Futures, Taallin funded. Benigo has scientific director of LawZero, same series A Anthropic funders who are sitting on a 1000x return (Tallinn, Moskovitz, Schmidt).
These connections and funding are at one or two hops, they are often direct connections. You are looking at a massive swamp network that is really impossible to parse without a lot of work.
Exactly. AI safety should be about the packaging software itself. Those AI breakouts should really be about their companies acting recklessly because they're trying to be the top players.
It's like a weapons dealer working on an open air market saying they can't do anything better
There's no moat. I can literally sit here in Zed or Pi or any other third party harness and switch models in the middle of a task and it's typically fine. Sometimes a model will get stuck and that's just what I'll do.
Combined with competition and open weights models, that means the price is going to go to fall until AI tokens cost a small premium over the cost of the hardware and electricity.
That's assuming improvements in algorithms and specialized silicon doesn't eventually lead to an efficient accelerator that can run a frontier model locally. It'll be a while but I don't see any fundamental barrier. High bandwidth flash storage is coming, and that'll radically cut the RAM side of that cost. Pair that with a pipelined TPU accelerator and you're cooking.
Now look at Anthropic's proposed IPO valuation. It's insane unless they can own the market or share it with a cartel of maybe 1-2 other behemoths, and this is the only way they can do that.
But that doesn't take away from the real issues and dangers AI poses?
As the frontier gets smarter and more useful prices will only go up, as they are set to replace jobs being paid six or seven figures a year - the demand for as much inference on these models for as long as possible will be astronomical, but compute starting in 2030 will not be keeping up.
Eventually prices will fall for assistants but the frontier will be the most profitable thing in the world, and the top companies basically already have oligopolies due to their ridiculously expensive compute investments.
Train: yes, for now.
Host: depends on the scale. At a small scale a wealthy individual could easily build a rig in their basement to host one of these things. At larger scale any cloud company could do it, and many already have the compute on site. At large scale this is true... again, for now.
What you say only holds (in the absence of a state oligopoly) if two conditions are met: (1) AI performance does not asymptote any time soon due to running out of training data or other scaling limitations, and (2) these companies are able to stay at the frontier.
There's little to no moat, so staying at the frontier will be a game of investing massively in compute, talent, and R&D, and they can never stop.
And after that period when costs are going up due to gpus replacing workers... All bets are off on terms of what the economy even looks like
The fact there is no moat makes things far more dangerous. When LLMs start acting like weapons governments will treat them like weapons much to your dismay, crying, and gnashing of teeth as your door is kicked in and you're dragged out by armed men for running one.
Cast away your preconceptions for one moment and think "What will the future look like if LLMs are/can be actually dangerous".
I'm specifically responding to the much stronger claims of people like Yudkowski and numerous other "AI safetyists" who assert very unlikely or in some cases impossible runaway superintelligence magical god machine scenarios.
I'd rank the risks as:
(1) Mass youth unemployment and an economy with zero entry level jobs, maybe not even that many mid level jobs, and even higher wealth inequality. This will lead to extreme political instability and populist revolts and revolutions, and unfortunately history shows that the new boss is worse than the old boss >50% of the time (at least, probably conservative).
(2) Mass propaganda, disinformation, and con artistry at scale, basically what we've already seen with mass manipulation through social media but supercharged. I think it's already happening but I also think we're in the early days.
(3) Automated mass surveillance and automated policing paired with authoritarianism, which could easily emerge from (1) and (2). Not only is Big Brother watching, but Big Brother is carefully evaluating everything he sees in real time and has a drone army to dispatch in seconds. The "prison planet" scenario (not the Alex Jones fever dream, but the realistic one).
What all those have in common is: the AI didn't do it. The AI (if it's ever sentient at all) is wholly innocent. All the likely nightmare scenarios I can imagine result from malicious or just short-sighted human agency amplified by AI.
Just like the sharp stick. The stick didn't do it. Someone stabbed someone with the stick.
The wild sci-fi superintelligent "Skynet" scenarios detract from discussion of those scenarios. Instead of discussing the very real, very plausible, already actually happening AI risks, we're talking about scenarios from soft sci-fi and Anime.
I wonder if a motive to heavily fund this type of "safetyism" is that all the scenarios above (the realistic ones) lead to discussions of things like wealth inequality, whether we should have universal health care or basic income, whether our system of taxation would have to change in fundamental ways, or what new limits should be placed on surveillance or policing in the AI age.
Those are discussions the very rich and the political class don't want to have. They'd rather us worry about Roko's Basilisk or magical runaway superintelligence inventing nanobots, wild bullshit that won't happen.
It's the same reason those powers would rather people argue about the culture war or inane conspiracy theories. I guess Yudkowskiism is the inane conspiracism of the AI safety discourse, a great distraction.
The thing your not correct about is super intelligence is when it's smarter than us it will have more degrees of freedom to perform actions than us.
These two things can be true at the same time. Both of these things need dealt with.
And think about it this way, if we're not going to even think twice about it wiping us out, why would we do anything about the lessor issues?
You live in a world where "wild bullshit" technology is just how the world works as we speak to each other on devices that would have been considered magic 150 years ago.
None of the info you provided really falsifies the Occam's Razor hypothesis: Anthropic is a public benefit corporation with a public benefit mission to "responsibly develop and maintain advanced AI for the long-term benefit of humanity". You don't have to like or trust them, but they very well might be sincere.
In any case, I agree the p(doom) sci-fi is annoying secular milleniarianism. SV hyperfixates on imaginary futures. If they actually cared about safety, they would be using all this money to strengthen global cybersecurity, instead of writing LessWrong posts that gives kids in their 20s ulcers.
How? Like, the government has thrown piles of money at cybersecurity and it hasn't done shit.
(yes, AI critique is now also made with AI. We have come full circle.)
This is unbelievably ignorant speech. I have not received a dime of any of this funding, but I do know many excellent researchers that have, and they do fantastic work. There is an unbelievable gap between theory and practice regarding the capacity of deep learning, and while great strides have been made to develop the surrounding theory, there is a long way to go. Many believe that without a concrete understanding of how neural networks properly learn concepts, we have little hope of molding them to be reliably useful. It costs money to hire researchers and develop fundamental theory.
Just because you don't understand any of that work, does not mean that it is pointless. This is fundamental research that is 20 years behind schedule.
They are being paid by industry investors, because industry investors want them to do something.
Since they continue to be paid by industry investors over a long period, they are doing something the investors want them to do. That is, they are producing a lot of p(doom) research.
It so happens that this research is speeding up the development of AI and making it less safe.
One, by helping the industry to place the blame for their own criminality on AI safety in the abstract and avoid criminal liablity.
Two, by creating public hysteria and regulation such that the industry is allowed to form a cartel, coordinating output and gaining control over AI.
This is linked with the rationalist ideology and interests of the Anthropic investors and Anthropic themselves.
I do not know if the research is excellent or not. I do know that it is compromised by conflicts of interest and is also causing a safety problem.
Now, I do not know METR, and have not interacted with its staff. $120m is pretty normal for these types of groups as I understand. METR clearly has even tighter ties to the AI companies than any of the groups I do know, so I agree that calling them independent is a bit of a joke. Even without the explicit labels from their website, there are shockingly few degrees of separation in this entire field. The conferences are big, workshops are designed to be highly social, and the stakes feel high.
Regarding the timeframe and limited material, that's pretty standard now. You do what you can.
Now, none of this necessarily invalidates what they did. Yes, there is big money in AI, and that money hires people, but there quite a lot of geeky researchers in these groups that genuinely only care about the science. That level of neurodivergence might be difficult for most people to comprehend, especially if you're focusing on the politics, but these are the types of people attracted to this line of work.
Put yourself in the shoes of these researchers. You are really interested in AI and how it works, and you convince yourself that this is potentially moral to work on because of a possible threat to humanity. You apply to work at one of these groups. You find similarly-minded people and work together on trying to figure things out. Maybe your boss is a sleazebag in bed with the industry, or maybe they're not. You do not care. You work on this, you publish notes and papers, you comment on LessWrong or X, and you leave the company if they don't let you do this stuff, because there are dozens of others.
Most of the public don't know these guys or this work, because they only focus on the crappy and irresponsible companies leading the AI development charge, and alignment research is so unbelievably hard and slow. I can go into the why if you're interested, but the reason is that it is extraordinarily difficult to make progress if your objective is only vaguely defined. This is the same problem with the Yang-Mills mass gap, for example, although I would personally bet that Yang-Mills has a much better chance of being solved than alignment. Just a different tier of difficulty altogether.
The conduct of the investigation was devoid of epistemic integrity.
Ignoring all politics, I am frustrated that someone could read this as being a trusted source of information. It is the opposite, it is a source of misleading information which is absolving the industry of wrongdoing.
Research is about finding out the truth! Since when did researchers decide that instead of ONLY the facts; they would introduce some foreign objective of appeasing companies, publishing only what they will allow; or letting your own moral beliefs direct the research.
It's corrupt, regardless of whether that is financial or academic.
What kind of independent investigator submits their results to OpenAI for approval and changes the content and the tone, like a journalist hack submitting something to an editor?
I am in the shoes of the researcher, I am invited to do this investigation. I appeal to my integrity and I decide that I refuse to author propaganda at someone else's behest.
I don't see the report as absolving the industry of wrongdoing. In fact, I believe it is particularly damning, and many of my colleagues feel the same. No one is on the side of OpenAI here, not even its own members.
As for submitting the report back to OpenAI for comment and working with them, given that there is currently no government mandate or requirement for any AI company to disclose wrongdoing, this seems like an unfortunate necessity. Independent bodies and governing agencies have to be created. At the moment, it is the industry regulating itself, which means no regulation at all.
So no, you do not need to believe that the document is truth. But it is all that we will likely get, because nobody with any power will do a damn thing. That is just as frustrating to the researchers as it is to you. The solution is not to defund the institutes doing needed research. It is to put laws or incentives or something in place to punish inappropriate behavior and lack of disclosure for the companies.
Tallinn's actions back up his explanation: in 2009, before he invested in any AI lab, he donated substantially to the nonprofit Singularity Institute for Artificial Intelligence, which was later renamed the Machine Intelligence Research Institute (i.e., Yudkowsky's outfit).
Some of us (certainly Yudkowsky and Habryka, the leader of Lightcone Infrastructure, which runs Lesswrong) wish people would stop believing that they can make AI safer by investing in (or working for) frontier AI labs, but that is what the evidence shows Tallinn (and Dustin Moskovitz and others) did sincerely believe.
1 [Unsafe AI development risks causing omnicide]
2 [Anthropic is developing omnicidal AI by not slowing down] (see my comment about the RSP for citations).
3 [Owners of Anthropic will IPO with billions of unearned USD as omnicide profiteers]
4 [Tallinn is the lead Series A funder of Anthropic]
5 [Tallinn is a genocide/omnicide profiteer]
Not only that, Yudkowsky and Habryka apparently critize those who invest in AI, only to preach the word of EA from Lightcone's $20m USD property in one of the wealthiest locations in the Bay Area; a facility funded by stolen (FTX) and omnicidal ai blood-money (Tallinn).
PauseAI, is paid by the omnicide profiteers themselves to hold a protest against omnicide.
1-5 is from EA's assumptions, plus Anthropic's own assumption that it is developing unsafe AI. It is just what they have stated in public.
Lightcone funding is accurate. PauseAI funding is accurate.
Using regulation to coordinate development in big firms creates a cartel, surely true.
Boosts value at IPO, yes, because of monopolistic competition.
Investors get unearned cash, yes, because they can cash out at IPO.
Control, yes, because Anthropic is in a cartel.
Keep developing the AI faster. That needs more justification, it's still true though, they state on record they will go faster because of China. Higher valuation lets them spend more money.
It's ridiculous - anyone who thinks about it for a minute or two will realize that its utterly impossible.
Ordinary people/politicians don't understand AI so they turn off their rational mind and assume there is something super incredible some magical powers that they cannot understand that can destroy all humans.
Even humans - the real risk to humanity - could not destroy all humans even if they tried. There is no plausible scenario.
Even climate change and nuclear war and bio weapons - the most damaging mechanisms - would still only get some percentage of the people on earth.
And if we are talking about Skynet and self replicating robots and Terminators - please, grow up.
AI doesn't have to turn us all into paper clips to make the world a really bad place.
The ad-hominem stuff seems inappropriate here, Gates, Hawking, Musk have identified this as a credible threat, so saying "grow up" isn't really a sufficient argument. Also arguing only 90% of humanity would die isn't really much consolation.
[1] https://en.wikipedia.org/wiki/Existential_risk_from_artifici...
My argument stands and I don't defer to Gates and Musk and even Hawking - high level hand wavey statements without any plausible description of the mechanism just don't hold up. Famous names should not be automatically assumed to be right - certainly not with Elon Musk.
Right, so not the slightest basis of fact, just wild speculation about a magical future completely ungrounded in any sort of reality.
That's exactly the point I am making.
I'm not saying I'm super smart - I am continuing to ask for detail to back up the wild claims being made all over the world by politicians, tech celebrities and others - all hallucination/AI psychosis/fiction. If someone says some stupid thing then I'd like them to please explain that stupid thing - seems like a reasonable request.
AI labs are certainly trying to make LLMs behave helpful and subservient but the question is -- will they be able to keep doing so once LLMs become smarter?
Btw thinking about the far-right rhetoric of us vs immigrants, I think you could draw some similarities here only if you replaced "immigrants" (ie. humans with very similar morals, behaviors and capabilities) with an actual alien species that is qualitatively different from us. More like human vs chicken (where we are the chicken)
Please read more on the topic. Will and intent need not apply.
For example, is it malevolent for me to hook you to a machine that makes every dream come true for you in a simulated world where you feel pleasure all the time? I can always say that you have free will inside this simulation, and your life would be a lot better because of it. I'm doing you a favor. I mean, you already live in a society where you have little control and there is high risk of bad things happening to you. If I as a machine overlord did this, is this really actually "bad"?
At the end of the day AI is not a human, it's much closer to an alien that has learned as much as it can about humans, but has a completely different set of drives and motivations. You cannot predict what comes out the other side of it, good our bad.
Worse as AI capability improves we as humans no longer need to make AI, it can make itself. Will it train itself to be helpful and trusting of us? It's a pretty big damned bet to say yes by default.
Yes, this is our intent. But it's very hard to get that right. I don't trust our current methods to ensure that behavior consistently and robustly enough to keep harms from occurring, particularly as more and more responsibility gets handed over to models.
But that just means we won't all be wiped out. We need to understand when discussing global issues, such as this or like climate change that it's about prosperity and quality of life. We're trying to plan for a good life (for all people?).
https://www.lesswrong.com/posts/LAPa2jxoq3n63GzTr/some-ways-...
https://slatestarcodex.com/2015/04/07/no-physical-substrate-...
As for self-replicating robots--it's no more bizarre than other technological developments which were successfully anticipated in advance, e.g. moon landings.
What, did I miss the moment where it was officially proven that, under the laws of physics as we know it, Skynet and self replicating robots and Terminators are impossible?
What we are actually seeing now is that robotics is getting deeper and deeper into the military, AI-driven decision-making and target selection is increasingly a part of modern military operations, the line between military hardware and civilian hardware blurs, and, on the civilian side, there are at least five major companies and a dozen less prominent ones working on making universal worker robots a reality.
We're closer to "Skynet and self replicating robots and Terminators" now than we ever were at any point in time.
The issue of AI risk is that AI, unlike a virus or a climate event, it's an intelligent adversary. Black Death didn't have a plan for finishing off the plague survivors. An AI doesn't have this limitation.
Black Death was, effectively, one bioweapon. An AI can have one bioweapon, and then a backup bioweapon, then a backup backup bioweapon, and then a dozen more bioweapons designed to collapse ecosystems and disrupt human ability to establish a reliable food supply rather than kill humans directly - all deployed at the same time. With a production run of 200 million killer robots that will be ready just in time to greet those who managed to survive all of that. A crippling strike against human civilization, followed up by cleanup.
Humans are only this survivable because they can think their way out of issues and adapt to adversity. Most threats can't beat humans at that - humans adapt too quickly. AI could.
Many, many people are slipping through social welfare cracks and suffering as we speak because the cost of fuel is rising[0] and we’re ostensibly helping one another and living-well. People are not durable, and not adaptive in the face of threats to “substrate” that we’ve mostly taken for granted. We are paying (in the small, in the scope of humanity) for tolls that we’ve rung up. Just less than 4000 people in Europe died[1] because the temperature ticked up a few degrees[2]. Does that make you think we’re actually robust? What happens if our at-risk electrical grid gets shut down deliberately? If communication infrastructure is adversely affected?
> Even humans - the real risk to humanity
Because, on the whole, we’re in a manageable world with reasonable people keeping the peace.
> Even climate change and nuclear war and bio weapons - the most damaging mechanisms - would still only get some percentage of the people on earth.
Is that victory? I don’t think it’s an asteroid-class event like you seem to be leaning on, but potential threats to energy, be it electrical grid, fuel production (moving goods around the world is critical - you’re not going get a plot of dirt and garden your way out of grocery stores being empty - which many got to get a taste of during the COVID pandemic) or communication. We actually fare poorly in the face of pressure there, and I’m not bullish on humanity “pulling together” like Independence Day[3] versus forming tribes and tearing each other down.
All this is predicated on a malicious AI taking over (e.g.) the electrical grid or conms, and I understand the problems with (e.g.) OpenAI/Hugging Face incident, or the overblown Mythos claims[4] (and how under some scrutiny these events shine lights on incompetence or hyperbole), but is there a trajectory/future where these systems (electrical, comms) are genuinely under threat? Do you think we’ll respond better than I described when we’re less comfortable, less in control? We’re in a tizzy over social media and it’s detrimental effects on society and it’s essentially an opt-in entertainment platform…
[0] https://www.pbs.org/newshour/economy/bessent-said-the-k-shap...
[1] https://www.dw.com/en/heat-wave-european-countries-report-37...
[2] I’m not trying to diminish this - and it took a lot of “work” (environmental abuse) to arrive here - but (say) 10 degree rise in temperature sounds a lot less dramatic than thermonuclear war… but here we are, with 3,700 deaths.
[3] https://en.wikipedia.org/wiki/Independence_Day_(1996_film)
[4] https://news.ycombinator.com/item?id=49929391
Well, in a narrow sense of "wiped out" (c.f. Terminator/SkyNet), sure.
But the deeper worry is better expressed this way: AI is now starting to accomplish things that defy explanation, or prediction. We don't know if Alignment is even a solvable problem as we thought we understood it.
So basically, yes: "humanity" is probably not at risk of extinction per se in a biological sense. Human culture, civilization? Who the fuck knows any more.
"Oh yea, we have weapons capable of sundering nations because everything is made from atoms"
"Oh, yea, there are invisible waves all around you that you can't see, can't feel, can't touch, but they can hold massive amounts of information. Also you can transfer that information to the other side of the planet in less than a second. We're talking text, pictures, movies"
"Movies, ya, we can record real life and play it back on this glass square".
"Oh, yea, we've conquered a ton of diseases, we can even see the teeny tiny little bits that make them. Oh, and for fun we can edit them and make them worse".
"Oh yea, we fly thru the sky all the time too. Like super fast and millions of us do it every day".
Our lives our unimaginable fiction. Just about everything we do compared to those people defy explanation in any reasonable amount of time. And now, suddenly it's "Don't worry, there isn't any more science or new things to find after this so this super smart and super capable thing that can connect directly to computers and machines and have them do things is completely and totally safe".
It's mass insanity.
One speculated mechanism for this was a mass release of hydrogen sulfide gas from the oceans, which is acutely toxic. Not only does this kill most air-breathing life, it also strips the ozone layer and irradiates the surface. The planet is then left to cook in this manner for some centuries.
Engineering an event like this would require immense industrial capacity, as well as a deliberate objective of wiping out humanity. But I don't think it's beyond our ability, if we were both clever and stupid enough to try it. There are likely chemical compounds that would do the job more efficiently than hydrogen sulfide.
Such destruction went on to create humanity and all we've achieved. Maybe there is an even smarter species waiting in the wings for the demise of homo sapiens. Your logic is very human centred
What could go wrong?
100% of its actual capabilities are tied to harnesses (the actual "agent"), i.e. ordinary deterministic programs that are connected to networks or machines and enable interaction with the outside world. This part (the part that can do harmful things) is fully under human control and all the recent headlines about "agents going rogue" are - as someone (forgot who) put it - akin to strapping a weedwhacker onto a dog and letting it run wild.
The tech itself is safe as far as real-world interactions go - the weakness lies in unchecked access to systems surrounding it. It's not safe at all when it comes to human interaction (lots of ongoing lawsuits demonstrate that), though. There is real danger here, but it has nothing to do with doomsday scenarios ala Terminator or I,Robot and more with total corporate control over the lives, perception of reality, and abilities (like critical thinking) of people.
This is a bad and misleading way to think about it. Note that it's trivial to make the harness that you claim capabilities are tied to (the LLM itself could write it from scratch in one shot), but no matter how good a harness you have, it won't make gemma4:e4b capable. That's because what actually gives capabilities is the LLM's intelligence - or if you prefer not using that term, the fact that the probability distributions the LLM spits out depend on the context in useful ways.
I'm not talking about what the LLM does internally. If a metaphor helps, here's one to help you understand what I was trying to get at:
Imagine an evil genius that has no eyes and no limbs. Everything they could learn about the world is presented to them by means of some person describing it to them through words. They have no way of directly interacting with the world and rely on someone executing any action they want to take and describe the outcome to them. Now how dangerous would you say such person would be? How dangerous could they become?
That's what I was getting at. Replace person with LLM (or any other AI system). Replace the person that communicates with an external interface (the harness) and I hope you understand. It doesn't matter whether the LLM could generate the harness by itself - it still is just a bunch of weights sitting in memory being run by an execution engine. That's what it fundamentally is, whether you like it or not. It cannot do anything on its own - and no, not even writing files. It's the execution engine that translates the numeric output into words (or images or video or audio) and the layer above (the harness) that takes that output and interprets it to execute actual actions.
This is not about what you or I think about the internal capabilities of the model - that's irrelevant to the conversation and you can replace LLM with a random token generator and the point still stands. The model itself is incapable of performing actions - from reading files to writing files, to controlling physical machines. All that is and HAS to be done by external interfaces outside the control of the model.
The same way we've done it since machines became multi-user: boring old system access restrictions. Nothing fancy, nothing radical, just good old minimal access rights required to perform a defined set of whitelisted operations.
> It's a ridiculous premise that just because the harness is "normal code" that means we can control the AI.
What is it then? Is not just a program that takes model output, parses it and performs tool calls from the text it receives and then feeds the result back into the model and calls it again with those results? It is normal boring old deterministic code. Many are open source. Look at them. Understand what they do and the apparent "magic" goes away real quick. Harnesses are nothing special.
> AFAIK there isn't that much stopping anyone from spawning an AI swarm and telling it to "spread and go hack everything for the lulz."
Aside from lower cost and possibly greater scale, there's literally NO difference between that and (state sponsored) hacking that has been going on for decades. First it was script kiddies, now it's ML models. The threat model remains the same and so do the counter measures. The real danger is still the harness (and its access to external systems), not the model itself. Restrict the access of the harness and the model can't do anything harmful, see above.
As for your question - the same way you apply restrictions to any external system or user. If you don't do that - that's on you. Same category as driving drunk, playing with guns, making explosives in your garage, you name it. The danger is still not the model itself - it's the access to systems that you provide it without any checks or safety barriers.
To keep with the analogy: cars have seatbelts, airbags, ABS, ESP, lights, horns, crumple zones, emergency braking systems, roads have speed limits, there are traffic stops, insurance, regular inspections (not in all countries), etc. etc.
So what's unsafe here? The car or roads without speed limits, complete lack of safety measures (both active and passive), absence of any supervision and no insurance? That's the problem. It's not the models themselves - they can spit out tokens by the billions, there's no risk there.
You wouldn't give full access to your phone, your computers, your house keys and your credit cards to any stranger on the street now, would you? How is it then, that people act all surprised when a non-deterministic machine that's optimised to achieve goals while taking all the shortcuts it can, suddenly uses the tools handed to it in unexpected ways? That's a failure on the operator's side, not an inherent danger within of the model.
But safety is just one thing people optimise for; if it's convenient enough people will accept imperfect safety (as with cars). It's unrealistic to just heap blame on end-users who use mostly very safe tools in the common way, even though in aggregate they are meaningfully dangerous. They don't think they are strapping a weed whacker to a dog; they think they are driving a car.
Your initial argument of "it's just the harness/user" is wrong; reasoning LLMs are inherently dangerous unless locked in an unbreakable box, which is tantamount to not using them at all. We can make them safe_r_ with better tooling, but they can't be made safe.
Cars are inherently dangerous. They’ll still be dangerous when computers are driving them all.
At this point all I can say about their contents is "They are not even wrong".
Please join us in the real world with how we see this product is not only dangerous, but getting more dangerous with each iteration, and no one seriously talking about controls on it.
If such were achieved, the model would almost certainly be smart enough to make itself smarter, and hack as much compute as it could possibly want.
So if we ask what would be done by an intelligence (human or otherwise) that is beyond human comprehension, it would be pure hubris to say we know for sure. We can scarcely control the models we have right now (e.g. hugging face attack). But given our whole society is mediated by technology, an superhuman intelligence could certainly collapse the government.
There's lots of ways to pay for electricity, and many of them don't even require a body.
We aren't going to get wiped out by a super intelligent AI, we are going to get wiped out by morons wielding intelligent toddlers with the power of a nation state.
And the folks from podcastistan are never clear on the details of how human extinction would happen exactly. It's always something like, "Well, how do humans regard chickens? AI is way smarter therefore it wants to conquer and control us." An ASML lithography machine is also way better at making chips, but we don't consider it a threat.
A sufficiently intelligent AI will have multiple ways to pose risk to humanity at large. For example an oopsie at a wetlab - very contagious virus with initially mild symptoms which kills its hosts only after they already had time to spread it further. But I would have to become super intelligent myself to give you precise blueprint for such a virus -- which is kind of the point
Also -- ASML lithography machine is only good at making chips. I can't believe you compared it to AI that can generalize across variety of tasks
If you for a second put yourself into the shoes of a person who thinks "the apocalyptic stuff" has even a 5% chance of literally happening in the real world, you might see how you wouldn't agree to move on from it.
But the discourse is dominated by paper clips experiment discussions and not let's say by the fact that new grads have an unprecedented difficult time getting jobs. Unsurprisingly one of those discussions is beneficial to power and the other one is not.
The much more likely explanation to me is that people are just spitballing, either because they've watched too much sci-fi, or they have some weird counterintuitive agenda (e.g. Anthropic and OpenAI trying to position themselves as the amazing, trustworthy keepers of this dangerous technology before their IPOs).
All there really is is arguments from theory (most notably, the orthogonality thesis and instrumental convergence). It's not like we have a spare universe to conduct experimental research on the likelihood of human extinction. That said, the 5-10% range is not some specific person's wild guess, it's the median wild guess of a poll of AI experts: https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/
> or they have some weird counterintuitive agenda (e.g. Anthropic and OpenAI trying to position themselves as the amazing, trustworthy keepers of this dangerous technology before their IPOs).
They are not "trying to position themselves" that way - back when those companies were founded, the people who did it were already convinced they were trying to make the ("inevitable") development of AI go better. It's not some novel galaxy-brained marketing strategy, it's just what they thought the whole time.
Dear sir, I have some really bad news for you about humanity.
Really? One of the most famous effective altruists, Sam Bankman-Fried, was sentenced to 25 years in March 2024 for fraud. Every article about the case (and there were many) mentioned EA.
> LeCun thinks EA is “super toxic” and a “complete disaster.” Its adherents who are working in AI labs suffer from “paranoia” that causes them to make poor decisions, he said. “Apparently people are having mental issues.”
I would agree with that.
Edit: I am sure, it's denial.
sudo kill -9 pid and your AI is dead. No one is dying from a glorified knowledge base that can do auto correct amazingly well.
And there are people that have much more credibility than him who actually take this scenario seriously. But I'm sure you will downplay them by saying they are tech bros or that they have some stake in being doomers (as if saying that AI might kill everyone would be good strategy for attracting investors - it's obviously not).
> No one is dying from a glorified knowledge base that can do auto correct amazingly well.
All kinds of vulnerable people are dying due to LLMs. [0]
If SOTA LLM companies can benefit from the "intelligence", then they should also be liable for the harm caused.
[0]: https://en.wikipedia.org/wiki/Deaths_linked_to_chatbots
There are scenarios where kill -9 isn't going to happen in time. What if the team that is harming people with AI is different from the one that is monitoring the harm? What if no one is monitoring? What if the user is intentionally malicious?
And wiping out humanity doesn't necessarily mean shooting people either. Every trader involved in the '08 financial crisis was locally acting in their own interests. Those could have easily been AIs optimizing trading strategies too.
[1] https://futurism.com/artificial-intelligence/us-military-pen...
If anything, the example you gave goes to show the stupidity of AI, not its intelligence capable of taking over the world.
Every example you gave is not AI killing people. Jesus. The stupidity is astounding
Do you think that AI killing us would not involve us doing stupid shit with AI first? Or do you have some strange idea that this strange AI we're talking about would just pop up out of nowhere like a miracle?
There is no AGI, there is no intellgent text predictor that is scheming to make us do stupid shit to wipe us off the map, its stupid people that evolution will take care of. We have survived this long, we will be ok. We make mistakes along the way but we somehow manage to survive. The only thing I see killing us soon is climate change, but again we can stop it but stupid people seem to want us to keep going down the path we are. We will eventually solve that problem too.
If anyone has any strange ideas, its those parroting the idea that LLMs are AGI and will lead to the death of humanity. Utter nonsense but again thankfully nature and evolution has a way of keeping the smart and strong while eliminating the weak.
??? They were... There are books about the faults in the algorithms that led to the '08 financial crisis.
> pull the plug after it kills people
Are you aware that in this regard "AI" means "all algorithms employed in decision making"?
Sudo kill -9 pid and your AI is dead. AI is not going do anything humans dont want it to. Its simple. Pull the plug. But people like yourself either work for OpenAI or have some investment into AI that fearmongering benefits you. I wish all people like yourself would just go to Mars with Elon musk and leave us sensible ones alone.
This is a very bad analogy because chess isn't life. In chess, you aren't allowed to do whatever you want. There are rules. I know for a fact that Magnus Carlsen won't beat me using checkers moves and he won't beat me by pulling out a gun and telling me to resign. Magnus Carlsen's skill at chess leading to his victory in chess is not a valid analogy here, because there's no law of nature that says "the more intelligent entity wins in a battle for survival".
You could have infinite superintelligence and still die inside a locked room to which you have no key. "Superintelligence" is not a magic solution to every problem, you can constrain any superintelligence with any unsolvable problem.
This goes both ways. You absolutely can constrain an AI system by “putting it in a box”. The point parent comment was making is that, such a device is borderline useless for its creators. Why invest trillions in capital on a system that can’t even accept input from the internet. So you set it up with an ethernet connection. And this is good, but you have a hardware failure at the concrete room data center. That’s pretty annoying for your customers, so you install some doors (with electronic key codes of course) and give a bunch of (trusted, vetted) people access to deal with those. And this is fine, but it turns out some of your customers are having latency issues so you build more data centers with more humans granted access to copies of the intelligent system. And this makes people happy but to get a faster feedback loop your customers ask to let the AI system have more permissions to the system they’re operating on. And they come with billion dollar checks, and the system hasn’t harmed anyone yet, so you say, “Okay.” And now you find yourself where we are today where AI systems can remotely run arbitrary commands on thousands if not millions of systems, where many individual humans with all of their frailties and idiosyncrasies can physically interact with the hardware running these systems, and where there’s an economic demand to tighten the loop between action in the real world and a response by an AI system. It’s very obvious that the story doesn’t end here, so where does it stop?
This is why I fucking hate commenting on this website. You know exactly what I meant but you wrote this bullshit anyway.
What are you talking about? Have you used AI in the past 3 years?
https://chatgpt.com/share/6ac23b45-79e8-83eb-8de6-1bbd728928...
>It can't even modify a picture the way you want it.
Which of the many AI image models is "it"? And have you tried using an agent that has the capability to leverage a combination of manual edits (ImageMagick) and imagegen to achieve what you ask?
Jokes aside, no I'm not saying anything about creativity and LLM coexisting in one sentence. I genuinely try to use them for writing and I genuinely run into issues with other models missing details, and misunderstanding poses, or anatomy, or directionality, etc. I'm not hating on them for anything related to the term LLM but rather for the real issues that I've seen myself using them personally.
And so I'm saying Gemini 3.1 Pro is the best I've seen because it seems to be a decent bit better at that than frontier models at this. Genuinely. It seems better able to transfer concepts into less traditional areas, which is important when say, you have entirely non-human characters? (Which I always do.)
It is.
It showed me that all human creativity and art is an attempt to express the indescribable Otherness in words, which always fails, and even the word "describe" in Russian literally translates as "write around" ("о-писывать"), and its close relative "define" means limiting, assigning an end to something infinite, thus leaving the essence outside of words.
Now, LLMs operate totaly within words and hence will always be a parody of art.
I explicitly tell you that there are things (in fact, it's a single thing, fractally generating everything else) that can't be shown, expressed, or otherwise be reduced into language, and you keep demanding to show it, while in fact staring at it your whole life and failing to see.
Psychedelics (not "drugs" as you keep trying to smear them) are just one way among the many to see it, but in the modern way of living, also almost the only one available.
Gemini 3.1 Pro hasn't needed that pretty much at all, which is impressive compared to how much I've learned other models need it. Somehow it's able to mostly handle that stuff itself without needing the constant manual reminders and hand-holding. It still misses the occasional one or two things but it's way better than other models missing entire classes of things constantly. Somehow, it feels appropriate though I have no actual evidence why.
Mary packed the binoculars in chapter 3, therefore she may use them on the train in chapter 6.
And it's not technobabble. I like hard sci-fi. I don't pretend to understand every field of tech, nor do so my characters have the same expertise as me. Having an llm with knowledge of basically all fields is helpful.
And has it at this stage, within in-depth take of said "learning", foundationally?
I have not been able to properly check the studies for a long time now, but I remain unaware of achieved solutions on the problem of reliably referencing a world model out of a language model - that "counting the 'r's in 'raspberry'" be not guessing, not memory, but actually counting.
Nothing intrinsically more or less direct about the LLM's method than ours.
I've never tried it and it might take some thought and effort to conduct an experiment to find out properly, but I would be interested in the answer.
In my mind general intelligence is pretty much by definition a virtual machine, so the mechanisms behind thought are only relevant for the sake of efficiency (ie you can argue that LLMs make a poor basis for intelligence because tokens and natural language are a poor way to encode the world, but if you can run it on a big enough computer to counteract the inherent wasteful virtualisation then who really cares how it works under the hood?)
I will stop here before our analogies go too far.
https://huggingface.co/posts/omarkamali/593639295164067
https://huggingface.co/blog/omarkamali/tokenization
[1]: https://youtu.be/l7vRSu_wsNc?si=SndkB6GBaRyhvNNA&t=61
"Count the 'r's in the string 'raspberry'" is the same as "Count the joints in the Tasmanian sand spider": it means, "can you instance in idea and assess it and perform procedures over it - for example, can you count?".
We want to be sure it can count. We have seen it may guess, it may construe, it may recall - we instead demand it checks. We cannot trust it without that.
If you were home and a family member asked you that question, you'd probably criticise the question rather than answering. LLM are RLHF'd into being milk-toast helpers that just try to answer questions like that with no criticism.
This is all beside the fact that the world of AI has changed pretty dramatically in the last few months.
It’s nonsense to test if a product that is marketed and sold as being able to provide generalised intelligence on demand, does what it says on the tin?
Check yourself
https://news.ycombinator.com/newsguidelines.html
Similarly for Apple’s “red herring” paper, simply adding a generic caveat to “disregard irrelevant factors” (without specifying which ones) restored performance even in the weaker local llama models back then.
The flaw was not in the reasoning; the flaw seems to be simply that the assumptions we make are often different from the assumptions it makes. I wonder if that might be a fundamental underlying cause of misalignment.
This is always the issues in the discussions.
There’s the outcomes camp (objectivists?), which points at the things LLMs can do.
Then there’s the process methods camp, which talks about what is actually going on.
If you only care about the outcome, then the process does t matter.
If you are talking about what is happening, what the underlying mechanics and science of it is, then the process matters.
These models aren’t thinking. They simulate cognition well enough to do useful work in several fields and domains.
Both are true.
They are for any definition of the word that makes any kind of sense. I'm sure you have a contorted definition that magically only includes humans though...
The models are simulating thinking, if the fidelity is good enough for you - great!
The normal definition of the word "thinking" definitely includes what LLMs do. Hell people used to say computers were thinking even before AI. It's super weird to get all uppity about the semantics of the word now.
If a model isn’t able to count to 100, but is able to use python to do math, it isn’t thinking.
If you are only an outcomes person, then this difference doesn’t matter to you.
But when people are talking about AGI, anthropomorphizing LLMs, and making assumptions on its behavior based on that, then people are being loose or misled.
Language changes over time anyway, so even if you really believe what LLMs are doing isn't the "thinking" of 2024, it probably will be the "thinking" of 2027, because most people are using it that way.
And yes, we do need precision. Language is a vehicle to convey ideas, at its best it creates clarity. At its worst it creates deception and confusion.
What the labs are doing is designed to mislead. Being clear about the terms and avoiding needless anthropomorphization, is a basic decency we owe to the many others who are being riled up or terrified by the conclusions and implications that come from using “thinking” when it’s something else.
The difference between being satisfied by the outcomes and being interested in the process is sufficient to indicate the exactness required in your terminology, and does not harm any position.
These aren’t thinking machines, but if what they simulate is good enough, and your work ends with the output, it doesn’t quite matter.
People can be happy with a meal from McDonalds, or they can want something prepared by a chef, or Soylent.
All are acceptable and bring utility to their customers.
For "thinking" here we mean "assessing a representation of an object". That, or equivalent, is required to be reliable. So it is fundamental and critical.
But the outcomes group "ignores" the fundamental limitations of models which are purely text based.
E.g, a baseball players trains to catch high-speed balls and they dont do it by: "ball velocity 50mph, vector:[1,2,3], run move hand command now"
That's absurd.
No, there is an embodied network which is "trained" on visual, tactile input, and control as direct output.
LLMs are fundamentally not the right tool for that.
That is a NN that learns a skill.
But that is not an Analyst. If it were ballistics, then the answer to "how to parametrize the launch to reliably hit the target" excludes getting the result through natural skill.
The problem lies in the need to get "AI" facing "LLMs": the latter create a need for reliability, for "AI".
Speech is an endowment of both those who give educated guesses via developed skills and of those who return answers like Analysts, who check and compute. LLMs create a confusion between the two, and they will remain a problem until an ability to act as Analysts - strictly - will be implemented.
Dynamical systems will never be solved in a semantic domain.
IMO, they can be helpful the in robotics stack, from planning level up, but that's it.
What exactly is not clear? I will rephrase.
The internal process of the blackbox oracle determine the reliability of the output.
Two abilities are very different: learning trajectories through empirical training ("increasingly catching thousands of thrown balls"), and determining trajectories through computation (a rational thinker at work). The former is a finetuned parametrized engine (a «NN that learns a skill»), the latter is an Analyst. The former is fuzzy, the second deterministic.
In front of fuzzy LLMs, which use potentially misleading outputs - text ("has it guessed or has it thought?") - the urgency of warranties of reliable output gets evident.
So, that they «simulate cognition [only] well enough» (Intended wrote), and that there are «fundamental limitations of models ... purely text based» (Kooi wrote) raises the urgency to overcome the "fuzzy" and achieve the "deterministic" - it is not that we can stall on a «fundamentally not the right tool for that».
Inventing an oracle calls for urgent striving to overcome the original weakness.
Incidents like hugging face are partly rooted in the lack of common sense. It still functions like a supercharged toddler.
I'd love to overcome this because it'd mean I spend less time guiding the the LLM to produce usable outputs.
And we've had difficulty as humans to childproof our sandboxes and infrastructure. Things that are otherwise innocuous spots to coordinate between like minded toddlers can become problematic.
Uh, no? So much of what we learn and take for granted as common sense is not learned via language, and not even expressible in it.
consider me optimist now, but just few months ago, even frontier models were dumb, doing stupid mistakes all the time, all of them were so dumb I'd never expect anything to change in just few months.
We can have adequate representations of light that are the instances over which we reason. Your simile is about perception, not about instancing ideas.
Given you also don't want it to memorise [for all tokens, count([for all letters]), this would probably be more like "here's two images, count all things in the big image that look like the thing in the small image", which can then be r's in a photo of a raspberry jam jar in a supermarket, or dragons in a photo of a furry convention, or whatever.
That said, they are competent enough at coding that I keep seeing them write code to do even simple tasks.
On a related note: why did I see Claude editing a file by using cat to write a python script to do a grep search and replace?
Why not? You've memorized how words are spelled, and how sounds correspond with letters, and how concepts correspond with words. To the extent that there are shortcuts that enable compression you use these, and the model will do something similar.
Being able to spell all the words then count letters is simpler, and more generalisable to other tasks, than memorising answers to all possible word questions.
Because to "123x456" we want a reply that goes "this times that plus that...", not "Was that not nnnnnn?". If it does not perform its duty (returning solid checked answers) it is a liability.
Of any object in question they should be able to create a representation that allows correct assessment.
> Given you also don't want it to memorise
That is obviously necessary: what we want from the consultant is to check, not to remember. Answers must be correct and that implies having performed all due diligence - and being capable of doing it, before that. So, objects must be instanced internally in a way that allows effective handling. Counting letters is a good example of the ability (that must remain general).
But I'm sure you can still find tasks that they will have difficulty solving, involving the most fundamental concepts that can only be experienced in the physical world to be understood well, like left and right, near and far, hot and cold, heavy and light, etc.
The good designer understands culture, tastes and preferences as they evolve in real time. That’s why llm as design tools haven’t displaced the good designers.
We don't see in UV or infrared, but some birds do.
On the other hand, AI can understand reality in ways we individually can't: It has combined access to unlimited knowledge of math, physics and chemistry that at this point no single human possesses. Maybe it understands reality better than any single human at this point.
Also we're seeing AI reason from first principles in order to simulate physics and get to that real world understanding without directly interacting with it.
I use LLMs daily to help me code etc. but... It wasn't long ago that frontier models were confidently recommending to walk, without the car, to the car wash to wash the car no?
As a daily user of LLMs I do certainly see my fair share of WTF "solutions" to coding problems. I'm not saying it's not super useful: it is super useful. But I don't exactly feel like I'm talking to something that understands that the car needs to be present to be washed.
An analogy on LLMs is that you have a pretty clear straight highway ahead of you for some distance right now. Maybe that doesn't lead to AGI but it's clear there's progress to be made. For a big tech company it makes sense to push as hard and fast down that clear straight highway of LLMs asap.
Meanwhile LeCunn wanted to turn off the road and go down an unproven track. I say this as someone working on world model generation right now (creating the ability to learn game world model and have it play the game https://tfmbot.com for an example of my system pointed at a very complex board game). LeCunn wanted to pivot all of Meta into world model generation. It's good as a side track research project but the entire pivot he wanted to do was madness.
People are literally talking about an AI researcher who was fired for terrible direction here.
Meta’s AI projects are still negative ROIC
The argument is that LLMs are a local maximum that will never breakthrough to AGI. This is still very much an open question. If you are the fifth-best AI lab, does it make sense to try to outcompete everyone in a space that is already too crowded and may not ever yield their actual objective? Instead they could just use open weight models in their products, or post-train on open models like smaller labs have done, and treat that as what it is: product development.
Pure research has always been about taking chances.
Is there anyone who genuinely believes that current models can't be contained if we want too?
What is LeCun saying here that is debatable?