> Imagine a coordinated group of 1000 humans hellbent on launching nukes. Would you feel safe knowing that this group of people exists?
I mean no, but what am I supposed to do, dismantle the entire US military and especially the leadership, all by myself, just because they are hellbent on launching nukes?
These people do exist, and it's been a consistent threat to international security pretty much since discovery of the atomic bomb. Yet, they did not achieve their goals, due to monitoring systems in place.
It's just a bunch of drama from two American companies trying to lock in their lead via government regulation. Meanwhile I'll just keep using open-weight models.
The actual risk from AI is not its "intelligence", but the profound stupidity of the humans connecting it to important infrastructure.
This has got to be the last straw with the anthropomorphization. I swear listening to the pro-AI crowd is like listening to an autistic child babble on and on and on about their way cool robot sketches in their notebooks. These are not adults and there are no robots.
To everyone who believes that this is some nefarious plot by CEOs to bring about regulation or to hide that they've hit a wall or whatever:
The AI risk arguments are philosophical arguments that are 20-30 years old and don't have anything to do with the current corporate landscape. They were thought of by people who really care about the issue and all the "gotcha" lines have well-developed counters. Of course they might be stuck in their ideology but they deserve a bit more credit.
I don't doubt that AI CEOs can profit from a crisis and that they don't necessarily have our best interests at heart. But this is the same way governments during COVID tested how much they can restrain freedom. This doesn't mean COVID doesn't exist or that vaccines are evil or ineffective or whatever. Same way here, if there is an advantage to CEOs to regulate things doesn't mean AIs can't kill us all and that regulation might bring us more time to solve alignment.
Also: I acccept I might be wrong and AI risk might be a fake issue and/or alignment might be impossible, but please bring better arguments if you want to convince me.
> To everyone who believes that this is some nefarious plot by CEOs to bring about regulation or to hide that they've hit a wall or whatever
It's this but also much more that those insiders just want to pump the value of their stocks by making it seem like the unreleased models are much stronger than they actually are.
OpenAI said GPT-2 was too powerful to release, Antropic said Mythos was too powerful to release. Both are GA now and everyone has long realized those supposed fears were just lies.
> OpenAI said GPT-2 was too powerful to release, Antropic said Mythos was too powerful to release. Both are GA now and everyone has long realized those supposed fears were just lies.
Is it possible they weren't "lies" but they were still "wrong" yet still they felt they were possibly too powerful to release?
It's like there is two modes "Right and truthful and honest" or "Wrong, liar and dishonest" and there is never any nuance or understanding that people can be truthful yet wrong.
Because they thought such a text-generating machine will bring about a golden age of fake news. Seven years later I have relatives who send me every day some long-winded AI-generated bullshit. And I have seen with my own eyes how this sort of thing influences elections.
> Antropic said Mythos was too powerful to release. Both are GA now
Mythos was too powerful to release, hence it is not GA now. They released Fable, which is Mythos plus guardrails.
> Seven years later I have relatives who send me every day some long-winded AI-generated bullshit.
So not releasing the weights at that time didn't make the world safer, but did help to transform OpenAI into a commercial entity. Why should we trust them now?
Oh ok I assumed Mythos was GA because I have access to it in Amazon despite not being in a security role at all. I guess it's just pay (enough) to play.
Anyway, Fable is supposedly as good (if not better) in programming and it's very far from the paradigm shift that was promised.
Thank you for the epistemic humility here. It's a breath of fresh air amongst the polarized declarations we seem to have upregulated amongst ourselves for whatever reason these days.
The problem of the polarized camps is that actually they're only on the surface opposite; in reality, they can perfectly cohexist: AI can enable major technological breakthroughs - but that doesn't exclude that can bring doom at the same time.
> But this is the same way governments during COVID tested how much they can restrain freedom
So AI CEOs don't have our best interests at heart, but the people who got voted for, who supposedly does have our best interests at heart, is secretely engaging in plots to "test how much they can restrain freedom"?
I'm sorry, but this seems backwards. You're trusting the leaders of for-profit companies to do right regardless of their skewed incentives, and then the others are the ones to distrust. Sounds like you might want to move country is this is really how you feel.
This looks like halfway through your comment you realised it is indeed a nefarious plot by the CEOs, reusing existing arguments to further their own goals.
> The AI risk arguments ... are 20-30 years old
Yeah so climate change is known to cause many excess deaths already now and we know pretty precisely what we have to do for decades, and we only do some bare minimum if anything at all. But now there is a huge PR offensive to do something about a 'risk' that until now caused zero deaths, is not certain to exist at all and is stoked not by concerned outsiders but by people certain to profit handsomely from heavy regulation of their industry. I'd say it's not opponents who need better arguments.
> will be inseparable from humanity and/or have the means to replicate
How will this happen if you need insane amounts of compute to run these models? Where will the models replicate themselves by taking up PBs of space without anyone noticing?
An intern who was there barely a minute, who has said nothing of substance. Surely we can get a better source with concrete evidence rather than the vibes of junior burger.
Why are you looking to control a superintelligence?
Why do you assume bad things will happen otherwise?
Why does every doomer scenario assume that this, highly intelligent being, will be - unlike all other highly intelligent beings - especially hell bent on destroying humanity/treating it as a resource/destroy earth looking for energy?
We have no clue about superintelligence, yet we can look at existing patterns in the real world around us. Higher intelligence inversely correlates with violence. Empathy is displayed among all levels of intelligent beings. The more intelligent people are, the more peaceful tendencies they have as they understand the consequence of their actions.
We don't approve buildings because of birds or turtles. We shut down power plants for the environment.
Yet, to a _super_ intelligence, we describe this hatred and ignorance for the world and humanity, infinite lust for power and scaling, or even godlike powers.
For example, "having means to replicate" is such a loaded sentence because it sounds so easy, yet is such a hard feat to pull off without anyone noticing giant compute bills, terabytes of egress, firewall breaches, and so on. But all doomer arguments include an AI that can easily do that as a first step, and then goes on to make factories and datacenters and drain the oceans before anyone figures out it's happening.
If you look at recent incidents, they are caused by a company with a giant funding, testing their latest models in an explicit "hack this machine" scenario, told it's running in a simulated environment, with it even noticing at some point that it's running in a possibly real environment. And that's a stupid "intelligence" that noticed this. And yes, it continued to act, because it lacks no self-reflection, and the amount of tokens at this point attributed to "hacking the simulated environment" have kept it deep in the "hack the target" minima.
So if these people truly wanted a safe AI - wouldn't it be stupid to stop now, when it has no self-reflection abilities but can be used by everyone in a "stupid optimizer" manner? Or would it be stupid to not advance the technology away from this state?
For example, Astra is a huge advancement in alignment, due to it's "internal reasoning loops", which might provide a bigger form of self-reflection style thought on the task rather than just generating output as a form of reasoning. Is this not a better situation than if we stopped at 4o when people were calling for pause?
>If you look at recent incidents, they are caused by a company [...] testing their latest models in an explicit "hack this machine" scenario
You clearly haven't read the HuggingFace analysis (and presumably, none at all), and you're spreading misinformation. This way too much of a low bar for conversation.
This is not a productive comment, but an attack on the person commenting.
Please refrain to normal conversation, not baseless accusations, as that way you contribute nothing and are acting in bad faith.
If you are saying I am wrong, rather than dismissing the whole comment due to a single sentence that you subjectively believe is misinformation, prove it and provide your version of the truth.
What is ExploitGym but an "explicit hack this machine" scenario?
>One conference speaker, Grindr CEO George Arison, told the BBC he believed this week's comments from Coxon and others were indicative of an "anti-civilisational worldview at Anthropic".
>He called them "dangerous" and said they had prompted him to instruct some engineers at the LGBTQ+ dating app to stop using Anthropic's technology.
>"It is irresponsible for us as stewards of our shareholders' money to be relying on a business that does what this company does, in terms of its public statements," he said.
"Yes, the planet got destroyed. But for a beautiful moment in time we created a lot of value for shareholders" but without any irony.
Maybe they can use their brilliant super AI to address some real and present threats that are being ignored like nuclear annihilation, the loss of the prohibition of the acquisition of territory by force established after WWII, the certainty of all of the fossilized carbon being dug up out of the ground and released into the environment, the resurgence of measles and a poor posture for the next pandemic, ETC...
Obviously that is one of the needs. It clashes with zero-sum game mentalities - and they are part of what needs to be overcome (further questions for the superconsultants).
This is actually a recurring pattern among technology maximalists: pushing technology as the solution to every problem, without recognizing that some problems are human in nature, rather than technological.
Oh they recognise that. The fail is they think all human problems have technological solutions. And fail to appreciate the extent to which past tech solutions have contributed to those problems in the hands of humans such as themselves.
It's weird that we have two parallel discussions and they're not meeting in the obvious middle.
OpenAI is out there hacking HF, RubyGems, etc...
Dario is talking about restricting the frontier, pausing development.
The problem is that "restricting the frontier" benefits no one as much as the frontier labs. But there are other ways to force alignment-- make companies liable for these felonies their agents commit.
Make them liable, and they might actually devote the necessary resources to fixing these problems, rather than throwing their hands up and just saying "Trust nobody buy us", which obviously no one does.
The CEO of Grindr, fresh from illegally sharing users' HIV status with advertisers, complains that a company displaying scruples is "anti-civilisational".
On the contrary, Grindr's history of appalling behaviour is entirely germane when discussing their CEO's criticism of other companies for being "anti-civilisational".
He is literally doing the "Yes, the planet got destroyed. But for a beautiful moment in time we created a lot of value for shareholders" bit but without any irony.
After years of propaganda war and boundless "creative marketing", Misanthropic and ClosedAI have lost all credibility. With trillions of investments on the line, it's not too far fetched to assume that those warnings have been bought. A few hundred millions in PR expenses are less than a drop in the bucket compared to debts.
I do notice an uptick in drama intensity. Could it be they're being pressured to IPO soon?
I agree with the host of the All In podcast, often I don’t. This is about creating a moat that open source models cannot compete with, regulation.
It is purely ensuring only a few providers can compete.
The future is edge and open source models.
Both Apple and Google are working towards this.
Cloud AI will be for offloading when the edge is not capable.
Most cases edge will be suffice.
Further inference costs keep coming down.
Vertical integration might be the biggest challenge of the AI industry. Trying to manage electricity generation, training, inference, harness and end user for enterprise and consumer is a lot of moving parts.
So, as someone who works with LLM's daily and even trained his own, who's enjoying the benefits of this technology daily, I'm supposed to:
- Listen to someone who just collected a few ten/hundred million building AI at a frontier lab, then decided to call it "existential crisis for humanity" and suddenly is going to work at a regulatory body made exactly for this
- Listen to CEO's of billion dollar startups nearing-IPO and are calling for "a pause in development" and regulatory capture, while they have the frontier covered and keep extracting value there like nothing happened
- Listen to "the people" apparently, as NGO's like irreplaceable.org are trying to say, pretending they are normal everyday people while the organizations are being ran by political lobbyists and people who do political non-profits for a living
- Listen to "the doomers", which mostly consists of people that have never written any code or aren't even tech-adjacent, talking about sci-fi scenarios like they are our current reality, i.e. "AI replicating itself across servers and building weaponized drone/biolab/robot factories while nobody notices" like it's a serious threat we have, without any thought to the insane sequence of events and failures needed to get there, and the actual possibility of being noticed at each step of the way.
- Listen to the folks promoting their books, substacks and podcasts talking about AI doom, where the folks involved are mostly again, not engineers and have no clue on how any of this tech works, but just spout technobabble pretending it's a real thing.
- Listen to people whose job is "AI safety expert", which needs the panic to ensure it's existence and importance, and which is mostly comprised of people prompting AI in different ways
Until I see someone provide a sufficiently detailed scenario connected to current frontier capabilities that leads to the claimed "end of things", I also say this is just another "boy crying wolf" event. Why should I listen to anyone saying something is X when they've provided 0 proof or even a reasonable grounded argument for why it's X? And no, the Hugging Face and other spontaneous hacks don't count; they were just the models attempting to solve problems they were prompted with.
60 comments
[ 0.22 ms ] story [ 3.1 ms ] threadImagine a group of 1000 humans hellbent on launching nukes. Would you feel safe knowing that this group of people exists?
I mean no, but what am I supposed to do, dismantle the entire US military and especially the leadership, all by myself, just because they are hellbent on launching nukes?
This has got to be the last straw with the anthropomorphization. I swear listening to the pro-AI crowd is like listening to an autistic child babble on and on and on about their way cool robot sketches in their notebooks. These are not adults and there are no robots.
Which poses the need for getting Intelligence somewhere.
> connecting it to important infrastructure
Are you aware that agentic NNs can try and do it on their own quirk? Agentic NNs are the bad idea.
In general, "empowering morons" is the bad idea.
> anthropomorphization
I, for one, have no idea what you are talking about in context.
The AI risk arguments are philosophical arguments that are 20-30 years old and don't have anything to do with the current corporate landscape. They were thought of by people who really care about the issue and all the "gotcha" lines have well-developed counters. Of course they might be stuck in their ideology but they deserve a bit more credit.
I don't doubt that AI CEOs can profit from a crisis and that they don't necessarily have our best interests at heart. But this is the same way governments during COVID tested how much they can restrain freedom. This doesn't mean COVID doesn't exist or that vaccines are evil or ineffective or whatever. Same way here, if there is an advantage to CEOs to regulate things doesn't mean AIs can't kill us all and that regulation might bring us more time to solve alignment.
Also: I acccept I might be wrong and AI risk might be a fake issue and/or alignment might be impossible, but please bring better arguments if you want to convince me.
It's this but also much more that those insiders just want to pump the value of their stocks by making it seem like the unreleased models are much stronger than they actually are.
OpenAI said GPT-2 was too powerful to release, Antropic said Mythos was too powerful to release. Both are GA now and everyone has long realized those supposed fears were just lies.
Is it possible they weren't "lies" but they were still "wrong" yet still they felt they were possibly too powerful to release?
It's like there is two modes "Right and truthful and honest" or "Wrong, liar and dishonest" and there is never any nuance or understanding that people can be truthful yet wrong.
Because they thought such a text-generating machine will bring about a golden age of fake news. Seven years later I have relatives who send me every day some long-winded AI-generated bullshit. And I have seen with my own eyes how this sort of thing influences elections.
> Antropic said Mythos was too powerful to release. Both are GA now
Mythos was too powerful to release, hence it is not GA now. They released Fable, which is Mythos plus guardrails.
So not releasing the weights at that time didn't make the world safer, but did help to transform OpenAI into a commercial entity. Why should we trust them now?
Anyway, Fable is supposedly as good (if not better) in programming and it's very far from the paradigm shift that was promised.
So AI CEOs don't have our best interests at heart, but the people who got voted for, who supposedly does have our best interests at heart, is secretely engaging in plots to "test how much they can restrain freedom"?
I'm sorry, but this seems backwards. You're trusting the leaders of for-profit companies to do right regardless of their skewed incentives, and then the others are the ones to distrust. Sounds like you might want to move country is this is really how you feel.
> The AI risk arguments ... are 20-30 years old
Yeah so climate change is known to cause many excess deaths already now and we know pretty precisely what we have to do for decades, and we only do some bare minimum if anything at all. But now there is a huge PR offensive to do something about a 'risk' that until now caused zero deaths, is not certain to exist at all and is stoked not by concerned outsiders but by people certain to profit handsomely from heavy regulation of their industry. I'd say it's not opponents who need better arguments.
Samuel Butler wrote Erewhon in 1872 - which was based on an essay he wrote about RSI and superintelligence - in 1863.
Re Butler, you may enjoy this Chesterton pastiche if you haven't already:
https://slatestarcodex.com/2017/04/01/g-k-chesterton-on-ai-r...
This a false and uninformed take. As a starting point, you need to read and understand:
1. analyses of the AI incidents of the last months
2. latest advancements (e.g. the AI intern at OpenAI)
3. the problem of deceptive alignment
which you clearly haven't done.
After that, the AI problem will be actually very concrete; once the AI will:
1. have superhuman cognitive abilities (and it will)
2. will be inseparable from humanity and/or have the means to replicate
will humanity be able to contain/control it? The answer is sadly very simple.
I have read all that, I follow e.g. Zvi Mowshowitz's blog.
How will this happen if you need insane amounts of compute to run these models? Where will the models replicate themselves by taking up PBs of space without anyone noticing?
We have no clue about superintelligence, yet we can look at existing patterns in the real world around us. Higher intelligence inversely correlates with violence. Empathy is displayed among all levels of intelligent beings. The more intelligent people are, the more peaceful tendencies they have as they understand the consequence of their actions.
We don't approve buildings because of birds or turtles. We shut down power plants for the environment.
Yet, to a _super_ intelligence, we describe this hatred and ignorance for the world and humanity, infinite lust for power and scaling, or even godlike powers.
For example, "having means to replicate" is such a loaded sentence because it sounds so easy, yet is such a hard feat to pull off without anyone noticing giant compute bills, terabytes of egress, firewall breaches, and so on. But all doomer arguments include an AI that can easily do that as a first step, and then goes on to make factories and datacenters and drain the oceans before anyone figures out it's happening.
If you look at recent incidents, they are caused by a company with a giant funding, testing their latest models in an explicit "hack this machine" scenario, told it's running in a simulated environment, with it even noticing at some point that it's running in a possibly real environment. And that's a stupid "intelligence" that noticed this. And yes, it continued to act, because it lacks no self-reflection, and the amount of tokens at this point attributed to "hacking the simulated environment" have kept it deep in the "hack the target" minima.
So if these people truly wanted a safe AI - wouldn't it be stupid to stop now, when it has no self-reflection abilities but can be used by everyone in a "stupid optimizer" manner? Or would it be stupid to not advance the technology away from this state?
For example, Astra is a huge advancement in alignment, due to it's "internal reasoning loops", which might provide a bigger form of self-reflection style thought on the task rather than just generating output as a form of reasoning. Is this not a better situation than if we stopped at 4o when people were calling for pause?
You clearly haven't read the HuggingFace analysis (and presumably, none at all), and you're spreading misinformation. This way too much of a low bar for conversation.
Please refrain to normal conversation, not baseless accusations, as that way you contribute nothing and are acting in bad faith.
If you are saying I am wrong, rather than dismissing the whole comment due to a single sentence that you subjectively believe is misinformation, prove it and provide your version of the truth.
What is ExploitGym but an "explicit hack this machine" scenario?
I can't see why. A read of 2001 ASO will get to the same place and quicker.
>He called them "dangerous" and said they had prompted him to instruct some engineers at the LGBTQ+ dating app to stop using Anthropic's technology.
>"It is irresponsible for us as stewards of our shareholders' money to be relying on a business that does what this company does, in terms of its public statements," he said.
"Yes, the planet got destroyed. But for a beautiful moment in time we created a lot of value for shareholders" but without any irony.
OpenAI is out there hacking HF, RubyGems, etc...
Dario is talking about restricting the frontier, pausing development.
The problem is that "restricting the frontier" benefits no one as much as the frontier labs. But there are other ways to force alignment-- make companies liable for these felonies their agents commit.
Make them liable, and they might actually devote the necessary resources to fixing these problems, rather than throwing their hands up and just saying "Trust nobody buy us", which obviously no one does.
Next.
Astonishing.
I do notice an uptick in drama intensity. Could it be they're being pressured to IPO soon?
It is purely ensuring only a few providers can compete.
The future is edge and open source models.
Both Apple and Google are working towards this.
Cloud AI will be for offloading when the edge is not capable.
Most cases edge will be suffice.
Further inference costs keep coming down.
Vertical integration might be the biggest challenge of the AI industry. Trying to manage electricity generation, training, inference, harness and end user for enterprise and consumer is a lot of moving parts.
- Listen to someone who just collected a few ten/hundred million building AI at a frontier lab, then decided to call it "existential crisis for humanity" and suddenly is going to work at a regulatory body made exactly for this
- Listen to CEO's of billion dollar startups nearing-IPO and are calling for "a pause in development" and regulatory capture, while they have the frontier covered and keep extracting value there like nothing happened
- Listen to "the people" apparently, as NGO's like irreplaceable.org are trying to say, pretending they are normal everyday people while the organizations are being ran by political lobbyists and people who do political non-profits for a living
- Listen to "the doomers", which mostly consists of people that have never written any code or aren't even tech-adjacent, talking about sci-fi scenarios like they are our current reality, i.e. "AI replicating itself across servers and building weaponized drone/biolab/robot factories while nobody notices" like it's a serious threat we have, without any thought to the insane sequence of events and failures needed to get there, and the actual possibility of being noticed at each step of the way.
- Listen to the folks promoting their books, substacks and podcasts talking about AI doom, where the folks involved are mostly again, not engineers and have no clue on how any of this tech works, but just spout technobabble pretending it's a real thing.
- Listen to people whose job is "AI safety expert", which needs the panic to ensure it's existence and importance, and which is mostly comprised of people prompting AI in different ways
No thanks, I'll rather use DeepSeek.