Really well put. I hope this article can finally put to bed this very frustrating dismissal.
"Many debates about AI are complicated and hard to resolve, but this one isn’t: all of these stories are clearly false. Some evidence:
The AI experts who aren’t affiliated with big companies are just as concerned as the ones who are.
The AI company leaders giving these warnings were giving the exact same warnings before they had AI companies - and in many cases, before they worked in the industry, or before the industry existed.
If these corporate plots were real, they would be incredibly dumb."
Those people are bat shit crazy, some random blog is not going to change my mind on that. There are Ai experts who disagree with them, who are not in the big lab friend circle / echo chamber. There are security experts who are not concerned so much that they are telling everyone humanity is doomed if we don't... give the labs complete control over who can use Ai and for what... do you want to speedrun 1984?
Sam Altman has a history, Silicon Valley has a history, the corporate elites and oligarchs have a history. Why should I believe this instance is free of that history?
Perhaps Dario is advocating for things that would amount to a ban on open weights out of fear, that does not change the effect, and I do not believe the more savvy investors do not see the benefit for that regulatory control. Open weights (real competition, not tech cartels) will undermine their financial projections. There is no way I'm going to believe the people who have been doing this for decades are not doing it here too, especially with the most corrupt administration this country has had in my life.
2. Opening with “these people are batshit” is a pretty undeniable sign that you have no coherent theory of mind for them.
3. 1984 wasn’t about regulation of superintelligence. Governments are the only ones who can say “who can use [fissile material], and for what” — has that created 1984? It started 30 years before that, so it should’ve created 1984 by 1984!
4. No one’s banning “open weights”, they’re banning superintelligence development without any of the protections we find to be obviously essential for far, far less dangerous things. Perhaps they would’ve been open to something more drastic, but post GLM-5.3 that’s kinda moot. The cat’s out of the bag.
5. Investors don’t control Anthropic, and their control over OpenAI is complex and highly unusual to say the least.
6. “Some capitalists have done X therefor all capitalists force all companies they invest in to do X” is just not sound logic.
7. Finally: that isn’t what regulatory capture looks like! It looks like revolving doors and packing boards and shady policies, not publicly campaigning for new statutes.
2. I do not need a theory of mind to perceive (some of) the things they say are disconnected from reality. It also demeans our human abilities to handle these things. I'm not interested to live in a Nanny State.
3. it's about the ministry of truth, freedom of speech, Ai is closer to speech than nuclear science, which is a misleading analogy that relies on pre-supposing the Ai is super dangerous and inevitably wipe humans out.
4. there is nothing obviously essential other than humans should be staying in the loop more than they are
5. they do not need to control if they are in agreement, but that is besides the point
6. it's the same capitalists in many instances, and the patterns and trends are quite clear, especially with the biggest companies in the economy
7. Dario and Sam go to meet with the president, face to face, on a regular basis.
8. Sounds like you have too much attachment to these companies and their products to actually consider criticism in a fair light
9. If Ai is going to be so capable, why is the assumption it will eliminate us, rather than help make us better humans to ourselves and each other? Is that not something a "superintelligence" might do?
Many of your arguments rely on predeterminism, no one knows how this will play out.
> 9. If Ai is going to be so capable, why is the assumption it will eliminate us, rather than help make us better humans to ourselves and each other? Is that not something a "superintelligence" might do?
"Might" do.
Wanna risk it?
Even with the best will in the world, even with the best training, even with every advantage you could hope for, this is software, and the software is buggy.
> Why is p(doom) >> p(hope)? Why is there a presupposition that bad is the guaranteed outcome?
There is no such presupposition. Common p(doom) is 10%, though different estimates vary up to the 90%'s.
Would you play Russian roulette with a ten-barrel revolver?
> Why are you taking the side of these companies and people with such fervor?
I am not taking their side.
These companies need to be stopped.
It is strange to me that you think that anyone saying "this is an existential risk which needs to be stopped" is taking the side of the source of the risk.
1. Scott Alexander is the #3 most popular substack user in the "science" category, and has been following AI closely for many years on an academic level. Yes, he is a commentator; that is how media works. Yes, I think it's weird too, but :shrug:.
2. My point was that you don't understand the basics of their position. Just repeating "they're wrong" does not disabuse me of that notion. Also, anything about a "nanny state" is too far below the standards of this forum to reply to at all, sorry.
3. Superintelligence is not closer to speech than nuclear science, obviously. How often does speech lead to thousands, millions, or billions of deaths all on its own? And "their argument relies on part of their argument" isn't a rebuttal; if you read some of the papers, it is far from a presupposition.
4. I'm not sure what you're getting at here, sorry. Oh maybe this is a continuation of the above, rather than a reply to my #4? If so: yes, that's the core of the danger. You have successfully surmised one of the most serious threats.
5. Ok.
6. This just feels like a "many people are saying" argument. No, the patterns and trends are not clear: that's literally the basic question we're discussing right now.
7. That's still not what regulatory capture looks like, but also: no, they do not. Dario met him for the first time recently, and it was about intimidating him into compliance after they refused to make autonomous weapons for the pentagon. Altman has been there a few times, but A) not "on a regular basis", and B) again, that's not how regulatory capture works. The only AI CEO close to Trump is Musk, and even then it's obviously... complicated.
8. Literally the textbook definition of an ad-hominem lol. Never seen it stated so plainly and baselessly!
9. Great question! This is a large part of what's examined by the relevant literature, and they have many compelling arguments of various kinds. My favorite, layperson-targeted version is the blog post Optimality is the tiger, and agents are its teeth[1]
"Predeterminism" isn't a framework. I think the gist of what you're saying here is just "you never know for certain what's gonna happen so don't worry about it", which doesn't need rebuttal.
Yes, transparency and reproducibility sound great! It should be mandatory to conduct this research safely! We are 100% in agreement, in the end. We're just in disagreement about whether you can pattern-match this discussion to libertarian talking points from the early 2010s.
There is a straight forward reason here. Almost everyone, dare I say the 99%, look around and see blatant corporate-government corruption. They see an oligarchy using their wealth and power to get more of the pie each day, while escaping consequences when the misdeed.
Why should we give deference to the newest corpo kids on the block? They go hang out with Trump at what remains of the White House. Do you really think they are different from the rest of the lot?
You too see the corruption across the corporations and government that spans party lines, yes?
You're literally taking the people whom you identify as being in a giant conspiracy and saying their not-actually-human product trained to mimic their own ideas of human behaviour is safe, while disregarding all the people saying "this is not safe, we should do something about that".
This is the opposite of having faith in humanity.
I have (some) faith in humanity, that's why my p(doom) is only 10%, rather than the 95% it would be if I simply looked at, say, Elon Musk making a bot which called itself "MechaHitler" then being defended by the US executive branch when EU nations are cross about Grok making CSAM (or how slow we're being at doing anything about greenhouse gasses despite knowing about the problem for over a century), and in disregarded all the positive efforts and progress towards non-bad outcomes.
10% is still really bad.
Thinking of near-death experiences, I'm reminded of a religious joke: Man caught in a flood, waits in the branches of a tree. A dinghy comes by, he turns them away saying "god will save me". Then a motorboat, he turns them away saying "god will save me, find others who need help". Then a helicopter, same. He dies, goes to heaven, and asks god "Why didn't you save me?", to which god replies "What are you talking about, I sent two boats and a helicopter".
You want faith in humanity? P(doom)ers want to make the number smaller. Help them make the number go down, rather than denying there's something to be fixed.
Why are the researchers at the same level in China not freaking out? They have concerns and regulations, but they are not losing their minds about it? What is their perspective?
Here are a couple of videos from regular folks on the other side
There is work to be done, but it's not so close or so bad you need to let them give you anxiety. 10% is an arbitrary number with no foundation or walkthrough to how it happens. That boy wants his 15m of fame.
If you would like to help my effort improving things, please read and/or comment on this Pi issue. I'm trying to get all the harnesses to have better security posture against loss of control.
What we should not be doing is letting the big labs dictate the what and how for a technology that is (1) built on all human knowledge (2) impacts us all every day. We should all have a say in this and that will require chilling out, starting from provable facts, and having a rational debate.
Why should we trust AI company leaders? Sam Altman was quoted as saying, "We believe that the world should accept some bad things happening for the benefits of this technology and people having the agency." Does that sound like someone who considers safety a high priority?
There is more than one kind of corporate plot though. The article defends against the kind where some people inside a cooperation have some idea to make more money with unsavory methods, in an imagined backroom I suppose, and then use corporate resources to pursue that plan to achieve their goals.
In this case, we have a bunch of people who believe something. Some of them get jobs in that field. They notice their belief could be beneficial to their company (and make them more money), and so using corporate resources to amplify this viewpoint is well justified spend to help them achieve their goals.
In my view, there is little difference in the outcome between the two and probably the line is very blurry... do people in the 2nd case sometimes say the quiet part outloud? Do people in the first often believe they are only doing what is right and rational without finances considered?
Often these posts come straight from the top of AI companies with specific policy goals in mind. What good will come from understanding Amodei / Altman's "true" motives anyway?
1. Understanding motives is important. The fact that you’re questioning that should be a red flag in hindsight.
2. It seems your rebuttal to the article is “well maybe it’s a plot but just not one they’ve ever talked to each other about, but just happen to all secretly believe in the efficacy of”. Which, obviously: ???
3. We know who wrote this post, and it’s not from “the top of AI companies”. It’s from someone desperately trying to prevent the catastrophe that you’ll be up in arms about in 3-6 months.
I'm just laying reasons for thinking this does not mean there is no corporate plot to promote high p(doom) - it has no bearing on how much I agree with it and providing a framework for why this post might fail to convince people.. the narrative can still be reinforced via paid messaging which constitutes some kind of plot in my book.
I am getting some downvotes, so let me clear up a little:
Amodei and Altman are both openly advocating for increased regulation, and I do question their motives, because both of them are running companies with a ton of debt, and facing some very real competition from open models.
What I think they want is for politicians to ensure their companies' survival via some form of regulatory capture; making it illegal to use other models; enforce their twisted version of copyright; etc. Lacking enough political power is existential to them since they run such high costs.
Are they out there amplifying the viewpoint that what they are doing is dangerous and must be regulated?
Yes.
Would they do that if it was actually bad for their business?
NO - the CEOs specifically would virtually not be allowed to ask for regulation if they thought it was bad for investors for them to do that.
So what we have is AI companies openly saying they are doing dangerous work and must be regulated; and I infer they want that to protect them and not us.
FWIW, I think AI will just about obsolete us all and that is bad for almost everyone, but I still think it is transparently driven by profit and corporate agendas when the top leaders of these companies encourage regulation - they do not expect it to work out badly for them.
"AI Safety" is of course too broad to be a corporate plot, but there are some demands that people make in the name of AI safety that are obviously in the interest of frontier labs, like for example for NVidia to be banned from selling to China.
Good that this got posted here. IMO the leaders involved clearly do believe the AI Situation is more important than any "mere" sum of money. This does not contradict them wanting government power behind their vision of how the Situation should be handled.
24 comments
[ 5.3 ms ] story [ 42.4 ms ] thread"Many debates about AI are complicated and hard to resolve, but this one isn’t: all of these stories are clearly false. Some evidence:
The AI experts who aren’t affiliated with big companies are just as concerned as the ones who are.
The AI company leaders giving these warnings were giving the exact same warnings before they had AI companies - and in many cases, before they worked in the industry, or before the industry existed.
If these corporate plots were real, they would be incredibly dumb."
Sam Altman has a history, Silicon Valley has a history, the corporate elites and oligarchs have a history. Why should I believe this instance is free of that history?
Perhaps Dario is advocating for things that would amount to a ban on open weights out of fear, that does not change the effect, and I do not believe the more savvy investors do not see the benefit for that regulatory control. Open weights (real competition, not tech cartels) will undermine their financial projections. There is no way I'm going to believe the people who have been doing this for decades are not doing it here too, especially with the most corrupt administration this country has had in my life.
2. Opening with “these people are batshit” is a pretty undeniable sign that you have no coherent theory of mind for them.
3. 1984 wasn’t about regulation of superintelligence. Governments are the only ones who can say “who can use [fissile material], and for what” — has that created 1984? It started 30 years before that, so it should’ve created 1984 by 1984!
4. No one’s banning “open weights”, they’re banning superintelligence development without any of the protections we find to be obviously essential for far, far less dangerous things. Perhaps they would’ve been open to something more drastic, but post GLM-5.3 that’s kinda moot. The cat’s out of the bag.
5. Investors don’t control Anthropic, and their control over OpenAI is complex and highly unusual to say the least.
6. “Some capitalists have done X therefor all capitalists force all companies they invest in to do X” is just not sound logic.
7. Finally: that isn’t what regulatory capture looks like! It looks like revolving doors and packing boards and shady policies, not publicly campaigning for new statutes.
Scott is not an expert in AI. At best he can be described as a peculiar kind of cultural commentator or perhaps a political pundit.
2. I do not need a theory of mind to perceive (some of) the things they say are disconnected from reality. It also demeans our human abilities to handle these things. I'm not interested to live in a Nanny State.
3. it's about the ministry of truth, freedom of speech, Ai is closer to speech than nuclear science, which is a misleading analogy that relies on pre-supposing the Ai is super dangerous and inevitably wipe humans out.
4. there is nothing obviously essential other than humans should be staying in the loop more than they are
5. they do not need to control if they are in agreement, but that is besides the point
6. it's the same capitalists in many instances, and the patterns and trends are quite clear, especially with the biggest companies in the economy
7. Dario and Sam go to meet with the president, face to face, on a regular basis.
8. Sounds like you have too much attachment to these companies and their products to actually consider criticism in a fair light
9. If Ai is going to be so capable, why is the assumption it will eliminate us, rather than help make us better humans to ourselves and each other? Is that not something a "superintelligence" might do?
Many of your arguments rely on predeterminism, no one knows how this will play out.
"Might" do.
Wanna risk it?
Even with the best will in the world, even with the best training, even with every advantage you could hope for, this is software, and the software is buggy.
What I am against is oligarchy and government gatekeepers
Why is p(doom) >> p(hope)? Why is there a presupposition that bad is the guaranteed outcome?
Let's take a deep breath as a society and have a rational discussion based in fact instead of hyperbole and unprovable predictions.
There is no such presupposition. Common p(doom) is 10%, though different estimates vary up to the 90%'s.
Would you play Russian roulette with a ten-barrel revolver?
> Why are you taking the side of these companies and people with such fervor?
I am not taking their side.
These companies need to be stopped.
It is strange to me that you think that anyone saying "this is an existential risk which needs to be stopped" is taking the side of the source of the risk.
2. My point was that you don't understand the basics of their position. Just repeating "they're wrong" does not disabuse me of that notion. Also, anything about a "nanny state" is too far below the standards of this forum to reply to at all, sorry.
3. Superintelligence is not closer to speech than nuclear science, obviously. How often does speech lead to thousands, millions, or billions of deaths all on its own? And "their argument relies on part of their argument" isn't a rebuttal; if you read some of the papers, it is far from a presupposition.
4. I'm not sure what you're getting at here, sorry. Oh maybe this is a continuation of the above, rather than a reply to my #4? If so: yes, that's the core of the danger. You have successfully surmised one of the most serious threats.
5. Ok.
6. This just feels like a "many people are saying" argument. No, the patterns and trends are not clear: that's literally the basic question we're discussing right now.
7. That's still not what regulatory capture looks like, but also: no, they do not. Dario met him for the first time recently, and it was about intimidating him into compliance after they refused to make autonomous weapons for the pentagon. Altman has been there a few times, but A) not "on a regular basis", and B) again, that's not how regulatory capture works. The only AI CEO close to Trump is Musk, and even then it's obviously... complicated.
8. Literally the textbook definition of an ad-hominem lol. Never seen it stated so plainly and baselessly!
9. Great question! This is a large part of what's examined by the relevant literature, and they have many compelling arguments of various kinds. My favorite, layperson-targeted version is the blog post Optimality is the tiger, and agents are its teeth[1]
"Predeterminism" isn't a framework. I think the gist of what you're saying here is just "you never know for certain what's gonna happen so don't worry about it", which doesn't need rebuttal.
Yes, transparency and reproducibility sound great! It should be mandatory to conduct this research safely! We are 100% in agreement, in the end. We're just in disagreement about whether you can pattern-match this discussion to libertarian talking points from the early 2010s.
[1]: https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality...
Sadly, it won't.
I've made these points separately in HN threads to no apparent impact.
Why should we give deference to the newest corpo kids on the block? They go hang out with Trump at what remains of the White House. Do you really think they are different from the rest of the lot?
You too see the corruption across the corporations and government that spans party lines, yes?
Also, if they're all in bed with Trump, why does he not repeat the "it is dangerous" talking point? Why did his admin blacklist Anthropic?
These people can all be corrupt evil cartoon villains, but that should make you see more risk, not less.
Otherwise, I will be publishing some things within the next few months that I'm working on with some other people
In the meantime, it can help to walk back from the edge and the doomers, have some faith in humanity.
I recently has a near death experience, and since, one of the things I'm changing is to see the glass half full
You're literally taking the people whom you identify as being in a giant conspiracy and saying their not-actually-human product trained to mimic their own ideas of human behaviour is safe, while disregarding all the people saying "this is not safe, we should do something about that".
This is the opposite of having faith in humanity.
I have (some) faith in humanity, that's why my p(doom) is only 10%, rather than the 95% it would be if I simply looked at, say, Elon Musk making a bot which called itself "MechaHitler" then being defended by the US executive branch when EU nations are cross about Grok making CSAM (or how slow we're being at doing anything about greenhouse gasses despite knowing about the problem for over a century), and in disregarded all the positive efforts and progress towards non-bad outcomes.
10% is still really bad.
Thinking of near-death experiences, I'm reminded of a religious joke: Man caught in a flood, waits in the branches of a tree. A dinghy comes by, he turns them away saying "god will save me". Then a motorboat, he turns them away saying "god will save me, find others who need help". Then a helicopter, same. He dies, goes to heaven, and asks god "Why didn't you save me?", to which god replies "What are you talking about, I sent two boats and a helicopter".
You want faith in humanity? P(doom)ers want to make the number smaller. Help them make the number go down, rather than denying there's something to be fixed.
Why are the researchers at the same level in China not freaking out? They have concerns and regulations, but they are not losing their minds about it? What is their perspective?
Here are a couple of videos from regular folks on the other side
- https://www.youtube.com/watch?v=NnV_cWeoo5Q (linux security)
- https://www.youtube.com/watch?v=x3J2w7-9XC8 (chem/bio)
There is work to be done, but it's not so close or so bad you need to let them give you anxiety. 10% is an arbitrary number with no foundation or walkthrough to how it happens. That boy wants his 15m of fame.
If you would like to help my effort improving things, please read and/or comment on this Pi issue. I'm trying to get all the harnesses to have better security posture against loss of control.
https://github.com/earendil-works/pi/issues/10895
Here is my successful persuasion with OpenCode (spans a few issues/pull requests) https://github.com/anomalyco/opencode/pull/51337
What we should not be doing is letting the big labs dictate the what and how for a technology that is (1) built on all human knowledge (2) impacts us all every day. We should all have a say in this and that will require chilling out, starting from provable facts, and having a rational debate.
In this case, we have a bunch of people who believe something. Some of them get jobs in that field. They notice their belief could be beneficial to their company (and make them more money), and so using corporate resources to amplify this viewpoint is well justified spend to help them achieve their goals.
In my view, there is little difference in the outcome between the two and probably the line is very blurry... do people in the 2nd case sometimes say the quiet part outloud? Do people in the first often believe they are only doing what is right and rational without finances considered?
Often these posts come straight from the top of AI companies with specific policy goals in mind. What good will come from understanding Amodei / Altman's "true" motives anyway?
2. It seems your rebuttal to the article is “well maybe it’s a plot but just not one they’ve ever talked to each other about, but just happen to all secretly believe in the efficacy of”. Which, obviously: ???
3. We know who wrote this post, and it’s not from “the top of AI companies”. It’s from someone desperately trying to prevent the catastrophe that you’ll be up in arms about in 3-6 months.
Amodei and Altman are both openly advocating for increased regulation, and I do question their motives, because both of them are running companies with a ton of debt, and facing some very real competition from open models.
What I think they want is for politicians to ensure their companies' survival via some form of regulatory capture; making it illegal to use other models; enforce their twisted version of copyright; etc. Lacking enough political power is existential to them since they run such high costs.
Are they out there amplifying the viewpoint that what they are doing is dangerous and must be regulated? Yes.
Would they do that if it was actually bad for their business? NO - the CEOs specifically would virtually not be allowed to ask for regulation if they thought it was bad for investors for them to do that.
So what we have is AI companies openly saying they are doing dangerous work and must be regulated; and I infer they want that to protect them and not us.
FWIW, I think AI will just about obsolete us all and that is bad for almost everyone, but I still think it is transparently driven by profit and corporate agendas when the top leaders of these companies encourage regulation - they do not expect it to work out badly for them.