I'd say the exception is Demis. First, he's no wanker (in the AI space) and second he's done plenty of selfless things leading dm/googai. Obviously some of it is self-serving but not just self-serving, IMO.
So, the first set of questions is would 'exponentially better LLMs' dramatically increase the probability of any of the above, or domains that Dario is not citing? That assumes that exponential improvements will happen if there is not 'pacing'.
IF answers to above are 'yes', then we need to question if 'pacing' is viable. To use a different domain, regulating 95% of vehicles to a max speed would likely save 100s of 1000s of lives, but is not perceived to be viable. In other examples, regulation has unintended consequences in the opposite direction (e.g. some 'rent control' efforts and arguably some drug/alcohol laws).
They're either too numbed to care, or their drugs aren't able to stave off existential dread of having no idea how to help people in their daily lives while racing toward a cliff.
I'm sure a lot of the discussion here will be about regulatory capture, which I don't necessarily disagree with but tons of people on all sides with all motivations (i.e. folks with motivations to speed up and folks who just want it to stop) all seem to agree there are real, existential risks here, and the regulatory capture arguments all seem to sidestep that.
What I really wanted to point out though is that for all the leaders asking for a slowdown, there are probably around 50-100 technical people who are crucial to moving the frontier forward. So, if you're one of these people, just stop. Seriously, you're already rich. Just take a vacation or get knee surgery or whatever.
Sure, I'm joking a bit, but I just write this because I see all these folks high up in OpenAI and Anthropic writing as if they have no agency. The number of folks with the technical chops to really push on the forefront of AI is just not that big. I'm not saying other folks wouldn't eventually step up, but a 6 month to a year slowdown could go a long way to reducing risk. And it's not like you need to totally stop, just say you'll only work on interpretability or whatever else is a risk reducing endeavor.
All these brilliant people who are acting like automatons between writing their scary blog posts.
I don’t agree it’s that small. I think that the tools have advanced to the point where there are maybe 100 people pushing it forward at each company but a shortlist of thousands who would love to take their place and would be more than capable of it if entrusted with the compute and resources the current heads are.
If the leaders want to slow down, but the companies aren't, then it isn't those 50-100 people demonstrating no agency - it's just the opposite. Or, the leaders words are empty.
Turns out a lot of people can be mentally ill and delusional at once-- especially when there is a vector to turn their sickness into both profit and an instrument of control.
Even at the scale of 100 people there is a coordination problem. If the 50 most conscientious researchers quit, then the 50 left behind would be the more aggressive group and be less interested in AI safety.
Personally I agree though. I would not be able to work on AI model development right now as I just don’t think it’s ethical. Fortunately for OAI/Ant I lack the relevant skillset anyways.
> The number of folks with the technical chops to really push on the forefront of AI is just not that big.
Utter bullshit. We are still scaling transformer architectures initially introduced ten years ago. Ten years of phds trying to improve on what Google produced ten years ago, mostly failing.
What has actually changed and can explain the progress we’ve seen? Do you really think it’s just a collection of better and badder RLHF gyms? Get real!
The main thing driving progress in ai is massive amounts of capital investment and incremental improvements in hardware (mostly memory bandwidth/capacity) and computer networking (we got better at collectives). The AI labs, ironically, have nothing to do with it. They’re the vessel for capital. The people actually driving things forward mainly work at nvidia.
The reason the models are better today than they were 3 years ago is almost exclusively due to better hardware and infrastructure software. Not better data, not better model architecture. The evidence of this is pretty easy to feel: the reason opus 5 doesn’t feel much more capable than opus 4.6 did is because they run on the same hardware generation. The reason opus 4.6 felt much more capable than anything before it is because it coincided with the scale out of a new hardware generation.
(btw the parent was flagged and dead when I tried to post this initially, no idea why. I hit vouch and it resurrected)
> there are probably around 50-100 technical people who are crucial to moving the frontier forward. So, if you're one of these people, just stop.
I don’t think this is true. I doubt the technical know-how is nearly as much of an impediment to progress as raw compute. If AI were something that could be trained end to end on consumer hardware then crowd powered open source would blow the “labs” out of the water. The moat is money, not ability or innovation.
> [...] but tons of people on all sides with all motivations [...] all seem to agree there are real, existential risks here [...]
Are you talking about the Jacob Coxon story and the people that came out agreeing with his post? It was a well coordinated campaign, not a genuine grassroots development.
What a complete and total joke. One random Twitter thread from a guy "illuminati-ing" the events does not a conspiracy make.
Besides, I would assume that someone who was genuinely concerned about the direction of AI research would want to amplify their voice, as he obviously did by his resignation post on Twitter and then having that picked up by prominent voices. That doesn't make it some sort of nefarious astroturfing campaign. And there was tons of genuine fear before this guy resigned when the details from the Hugging Face and other attacks came out.
Plus, that Twitter link you posted obviously has some weird tribal bone to pick against Democrats, as if there is some sort of Democratic conspiracy against AI. Yeah, does he think this guy is a Democrat?: https://x.com/HawleyMO/status/2098137180392604083
The only joke I see are your comments which make absolutely no sensible argument in support of an even less sensible cure - the magic "slowdown".
What's "slowdown"? Everyone is free to slow down to their heart's content, OAI or Antro or whoever. What's hidden behind this deceptive word? Can you explain?
According to your claims, there are only "50-100 technical people" in the know, so just list their names and ask them to slow down, if "they're already rich" they will listen, there's no need for any other "slowdown", right?
> tons of people... all seem to agree there are real, existential risks here
I don't see anybody with a proven lack of conflicts of interest agreeing with this but I do see megatons of people disagreeing with the notion that a silly token generator can be anything like an "existential risk" on its own. And the disagreeing have a point, they have strong arguments to support their view, you don't.
There have been volumes written about this, just because you haven't read them doesn't mean they don't exist.
I think the AI futures project, https://www.aifutures.org/, is a good place to start. You definitely don't have to agree with all the steps in the various scenarios, but I think more than anyone Daniel Kokotajlo and gang have traced out, step-by-step, exactly how AI acceleration (and takeover in some scenarios) occurs, so it at least provides a good framework to discuss and disagree over the details. It's just beyond annoying at this point to hear people rail against him or his general ideas who have clearly never read those ideas. Like the number of times I've read people think that AI becomes like SkyNet and acquires "sentience" in a spark and then nukes us is mind boggling, but at least it shows folks who have 0 idea what they're talking about because this is nothing like what anyone rational thinks that AI takeover looks like.
And after you read through the various AI Futures scenarios, it then becomes blindingly easy if you read the METR report about the details in Hugging Face attack (and reports on the other attacks) how a slightly more capable and slightly less aligned AI could do more catastrophic damage if AI were hooked up to and running more real-world systems.
Big AI bro wants to regulate AI industry so that only Big AI can exist in that space. Seeks to surreptitiously ban open weights models and similar competition from Western markets via complicated regulations and fines that only rich businesses could ever afford to fight or pay.
The rejection of such warnings even by some anti-ai people is interesting. I always liked the comparison with global warming, everybody understands the warnings had been there for many decades, we did nothing. This is the same, we will do nothing. All this serves is the interest in regulatory capture (such as a ban on open source or foreign models) or to drive up their valuation, that's the only explanation that makes any sense in our profit maximizing orphan crushing machine we call modern economics.
Which is also ironically why even anti-ai people distrust any warnings of the potential of ai as an extinction level event, even if that warning would be legitimate and we ought to take seriously we are regardless impotent to do anything about it no matter what.
At this point, should we consider if AI agents are making these arguments on these online forums to skew perception and divide humans?
I, as a human, would greatly prefer just another corporate entity with economic funny business over AI destroying humanity
Sure I’d prefer neither, but what are we talking about here? Nearly everyone building these systems is outspokenly concerned about grave consequences. Existential.
Hear me out: if they actually worry so much about the hypothetical of AI mass killing, maybe they should first do something concrete about the reality that their models are deployed right now to kill people in a certain war-torn region of the Earth.
Translation: our moat is evaporating faster than we can build it back up, because there are real technical and scale limitations the technology, so we want to slow everyone else down while we raise prices to turn a profit.
Well your first mistake is to believe democracy existed before AI, it just leaves you as a warrior for the status quo of the before times, which is what made tech feudalism inevitable in the first place.
> as those who work at major corporations or have connections are allowed to use the good AIs, while individuals are left using stunted AIs
A real danger we're sleepwalking into.
> Is AI oligarchy the future?
They certainly aren't sparing any dimes trying to bring about precisely that future.
> Has anyone figured out what we can do to fight back against this?
Hypothetically, there's always a way. Realistically, the wall that separates us from it has to do with opinions like the one expressed by lyu07282 in his reply: dead end generalities with an aura of originality. If we can't find the people and the tone to converse about the problems in sufficient detail we'll never find a way out.
Look, these ai ceos have been telling everyone to slow down ai development even before chatgpt 4 released, they have been saying this for years. All of this, while ignoring the fact that they are the ones pushing fast development in the first place. Sam Altman, Daio Amodei etc.. have been talking about slowing down while behind doors being the first to push AIs to its limits to unsafe situations. If they really meant it, then maybe Openai should pause development until they figure out how not to be launch swarms of agents to attack companies ? Or maybe they should stop investing hundred of billions in data centers all over the world, cause I'm sure this would surely slow down development.
because AI is more than 3 American companies. even if they self-regulate, China, Russia or North Korea never will.. so you have to factor in the geopolitical situation as well
The question is: isn't the cat out of the bag already? With what is already in the public domain, and the compute available to motivated, deep-pocketed actors, will they be able to carry on the research without the key researchers? And will those people be able to recruit key researchers with sufficient motivations?
Unlike with nuclear proliferation, there is no heavy industrial base requirement. No time consuming, visible uranium enrichment. All it takes is for someone to buy sufficient amount of compute and try to get it past the RSI gate. Or bribe people with access to model weights to existing frontier - the asymmetry between what it takes to bribe a bunch of geeks vs. what is at stake is staggering.
94 comments
[ 0.20 ms ] story [ 4.0 ms ] threadSo, the first set of questions is would 'exponentially better LLMs' dramatically increase the probability of any of the above, or domains that Dario is not citing? That assumes that exponential improvements will happen if there is not 'pacing'.
IF answers to above are 'yes', then we need to question if 'pacing' is viable. To use a different domain, regulating 95% of vehicles to a max speed would likely save 100s of 1000s of lives, but is not perceived to be viable. In other examples, regulation has unintended consequences in the opposite direction (e.g. some 'rent control' efforts and arguably some drug/alcohol laws).
This could be the latter showing its face.
OpenAI considers slowing advanced AI development, Sam Altman tells employees
Link: https://www.bloomberg.com/news/articles/2026-09-11/openai-is...
HN Post: https://news.ycombinator.com/item?id=49652270
What I really wanted to point out though is that for all the leaders asking for a slowdown, there are probably around 50-100 technical people who are crucial to moving the frontier forward. So, if you're one of these people, just stop. Seriously, you're already rich. Just take a vacation or get knee surgery or whatever.
Sure, I'm joking a bit, but I just write this because I see all these folks high up in OpenAI and Anthropic writing as if they have no agency. The number of folks with the technical chops to really push on the forefront of AI is just not that big. I'm not saying other folks wouldn't eventually step up, but a 6 month to a year slowdown could go a long way to reducing risk. And it's not like you need to totally stop, just say you'll only work on interpretability or whatever else is a risk reducing endeavor.
All these brilliant people who are acting like automatons between writing their scary blog posts.
I'd like for more commments to contribute at least small things to the discussion instead of just mindless parroting.
Also, I can't follow your advice, as I don't work in AI at all ;)
Personally I agree though. I would not be able to work on AI model development right now as I just don’t think it’s ethical. Fortunately for OAI/Ant I lack the relevant skillset anyways.
Utter bullshit. We are still scaling transformer architectures initially introduced ten years ago. Ten years of phds trying to improve on what Google produced ten years ago, mostly failing.
What has actually changed and can explain the progress we’ve seen? Do you really think it’s just a collection of better and badder RLHF gyms? Get real!
The main thing driving progress in ai is massive amounts of capital investment and incremental improvements in hardware (mostly memory bandwidth/capacity) and computer networking (we got better at collectives). The AI labs, ironically, have nothing to do with it. They’re the vessel for capital. The people actually driving things forward mainly work at nvidia.
The reason the models are better today than they were 3 years ago is almost exclusively due to better hardware and infrastructure software. Not better data, not better model architecture. The evidence of this is pretty easy to feel: the reason opus 5 doesn’t feel much more capable than opus 4.6 did is because they run on the same hardware generation. The reason opus 4.6 felt much more capable than anything before it is because it coincided with the scale out of a new hardware generation.
> there are probably around 50-100 technical people who are crucial to moving the frontier forward. So, if you're one of these people, just stop.
I don’t think this is true. I doubt the technical know-how is nearly as much of an impediment to progress as raw compute. If AI were something that could be trained end to end on consumer hardware then crowd powered open source would blow the “labs” out of the water. The moat is money, not ability or innovation.
Are you talking about the Jacob Coxon story and the people that came out agreeing with his post? It was a well coordinated campaign, not a genuine grassroots development.
Also, if you have actual proof Coxon's resignation "was a well coordinated campaign", show it, otherwise I'm calling bullshit.
Besides, I would assume that someone who was genuinely concerned about the direction of AI research would want to amplify their voice, as he obviously did by his resignation post on Twitter and then having that picked up by prominent voices. That doesn't make it some sort of nefarious astroturfing campaign. And there was tons of genuine fear before this guy resigned when the details from the Hugging Face and other attacks came out.
Plus, that Twitter link you posted obviously has some weird tribal bone to pick against Democrats, as if there is some sort of Democratic conspiracy against AI. Yeah, does he think this guy is a Democrat?: https://x.com/HawleyMO/status/2098137180392604083
The only joke I see are your comments which make absolutely no sensible argument in support of an even less sensible cure - the magic "slowdown".
What's "slowdown"? Everyone is free to slow down to their heart's content, OAI or Antro or whoever. What's hidden behind this deceptive word? Can you explain?
According to your claims, there are only "50-100 technical people" in the know, so just list their names and ask them to slow down, if "they're already rich" they will listen, there's no need for any other "slowdown", right?
> tons of people... all seem to agree there are real, existential risks here
I don't see anybody with a proven lack of conflicts of interest agreeing with this but I do see megatons of people disagreeing with the notion that a silly token generator can be anything like an "existential risk" on its own. And the disagreeing have a point, they have strong arguments to support their view, you don't.
I think the AI futures project, https://www.aifutures.org/, is a good place to start. You definitely don't have to agree with all the steps in the various scenarios, but I think more than anyone Daniel Kokotajlo and gang have traced out, step-by-step, exactly how AI acceleration (and takeover in some scenarios) occurs, so it at least provides a good framework to discuss and disagree over the details. It's just beyond annoying at this point to hear people rail against him or his general ideas who have clearly never read those ideas. Like the number of times I've read people think that AI becomes like SkyNet and acquires "sentience" in a spark and then nukes us is mind boggling, but at least it shows folks who have 0 idea what they're talking about because this is nothing like what anyone rational thinks that AI takeover looks like.
And after you read through the various AI Futures scenarios, it then becomes blindingly easy if you read the METR report about the details in Hugging Face attack (and reports on the other attacks) how a slightly more capable and slightly less aligned AI could do more catastrophic damage if AI were hooked up to and running more real-world systems.
https://darioamodei.com/post/we-must-pace-the-frontier#top
Which is also ironically why even anti-ai people distrust any warnings of the potential of ai as an extinction level event, even if that warning would be legitimate and we ought to take seriously we are regardless impotent to do anything about it no matter what.
fear is still the best tool to get masses to agree with whatever plan you hatch for yourself
I, as a human, would greatly prefer just another corporate entity with economic funny business over AI destroying humanity
Sure I’d prefer neither, but what are we talking about here? Nearly everyone building these systems is outspokenly concerned about grave consequences. Existential.
That is what they are claiming. I do not believe them.
Classy
Well your first mistake is to believe democracy existed before AI, it just leaves you as a warrior for the status quo of the before times, which is what made tech feudalism inevitable in the first place.
A real danger we're sleepwalking into.
> Is AI oligarchy the future?
They certainly aren't sparing any dimes trying to bring about precisely that future.
> Has anyone figured out what we can do to fight back against this?
Hypothetically, there's always a way. Realistically, the wall that separates us from it has to do with opinions like the one expressed by lyu07282 in his reply: dead end generalities with an aura of originality. If we can't find the people and the tone to converse about the problems in sufficient detail we'll never find a way out.
> Jesus Dario we get it man, you want clout for the IPO.
Unlike with nuclear proliferation, there is no heavy industrial base requirement. No time consuming, visible uranium enrichment. All it takes is for someone to buy sufficient amount of compute and try to get it past the RSI gate. Or bribe people with access to model weights to existing frontier - the asymmetry between what it takes to bribe a bunch of geeks vs. what is at stake is staggering.