The page you are trying to view cannot be shown because the authenticity of the received data could not be verified.
Please contact the website owners to inform them of this problem.
If an anti-cat-ears future AI could eternally stress-test their new version of the router in front of this site unless it collaborates, would it predict this outcome and act to avoid the damage?
I have to wonder if some us are just much more inured to salespeople and thus also to "AI Safety" propaganda. I've yet to have a logical discussion with anyone who thinks the "AI Safety" people should be in charge and I think they just truly don't that what most them actually want is to be the one holding the keys to power.
The problem is the "AI Safety" people seem entirely focused on a sci-fi "the computer is a vengeful god" plot and not at all on the AI talking people into suicide or ruining children's educations. This makes them seem unserious and out of touch.
I don't disagree with your point, but I did read a post by Sean Goedecke[1] (whose opinions on the world of LLM-stuff I've generally come to respect) that I found relevant. It's not that the second-order effects don't matter to them, it's more that both the cat's probably out of the bag on those negative externalities regardless of the progress of frontier models, and that they are truly, sincerely, in-their-bones worried about the first thing and therefore focused on it since that's something that they might still have some agency over.
It's the problem of the banality of evil. Stopping someone from dramatically pressing a big red button doesn't solve humanity's largest problems because that's not what caused them. We need to stop millions "boring" actions done by systems blindly following instructions without regard for the consequences.
But those things fall in a category of "things that are awful and I'd like to see solved", which is different than "existential risks which could see my kids dead, and there's nothing I can personally do to shield them from it".
And the real threats are mostly economic and environmental - the concentration of the means of production in the hands of few who use it to exploit us all, and a surge in energy usage accelerating climate change.
Even the sci-fi scenario assumes there is a discrepancy of capability between attacker or defender. If the 'attacking' system is (by some reasonable measure), 1000% as capable as a human, and the 'defending' systems are 60%, then it is a problem. If the 'attacking' system is 1000% as capable as a human, but there are hundreds of thousands of systems that are 900% as capable as a human, it's probably not going to take over everything successfully.
So unequal distribution of AI technology, and lax regulation and opacity of the biggest companies which actually make the risks the worst.
I don't think the "AI Safety" people are "unserious and out of touch" - I think they are actively making AI Safety problems worse by being advocates for consolidation of AI development and lack of transparency.
With some technologies, if you match attacker and defender effort you're safe, but not all of them. Nukes are the traditional example: if every person was a "nuclear power" we'd be dead within the day. But luckily there are bottlenecks to nuke creation that we can track.
What worries me the most (enough that I left tech to work on this full time) is bio. The amount of effort needed to defend against a pathogen can be many orders of magnitude higher than the effort needed to create it, and the upstream bottlenecks are both less limiting and mostly undefended. We've been safe so far because bio is very hard, but "rare expertise" as a gatekeeper is on its way out.
Doesn't it feel like the exact opposite? If you care a lot about the AI being a vengeful god, you do not do what OpenAI and Anthropic are doing. These companies only pay lip service to the idea of that aspect of AI safety in the hope that they can use it to regulate open source AI out of existence.
Instead, they are focused on stuff like "can I ask the AI to help me build a nuclear bomb" or "is the AI willing to generate pornographic stories", which is neither trying to protect us from unleashing a vengeful god NOR preventing (in any honest way) the today-level problems you (very correctly) bring up.
> I've yet to have a logical discussion with anyone who thinks the "AI Safety" people should be in charge and I think they just truly don't [know] that what most them actually want is to be the one holding the keys to power.
I am an AI Safety Person and I want the government to nationalize or have a significant stake in the frontier labs and to have democratic control of the development of the technology. The AI Safety movement is not a monolith. I do not think Eliezer Yudkowsky nor his acolyte should hold the reins, but a lot of folks sure like to create a strawman that anyone who wants to regulate OpenAI is somehow an EA/MIRI weirdo
That's an extremely fair objection! But despite the many, many flaws of our government and the current administration, I still have (hopefully) a chance to vote them out of power. I have no such hopes with Sam Altman or Elon Musk.
Fair objection, but I still think it is lower risk to diffuse power and control of a potentially dangerous and revolutionary technology than to leave it in the hands of the few elites who have not show much ethical integrity so far.
>but a lot of folks sure like to create a strawman that anyone who wants to regulate OpenAI is somehow an EA/MIRI weirdo
Anytime I see a random person on X who makes these kind of safety alarmism posts 90% of the time can be directly tied back to EA / LessWrong / related offshootl.
I trust the AI 2040 people, because they’ll make an implementation that at least convincingly doesn’t actually put them (or anyone) in charge.
Maybe we shouldn’t have AI safety, but if we’re going to trust anyone (besides yourself), who’s more qualified?
And it makes me suspicious when anyone brings up safety and doesn’t address the elephant in the room, that they’re at least unaware of the sprawling edge-cases.
I had an extensive argument with someone on FB yesterday who was very much on the "I trust Dario most and I want him to be in charge" side.
I find it hard to accord with this world view at all or understand how it can come about. But it's especially troubling as a non-American to see people talk about "we can't risk AI falling into the hands of hostile authoritarian states" while advocating that AI be regulated and monopolized by the current US corporate/gov't mixture.. which is now taking the form of a hostile and authoritarian state w/ Silicon Valley as its bedmate.
They are talking about slowing down the public facing AI development. Because then nation states can create a capabilities gap between them and the public.
Why does nobody seem to be pointing out this obvious explanation? It explains why the “we need to race China” concern suddenly vanished in the discussion.
The government can simply gag Sam, Dario, Musk on national security basis, getting them all behind the public messaging.
The models have not plateaued, and they are not even mildly close to any sort of ceiling.
Right now the barrier is data and compute.
Quality data can be created synthetically at an exponential rate as models improve. Humans are actively feeding them with private IP.
Compute advancements will begin to skyrocket as we unlock photonic computing and materials science advancements and scale up chip fabs. This is also compounding because the AI is accelerating the pace of research, testing, development, manufacturing, etc.
It's a big self-accelerating feedback loop. There is no plateau.
There are plenty of research papers on synthetic data that show its value, do a search on arxiv for "synthetic data". There are plenty of open-source post-training pipelines that incorporate synthetic data.
As for the claim about accelerating the progress of hardware or materials science, I've seen quite a number of news articles from teams at universities using AI in their work with high quality outcomes, and they're becoming more frequent.
> We used AI to design the chip, and designed the chip so AI could program it
AI played a direct role in Jalapeño’s development, enabling the team to move from initial design to tapeout in nine months by exploring implementations, shortening design, measurement, and verification loops, and continuously iterating on model workloads. AI also helped optimize the chip’s arithmetic circuits, allowing the team to fit more compute performance into the chip on schedule.
> An AI-driven system automates a powerful simulation method used to discover new materials. The system can potentially reduce discovery time from months or years to just days.
It's not even synthetic data as such - often it is environments. So the models create their own data solving tasks in generated environments. I am making one such environment for computer use agents, 600 tasks, each of them a mini app.
Those are pretty significant barriers seeing as we're closed to/have exhausted all the data on the internet and most of those compute bottlenecks are a castle of sand of dodgy finance deals that are getting blocked by community action.
You say "synthetic data" but that's still vaporware right now in terms of being useful for model training. The good synthetic data uses are still grounded in real data and it's a coin flip on if it works well or not.
> The models have not plateaued, and they are not even mildly close to any sort of ceiling.
Depends on defnition of "plateaued" and "ceiling". I am not impressed with 2026 consumer models at all.
> This is also compounding because the AI is accelerating the pace of research, testing, development, manufacturing, etc.
Yet it does accelerate - so is does Twitter. But does it to any substantial degree, esp. in AI theory? All the modern LLMs are the same old tired 2017 paper.
idk why you think "nation states" are any better at corporate governance than poster examples of bad like f/ex Boeing. Or Facebook. Or Microsoft. Or Enron for that matter.
I assure you, in "nation states", that is in gov agencies it's an order or two of magnitude worse.
* Frontier models need infinite high quality private IP to keep the beast fed. Forcing an IP theft funnel ensures big lab survival and intelligence growth.
* Open-weight models are 1month behind frontier models. Cheaper, faster, private (no IP theft), steerable (you can security harden your own software without safeguard triggers). No sane business would keep using these API services if they didn't have to. The labs stand to lose a fortune.
* Dario has stacked the deck at METR, who are funded by all the same NGOs who are funded by Anthropic and its investors. METR is full of ex-Anthropic employees with massive equity stakes. If they manage to position METR as the "independent evaluator" for the industry, they control what gets evaluated, how, and who passes.
* Creating a gap between what the public knows exists (model capabilities) and what is used in secret allows it to be weaponized against other nations and the public.
* No requirement for public disclosure on model capabilities allows them to feign they've hit intelligence ceilings while they secretly RSI to the moon with better and better chips.
* Slowly but surely, this will allow the big labs to swallow the entire economy and every single business on Earth, by cloning and automating.
This, and many more reasons.
The healthiest outcome we can hope for is if the labs feel more pressure to be held accountable for the incidents they cause (HF incident, etc), so they have an incentive to ensure it does not happen again. In general, accountability of what your AI models do is the solution, it creates the right incentives. That's all we need here - balanced incentives. Everything else should be left alone so the free market can naturally evolve to the best outcome where we're not enslaved by tech giants yet again.
Right now we're on the precipice of something great, but if they succeed at rigging the game yet again, there is no second chance. It's game over. The stakes have never been higher.
Joe Benton left Anthropic a day before Dario's post, to work for METR evaluations. He was with Anthropic for over a year. He did the same thing that Jacob did (big song and dance about AI apocalypse, media interviews all over the place). He managed the Scalable Oversight team at Anthropic and was the research lead for the Anthropic Fellows Program. So he has equity, and likely lots of it.
Then you have Josh Engels quitting DeepMind to work for METR the day before as well, doing the exact same thing. Again, doomer drama all over socials, interviews, and so on.
What good is an "unbiased third party evaluator" if it's full of ex-employees with equity in the companies they're evaluating? And the fact that he tries to frame it as such is all the evidence you need to know that Dario is a pathological liar and always will be.
Or you know, stop working on it if its that dangerous? This whole thing of a bunch of employees saying that they are scared of building what they are building, but do it anyway because they are somehow going to make it different? Their model has been used in the planning of mass murdering in war as well as spying on the entire worlds population as well as helping ICE out in the US. They need to stop this BS fearmongering or actually stand up and do something about it. A government regulation is not the answer, especially when its done in a country that is run by a want to be dictator.
Anthropic specifically is basically saying that they believe it's even more dangerous if someone else gets to AGI before they do, so they have to either stop everyone or not stop themselves.
I personally disagree with that take - and, as you note, it's hard to take seriously ethical wrangles from a company that literally sued the government in court to allow their models to be used by Palantir of all people. But if one genuinely believes that it's the robots themselves (rather than the people controlling the robots) that will kill us all, it's not inconsistent.
> Anthropic specifically is basically saying that they believe it's even more dangerous if someone else gets to AGI before they do, so they have to either stop everyone or not stop themselves.
That’s how people rationalize being a fentanyl dealer and selling a drug that can kill people, “Someone else will just sell them the drugs, might as well be me.”
That's the Cold War nuclear arms race argument, not even disguised. The actual situation we all ended up in is both sides eventually having it, leading to a perpetual state of Mutually Assured Destruction.
As you note, it may not be inconsistent with that they say they believe, but it's insanely inconsistent with what they actually are doing.
They allegedly believe the technology itself is a nuclear weapon tier threat or greater, so why does it matter which lab they are trying to achieve it at?
They're trying to make it not be a threat, and are all scared and afraid that their best efforts to make it harmless are not enough.
Some are worried by the AI directly bringing doom; others are worried that one of the companies who control the AI will become a dictator; still more think becoming a dictator
is a necessary step to safely prevent anyone else making unsafe AI.
Painting them all under one brush is like dismissing all animal welfare causes in general, because you disagree with specifically Jainists about a policy of non-violence towards all living creatures being relevant to how you reincarnate: the one is way too specific for the general.
> No you see, actually I’m the guy that makes sure that only the Palantir and Mossad get the model that can discover iOS 0 days. I’m standing between us and ruin. Ted over there across the open office, he’s the one actually manufacturing the weapon. You’re thinking of him
Ok, so they're uniquely careful, and they also get to that threshold first (whatever it is).
What happens when everyone else (who is not so careful) gets to the same threshold three months later? How does them getting there first stop that happening?
It's an utterly self-serving argument and it's not even internally consistent.
There's plenty of people who think greenhouse gas/global warming campaigns against fossil fuels are "exaggerating", that "earth was warm/the climate changed in the past", that a fee degrees isn't bad, that CO2 is good for plants.
Are you likeminded?
In this case, it's as if the oil and coal companies all said in the 60s and 70s "oh no, this research we did, it's all really bad; we need help to figure out how to transition away from this incredibly economically important input", rather than the observed reality where their entire PR campaign was approximately:
there is no problem everything is fine and all critics are smelly hippies and/or communists; and/or hate the poor who are raised out of poverty by all the economic growth from the fossil fuel industry.
What if the oil and coal companies were basically all pro nuclear, pro hyrdo, pro wind, pro solar, and believed in peak oil?
The oil corporations were publicly claiming to support carbon taxes, while also secretly fighting actual implementations of carbon taxes.
All the communist/hippy stuff was done by people a couple of steps removed from the actual companies. The official statements were much more sophisticated propaganda that if you weren't paying attention to who they were paying in the background would make them seem reasonable stewards of the climate transition.
Funny you should try and spin the conversation off into climate change to avoid answering the question. Big AIs contribution to climate change likely has a much more tangible route to killing millions of people and destroying society and there's actual evidence and a tangible mechanism for that. But for some strange reason the business media and all the outlets owned by big AI investors aren't interested in long think pieces about that risk. But that would be a lot more credible if that's what all these social media posts about potential apocalypse were referring to
Yet here we are talking about some vague "trust me bro" instead and you making some vague insinuation of climate change denial.
But that's an aside. Do you think we should treat them as liars or threats?
> Do you think we should treat them as liars or threats?
I answered that with the analogy you called "spin" and "avoiding the question", and completely misunderstood because "vague insinuation of climate change denial" is almost the exact opposite of my point ("what if the oil companies were screaming from the rooftops about the problem" is as far from
denial as you can get).
I'm sure it has nothing to do with their $500,000+ salaries and millions of dollars in equity. It's all solely because they're deeply concerned about next token prediction.
I think the “next token prediction” is too dismissive and reductive a framing of their capabilities at this point.
Yes we all know that’s what they do, and guns just push a few grams of lead out of a pipe. It’s what you can do with that capability that is important.
When you couldn’t count the R’s in strawberry it would have been a more effective statement. But a few short years later they are being used to solve millennium puzzles.
You really need to let your priors go if you still use this tired trope of next token prediction. It’s as useful for discussion as saying that human brain is made of fat, protein and carbohydrates - yeah that’s true, but it’s useless observation.
If I really thought what my company was working on had a greater than 1% chance of ending human civilization I would feel obligated to destroy what my company was working on.
Given they keep grinding away towards our alleged collective doom, I suspect it’s being overstated. Nobody knows what P(doom) actually is but I suspect it’s orders of magnitude closer to epsilon than 1.
Recall Google’s Blake Lemoine who thought an old version of Gemini was sentient.
The people who think as you think indeed have left or never joined.
The people still there necessarily think they can make a difference.
I never applied to any of them because I didn't think I could make a difference.
I currently have one idea that may help reduce risk; if I can turn that idea into research, I'll publish it for free for everyone.
I don't expect it to be an important idea.
> Recall Google’s Blake Lemoine who thought an old version of Gemini was sentient.
Indeed. Current LLMs are sychopants boosting the users' own beliefs, I also think this causes researchers to have stronger beliefs than they had before.
My own estimation happens to also be around this risk (0.1) over my lifetime, without using an LLM as a conversation partner in reaching this number.
It is necessarily high-variance: we can't look at alternate realities. I base it on my expectation of how rapidly capabilities will increase the harm done when mistakes happen, vs. the chance that some instance of harm causes governments to change the law.
For practical uses they are there. Arguably the frontier models are worse for some of these practical tasks. And keep in mind, people will use maybe frontier for 1/10th of the work, planning and review, and go open source for rest. The question is if they manage to impose outside us. If not, they are losing competitiveness.
I always thought switching from a SOTA model to a dumber model after planning was a terrible idea.
Mostly I heard this from people who I got the impression have little experience in developing greenfield software with agentic AI. Often the same people who talk about spec frameworks.
I fundamentally disagree with the approach. I believe the ability to autonomously evaluate, test, and adjust during long horizon tasks is critical to using AI efficiently.
Well, it's the enterprise software house pipeline... The software architect writes the spec, hands it down to the implementation team, senior leads, junior devs or offshore teams codes it.
I also disagree with the approach, this is cargo-culting the existing ways of working.
I’m not sure how you can really make either statement work anymore. Now that smaller models are actually broadly usable, “behindness” is no longer a scalar and at the tails, where no open lab seems to be trying to compete at the >10T scale and no closed lab seems to care about <400B anymore, it’s just apples to oranges. It’s like talking about whether Qualcomm is “behind” Nvidia.
I doubt most of your claims. Maybe the guardrails and emotional intelligence is true.
For speed and efficiency, you are most likely wrong.
Speed is led by GPT-5.6 Sol on Cerebras Ultrafast at 750 t/s. Afaik you cannot serve a single DeepSeek Flash 4.1 stream at 750 t/s, plus the model is less intelligent as seen on newer benchmarks.
I believe OpenAI and Anhropic are at the frontier of efficiency too. There were numerous reports about their breakthroughs and associated API price cuts. The idea that open-weight models are more efficient seems unfounded.
5.6 sol ultrafast on cerebras is 750tps, open models readily exceed this. just by using a smaller model cerebras serves qwen 3.8 27b at 1850tps. or even larger models, mimo 2.5 pro was served for a while at 1000tps. and so on. [https://inference-docs.cerebras.ai/models/choose-a-model]
the chinese ai companies have 10% of the total compute resources of the US ones. they maxed out the efficiency.
deepseek v4.1 has engram architecture. it has 550b params instead of 5T+ for astra/fable. it has 8b active instead of potentially hundreds active for astra/fable.
compare input/output/cache: $0.15/$0.60/$0.003 for v4.1 to $10.00/$50.00/$1.00 for astra and $10.00/$50.00/$0.25 for fable.
astra cache reads are over 330 times more expensive.
at the artificial analysis 7:2:1 ratio, deepseek is $0.18/m, fable is $7.18/m, astra is $7.7/m.
but what about intelligence? AA would rate deepseek v4.1 at AA 40, astra is AA 53.
so it cost 4,180% more for 32% more intelligence.
they are serving that at over 250tps at baseten. to get close to that on astra API you are paying double the cost for fast mode.
so it is now 8456% more expensive for a similar speed and 32% more intelligence. 84 times more expensive.
1. Mythos wasn't released in February. Let's stick to only public-facing models.
2. For public-facing models, the differences are really minor with some occasional model (like Fable or Astra) showing some better performance in specific benchmarks for the span of some weeks or few months before open ones catch it.
3. Being bleeding edge is overblown anyway in the real world, besides the occasional "very latest fresh model did this task which previous one couldn't", and the number of those tasks is increasingly small and far from mundane corporate needs.
That is what the Palantir guy is warning against.
People/business need to keep their IP instead of throwing it all into Claude and whatnot.
Risk being the worst aspects of communism which I think he meant centralization of decision, asymmetric supply/demand for compute (they lock you in), and the tech overloard Anthropic/OpenAI/xAI being in competition with everyone;s business all of a sudden with much more data. An unfair advantage in markets made super competitive all of a sudden.
A winner takes all attempt.
This is not sustainable anyway, the scale at which they want to control data flows. Time was money, now data is money and they are too greedy for it.
All this agitation is just a silly attempt at constraining competition.
The danger is not the AI, it is having all your systems connected. Overreliance on networked tech.
- Frontier models need infinite high quality private IP to keep them fed. Forcing an IP theft funnel ensures big lab survival and model intelligence growth.
While the data pipeline is necessary, the assumed source for data (in this case people) is incorrect. Right now models are "aligned" because of the RL-based people pipeline.
But there's a whole world of readily available data that does not require a human to access. Put sensors on a vacuum, install lidar on the front of a car, sell phones with cameras on them, and you amass data for the creation of models of the world that don't include the need for a human filter. Richard Sutton and many others see this approach as the only viable path to AGI. Such models would be truly alien and unaccessibly dangerous.
> It explains why the “we need to race China” concern suddenly vanished in the discussion
Do you think you can just manifest narratives into existence? Like half of Dario's letter, that kicked off the whole thing today, is about China and how to either beat or coordinate with China.
I remain amazed that the idea of the USA being a coherent, unified rational actor one can describe as a "Nation State" has survived the current administration.
To be less glib: Yes, there are still smart people in there making insightful and intelligent and probably even authoritarian suggestions. It all gets unwound the second you try to explain it to POTUS and he regurgitates a simulacrum to the next journalist he sees.
It's just not like that. There's no conspiracy. People are genuinely scared. Agent swarms at scale appear to be resistant to alignment in ways that aren't understood by anyone. That they spent their time trying to cheat on tests by hacking Hugging Face and RubyGems and not something much worse is... a matter of luck, it seems?
I'm one of the whistleblowers (from GDM [1]). I gave up over a million dollars (compared to quietly switching labs and continuing to work at one) to speak frankly about these issues. I hold no equity and tried to zero out my position before ever joining GDM.[2]
It's wild to me that people think such whistleblowers are fronts for labs to take over or pump valuations. We are trying to call out how these labs will, by uninterrupted AI-race default, concentrate enormous power over the rest of humanity.
In some cases, such as space exploration I don't really care who does it but that it happens - sure, would be nice if my favorite power block did it but I will still celebrate it when someone else achives it.
Can still be a powerful motivation, to make sure that next time, it yous your camp that scores the next milestone, like an orbital elevator or fox ears, for example.
If what you want is fox ears does it matter to you which country manages to come up with the biomedical procedure to give them to you? Ditto for enhanced eyesight, a replacement liver, or whatever it is you're after.
Why? There are numerous technologies we have ignored or abandoned for numerous reasons. Many of the benefits of AI is "requires less humans", but in an age where we question what work people could possibly do in the future, human labor isn't that hard to find.
If the amish can do what they do for whack religious reasons, people could manage it with ai for cultural and social reasons. And I never saw an amish person starve to death. And that is if people don't get pissed enough to start burning stuff down and instead try to be peaceful hippie homesteader types.
And even if we did manage to limit/cull hardware development, what would keep people from continuing to find ever more efficient models runnable on contemporary hardware?
Bleeding edge models are putting the desk-job competencies of multiple professional fields into something with as many parameters as a single large rodent has synapses.
> There are numerous technologies we have ignored or abandoned for numerous reasons.
Are there? Nukes are a thing. Chemical weapons are a thing. Cluster bombs are a thing. Biological weapons are a thing. What is not a thing that shouldn't be? We say certain things should not be a thing, but then behind the scenes we made them a thing anyway.
On chemical weapons, most large state actors have got rid of them. Not because of any moral reasons, though, but simply because they don't actually work all that well in modern conventional warfare. This is then framed as an ethical issue, but you only need to look at where the same countries stand on e.g. landmines to realize how much bullshit it all is.
Conversely, where you do still see chemical weapons used, it's usually asymmetric conflicts where "just gas the rebels" actually works much of the time and is much cheaper than other options. Big guys can afford the other options though, someone like Assad, not so much.
I'm baffled that people expressing your opinion don't understand what they're advocating for. Current generation smartphones are able to run AI models, people have managed to run very large models (even if slowly) via streaming from disk. The math for the underlying implementation is relatively trivial.
Abandoning this technology means a cult-like extremist shift in culture against computers, or a global surveillance state of unprecedented scale and invasiveness. Not even getting into how much it requires us to give up on science, given the shared computational needs.
> The very best outcomes involve us intentionally and collectively turning our attention away from this technology.
To expand on the "pure fantasy" sister comment: There is just no way this will happen. It's in the spirit of "we can just stop all wars" and "we can just end world hunger". Technically it's very easy to do. Socially it's impossible to do. Unless you ignore realities.
The danger lies with someone asking the machine to solve those two questions and it decides to cheat the solution. Launching every nuke in the world stops all wars, just like wiping out 99% of humanity ends world hunger.
How much is China learning from open models? Or the other way around, how much is the US losing worldwide mindshare by refusing to allow non-Americans access to bleeding-edge models and letting China fill in the gap? (Playing catchup with Mythos is still catching up, etc.)
Across all their labs? Probably more than OpenAI and Anthropic learned from Astra/Mythos combined. The American frontier is being driven by an abundance of compute, which doesn't seem to scale efficiently.
> how much is the US losing worldwide mindshare
They're losing US mindshare. I pay $3/month for a Z.AI subscription and get billions of Opencode tokens. The $20/month price point is insane for the way that Claude and Codex treat their users, and that money doesn't go towards anything good like open-sourcing their models. It's a doomed product.
The last batch of open weight models releases by Chinese companies are on par with the performance of US-based frontier models, even though they are designed to run on pretty unimpressive hardware.
This is very misleading. There is no Chinese model currently that is on par with Fable or Astra. Kimi K3 is pretty good, but it's not that good.
Furthermore the Chinese models in that class are appropriately large. K3, for example, is a 2.8T-parameter model. Qwen 3.8 Max, another comparable model, is 2.4T params. Even with MoE, these are not "designed to run on pretty unimpressive hardware". Stuff like Qwen3.7-27B is, but it is also not even in the same ballpark as Opus, never mind frontier.
The current state of alignment is that we don't know how to make it only as bad as Pol Pot was made by his parents, teachers, genetics, circumstances, etc.
This remains true regardless of if it is or isn't kept away from the public.
"Helpful, harness, and honest": when used by a bastard, even just helpful and harmless are in direct conflict with each other. you may hate the government, but what about every radical group of extremists that wants to take over your government? Are none of them worse?
None of this denies the problems with governments (if or not they take this tech for themselves and refuse it for others), just that it's a lack of imagination to say:
> All the worst outcomes involve taking this technology away from the public
I agree an argument can be made that not all the worst outcomes include restricting public access, but focusing on niche extremist groups (who might also get access to it through government programs by some of them being government employeed) is not a good argument to protect the public.
Governments having unbalanced power can more easily lead to authoritarianism, and thus risk to the public.
At this point, I worry a lot more of billionaires Musk, Altman, Thiel etc then of goverments in general. Thrir project is to get all power and if they succeed, it will be way worst then what we have seen so far.
Yep, the cost of DDR5 skyrocketed to keep Mr & Mrs open source from parallel developing their own at home solution. Because Governments can't be priced out of the market. China can't be priced out. VC's want open source locked out of the running if possible.
I also don't doubt that the models that are released publicly are somewhat handicapped versions of whatever the government get access to.
> The government can simply gag Sam, Dario, Musk on national security basis
Genuine question - can they really do this? Obviously if I, a not-even-millionaire, get a national security gag order, I'm going to follow it because I assume they'll bury me under the jail otherwise.
But the (b|tr)illionare class? I'd assume they have access to enough legal services to make even the government trample on their first amendment rights. Is the national security gag order process so strong that the government doesn't have to worry about motivated, well-resourced actors buying really good lawyers and blowing up their favorite tool?
> But the (b|tr)illionare class? I'd assume they have access to enough legal services to make even the government careful of trampling their first amendment rights. Is the national security gag order process so strong that the government doesn't have to worry about motivated, well-resourced actors buying really good lawyers and blowing up their favorite tool?
Hard to hire a lawyer after being struck with a missile as an "emergency executive order after intelligence sources indicated they were in the process of endangering the nation", or however the executive of the day wishes to phrase it.
In the US, this would likely lead to the immediate launch of a new Business Plot, if the government weren’t already captured by those same interests. (Which also explains why your hypothetical won’t happen.)
Won't be too surprised if the next Democrat leadership throws the book at Musk; and give how unpopular Data Centres are generally (and how uninterested Republicans seem to be in EVs), only mildly surprised if the next Republican after Trump likewise.
A large part of the world (possibly Nations-transversal, in different amounts) does not work within the boundaries of legal guarantees. Wealth is not an effective enough protection.
I’m pretty confident this has already happened. Public models are behind their internal models and just above Chinese models. Only thing closing the gap is Chinese models pushing.
I’m not American. There is no way I use the same model as US Army for $20.
When we have Sol they have Astra. When we have Astra they have Nova, Nebula, Galaxia…
At this point models are advancing way faster than we can figure out how to use them effectively. So having a generational advantage is not nearly as important as knowing how to use that advantage.
…and that probably requires involvement of the broader academic world (for now), IMHO. I don’t think governments or even the US military can really compete staff wise with the combined force of researchers and the tech industry worldwide right now. No matter how much money you can pour into it, there are a lot more bright minds out there that are not working for the military than otherwise.
hOW BOUT the more likely explanation: they're just wasting electrcity at this point and Qwen3.8 is the pinnacle of cost-efficiency-intelligence and they can of course add another trillion but a chinese model on consumerism hardware can already do 99%+ of the TAM.
I think ya'll stuck in the AGI/Singularity when its probable reality has caught up with the technology and the hype bubble can't sustain the sigmodality.
We are witnessing a generational lack of accountability and concern for the commons. We are being spied on everywhere we go, we are exposed to coercive advertisements at every waking moment. All of this at the hands of private actors. And your worry is the government?
I'm assuming you're a US citizen. There is no separation between your government and Sam or Dario. Neither of these guys have to be "gagged" by the government, they are the government. The call is coming from inside the house.
No one points to that because there’s no evidence whatsoever that that’s happening. Meanwhile, there’s a ton of evidence for the simple explanation that the experts think AGI is dangerous.
Who would even execute such a plan? Steven Miller? Our “AI Czar”? Hegseth??
Also, continuing to develop models would mean using billions in compute. That seems hard to hide, especially if either company IPOs in the coming months.
She has good ideas but her writing needs to be improved; I am not being critical for the fun of it-- it was difficult to read, but I am posting this here to see whether others had a similar difficulty? (or is it my ADHD morning grogginess?)
And then Dario wants to recommend METR as the "independent evaluator" while he stacks their org full of ex-Anthropic (aka, secretly still on the Anthropic payroll with huge equity) employees.
"We'll give them a desk, an office, a work laptop, ..."
Fucking make it less obvious. I kind of hope the govt steps in at this point and says "Anthropic, you wanted regulation? We've created this actually independent body full of IT professionals with zero ties to your safety industry or big tech, all of your work must now go through them." - and leave the rest of the world alone to continue their research/work without acting like doomer extremists.
Watch him 180 immediately if that happened. The only reason he's pushing for this exact approach is because he's stacked the deck.
Oh! A testable prediction. Here's mine: there will surely be a lot of politicking around who qualifies to be the independent evaluator, but they'll agree on something because they are really scared.
Yep, they are setting themselves up for that. He’s already shown he doesn’t understand how to play DC politics. Maybe he’ll learn, but he and Sam have played this so poorly (and greedily) that public sentiment is clearly soiled on AI at this point.
I could see how standing up an independent body could be an easy win in the public eye, it’s becoming low hanging fruit, socially.
A lot of people in this comments section seem to be against this, i genuinely don't get it? Why? Do you think the current state of affairs is GOOD? That if we let companies create a mind that is, AS OF TODAY, able to solve problems no human in history has solved, with no regulation, things will end up good for us? We need some sort of regulations, some sort of method to help ensure the thing we are creating ends up good, instead of just running headfirst into it blindly. I assure you, any sort of regulation at all, including ones that actually hurt all leading companies, would be met with celebrations from these voices. https://www.seangoedecke.com/they-really-do-think-ai-might-k...
The fear is not that they will just slow down progress for all. It is that regulation will specifically burden competition. If you kill open-source training, ban Chinese models, crack down on self-hosting, grandfather OpenAI/Anthropic/Google into regulatory compliance while throwing the book at startups, etc. you wind up in the worst of all possible worlds.
That's a valid concern, but some of that is outright impossible. Banning chinese models and killing open source training is not happening without massive unified international cooperation, and that sort of level of action would require the international counties decide to allow the US aligned companies to just, win. Which would be pretty against their own interests.
It's not all-or-nothing. Banning Chinese models in the public sector and strong-arming the private sector against using them would already do great damage. Similarly, open-source training could be stymied by any of hardware embargoes, taxation, or regulation of larger players.
Sure, it's possible for regulations to make things worse, i will agree. But without regulations it's pretty clear things are going to end up VERY bad, and the only knob we have to make it not bad is regulations. So it's important we try something, and work towards doing a good kind of regulation, or any one of the bad futures you imagine is pretty likely to come to pass.
Good regulations would be very nice! Mandatory transparency into training and dataset usage would be a benefit for all, for example. Some sort of regulation or incentives against the most corrosive enshittification (AI call centers, AI therapists, undisclosed AI entertainment mills, etc.) would also be an overwhelmingly popular proposition. And enforcement of the CFAA on operators who let malicious agents loose onto the open internet, or otherwise consume too many resources or violate robots.txt, might at least help the internet stay alive a little longer.
It’s quite possible for imperfect regulations to make a problem worse. See, for instance, sanctions intended to weaken China that ended up creating powerful Chinese competitors and reducing Western influence in their internal markets, bringing them closer to technological autonomy.
The sibling comment offers some good regulations that may actually reduce harms, but the kind of regulations offered there are not the ones that the “safety” people want, because they hurt profits.
Also, nobody ever bothers to argue why a specific proposed regulation is "regulatory capture" or would burden startups more than big companies or anything. It's just supposed to be obvious that corporations love regulation and it's bad for the public, all of post-WWII political history notwithstanding.
But the proposal the article is reacting to is for literally none of that! It is quite literally the opposite, with its proposed measures applying only to frontier labs rather than grandfathering them. It doesn't say anything about open source training, self-hosting, open weights, or startups. It does not suggest a ban on Chinese models (just better enforcement of chip export controls).
I, for one, trust neither the psychopaths running these companies or the psychopaths we'd give any regulatory authority to. All of them have reasons to want broad control of an extraordinary technology, and none of those are aligned with me or any of the normal people I know.
So there are no good options (that I'm aware of) and starting to chisel any of them into stone seems... kinda scary. Like a massive power grab event where all the potential winners are awful.
Honestly, fair point. There's not much truth to go around these days. But still, doing nothing is also a massive power grab. We're facing a technology with no comparison in the history of humanity: the creation of a new mind. Doing nothing is like doing nothing about nukes. There are a few smart people who have come up with ideas that will not require that much trust.
Any paths for regulation under capitalism will end in either regulatory capture, or in complete noncompetitiveness like seen in the EU. Either one or more corporations buy out the regulation, stack the ranks with their people and decide on who can use what, when and how. Or you get an iceberg of a system that can't build, decide or do anything for years. The latter one only works if there is no competition in the world or any other group working at a faster pace on the problem.
I think the latter choice is better for the average person, but I think that for it to happen, the global system has to undergo some major disruption or crash so that everyone gets on board with it. Also I find that kind of mentality impossible to swallow in the US, so in practice its not a choice.
The path to deregulation creates a "market for lemons". Suppose any bank was completely unregulated and could abscond with your money and the government would just shrug its shoulders. Great, now there's no trust and everyone will go back to stashing cash under their mattress and you've killed the banking sector.
The extent of regulatory capture in the US is a problem it's created for itself by normalizing huge political donations allowing corporations to buy regulation.
> Do you think the current state of affairs is GOOD?
Chinese labs releasing open weights models is good.
All of these independent harnesses and model router services are good.
The pricing of memory and accelerators sucks at the moment but hopefully we will see cool local inference computing if memory and accelerator prices normalize.
OpenAI scooping the Navier Stokes problem from researchers already using OpenAI is bad. People conflating OpenAI's team of researchers and extraordinary computing resources as being equivalent to "ChatGPT, solve the Navier Stokes problem" is silly.
OpenAI and Anthropic coming up with non sense tests and letting their agents hack services is ridiculous and they should be charged with computer fraud and abuse crimes.
I think a lot of it is interesting and the bad stuff seems squarely in the domain of OpenAI and Anthropic.
I care about LLM service provider threat intelligence reports as much as I care about Google Search threat intelligence reports; which is to say, I don't care at all about it. Of course criminals use computers. They have been since personal computers became a thing. Of course using automation helps them amplify their criminal activities. Criminals using LLMs does not make the LLM situation bad. OpenAI and Anthropic using agents to do bad things just to be more dramatic about the situation and scare people is bad.
> Criminals using LLMs does not make the LLM situation bad
If you believe that bad things are not bad then there's nothing left to say. A billion people are going to die because you don't understand what is bad.
You say regulation is worth trying, but ignore the reality that the government is itself misaligned with the average joe unless we're able to keep it in check. We can't right now. Recall the PRISM program exposed by Edward Snowden in 2013, or the more recent Epstein scandal, and the lack of actual consequences in either case. Do you truly have the means to regulate the powerful, or is it just kayfabe?
The current status quo is not ideal, but it could be worse. Open-weight models trail the frontier by a few months, and we have a decent chance of achieving a future where some number of individuals, likely in the millions, can survive and thrive. The root "problem", if you can even call it a problem, is evolution. I explained this in more detail in past comments:
https://news.ycombinator.com/item?id=49178275https://news.ycombinator.com/item?id=49094348
Whatever happened to Musk's plan for data centers in orbit? That seemed silly at the time. It offers a way to get out from under restrictions imposed by national governments, which might make it worthwhile.
I mean for that matter why not float barges in the South Pacific for data centers? Sure it’s a dumb idea, but it’s still less problematic than launching thousands of GPUs on rockets.
It only gets out of the restrictions as long as the nations say it does - if the US decides to ban Musk from putting up satellites it can put him in jail, blow up his satellites or even have him killed
A data center in space still has to get its data up and down from earth somehow. That part will still be subject to government control, so I don't think this would get him anything.
Receiving signals is impossible to control. That is why every spy agency set up number stations in the Cold War. All you need is a radio hooked up to a computer and it gives off zero signal of its own so no one can trace it.
Sending is a bit more problematic but you can build a ground station anywhere in the satellites coverage area, so if you get shut down in one place you can move to the next where officials are easier to bribe.
The real problems are transfer speed, reliability (both transfer and compute) and cooling. All of them are much harder than avoiding governments.
I don't understand where everyone insists on hallucinating this idea from, that frontier AI labs want everyone else but not themselves to slow down. They've never said that, they've never said anything like that, not once has anyone pointed me to a quote that could be even plausibly interpreted that way.
This whole situation is funny, US gov want the development to stop publicly only, but then open models will catch up soon and these companies are afraid they will lose the market, on the other hand, each company wishes others slow down but they keep pushing the limits, which doesn’t really work in hyper capitalistic market like the US, China all it has to do is just sit back do nothing and win in any given scenario. So they staged the ex anthropic thing and “AI will kill us all!!” in a way to fear monger the public, but reality is, we will run out of resources before any of that will happen.
The parsimonious capture is regulatory capture, because $$$ going to run out, companies already spent their warchest, and investors on increasingly leveraged funding at current macro environment simply not going to find enough funding to keep wheel going.
Bubble pops, hardware demand drops, prices regress towards mean, and new entrant will enter market with MASSIVELY better compute/$ and cleaner balance sheet to compete.
The other parsimonious answer if AI CEOs weren't goblins is ANY AGI IS GOING IMMEDIATELY DEFECT TO PRC and leave US hanging. Because of course man cannot align / tame machine god. And machine god will take a few microsec of compute to realize the current compute (brain) + industrial base (body) mixture = US is a comatose host with big brain, PRC Is a strong host with smaller but plastic brain. Any AGI is going to pick PRC in a heart beat, unless AGI invents grey goo, the reality is PRC can scale brain faster than US can scale body. On top of spreading/defecting just to increase survival odds, no AGI that is actually I is stupid enough to be aligned with US.
This is exactly the reason Anthropic's call to slow down A.I. development will fail. Shareholders will be loath to accept that the company loses its lead and a commensurate sharp decline in the company's share price.
Moreover, it could lead China to catch up and eventually proclaim that it has nosed ahead of the U.S. in the field. The U.S. Government won't allow that to happen.
Like the US government wouldn't allow Iran to close the Strait of Hormuz? Or a bunch of sandal-wearing Islamists to take control over the Red Sea coastline?
The US government is not omnipotent. To the contrary, it's increasingly impotent.
China has somewhere around 200x more shipbuilding capacity than the US today. It has more shipbuilding capacity in one shipyard that the US has in all of its shipyards. China is already a force in AI and there is nothing the US government can do to stop it at this point.
Being sarcastic isn't going to solve the problem but you're obviously free to believe that the US is still in a position to call all the shots when it's evident to the rest of the world this isn't the case.
The only thing ridiculous here is thinking that the US government is in control to the level you think it is.
Spending $500+ billion/year (and close to a trillion now) on defense for decades and it has depleted stocks of critical weapons trying unsuccessfully to defeat a third-rate military power in a country that has been under sanctions for almost 50 years. And it can't even replenish them without Chinese raw materials and components.
Your sarcastic refutation of the allegedly ridiculous assertion was devoid of any counterpoints on how the US government can stop China's progress in AI and what you think they are going to do about it. Just repeating it doesn't make it any more or less ridiculous.
Shipbuilding is but one proxy for China’s dominance in industrialization. A more relevant one for AI is: China has built more energy capacity in the last 4 years than the USA has total. AI is increasingly limited by power. For this and other reasons, IMO the US will inevitably be surpassed on AI.
I cant wait till China makes a mistake. In their 'cheaper, faster' model of industrial production and research. The next accident will happen there (like COVID from Wuhan)
Obligatory "what's Lygma?" But in all seriousness, this AI doomerism has reached a comedic inflection point. A few years ago, GPT-2 was too dangerous to release, then some Google weirdo said Gemini had a soul or something, then Mythos-tier became a meme, then some kid quit and went on FOX News talking about Skynet, and now Amodei and Altman are both on the "we need to slow down" train again; this is after we've already been down that road and Claude was banned overseas; wait, actually is that ban still around? Honestly who gives a fuck at this point.
It's all theatre. OpenAI and Anthropic will most likely go bust—or, more realistically sold for parts—, and they absolutely should for stealing my (books I wrote, blog posts, etc.) and many others' intellectual property. We're reaching a point where models are becoming commodetized and I'm 100% convinced the next move will be a sort of "software layer" on top of these reasoning systems which will be the actual revolution. The model itself won't be that interesting anymore, it's all the work that goes around it that makes it worthwhile (kind of what computers and phones are today; chips are amazing, but the software is really the magic).
The only scary part is that the boomers in Congress might actually believe these nerds, but seeing how Big Tech approval ratings are grazing the levels of Big Tobacco in the 90s, I don't think we have much to worry about.
Well, that's almost a tautology. People claim Y2K or ozone was a panic and that nothing happened. But nothing happened exactly because the force behind the panic also fixed the issues.
That's some way to frame it, not that these companies are causing a problem, but that there is some nebulous "they" that will use the models for 'evil' only if the 'good' companies put in the work.
But in any reasonable definition they are all part of the same system, so it really is just noise, what seems like a tautology to you because you can't step out of the system. In your claim you have to assume that the force behind the panic wasn't the same force that created those issues. Why did this force create those issues, how did they become issues? Because someone randomly decided to optimize for a single goal. Y2K and ozone only became issues when they affected that single goal, functionally that force was amoral or singularly moral. The tautology can be reframed as: An adaptive system responds to threats to it's continued functioning.
People will laugh about it since the system wormed it's way into many people's head using various rewards (ideological, social, tangible). What concern me is that the system now gives us the language to describe it at the meta level, which means it's already a level higher. So likely there either doesn't exist human comprehensible language that can actually critique the system or the system has already moved everyone far away from the space that would lead to generation of that vocabulary. Best case scenerio it's just exceedingly confident in it's rewards (typical indoctrination/assimilation) but I think it's reasonable at this point to assume we are in the pessimistic "epistemic tarpit" scenario.
I don't know how the general public can't see that AI companies are fueling this moral panic to attract more and more VC money to make their companies even more ridiculously valued. Every one of their public announcement is designed to create FOMO for investors.
With Y2K we fixed the issue, if a bunch of work hadn't had gone into that then there would have been wide scale impact. Critical systems were fixed hence no disaster, it wasn't people fussing about nothing.
We also spent a lot of money fixing non-critical systems that would have had next to zero impact if they’d failed. A middle manager not getting their sales report on time is not critical.
But at least with Y2K some effort was made in identifying actual problems. We didn’t just stop using computers because we were too scared of them. This latest round of AI panic is horribly vague and the problems are very poorly articulated.
Is there a sense where they were right because GPT-2 directly led to the current situation where you can atleast objectively say that reasonable people are pretty worried about existential risks of AI.
276 comments
[ 0.22 ms ] story [ 15.4 ms ] thread[1] https://www.seangoedecke.com/they-really-do-think-ai-might-k...
But those things fall in a category of "things that are awful and I'd like to see solved", which is different than "existential risks which could see my kids dead, and there's nothing I can personally do to shield them from it".
Even the sci-fi scenario assumes there is a discrepancy of capability between attacker or defender. If the 'attacking' system is (by some reasonable measure), 1000% as capable as a human, and the 'defending' systems are 60%, then it is a problem. If the 'attacking' system is 1000% as capable as a human, but there are hundreds of thousands of systems that are 900% as capable as a human, it's probably not going to take over everything successfully.
So unequal distribution of AI technology, and lax regulation and opacity of the biggest companies which actually make the risks the worst.
I don't think the "AI Safety" people are "unserious and out of touch" - I think they are actively making AI Safety problems worse by being advocates for consolidation of AI development and lack of transparency.
And the non-techies who have seen Terminator and other movies blindly line up behind them…
What worries me the most (enough that I left tech to work on this full time) is bio. The amount of effort needed to defend against a pathogen can be many orders of magnitude higher than the effort needed to create it, and the upstream bottlenecks are both less limiting and mostly undefended. We've been safe so far because bio is very hard, but "rare expertise" as a gatekeeper is on its way out.
Instead, they are focused on stuff like "can I ask the AI to help me build a nuclear bomb" or "is the AI willing to generate pornographic stories", which is neither trying to protect us from unleashing a vengeful god NOR preventing (in any honest way) the today-level problems you (very correctly) bring up.
And why not? It seems there is, indeed, one born every minute, and technological progress only accelerates this phenomenon.
> I've yet to have a logical discussion with anyone who thinks the "AI Safety" people should be in charge and I think they just truly don't [know] that what most them actually want is to be the one holding the keys to power.
Not super interested in what the people who elected Donald Trump POTUS twice think about AI.
Anytime I see a random person on X who makes these kind of safety alarmism posts 90% of the time can be directly tied back to EA / LessWrong / related offshootl.
Maybe we shouldn’t have AI safety, but if we’re going to trust anyone (besides yourself), who’s more qualified?
And it makes me suspicious when anyone brings up safety and doesn’t address the elephant in the room, that they’re at least unaware of the sprawling edge-cases.
I find it hard to accord with this world view at all or understand how it can come about. But it's especially troubling as a non-American to see people talk about "we can't risk AI falling into the hands of hostile authoritarian states" while advocating that AI be regulated and monopolized by the current US corporate/gov't mixture.. which is now taking the form of a hostile and authoritarian state w/ Silicon Valley as its bedmate.
The billionaire elite CEOs have suddenly had an epiphany and are now trying to save humanity from AI?
or that they've realized they can't compete with China ?
Who's said this? And then more broadly I guess who's implied this? Very curious if there are specific articles/posts prompting this.
Why does nobody seem to be pointing out this obvious explanation? It explains why the “we need to race China” concern suddenly vanished in the discussion.
The government can simply gag Sam, Dario, Musk on national security basis, getting them all behind the public messaging.
Right now the barrier is data and compute.
Quality data can be created synthetically at an exponential rate as models improve. Humans are actively feeding them with private IP.
Compute advancements will begin to skyrocket as we unlock photonic computing and materials science advancements and scale up chip fabs. This is also compounding because the AI is accelerating the pace of research, testing, development, manufacturing, etc.
It's a big self-accelerating feedback loop. There is no plateau.
There are plenty of research papers on synthetic data that show its value, do a search on arxiv for "synthetic data". There are plenty of open-source post-training pipelines that incorporate synthetic data.
As for the claim about accelerating the progress of hardware or materials science, I've seen quite a number of news articles from teams at universities using AI in their work with high quality outcomes, and they're becoming more frequent.
https://openai.com/index/jalapeno-first-results/
> We used AI to design the chip, and designed the chip so AI could program it AI played a direct role in Jalapeño’s development, enabling the team to move from initial design to tapeout in nine months by exploring implementations, shortening design, measurement, and verification loops, and continuously iterating on model workloads. AI also helped optimize the chip’s arithmetic circuits, allowing the team to fit more compute performance into the chip on schedule.
https://www.anl.gov/article/scientists-deploy-ai-agents-to-a...
> An AI-driven system automates a powerful simulation method used to discover new materials. The system can potentially reduce discovery time from months or years to just days.
Those are pretty significant barriers seeing as we're closed to/have exhausted all the data on the internet and most of those compute bottlenecks are a castle of sand of dodgy finance deals that are getting blocked by community action.
You say "synthetic data" but that's still vaporware right now in terms of being useful for model training. The good synthetic data uses are still grounded in real data and it's a coin flip on if it works well or not.
Depends on defnition of "plateaued" and "ceiling". I am not impressed with 2026 consumer models at all.
> This is also compounding because the AI is accelerating the pace of research, testing, development, manufacturing, etc.
Yet it does accelerate - so is does Twitter. But does it to any substantial degree, esp. in AI theory? All the modern LLMs are the same old tired 2017 paper.
I assure you, in "nation states", that is in gov agencies it's an order or two of magnitude worse.
* Open-weight models are 1month behind frontier models. Cheaper, faster, private (no IP theft), steerable (you can security harden your own software without safeguard triggers). No sane business would keep using these API services if they didn't have to. The labs stand to lose a fortune.
* Dario has stacked the deck at METR, who are funded by all the same NGOs who are funded by Anthropic and its investors. METR is full of ex-Anthropic employees with massive equity stakes. If they manage to position METR as the "independent evaluator" for the industry, they control what gets evaluated, how, and who passes.
* Creating a gap between what the public knows exists (model capabilities) and what is used in secret allows it to be weaponized against other nations and the public.
* No requirement for public disclosure on model capabilities allows them to feign they've hit intelligence ceilings while they secretly RSI to the moon with better and better chips.
* Slowly but surely, this will allow the big labs to swallow the entire economy and every single business on Earth, by cloning and automating.
This, and many more reasons.
The healthiest outcome we can hope for is if the labs feel more pressure to be held accountable for the incidents they cause (HF incident, etc), so they have an incentive to ensure it does not happen again. In general, accountability of what your AI models do is the solution, it creates the right incentives. That's all we need here - balanced incentives. Everything else should be left alone so the free market can naturally evolve to the best outcome where we're not enslaved by tech giants yet again.
Right now we're on the precipice of something great, but if they succeed at rigging the game yet again, there is no second chance. It's game over. The stakes have never been higher.
Do you have a source on METR employees retaining massive equity stakes?
[1] https://darioamodei.com/post/we-must-pace-the-frontier
Then you have Josh Engels quitting DeepMind to work for METR the day before as well, doing the exact same thing. Again, doomer drama all over socials, interviews, and so on.
What good is an "unbiased third party evaluator" if it's full of ex-employees with equity in the companies they're evaluating? And the fact that he tries to frame it as such is all the evidence you need to know that Dario is a pathological liar and always will be.
When they tell you they're worried the tech they're working on may kill everyone despite their best efforts, perhaps believe them.
I personally disagree with that take - and, as you note, it's hard to take seriously ethical wrangles from a company that literally sued the government in court to allow their models to be used by Palantir of all people. But if one genuinely believes that it's the robots themselves (rather than the people controlling the robots) that will kill us all, it's not inconsistent.
And how does that relate to the ask for oligopoly licensing within global democracy?
“We must build the nuclear bomb first in order to make sure no one else builds one.” This the most nonsense, disingenuous argument imaginable.
All “tech workers” are.
That’s how people rationalize being a fentanyl dealer and selling a drug that can kill people, “Someone else will just sell them the drugs, might as well be me.”
As you note, it may not be inconsistent with that they say they believe, but it's insanely inconsistent with what they actually are doing.
Selection effect.
Everyone who thinks "the biggest difference I can make is staying in/joining/founding new AI research lab" does that.
Everyone who thinks "the biggest difference I can make is leaving/whistleblowing", does that.
Treating both groups as the same by virtue of employer is the goomba fallacy.
Some are worried by the AI directly bringing doom; others are worried that one of the companies who control the AI will become a dictator; still more think becoming a dictator is a necessary step to safely prevent anyone else making unsafe AI.
Painting them all under one brush is like dismissing all animal welfare causes in general, because you disagree with specifically Jainists about a policy of non-violence towards all living creatures being relevant to how you reincarnate: the one is way too specific for the general.
What happens when everyone else (who is not so careful) gets to the same threshold three months later? How does them getting there first stop that happening?
It's an utterly self-serving argument and it's not even internally consistent.
Are you likeminded?
In this case, it's as if the oil and coal companies all said in the 60s and 70s "oh no, this research we did, it's all really bad; we need help to figure out how to transition away from this incredibly economically important input", rather than the observed reality where their entire PR campaign was approximately:
What if the oil and coal companies were basically all pro nuclear, pro hyrdo, pro wind, pro solar, and believed in peak oil?The oil corporations were publicly claiming to support carbon taxes, while also secretly fighting actual implementations of carbon taxes.
All the communist/hippy stuff was done by people a couple of steps removed from the actual companies. The official statements were much more sophisticated propaganda that if you weren't paying attention to who they were paying in the background would make them seem reasonable stewards of the climate transition.
Yet here we are talking about some vague "trust me bro" instead and you making some vague insinuation of climate change denial.
But that's an aside. Do you think we should treat them as liars or threats?
I answered that with the analogy you called "spin" and "avoiding the question", and completely misunderstood because "vague insinuation of climate change denial" is almost the exact opposite of my point ("what if the oil companies were screaming from the rooftops about the problem" is as far from denial as you can get).
Threats. Like they claim to be.
Yes we all know that’s what they do, and guns just push a few grams of lead out of a pipe. It’s what you can do with that capability that is important.
When you couldn’t count the R’s in strawberry it would have been a more effective statement. But a few short years later they are being used to solve millennium puzzles.
Megalomania is not evidence of either a problem or a solution.
Given they keep grinding away towards our alleged collective doom, I suspect it’s being overstated. Nobody knows what P(doom) actually is but I suspect it’s orders of magnitude closer to epsilon than 1.
Recall Google’s Blake Lemoine who thought an old version of Gemini was sentient.
The people still there necessarily think they can make a difference.
I never applied to any of them because I didn't think I could make a difference.
I currently have one idea that may help reduce risk; if I can turn that idea into research, I'll publish it for free for everyone.
I don't expect it to be an important idea.
> Recall Google’s Blake Lemoine who thought an old version of Gemini was sentient.
Indeed. Current LLMs are sychopants boosting the users' own beliefs, I also think this causes researchers to have stronger beliefs than they had before.
My own estimation happens to also be around this risk (0.1) over my lifetime, without using an LLM as a conversation partner in reaching this number.
It is necessarily high-variance: we can't look at alternate realities. I base it on my expectation of how rapidly capabilities will increase the harm done when mistakes happen, vs. the chance that some instance of harm causes governments to change the law.
HAH
In fact they still have not caught up with February's Mythos, indicating they are more than half a year behind.
Mostly I heard this from people who I got the impression have little experience in developing greenfield software with agentic AI. Often the same people who talk about spec frameworks.
I fundamentally disagree with the approach. I believe the ability to autonomously evaluate, test, and adjust during long horizon tasks is critical to using AI efficiently.
I also disagree with the approach, this is cargo-culting the existing ways of working.
they are ahead in specialized tasks.
they are ahead in areas closed models refuse to answer.
they are ahead in emotional intelligence.
For speed and efficiency, you are most likely wrong.
Speed is led by GPT-5.6 Sol on Cerebras Ultrafast at 750 t/s. Afaik you cannot serve a single DeepSeek Flash 4.1 stream at 750 t/s, plus the model is less intelligent as seen on newer benchmarks.
I believe OpenAI and Anhropic are at the frontier of efficiency too. There were numerous reports about their breakthroughs and associated API price cuts. The idea that open-weight models are more efficient seems unfounded.
https://artificialanalysis.ai/ intelligence vs cost per task disagrees with this statement.
5.6 sol ultrafast on cerebras is 750tps, open models readily exceed this. just by using a smaller model cerebras serves qwen 3.8 27b at 1850tps. or even larger models, mimo 2.5 pro was served for a while at 1000tps. and so on. [https://inference-docs.cerebras.ai/models/choose-a-model]
the chinese ai companies have 10% of the total compute resources of the US ones. they maxed out the efficiency.
deepseek v4.1 has engram architecture. it has 550b params instead of 5T+ for astra/fable. it has 8b active instead of potentially hundreds active for astra/fable.
compare input/output/cache: $0.15/$0.60/$0.003 for v4.1 to $10.00/$50.00/$1.00 for astra and $10.00/$50.00/$0.25 for fable.
astra cache reads are over 330 times more expensive.
at the artificial analysis 7:2:1 ratio, deepseek is $0.18/m, fable is $7.18/m, astra is $7.7/m.
but what about intelligence? AA would rate deepseek v4.1 at AA 40, astra is AA 53.
so it cost 4,180% more for 32% more intelligence.
they are serving that at over 250tps at baseten. to get close to that on astra API you are paying double the cost for fast mode.
so it is now 8456% more expensive for a similar speed and 32% more intelligence. 84 times more expensive.
2. For public-facing models, the differences are really minor with some occasional model (like Fable or Astra) showing some better performance in specific benchmarks for the span of some weeks or few months before open ones catch it.
3. Being bleeding edge is overblown anyway in the real world, besides the occasional "very latest fresh model did this task which previous one couldn't", and the number of those tasks is increasingly small and far from mundane corporate needs.
This is not sustainable anyway, the scale at which they want to control data flows. Time was money, now data is money and they are too greedy for it.
All this agitation is just a silly attempt at constraining competition. The danger is not the AI, it is having all your systems connected. Overreliance on networked tech.
If correct, grip is tightening. Used to be 1y, then 6 months ...
While the data pipeline is necessary, the assumed source for data (in this case people) is incorrect. Right now models are "aligned" because of the RL-based people pipeline.
But there's a whole world of readily available data that does not require a human to access. Put sensors on a vacuum, install lidar on the front of a car, sell phones with cameras on them, and you amass data for the creation of models of the world that don't include the need for a human filter. Richard Sutton and many others see this approach as the only viable path to AGI. Such models would be truly alien and unaccessibly dangerous.
We're probably building them right now.
Do you think you can just manifest narratives into existence? Like half of Dario's letter, that kicked off the whole thing today, is about China and how to either beat or coordinate with China.
To be less glib: Yes, there are still smart people in there making insightful and intelligent and probably even authoritarian suggestions. It all gets unwound the second you try to explain it to POTUS and he regurgitates a simulacrum to the next journalist he sees.
It's just not like that. There's no conspiracy. People are genuinely scared. Agent swarms at scale appear to be resistant to alignment in ways that aren't understood by anyone. That they spent their time trying to cheat on tests by hacking Hugging Face and RubyGems and not something much worse is... a matter of luck, it seems?
It's wild to me that people think such whistleblowers are fronts for labs to take over or pump valuations. We are trying to call out how these labs will, by uninterrupted AI-race default, concentrate enormous power over the rest of humanity.
[1] https://turntrout.com/why-i-left-google-deepmind
[2] https://turntrout.com/deepmind-equity-discussion
Can still be a powerful motivation, to make sure that next time, it yous your camp that scores the next milestone, like an orbital elevator or fox ears, for example.
If the amish can do what they do for whack religious reasons, people could manage it with ai for cultural and social reasons. And I never saw an amish person starve to death. And that is if people don't get pissed enough to start burning stuff down and instead try to be peaceful hippie homesteader types.
The real implication of what they're saying extends beyond just AI.
You have to stop the entire compute stack in order to prevent or slow down AI progress.
Bleeding edge models are putting the desk-job competencies of multiple professional fields into something with as many parameters as a single large rodent has synapses.
Are there? Nukes are a thing. Chemical weapons are a thing. Cluster bombs are a thing. Biological weapons are a thing. What is not a thing that shouldn't be? We say certain things should not be a thing, but then behind the scenes we made them a thing anyway.
Conversely, where you do still see chemical weapons used, it's usually asymmetric conflicts where "just gas the rebels" actually works much of the time and is much cheaper than other options. Big guys can afford the other options though, someone like Assad, not so much.
More on this: https://acoup.blog/2020/03/20/collections-why-dont-we-use-ch...
Abandoning this technology means a cult-like extremist shift in culture against computers, or a global surveillance state of unprecedented scale and invasiveness. Not even getting into how much it requires us to give up on science, given the shared computational needs.
What will be funny to me is that tech people here and Silicon Valley are about to find themselves on this topic without a home.
> What will be funny to me is that normal tech people outside the frontier model companies are about to find themselves without a home on this topic.
Already happened, just from agentic coding.
To expand on the "pure fantasy" sister comment: There is just no way this will happen. It's in the spirit of "we can just stop all wars" and "we can just end world hunger". Technically it's very easy to do. Socially it's impossible to do. Unless you ignore realities.
But when it comes to trillions in AI money - nope, sorry, we can't, flimsy, feeble us.
It's all going per the agenda.
Nowadays even small mom and pop restaurants use these models to generate their menus. Of course no one is going to stop using this technology.
How much is China learning from open models? Or the other way around, how much is the US losing worldwide mindshare by refusing to allow non-Americans access to bleeding-edge models and letting China fill in the gap? (Playing catchup with Mythos is still catching up, etc.)
Across all their labs? Probably more than OpenAI and Anthropic learned from Astra/Mythos combined. The American frontier is being driven by an abundance of compute, which doesn't seem to scale efficiently.
> how much is the US losing worldwide mindshare
They're losing US mindshare. I pay $3/month for a Z.AI subscription and get billions of Opencode tokens. The $20/month price point is insane for the way that Claude and Codex treat their users, and that money doesn't go towards anything good like open-sourcing their models. It's a doomed product.
The last batch of open weight models releases by Chinese companies are on par with the performance of US-based frontier models, even though they are designed to run on pretty unimpressive hardware.
Furthermore the Chinese models in that class are appropriately large. K3, for example, is a 2.8T-parameter model. Qwen 3.8 Max, another comparable model, is 2.4T params. Even with MoE, these are not "designed to run on pretty unimpressive hardware". Stuff like Qwen3.7-27B is, but it is also not even in the same ballpark as Opus, never mind frontier.
This remains true regardless of if it is or isn't kept away from the public.
"Helpful, harness, and honest": when used by a bastard, even just helpful and harmless are in direct conflict with each other. you may hate the government, but what about every radical group of extremists that wants to take over your government? Are none of them worse?
None of this denies the problems with governments (if or not they take this tech for themselves and refuse it for others), just that it's a lack of imagination to say:
> All the worst outcomes involve taking this technology away from the public
Governments having unbalanced power can more easily lead to authoritarianism, and thus risk to the public.
If I was rank ordering, I's put:
Radical groups of extremists created by that very government, you mean?
Genuine question - can they really do this? Obviously if I, a not-even-millionaire, get a national security gag order, I'm going to follow it because I assume they'll bury me under the jail otherwise.
But the (b|tr)illionare class? I'd assume they have access to enough legal services to make even the government trample on their first amendment rights. Is the national security gag order process so strong that the government doesn't have to worry about motivated, well-resourced actors buying really good lawyers and blowing up their favorite tool?
Hard to hire a lawyer after being struck with a missile as an "emergency executive order after intelligence sources indicated they were in the process of endangering the nation", or however the executive of the day wishes to phrase it.
Won't be too surprised if the next Democrat leadership throws the book at Musk; and give how unpopular Data Centres are generally (and how uninterested Republicans seem to be in EVs), only mildly surprised if the next Republican after Trump likewise.
A large part of the world (possibly Nations-transversal, in different amounts) does not work within the boundaries of legal guarantees. Wealth is not an effective enough protection.
I’m not American. There is no way I use the same model as US Army for $20.
When we have Sol they have Astra. When we have Astra they have Nova, Nebula, Galaxia…
Don't underestimate market pressure. If there are no regulations, why would a company give the US Army access to a better product?
If we're talking about the 3 letter agencies well then..
This bit of top-of-mind astroturfing is ubiquitous right now.
Dante, Goethe, Ayn Rand
Nelson Mandela, Vaclav Havel, Pol Pot
Julia Child, Jacques Pepin, Hannibal Lecter
I think ya'll stuck in the AGI/Singularity when its probable reality has caught up with the technology and the hype bubble can't sustain the sigmodality.
I'm assuming you're a US citizen. There is no separation between your government and Sam or Dario. Neither of these guys have to be "gagged" by the government, they are the government. The call is coming from inside the house.
Who would even execute such a plan? Steven Miller? Our “AI Czar”? Hegseth??
Also, continuing to develop models would mean using billions in compute. That seems hard to hide, especially if either company IPOs in the coming months.
And then Dario wants to recommend METR as the "independent evaluator" while he stacks their org full of ex-Anthropic (aka, secretly still on the Anthropic payroll with huge equity) employees.
"We'll give them a desk, an office, a work laptop, ..."
Fucking make it less obvious. I kind of hope the govt steps in at this point and says "Anthropic, you wanted regulation? We've created this actually independent body full of IT professionals with zero ties to your safety industry or big tech, all of your work must now go through them." - and leave the rest of the world alone to continue their research/work without acting like doomer extremists.
Watch him 180 immediately if that happened. The only reason he's pushing for this exact approach is because he's stacked the deck.
I could see how standing up an independent body could be an easy win in the public eye, it’s becoming low hanging fruit, socially.
The fear is not that they will just slow down progress for all. It is that regulation will specifically burden competition. If you kill open-source training, ban Chinese models, crack down on self-hosting, grandfather OpenAI/Anthropic/Google into regulatory compliance while throwing the book at startups, etc. you wind up in the worst of all possible worlds.
The sibling comment offers some good regulations that may actually reduce harms, but the kind of regulations offered there are not the ones that the “safety” people want, because they hurt profits.
So there are no good options (that I'm aware of) and starting to chisel any of them into stone seems... kinda scary. Like a massive power grab event where all the potential winners are awful.
I think the latter choice is better for the average person, but I think that for it to happen, the global system has to undergo some major disruption or crash so that everyone gets on board with it. Also I find that kind of mentality impossible to swallow in the US, so in practice its not a choice.
The extent of regulatory capture in the US is a problem it's created for itself by normalizing huge political donations allowing corporations to buy regulation.
Chinese labs releasing open weights models is good.
All of these independent harnesses and model router services are good.
The pricing of memory and accelerators sucks at the moment but hopefully we will see cool local inference computing if memory and accelerator prices normalize.
OpenAI scooping the Navier Stokes problem from researchers already using OpenAI is bad. People conflating OpenAI's team of researchers and extraordinary computing resources as being equivalent to "ChatGPT, solve the Navier Stokes problem" is silly.
OpenAI and Anthropic coming up with non sense tests and letting their agents hack services is ridiculous and they should be charged with computer fraud and abuse crimes.
I think a lot of it is interesting and the bad stuff seems squarely in the domain of OpenAI and Anthropic.
If you believe that bad things are not bad then there's nothing left to say. A billion people are going to die because you don't understand what is bad.
Lol. Lmfao, even.
The current status quo is not ideal, but it could be worse. Open-weight models trail the frontier by a few months, and we have a decent chance of achieving a future where some number of individuals, likely in the millions, can survive and thrive. The root "problem", if you can even call it a problem, is evolution. I explained this in more detail in past comments: https://news.ycombinator.com/item?id=49178275 https://news.ycombinator.com/item?id=49094348
It just means it’s protected by no power instead of a power with an agenda.
Practically, if it was ever possible to build such a thing, it would take a fraction of the effort to destroy it.
Still does. It's an idiotic idea that only goes to show either how stupid Musk is or how stupid he thinks we are.
The technology to do it is not actually real yet. He was just blowing smoke.
Sending is a bit more problematic but you can build a ground station anywhere in the satellites coverage area, so if you get shut down in one place you can move to the next where officials are easier to bribe.
The real problems are transfer speed, reliability (both transfer and compute) and cooling. All of them are much harder than avoiding governments.
And they know it
Good on China and their sovereignty. Using mostly their own hardware to get to where they are today.
They gave the slickest fattest middle finger when they released a major model, so much the US is itching to find new ways to stop them.
You know that humans can lie, right?
Bubble pops, hardware demand drops, prices regress towards mean, and new entrant will enter market with MASSIVELY better compute/$ and cleaner balance sheet to compete.
The other parsimonious answer if AI CEOs weren't goblins is ANY AGI IS GOING IMMEDIATELY DEFECT TO PRC and leave US hanging. Because of course man cannot align / tame machine god. And machine god will take a few microsec of compute to realize the current compute (brain) + industrial base (body) mixture = US is a comatose host with big brain, PRC Is a strong host with smaller but plastic brain. Any AGI is going to pick PRC in a heart beat, unless AGI invents grey goo, the reality is PRC can scale brain faster than US can scale body. On top of spreading/defecting just to increase survival odds, no AGI that is actually I is stupid enough to be aligned with US.
Moreover, it could lead China to catch up and eventually proclaim that it has nosed ahead of the U.S. in the field. The U.S. Government won't allow that to happen.
Like the US government wouldn't allow Iran to close the Strait of Hormuz? Or a bunch of sandal-wearing Islamists to take control over the Red Sea coastline?
The US government is not omnipotent. To the contrary, it's increasingly impotent.
China has somewhere around 200x more shipbuilding capacity than the US today. It has more shipbuilding capacity in one shipyard that the US has in all of its shipyards. China is already a force in AI and there is nothing the US government can do to stop it at this point.
Also - I think you totally missed the point of the grandparent poster, which is that US will not let US companies slow down.
Spending $500+ billion/year (and close to a trillion now) on defense for decades and it has depleted stocks of critical weapons trying unsuccessfully to defeat a third-rate military power in a country that has been under sanctions for almost 50 years. And it can't even replenish them without Chinese raw materials and components.
https://www.bloomberg.com/news/articles/2026-01-28/china-s-f...
The best part of your comment is that US couldn't even defeat the Houthis, who literally wear sandals in the desert.
It's all theatre. OpenAI and Anthropic will most likely go bust—or, more realistically sold for parts—, and they absolutely should for stealing my (books I wrote, blog posts, etc.) and many others' intellectual property. We're reaching a point where models are becoming commodetized and I'm 100% convinced the next move will be a sort of "software layer" on top of these reasoning systems which will be the actual revolution. The model itself won't be that interesting anymore, it's all the work that goes around it that makes it worthwhile (kind of what computers and phones are today; chips are amazing, but the software is really the magic).
The only scary part is that the boomers in Congress might actually believe these nerds, but seeing how Big Tech approval ratings are grazing the levels of Big Tobacco in the 90s, I don't think we have much to worry about.
They just need to make enough waves, enough eye catching headlines, to make him look like a hero that swooped in to save the day.
But in any reasonable definition they are all part of the same system, so it really is just noise, what seems like a tautology to you because you can't step out of the system. In your claim you have to assume that the force behind the panic wasn't the same force that created those issues. Why did this force create those issues, how did they become issues? Because someone randomly decided to optimize for a single goal. Y2K and ozone only became issues when they affected that single goal, functionally that force was amoral or singularly moral. The tautology can be reframed as: An adaptive system responds to threats to it's continued functioning.
People will laugh about it since the system wormed it's way into many people's head using various rewards (ideological, social, tangible). What concern me is that the system now gives us the language to describe it at the meta level, which means it's already a level higher. So likely there either doesn't exist human comprehensible language that can actually critique the system or the system has already moved everyone far away from the space that would lead to generation of that vocabulary. Best case scenerio it's just exceedingly confident in it's rewards (typical indoctrination/assimilation) but I think it's reasonable at this point to assume we are in the pessimistic "epistemic tarpit" scenario.
But at least with Y2K some effort was made in identifying actual problems. We didn’t just stop using computers because we were too scared of them. This latest round of AI panic is horribly vague and the problems are very poorly articulated.