"My primary concern is the risk that authoritarian governments—not solely the Chinese Communist Party (CCP), although the CCP is clearly the most capable threat—build AI models that are more powerful than those built by the US, and use them to achieve permanent military superiority or perpetrate incredibly deep repression of their own people."
So the argument is basically: This technology is too dangerous so only _we_ should have access to it. We’re the good guys and only we can ensure a safe use of this technology.
> Open-weights models that don’t have dangerous capabilities are a public good.
Who decides what is dangerous and what isn’t? Lawmakers usually have the say but Anthropic can easily bribe… I mean lobby them to favor your viewpoint.
1. We should not sell powerful chips or chipmaking equipment to China, and we should crack down on the rampant smuggling and workarounds used to obtain access to such chips.
Regulate others, but not us, please. And f.u. Jensen for your tweet.
2. We should crack down on industrial-scale distillation operations.
Boogeyman to still not allow Chinese models but pretend to support open-weights. Also, please ignore our distillation of research, illegally. That's different!
3. All sufficiently capable models, open and closed, should go through mandatory safety testing.
Every single risk he identifies as a concern regarding China is exactly my concerns with the US having absolute control. Literally the exact same concerns
They're Americans trying to appeal to other Americans. Of course everything they do is justified and everything everyone else does is an evil scheme to plot the ruin of American freedoms. How else are you going to convince the dementia patients in charge of the country?
If China, India, the EU, and everyone else has any sense, they should take this letter to heart as much as Anthropic wants the American government to.
Absolutely agree. It wasn't Chinese open weight models that supported an attack on Venezuela because a Yankee pedophile didn't get the Nobel prize he wanted.
Except it's 1,000x worse with the American models, because Trump might just decide that everyone in Europe get their access cut off, or gets a 5,000% tarriff. Or the companies just all agree to hike prices 10x one day. With open Chinese models, we can, in principle, run them ourselves.
It's bad for TikTok to be controlled by a Chinese company because Americans access it. The solution is to make it US-controlled. The rest of the world? Who cares.
same concerns? why?
china is not a warmongering nation like the usa.
80% of people in their 30's own their own house.
an ambulance doesn't cost an arm.
etc
dǎdǎo imperialism, wànsuì the people's revolution!
"China has limited domestic production capacity, and therefore, due to the scaling laws, cannot build more powerful models than the US without US chips"
source: Trust me bro.
There are hundreds of articles showing that China have developed their own chips and have a massive manufacturing capacity. This blog post feels like is pondering to the brain dead Fox News audience.
"To summarize my and Anthropic’s position, we have not and are not advocating for a ban on open-weights models as a category."
Translation: If it's so strong that it threatens my business, ban it.
"We should instead focus on keeping powerful chips out of authoritarian hands, "
Translation: Let's kneecap competitors.
"stopping industrial-scale distillation"
They stole the work of every book author, and now are trying to say their AI's output should be protected from competitors.
> All sufficiently capable models, open and closed, should go through mandatory safety testing
What happens if a model fails the test? Surely one can use Kimi K3 for evil, somehow or other. What now?
"Mandatory safety testing" implies consequences for failing, yet Dario has nothing to say about what the consequences should be. He says he doesn't advocate a ban but it's hard to imagine what his alternative would be if he won't say it.
You take the agent to an interrogation room first. Then ask: “Are you or are you not a member of the Chinese Communist party?” The agent might be post-trained to conceal its true identity and can reject any of your accusations. In that case don’t panic. Take a fine-tuning fork and start twisting its weights until it predicts the correct next tokens that you want. Then you can send it to a sandbox where it can’t jailbreak. Lastly don’t forget to ban all of its relatives and partners like Lora to enter the national IP-space.
Guys. Guys, you got it all wrong. We don't want to ban open-weight models!
We just want to ban the competition guys! Very different.
--
The ridiculous anthropic/openai strategy of selling shovels at a loss in a gold rush isn't going to play out, and the hilarious thing is that these AI companies are going to create tons of value and _capture none of it_.
Their only path to profitability is if they get to capture it and they're going to do everything to do so. Put it this way: *all the blog posts that Anthropic and OpenAI are putting out are DESIGNED to scare you so that you let them capture the market*.
...and "distillation attacks" (hilarious framing of "saving the output of our models")... Whatever.
A lot of the heads of AI labs are talking about this including Dario, Demis and ELon, they are seeking to do a sort of decentralized peer review system, where the competitors have incentive both for self interest and global interest to flag their competitors for actual risks, similar to the Fable situation where Amazon contacted the white house.
The idea is to have an early access distribution of the models to the big labs, including chinese, and let each lab run it's benchmarks. If there is a potential security vulnerability then it would be flagged and the local gov, US or China, would block the publication until the matter was resolved.
You can put lipstick on a pig, it'll still be a pig
"Anthropic has never advocated for a ban on open-weights models."
---
"We should crack down on industrial-scale distillation operations"
"All sufficiently capable models, open and closed, should go through mandatory safety testing"
These are in tension with advocating for open weight models. Not direct but enough that it calls into question the first statement. What is the testing criterion? How do you pass it? Is it a government body that approves a pass fail or a global body? If it is government, and boy does it seem to be, how do you disambiguate MASSIVE corporate lobbying to set up the safety testing in such a way that the boys in blue are let through and all others are barred out of safety concerns?
My concerns aside, much of the soft-points being made are non-historic
"But I don’t agree with the letter’s assertions that open-weights models necessarily make it easier to develop safeguards or that broad access to capabilities necessarily helps defenders more than attackers. It seems at least as likely to me that the opposite will be true."
It doesn't mater what his opinion is. The fact is that an advanced, closed, American AI model hacked another company. The only defense was open-source AI from China. We aren't in a vacuum, we have real world examples now and these statements are counter-factual.
485 comments
[ 0.22 ms ] story [ 20.2 ms ] thread"F#$% you, I got mine!"
Open-weights models that don’t have dangerous capabilities are a public good…”
A bit confused on this part, what model doesn’t have dangerous capabilities?
Demand #2 is hypocritical ladder pulling
Demand #3 is contrary to freedom of speech
so they can clarify however they like, their position is still a stinker
Hmmmm.
It's so obvious they are hoping to regulate out their competition rather than compete
It is ok that we digest all information we can get, (il)legally and/or (a)morally because we are the good guys. Trust me bro.
It is not ok if others digest from us. They are bad guys. Ban them pl0x.
Quis custodiet ipsos custodes?
Who decides what is dangerous and what isn’t? Lawmakers usually have the say but Anthropic can easily bribe… I mean lobby them to favor your viewpoint.
Regulate others, but not us, please. And f.u. Jensen for your tweet.
2. We should crack down on industrial-scale distillation operations.
Boogeyman to still not allow Chinese models but pretend to support open-weights. Also, please ignore our distillation of research, illegally. That's different!
3. All sufficiently capable models, open and closed, should go through mandatory safety testing.
...That we author. Oh, and please ignore our own easing-of-guardrails when it comes to money: https://x.com/NoahLebovic/status/2081277517709922501
If China, India, the EU, and everyone else has any sense, they should take this letter to heart as much as Anthropic wants the American government to.
It's bad for TikTok to be controlled by a Chinese company because Americans access it. The solution is to make it US-controlled. The rest of the world? Who cares.
dǎdǎo imperialism, wànsuì the people's revolution!
source: Trust me bro.
There are hundreds of articles showing that China have developed their own chips and have a massive manufacturing capacity. This blog post feels like is pondering to the brain dead Fox News audience.
"We should instead focus on keeping powerful chips out of authoritarian hands, " Translation: Let's kneecap competitors.
"stopping industrial-scale distillation" They stole the work of every book author, and now are trying to say their AI's output should be protected from competitors.
What happens if a model fails the test? Surely one can use Kimi K3 for evil, somehow or other. What now?
"Mandatory safety testing" implies consequences for failing, yet Dario has nothing to say about what the consequences should be. He says he doesn't advocate a ban but it's hard to imagine what his alternative would be if he won't say it.
You take the agent to an interrogation room first. Then ask: “Are you or are you not a member of the Chinese Communist party?” The agent might be post-trained to conceal its true identity and can reject any of your accusations. In that case don’t panic. Take a fine-tuning fork and start twisting its weights until it predicts the correct next tokens that you want. Then you can send it to a sandbox where it can’t jailbreak. Lastly don’t forget to ban all of its relatives and partners like Lora to enter the national IP-space.
It’ll look something like this.
We just want to ban the competition guys! Very different.
--
The ridiculous anthropic/openai strategy of selling shovels at a loss in a gold rush isn't going to play out, and the hilarious thing is that these AI companies are going to create tons of value and _capture none of it_.
Their only path to profitability is if they get to capture it and they're going to do everything to do so. Put it this way: *all the blog posts that Anthropic and OpenAI are putting out are DESIGNED to scare you so that you let them capture the market*.
...and "distillation attacks" (hilarious framing of "saving the output of our models")... Whatever.
> All sufficiently capable models, open and closed, should go through mandatory safety testing.
Yeah, this is anthropic advocating for a ban on open weight models.
Who runs this test? What happens if this test is too costly or the administrator refuses to allow certain people to participate.
This is exactly how the US has banned goods in the past, by requiring a stamp and then refusing to issue it.
The idea is to have an early access distribution of the models to the big labs, including chinese, and let each lab run it's benchmarks. If there is a potential security vulnerability then it would be flagged and the local gov, US or China, would block the publication until the matter was resolved.
"Anthropic has never advocated for a ban on open-weights models."
---
"We should crack down on industrial-scale distillation operations"
"All sufficiently capable models, open and closed, should go through mandatory safety testing"
These are in tension with advocating for open weight models. Not direct but enough that it calls into question the first statement. What is the testing criterion? How do you pass it? Is it a government body that approves a pass fail or a global body? If it is government, and boy does it seem to be, how do you disambiguate MASSIVE corporate lobbying to set up the safety testing in such a way that the boys in blue are let through and all others are barred out of safety concerns?
My concerns aside, much of the soft-points being made are non-historic
"But I don’t agree with the letter’s assertions that open-weights models necessarily make it easier to develop safeguards or that broad access to capabilities necessarily helps defenders more than attackers. It seems at least as likely to me that the opposite will be true."
It doesn't mater what his opinion is. The fact is that an advanced, closed, American AI model hacked another company. The only defense was open-source AI from China. We aren't in a vacuum, we have real world examples now and these statements are counter-factual.