I’m still kind of shocked that Dean Ball can tweet such incendiary stuff about OpenAI policy. Like presumably OpenAI would prefer it if their staff don’t pick fights with Trump administration officials.
The only good arguments I see against open weight AI also apply to closed AI. And regardless, the box is open, nobody can stop it even if stopping it was a good thing.
This post does not mention safety at all. What's to stop bad actors from fine tuning open weights to run fully automated genius-level scams personally targeting basically everybody?
You could literally develop a hyper-intelligent advisor on how to kill people or perform dangerous hacks using ablated 'open source' AI. You can do this right now, this very moment, and have an extremely adept advisor on how to do really, really bad things.
>Much of the angst around China's models centers on "losing the AI race". But what's the goal of this race? Is it to develop the best model? To sell the most tokens? To destroy humanity first?
Some people would say it is reaching some sort of singularity. Even if you don't buy into a more sci-fi interpretation of this, there are pretty grounded arguments one could make that there is some sort of "goal" in AI development that, if realized, would effectively make it a superweapon. Altman has been pretty vocal about his expectation that this will eventually happen and that it is his goal to be the guy to produce it. Even if it isn't some superintelligence, the ability for a machine to do something like, say, exploit cybersecurity weaknesses, is pretty worrying for entities like governments. It's the pretense used when we saw the US government ban a US model recently.
Again, you don't have to buy that "the singularity" is a real thing, but it's not hard to see that some people think some version of this is real and it is exactly what is being referred to as the goal in an "AI race."
> It's theoretically possible for a bad actor to embed hidden adversarial behavior in a model. But if this happens, it serves the interests of responsible actors to find these exploits as soon as possible, and the best way to do this is to let anyone who wants to inspect them.
This is a bad argument. It isn't trivial to tell if the weights have poisoned:
I'm not arguing in favor of closed-AI; I'm simply saying poisoning may be subtle.
I was gonna say that at least the frontier labs may be motivated to not-poison their own models, but then Anthropic just attempted to poison Fable's LLM training capability so... sigh. I'll try not to derail.
They don't have to convince developers, they just have to convince the general population. I've had several discussions about open source with people who don't care about coding and it's hard to untangle the misinformation they receive from the media. They get talking points from authorities that they don't understand so they will always double down on it. For example, rhetoric relating open source to communism is humiliating to discuss. For proprietary evangelists, the cruelty is the point.
Just to add on to other comments: reasonable people can disagree about the degree of safety concern with near to medium term AI. But to not address the arguments at all is, in my opinion, a serious mark against the value of this article.
Photoshop SAAS web app = closed model like GPT, Opus/Fable etc.
There is nothing "open source" about the Chinese models in question. All they're doing is allowing you to run their binary yourself instead of through their API.
If you want actual open source then you would need to look at like OLMo 3
It's a matter of degree. If open source means "everything you need to reproduce this" then maybe an open source application should include a detailed architecture document and course in the c language. Oppositely, you can certainly use models to reproduce model - the Chinese models themselves were supposedly created by interactions with the American models so you can use them/
The more I see this argument, the sillier it becomes to me. The point of "open source" in software is something like "the ability to deconstruct the thing line by line for purposes of predicting what it will do."
But you can't fully do that for generative AI anyway? What's the point?
Now, if you wanted something more robust, then you probably should have supported Stallman's ideas and insisted on Free Software in the first place.
This is a facile and pointless discussion. The hard work in building source code is paying millions of dollars to software engineers to write it. "Open source" allows others to benefit from this expenditure. The hard work in building an ai model is spending millions of dollars on training. "Open source" models allow others to benefit from this expenditure. Regardless of whether you had access to the complete training data and architecture of a model, you would never retrain from scratch in order to make a modification to the model. You would fine-tune in whatever behaviour you wanted. Just like open source software is when you are allowed to modify it, open source models are when you are allowed to fine tune like this.
Controlling how to train, not over fitting, making sure training data is good, and being able to do distributed training are all reasons why open weight isn't like open source. You can't fork during pre-training with open weight-only models.
When talking about "open source" models no one is talking about the legal framework and textual doctrine of open source. They simply want to know: "can I run this on hardware I own or no?" That's it. It's simply about control. If you don't think there's a difference between models you can download and run on hardware you own, regardless of how big it is, vs. models you can only talk to through an API, you are seeing the forest for the trees.
You are also indirectly helping the closed SOTA model toll-keepers by taking the focus away from their mass copyright theft and onto some pointless textbook definition.
You can't reconstruct the same LLM weights even with the same training data.
You can fine tune an open weights model.
Much of the value in modern LLMs is in the RL envrionments, not the pre-training data. That is a very different piece of software to an LLM, and the LLM weights are useful without it.
The way Anthropic is playing the "AI is scary and dangerous and only we can be trusted with it" game is clearly aimed at regulatory capture, which would only be good for Anthropic. It's worrying how cozy some of the tech leaders have been with the Trump administration, which is also clearly aimed at regulatory capture that serves the interests of the biggest tech companies and no one else.
The US has already been set back a decade or more by anti-science and anti-intellectual purges of agencies and formerly non-partisan positions. A ban on open models would be pretty much game over for the US as a tech leader. But, Anthropic would make a lot of money for a couple of years, so, who can say what's right?
OpenAI exec scaremongering about "A nonliving, invisible, dangerous, and infinitely self- replicating agent escaped from a Chinese lab" mere 4 days before https://news.ycombinator.com/item?id=48997548 is hilarious
Conversely, “we should stop X” should come with some idea of how to feasibly do so.
In any case, I’d summarize my position as “you cannot stop open source AI, and attempts at it will inevitably empower bad actors relative to good ones.”
I would be more inclined to agree if American companies were not in a competition to be the biggest flameout and lose the most money. How is China able to afford to train models that US companies need to burn billions of dollars for?
In general, if American capitalism weren't so broken I would advocate for it as the strongest option, but American corporations are dead set on destroying it. The US may end up as "democratic socialist" if there is no capitalist reform.
From a neutral standpoint, governments usually have more general, comprehensive analytics, as well as greater context and insight into the risks of AI usage, and a possibly better methodology to control access to or use of open-source AI (perhaps by analyzing digital activities and consequences), because they learn from failures. On the other hand, open source creates major opportunities, and it’s hard to accept bans on something that truly benefits users acting for the common good. However, it also benefits unethical or opportunistic users—for whom regulation is indeed necessary.
The most compelling argument against Open Source AI: it is analogous to "open sourcing" nuclear weapon technology. Or an how-to guide for making meth/bombs/bioweapons. There is a high risk that we will one day achieve ASI or something close to it. When that day nears, if even one open-source AI user handles their AI in an irresponsible way, we are all FUBARed. In the same way that one irresponsible person with a nuclear bomb can FUBAR the entire world, hence why "open sourcing nuclear weapon tech" is not even a subject of debate.
You don't have to believe that ASI is 100% likely. People like Hinton believe there's a 50% chance of existential risk from AI, but you don't have to go that far either. If there's even a tiny chance of ASI posing a threat to humanity, that's reason enough to lock it down
Just because encryption bans are badly conceived or implemented doesn't mean they are wrong.
Encryption has military value.
The point is that the good outweighs the bad. Making the government seem like buffoons for attempting to prevent military technology like dual use crypto is not in our best interest.
The same government upheld the right to free speech, so this balancing act is widely observed.
The government is not evil, and yes they make mistakes, but they also not infrequently protect us.
LLMs are trained on public data (as well as illegally obtained data see: Anthropic 1.5B settlement). LLMs are nothing without the huge corpus of human data that powers them. There is an argument that research of this kind should be restricted to governments and regulated universities rather than opaque public companies with competing incentives. Or research should be stewarded by genuine non-profit collectives with democratic leadership. e.g. like internet standards, telecom, etc
I do not think we can trust private companies, no matter the virtue signaling they put forth into the world, to effectively regulate themselves and inform the public and scientific communities about risks. Their ongoing conflict of interest poses serious credibility risks.
38 comments
[ 3.1 ms ] story [ 43.0 ms ] threadSome people would say it is reaching some sort of singularity. Even if you don't buy into a more sci-fi interpretation of this, there are pretty grounded arguments one could make that there is some sort of "goal" in AI development that, if realized, would effectively make it a superweapon. Altman has been pretty vocal about his expectation that this will eventually happen and that it is his goal to be the guy to produce it. Even if it isn't some superintelligence, the ability for a machine to do something like, say, exploit cybersecurity weaknesses, is pretty worrying for entities like governments. It's the pretense used when we saw the US government ban a US model recently.
Again, you don't have to buy that "the singularity" is a real thing, but it's not hard to see that some people think some version of this is real and it is exactly what is being referred to as the goal in an "AI race."
> It's theoretically possible for a bad actor to embed hidden adversarial behavior in a model. But if this happens, it serves the interests of responsible actors to find these exploits as soon as possible, and the best way to do this is to let anyone who wants to inspect them.
This is a bad argument. It isn't trivial to tell if the weights have poisoned:
``` https://www.thedeepview.com/articles/microsoft-how-to-spot-a... https://futurism.com/future-society/easy-poison-open-weight-... https://semgrep.dev/blog/2026/ai-supply-chain-problem/ ```
I'm not arguing in favor of closed-AI; I'm simply saying poisoning may be subtle.
I was gonna say that at least the frontier labs may be motivated to not-poison their own models, but then Anthropic just attempted to poison Fable's LLM training capability so... sigh. I'll try not to derail.
Will there ever be a way to fully audit Chinese models?
Photoshop source code + OSI license = open source
Photoshop binary = open weight
Photoshop SAAS web app = closed model like GPT, Opus/Fable etc.
There is nothing "open source" about the Chinese models in question. All they're doing is allowing you to run their binary yourself instead of through their API.
If you want actual open source then you would need to look at like OLMo 3
https://allenai.org/
But you can't fully do that for generative AI anyway? What's the point?
Now, if you wanted something more robust, then you probably should have supported Stallman's ideas and insisted on Free Software in the first place.
You are also indirectly helping the closed SOTA model toll-keepers by taking the focus away from their mass copyright theft and onto some pointless textbook definition.
You can't reconstruct the same LLM weights even with the same training data.
You can fine tune an open weights model.
Much of the value in modern LLMs is in the RL envrionments, not the pre-training data. That is a very different piece of software to an LLM, and the LLM weights are useful without it.
The OP actually acknowledges this in the footnotes. Perhaps we don't need to re-litigate this on every post?
Training a model requires two or three orders of magnitude more investment than the typical OSS/Creative Common contributor can reasonably afford.
It reminds me a bit of Open Source hardware (CPU, GPU).
The US has already been set back a decade or more by anti-science and anti-intellectual purges of agencies and formerly non-partisan positions. A ban on open models would be pretty much game over for the US as a tech leader. But, Anthropic would make a lot of money for a couple of years, so, who can say what's right?
In any case, I’d summarize my position as “you cannot stop open source AI, and attempts at it will inevitably empower bad actors relative to good ones.”
America is a corporate hellhole.
In general, if American capitalism weren't so broken I would advocate for it as the strongest option, but American corporations are dead set on destroying it. The US may end up as "democratic socialist" if there is no capitalist reform.
oh no, not communism.
You don't have to believe that ASI is 100% likely. People like Hinton believe there's a 50% chance of existential risk from AI, but you don't have to go that far either. If there's even a tiny chance of ASI posing a threat to humanity, that's reason enough to lock it down
Encryption has military value.
The point is that the good outweighs the bad. Making the government seem like buffoons for attempting to prevent military technology like dual use crypto is not in our best interest.
The same government upheld the right to free speech, so this balancing act is widely observed.
The government is not evil, and yes they make mistakes, but they also not infrequently protect us.
And I, for one, like being protected.
I do not think we can trust private companies, no matter the virtue signaling they put forth into the world, to effectively regulate themselves and inform the public and scientific communities about risks. Their ongoing conflict of interest poses serious credibility risks.