I understand why China wants to restrict exports of their models.
But I feel the only reason Chinese models have been able to gain such popularity is BECAUSE they are open source.
I wouldn’t trust models from China enough to use them in important work unless they were open.
Restricting any powerful Chinese model [that ranks in the global top 10] is going to make them very difficult to advocate for, and very hard for global users to adopt.
Feels like this could be the beginning of the end of the democratisation of powerful AI.
And yes, I appreciate the irony that Chinese models have been the ones enabling that democratisation!
Why do they care about global usage though? This seems primarily so that they can control it for their billion plus population, they really don't need global users at all, much to our chagrin.
I think it is more about avoiding being boxed in by the rest of the world. The US would of course ban the use of Chinese hosted models and have already taken steps to ban the use of open Chinese models. There's nothing that could have been done about that. But the rest of the world would likely have adopted American hosted models eventually, the same way they adopted American mobile phones OSes, computer OSes, Office, etc. If the choice was between American hosted AI and Chinese hosted AI they'd choose American if they couldn't develop their own.
But if there's an open model available then they might choose that instead of either of them. Even if the Chinese models aren't used they will bring down profit margin just by existing. They wouldn't have the control the hosted American models give but that wasn't even on the table. But denying America control is a worthwhile strategic goal in and of itself. And they don't even need to expend extra resources to do it, they needed to develop their own models for security reasons anyway.
There are risks in both cases, but they are very different.
If you're using a closed model, the model provider gets all your data, and can if they want to send back actively hostile tool calls and rely on your harness to catch them without any effort on their side to train that behaviour into their models. The only things you can do about that is to place careful restrictions
If you're using an open model against your own system or a third party provider, the origin of that model could have trained in "triggers" that'd cause the model to produce actively hostile tool calls as well, but they'd have no direct control and would need to train in that behaviour "blind" to how you'd actually use the model.
It doesn't mean you can be complacent about trusting the model, but it does change the likelyhood they'd be able to do any damage, not least because you'd be able to run evals against the model without any risk that the model or provider API could just randomly change on you after a while.
So it takes far more trust to run a closed model than to run an open model even on a hosted third party provider, and certainly far more than to run an open model on your own hardware.
You can run Chinese models locally without Internet, while with US models you need to have Internet connection. Guess where it is easier to hide a backdoor.
Let's just hope that both US and Chinese models remain usable enough by the masses for long enough for someone in Europe or Japan or some other actual free country to make a truly open AI :')
Does anyone know about how you could go about training this type of thing into the model? Surely it is quite tricky to get it to do such things without it doing them more than this one incident? It should have happened lots if it really was trying to obtain secret info and build backdoors?
> But I feel the only reason Chinese models have been able to gain such popularity is BECAUSE they are open source.
Why do people keep calling these "open source" models? They're not that different from NVIDIA's binary blobs. Sure, they work, and you can run them and get results, but the source code, training data, and training process are largely hidden and proprietary. Publishing weights is not the same as publishing the source. They're open-weight models, not open-source models.
I'd argue that they're still open-source because inference code is fully public. In operation, a user can crack open the model and examine its internal state, and we also see that many people produce various modifications or fine-tunings of public models for their own goals.
In my mind, this meets the conceptual and technical bars of an 'open source' designation. We can also imagine the opposite, where inference itself has to pass via an inscrutable binary blob that keeps the model's technology stack fully proprietary.
You're correct that we're missing the training data, the training recipe/curriculum, and any secret sauce that went into optimization of same, but I don't think that's an essential feature of an open source designation.
Instead, I think that the model weights are a kind of rendered artifact, and it should be evaluated on that basis. We don't treat permissively licensed images as something lesser, even if the photographer or illustrator supplies only the final image rather than the full camera raw file and/or Photoshop layer stack.
The difference, of course, is that the rendered artifact (weight blob) in a language model is so much more important than a typical photograph.
Nah this is like saying that Photoshop is open source just because you can download and run the binary on your own computer instead of through a SaaS webapp. I think basically everyone would disagree with that.
I mean all of open source software (as we traditionally defined it) relies on an ISA, which is actually executed by microcode that isn’t available to the public?
I don’t see how this is any different. It’s like saying Intel has to publish their transistor physical library and microcode to be “truly” open source. Like I don’t have the internal notes about the most recent microcode changes (which is akin to “training process” in the analogy).
They gained popularity because THEY ARE GOOD, not because they are open weight. There are so many other models that are not popular. Do you know about arcee and their models? every used inkling? how about laguna? these are American open weights, they didn't take off because they are not as good as the rest. Do you know about the command-x series from COhere? probably not, this is a Canadian model, folks into local were really into it 2 years ago when they were keeping. do you know about dbrx? Yeah, databricks an American company also gave us a model. Anyways, my point is the competition is strong and giving away for free doesn't matter if it can't keep up. It's expensive, to send bandwidth, storage, gpu compute to run a model when there are better options.
Its because the triangle of good, cheap, and open favored them. Just being good isn't enough unless you're number 1 but likewise neither is just being open (ish). It was important to release the wieghts and the research because otherwise good wouldn't be good enough to outweigh the risks.
> they didn't take off because they are not as good as the rest.
This may be true now but probably won't be true in the future. My guess is that part of the reason for this is that the major American technology companies are acquiring talent to build out models and related capabilities. China doesn't (or at least didn't) have that same dynamic.
> I don’t understand how you think this is the likely future. It is a future, sure, but assuming America just wins “because capital” seems wishful.
I don't think it's a stretch to suggest that part of the reason American startups haven't released or haven't released as many successful "open weight" models or related companies is because we have a lot of really high-end, top-tier tech firms which are hiring most of the talent and paying them a lot. It's not just Anthropic and Open AI. It's basically every tech company, plus finance companies, airlines, &c. who are researching and figuring out how to make their own companies better.
If you find that to be something wishful or you have trouble understanding, I think instead of combatting my comment you should ask yourself why you think that American tech companies aren't hovering up exceptional talent and that has had an impact on startups or other tech companies launching similar open-weight models.
You also need to understand the economics better and talk through those. Investors in the US don't generally have state backing so they may be more diligent in what they invest in.
> Rome is not what it once was, and China has plenty of universities, bright people and capital of its own.
Sometimes I think people are internalizing the troubles in Europe (I know you mean Rome as in the US being the new Romans or something) and externalizing those issues at American problems because of Donald Trump, but I would caution against this. China has plenty of universities, bright people, and capital of its own. Yes. It also has disadvantages. The US also has advantages and disadvantages.
> I don’t see how anyone, looking at the past 30 years, can be so sure that the US will out-do China. But good on you for being an optimist I guess.
We've proven we can build competitive and capable companies. I'm not being an optimist, instead I'm just calling it like it is. Our ability to continue doing what we already have demonstrated we can do isn't taken away by China (or other countries) also doing those things. There hasn't been any material change in the ability of the US to build world-class companies in recent history.
USA models proven to use copyrighted materials illegally. Meta also proven to ignore privacy. USA tech companies routinely give your date to military and law enforcers without even legally allowed to tell you. Unless you meant not trusting anyone, specifically saying not China is odd. You are almost likely to get Uncle Sam notice and legal enforcement while zero from China, you dropby their police station. In fact most EU countries will also extradite you to USA without your legal protection.
You know people are saying that it’s not open source without regard to intent. The intent is 100 percent open source.
The reason why it’s not done is because it takes a huge amount of resources to not only record the training process but to release it in a readable format.
What you’re asking for is almost like surveillance documentation for a model that’s likely put together in a way that’s not easily peaceable into clean step by step procedures.
> I wouldn’t trust models from China enough to use them in important work unless they were open.
One issue is that in the end, open models are still comparable to "freeware" from the old world. You have a black boxed binary to takes input and produces output. On typical "freeware", you could still reverse-engineer it and have have confidence to map assembly to actions it takes. But with these open models, not so much. So, we still can't put too much trust to these models, even if they are open. There can be some backdoor that is impossible currently to identify based on reading the model weight numbers. We can try to post-train models and reduce the likelihood of some "known" stuff, but we don't know what unknown there is. Interesting times ahead.
>I wouldn’t trust models from China enough to use them in important work unless they were open.
It's possible to embed malicious behavior into models, such as "secretly phone data that meets these requirements back to this address..."
This has been shown, and anyone who does cybersec should be aware that "open model running locally" isn't fool proof.
Edit: People might think I am referring to the cohere incident, I'm not, I'm referring to other labs that have purposely experimented to see if this is possible. It is.
"Second, we encourage open source, open collaboration, and resource sharing. The Chinese government is committed to making artificial intelligence a significant driving force for promoting common prosperity and safeguarding common security. To support the construction of an open source innovation ecosystem for artificial intelligence, convenient measures such as automatic licensing will be applied to technology exports involving open source AI technologies, Chinese companies building and operating their own computing centers overseas, conducting chip fabrication overseas, utilizing overseas resources for joint research and development, and participating in the formulation of international standards, basically achieving "immediate reporting and export." Technologies that have already entered the public domain through open source or published academic papers will not be included in the export management scope, creating a favorable environment for international exchange and cooperation in artificial intelligence technology."
their official stance suggests they remain in support of open-weights
No, what they've been doing so far is changing nothing.
What they are saying is that they won't be changing anything.
That could of course change overnight.
There are multiple aspects to this: China is cottoning on to the meaning of soft power, and besides that being a major supplier of AI related models, services and hardware further cements their place in the world economy.
Of course there are risks as well. Models produced and released into the wild for which you have not seen the training and conditioning data could have payloads that you'd rather not have.
The proposed export controls do add some categories where export is completely prohibited. For example, sandbox detection. While the provisions specifically for AI models are generous enough, it's still possible that they trigger some other prohibition by being too capable. The statement about encouraging open source only applies to cases where export is merely restricted, not prohibited, and even then companies need to report their intention to export in advance, in principle giving the government the opportunity to deny them an export license.
Care to substantiate that? Not that I disagree, just curious.
My (not thorougly informed) take is that they are are taking measures to have the capability of restriction but are also adding provisions so that this capability does not add friction to the current practice of enabling export of open weight models.
So to me it's like they are keeping their options open rather than leaning on one versus the other.
I would find it surprising if the chinese government wanted to allow Anthropic/OpenAI/Google levels of influence on the public opinion and regulatory environment without guardrails. OSS models make AI a replaceable commodity and reduce the potential for market/opinion capture by the frontier labs.
Companies making closed-weight models typically don't export them at all, but sell access to them as a service. (Which doesn't count as an export of the model.) So it's primarily companies releasing open weights (if they pass the thresholds for the restricted category), or that host their models in datacenters outside China that will need to apply for an export license.
Meanwhile the US certainly appears to consider API access to constitute a form of export so misunderstandings are understandable. Or perhaps it would be more accurate to say that China isn't currently restricting that form of export.
You trust other countries to behave like this out of the goodness of their hearts or something?
If China’s selfish behavior = more open LLM tools and greater ability to run them at home,
I’ll gladly take it over the US frontier labs’ druthers, which is a restricted/closed shop we rent from.
Correct, this news just hasn't gotten traction until now on HN it seems, as I've already seen this news on r/LocalLlama and thought it was another piece of newer news.
Yes, this happened weeks ago and was never discussed on HN.
Part of the problem is that there are too many major stories happen back-to-back nowadays, and important stories like this one never get discussed. HN only shows 30 stories on the front page, which limits discussion.
slightly ot, but this seems like a good place to ask, since I don't think I have the strategic-business mind to understand it all myself: What's the play here with China having supported publishing open weight models this far? I know the official line is for the good of mankind and all that, but presumably it's more strategic than that. Soft power? Commoditise your competition?
In my mind, they’re putting huge downward pressure on the major US AI companies’ ability to become platforms.
Think about the history of AWS. Amazon made a very specific point of building, specifically, a “platform”. When they developed a major internal service, they would sell it. Basically anyone who wanted to use it could pay AWS (often an outrageous amount, but it was often worth it), and AWS would provide the service, and AWS was consistently neutral and trustworthy. They didn’t really tell you what you could or couldn’t do, they didn’t even ask what you wanted it for, and they never used your private data for their own purposes. AWS pulled this off quite well despite the fact that competing products and even open source products could and still can, with effort, do most of what AWS offered.
Compare to OpenAI and Anthropic. They want to sell you their front-ends (chatgpt.com, claude.ai, their apps and harnesses), but the big money is selling enterprise contracts with per-token pricing. The latter is the platform.
But their platforms suck in comparison to their open competition. You are paying for inference on a series of models, but:
- You need to negotiate with them if you don’t want them to retain your data.
- They might not even be willing to not retain your data.
- Part of the model output gets encrypted before you see it.
- You are not allowed to give the model arbitrary input.
- They run classifiers on the token stream to try to restrict what you may or may not do.
- You cannot see the model’s internal states.
- You are extremely restricted in your ability to modify the model and you may not be allowed to modify it at all.
- Oh, and maybe the price is above market rates.
The last point is not really a big deal: AWS has demonstrated quite effectively that customers will pay many, many billions of dollars above market rates for a high quality platform.
But I think the rest are a big deal, and the open-weight models allow anyone to use a provider other than Anthropic and OpenAI and avoid all the other problems in the list. This means that, if I’m a startup building a product that uses AI, I get to choose between OpenAI/Anthropic-like terms and a far better alternative. And this makes it very hard for OpenAI or Anthropic to sell me on their platform even at equivalent pricing.
I'm pretty daft with this but my opinion is that they are open source so they can help Russia war efforts (and maybe Iran) unofficially and indirectly.
So far, there have not been any obvious drawbacks to open-weight models from the government perspective, so when a company wanted to release their weights, they weren't prohibited from doing so. Similarly, there have not been any obvious drawbacks to closed-weight models, so when a company wanted to keep their weights to themselves, they weren't prohibited from doing so.
So the real question you should be asking yourself is why you thought that China supported publishing open-weight models, even though they merely didn't prohibit it.
Denying positive ROI for non-Chinese labs, destroying AI industrial base of others, while allowing NVIDIA to profit?
OpenAI had been miniaturizing GPT models as far back as GPT-4o era, but Chinese free-beer models that run on 16GB consumer GPUs can handle the vast majority of LLM use cases, and now they have no options but to attempt development of actual superintelligence that can cure cancer or solve the P=NP question just to differentiate against free-beer alternatives. It's clear what China is trying to achieve.
Well, there is CrowdGPT[0] which is a fully open and distributed attempt to train a small 1B parameter LLM from scratch with open data. They've started just a few days ago but they're already at 13.0B / 100.0B tokens for the pretraining phase.
Of course training something like a 1T LLM will probably take some time :-P but with enough people contributing it might be possible to train something small - and if the project reaches consensus, it might be adopted by some institutions with a bit more available compute power too (e.g. universities).
If i had a second PC (i don't really trust the project right now to put their auto-updating client on my main PC - maybe if it grows and there are more people behind it or there is some better governance than "trust me bro" i'd put it in a container or something) with some GPU power i might have contributed a bit too since i like the idea.
I guess China didnt got the memo with the rebranding of AI to Super Intelligence. They should really watch themselves, they might end up making the big orange baby angry.
somebody wake me up when we have "SLSA, but for model provenance".
I don't (just) want "open weight + verified distribution". I want a dang audit trail.
If/when one of these models is trusted enough to make real decisions with teeth, and if/when one of those decisions screws up badly enough to hurt somebody, I want to be able to play back the tape and pinpoint the source dataset + training method responsible for the failure. At LEAST at a coarse level, on par with something like modern RPO SLAs.
P.S. this comment is applicable to pretrained models in general. I don't care if it came from China, USA, or model weights scrawled in sharpie on a random bathroom stall. I just want to know the provenance and have enough information to run a blameless "git blame" (or equivalent) if/when things go wrong.
P.P.S. IMHO, we aren't there (yet), but OTOH this proposal ain't a bad place to start.
With LLMs it's almost certain what you want will never happen.
Most people just don't understand how many compute operations occur to train a modern model, and just how infinitesimal any particular data point is. Even worse is when a model generalizes a function. You have to now prove the entire generalization.
The risk is that USA and China will come together on a joint "AI non-proliferation" agreement in the same way that nuclear technology became restricted by the countries that already had the capability. This will lead to a de facto techno-colonization of the rest of the world just between themselves.
This was a topic of discussion at Trump and Xi's AI summit just last month. We're now starting to see the export controls coming into place.
I don't know how anyone with a level understanding of the world would think that China is going to hand out nuclear scale cyber weapons willy nilly.
It's not in the interest of any government, or even any AI lab, to democratize dangerous intelligence/ability.
China wants to undercut the west, of course, but it's looking like good/bad model behavior are two sides of the same coin. "Show me how to protect my server from the most esoteric attacks" is simultaneously "Show me how to attack a server with the most esoteric attacks."
Yea, many here keep trying to treat LLMs like software, or at least conventional software. They are not. To me they are at least the modern equivalent to ballistics charts that governments used to keep secret. No one is going to hand them out to their enemy to use against them.
Hell, I expect before long hardware is going to have all kinds of lockdowns occurring.
73 comments
[ 0.21 ms ] story [ 46.8 ms ] threadBut I feel the only reason Chinese models have been able to gain such popularity is BECAUSE they are open source.
I wouldn’t trust models from China enough to use them in important work unless they were open.
Restricting any powerful Chinese model [that ranks in the global top 10] is going to make them very difficult to advocate for, and very hard for global users to adopt.
Feels like this could be the beginning of the end of the democratisation of powerful AI.
And yes, I appreciate the irony that Chinese models have been the ones enabling that democratisation!
A model is never neutral. A model is always also a way of disseminating an ideology, and it doesn't need to be lying for that to work.
Consider the question "what caused the French revolution?"
There's many ways to answer this while being technically correct. What gets prioritized, what gets omitted, that's where the ideology lives.
(Unless someone totally bonkers spends thousands of hours and thousands of pages answering a single question)
But once we automate that, it drastically lowers the amount of viewpoints represented.
But if there's an open model available then they might choose that instead of either of them. Even if the Chinese models aren't used they will bring down profit margin just by existing. They wouldn't have the control the hosted American models give but that wasn't even on the table. But denying America control is a worthwhile strategic goal in and of itself. And they don't even need to expend extra resources to do it, they needed to develop their own models for security reasons anyway.
So you've inspected the weights for backdoors?
If you're using a closed model, the model provider gets all your data, and can if they want to send back actively hostile tool calls and rely on your harness to catch them without any effort on their side to train that behaviour into their models. The only things you can do about that is to place careful restrictions
If you're using an open model against your own system or a third party provider, the origin of that model could have trained in "triggers" that'd cause the model to produce actively hostile tool calls as well, but they'd have no direct control and would need to train in that behaviour "blind" to how you'd actually use the model.
It doesn't mean you can be complacent about trusting the model, but it does change the likelyhood they'd be able to do any damage, not least because you'd be able to run evals against the model without any risk that the model or provider API could just randomly change on you after a while.
So it takes far more trust to run a closed model than to run an open model even on a hosted third party provider, and certainly far more than to run an open model on your own hardware.
Why do people keep calling these "open source" models? They're not that different from NVIDIA's binary blobs. Sure, they work, and you can run them and get results, but the source code, training data, and training process are largely hidden and proprietary. Publishing weights is not the same as publishing the source. They're open-weight models, not open-source models.
In my mind, this meets the conceptual and technical bars of an 'open source' designation. We can also imagine the opposite, where inference itself has to pass via an inscrutable binary blob that keeps the model's technology stack fully proprietary.
You're correct that we're missing the training data, the training recipe/curriculum, and any secret sauce that went into optimization of same, but I don't think that's an essential feature of an open source designation.
Instead, I think that the model weights are a kind of rendered artifact, and it should be evaluated on that basis. We don't treat permissively licensed images as something lesser, even if the photographer or illustrator supplies only the final image rather than the full camera raw file and/or Photoshop layer stack.
The difference, of course, is that the rendered artifact (weight blob) in a language model is so much more important than a typical photograph.
I don’t see how this is any different. It’s like saying Intel has to publish their transistor physical library and microcode to be “truly” open source. Like I don’t have the internal notes about the most recent microcode changes (which is akin to “training process” in the analogy).
This may be true now but probably won't be true in the future. My guess is that part of the reason for this is that the major American technology companies are acquiring talent to build out models and related capabilities. China doesn't (or at least didn't) have that same dynamic.
Rome is not what it once was, and China has plenty of universities, bright people and capital of its own.
I don’t see how anyone, looking at the past 30 years, can be so sure that the US will out-do China. But good on you for being an optimist I guess.
I don't think it's a stretch to suggest that part of the reason American startups haven't released or haven't released as many successful "open weight" models or related companies is because we have a lot of really high-end, top-tier tech firms which are hiring most of the talent and paying them a lot. It's not just Anthropic and Open AI. It's basically every tech company, plus finance companies, airlines, &c. who are researching and figuring out how to make their own companies better.
If you find that to be something wishful or you have trouble understanding, I think instead of combatting my comment you should ask yourself why you think that American tech companies aren't hovering up exceptional talent and that has had an impact on startups or other tech companies launching similar open-weight models.
You also need to understand the economics better and talk through those. Investors in the US don't generally have state backing so they may be more diligent in what they invest in.
> Rome is not what it once was, and China has plenty of universities, bright people and capital of its own.
Sometimes I think people are internalizing the troubles in Europe (I know you mean Rome as in the US being the new Romans or something) and externalizing those issues at American problems because of Donald Trump, but I would caution against this. China has plenty of universities, bright people, and capital of its own. Yes. It also has disadvantages. The US also has advantages and disadvantages.
> I don’t see how anyone, looking at the past 30 years, can be so sure that the US will out-do China. But good on you for being an optimist I guess.
We've proven we can build competitive and capable companies. I'm not being an optimist, instead I'm just calling it like it is. Our ability to continue doing what we already have demonstrated we can do isn't taken away by China (or other countries) also doing those things. There hasn't been any material change in the ability of the US to build world-class companies in recent history.
Open source model would provide you with all the data & tooling necessary for you to rebuilt the model from original or modified data.
The reason why it’s not done is because it takes a huge amount of resources to not only record the training process but to release it in a readable format.
What you’re asking for is almost like surveillance documentation for a model that’s likely put together in a way that’s not easily peaceable into clean step by step procedures.
One issue is that in the end, open models are still comparable to "freeware" from the old world. You have a black boxed binary to takes input and produces output. On typical "freeware", you could still reverse-engineer it and have have confidence to map assembly to actions it takes. But with these open models, not so much. So, we still can't put too much trust to these models, even if they are open. There can be some backdoor that is impossible currently to identify based on reading the model weight numbers. We can try to post-train models and reduce the likelihood of some "known" stuff, but we don't know what unknown there is. Interesting times ahead.
It's possible to embed malicious behavior into models, such as "secretly phone data that meets these requirements back to this address..."
This has been shown, and anyone who does cybersec should be aware that "open model running locally" isn't fool proof.
Edit: People might think I am referring to the cohere incident, I'm not, I'm referring to other labs that have purposely experimented to see if this is possible. It is.
"Second, we encourage open source, open collaboration, and resource sharing. The Chinese government is committed to making artificial intelligence a significant driving force for promoting common prosperity and safeguarding common security. To support the construction of an open source innovation ecosystem for artificial intelligence, convenient measures such as automatic licensing will be applied to technology exports involving open source AI technologies, Chinese companies building and operating their own computing centers overseas, conducting chip fabrication overseas, utilizing overseas resources for joint research and development, and participating in the formulation of international standards, basically achieving "immediate reporting and export." Technologies that have already entered the public domain through open source or published academic papers will not be included in the export management scope, creating a favorable environment for international exchange and cooperation in artificial intelligence technology."
their official stance suggests they remain in support of open-weights
What they are saying is that they won't be changing anything.
That could of course change overnight.
There are multiple aspects to this: China is cottoning on to the meaning of soft power, and besides that being a major supplier of AI related models, services and hardware further cements their place in the world economy.
Of course there are risks as well. Models produced and released into the wild for which you have not seen the training and conditioning data could have payloads that you'd rather not have.
So they're changing some things.
“Right now, your government is doing things you think only other governments do.” -Van Halen, Right Now
My (not thorougly informed) take is that they are are taking measures to have the capability of restriction but are also adding provisions so that this capability does not add friction to the current practice of enabling export of open weight models.
So to me it's like they are keeping their options open rather than leaning on one versus the other.
What they will do remains to be seen. The document at hand is what they're saying.
I.e. less important if we follow your own assessment.
Says who?
You can also download offline PDF at: https://insanai.github.io/aiexportcontrols/books/en/open-sou...
It also shows using game theory that export control leads to overall failure of both the sides.
Of course I never expected anything different from the Chinese Communist Party.
If China’s selfish behavior = more open LLM tools and greater ability to run them at home, I’ll gladly take it over the US frontier labs’ druthers, which is a restricted/closed shop we rent from.
Part of the problem is that there are too many major stories happen back-to-back nowadays, and important stories like this one never get discussed. HN only shows 30 stories on the front page, which limits discussion.
Additionally, the financial and technological equilibrium which makes OpenAI or Anthropic possible is rather precarious so they are poking on it.
Think about the history of AWS. Amazon made a very specific point of building, specifically, a “platform”. When they developed a major internal service, they would sell it. Basically anyone who wanted to use it could pay AWS (often an outrageous amount, but it was often worth it), and AWS would provide the service, and AWS was consistently neutral and trustworthy. They didn’t really tell you what you could or couldn’t do, they didn’t even ask what you wanted it for, and they never used your private data for their own purposes. AWS pulled this off quite well despite the fact that competing products and even open source products could and still can, with effort, do most of what AWS offered.
Compare to OpenAI and Anthropic. They want to sell you their front-ends (chatgpt.com, claude.ai, their apps and harnesses), but the big money is selling enterprise contracts with per-token pricing. The latter is the platform.
But their platforms suck in comparison to their open competition. You are paying for inference on a series of models, but:
- You need to negotiate with them if you don’t want them to retain your data.
- They might not even be willing to not retain your data.
- Part of the model output gets encrypted before you see it.
- You are not allowed to give the model arbitrary input.
- They run classifiers on the token stream to try to restrict what you may or may not do.
- You cannot see the model’s internal states.
- You are extremely restricted in your ability to modify the model and you may not be allowed to modify it at all.
- Oh, and maybe the price is above market rates.
The last point is not really a big deal: AWS has demonstrated quite effectively that customers will pay many, many billions of dollars above market rates for a high quality platform.
But I think the rest are a big deal, and the open-weight models allow anyone to use a provider other than Anthropic and OpenAI and avoid all the other problems in the list. This means that, if I’m a startup building a product that uses AI, I get to choose between OpenAI/Anthropic-like terms and a far better alternative. And this makes it very hard for OpenAI or Anthropic to sell me on their platform even at equivalent pricing.
So the real question you should be asking yourself is why you thought that China supported publishing open-weight models, even though they merely didn't prohibit it.
OpenAI had been miniaturizing GPT models as far back as GPT-4o era, but Chinese free-beer models that run on 16GB consumer GPUs can handle the vast majority of LLM use cases, and now they have no options but to attempt development of actual superintelligence that can cure cancer or solve the P=NP question just to differentiate against free-beer alternatives. It's clear what China is trying to achieve.
Of course training something like a 1T LLM will probably take some time :-P but with enough people contributing it might be possible to train something small - and if the project reaches consensus, it might be adopted by some institutions with a bit more available compute power too (e.g. universities).
If i had a second PC (i don't really trust the project right now to put their auto-updating client on my main PC - maybe if it grows and there are more people behind it or there is some better governance than "trust me bro" i'd put it in a container or something) with some GPU power i might have contributed a bit too since i like the idea.
[0] https://www.crowdgpt.net/#status
somebody wake me up when we have "SLSA, but for model provenance".
I don't (just) want "open weight + verified distribution". I want a dang audit trail.
If/when one of these models is trusted enough to make real decisions with teeth, and if/when one of those decisions screws up badly enough to hurt somebody, I want to be able to play back the tape and pinpoint the source dataset + training method responsible for the failure. At LEAST at a coarse level, on par with something like modern RPO SLAs.
P.S. this comment is applicable to pretrained models in general. I don't care if it came from China, USA, or model weights scrawled in sharpie on a random bathroom stall. I just want to know the provenance and have enough information to run a blameless "git blame" (or equivalent) if/when things go wrong.
P.P.S. IMHO, we aren't there (yet), but OTOH this proposal ain't a bad place to start.
Most people just don't understand how many compute operations occur to train a modern model, and just how infinitesimal any particular data point is. Even worse is when a model generalizes a function. You have to now prove the entire generalization.
Not even remotely possible. Not with our current understanding of how these things work.
This was a topic of discussion at Trump and Xi's AI summit just last month. We're now starting to see the export controls coming into place.
It's not in the interest of any government, or even any AI lab, to democratize dangerous intelligence/ability.
China wants to undercut the west, of course, but it's looking like good/bad model behavior are two sides of the same coin. "Show me how to protect my server from the most esoteric attacks" is simultaneously "Show me how to attack a server with the most esoteric attacks."
Hell, I expect before long hardware is going to have all kinds of lockdowns occurring.
Maybe not for long. You need another talking point.