My immediate midwit take is: doesn't matter if it helps Anthropic/OpenAI if it helps DeepSeek more, relatively. Making open-weights models even cheaper and easier to run expands that "market" and increases competitive pressure on the Big Two, who still have to charge money.
My guess would be that if they "help" western labs becoming better, then any break-throughs they (western labs) make after that, is also a benefit to the Chinese labs - if they can distill the models.
Basically, western labs are in it for the money / commercial monopoly. Chinese labs are in it for the tech? As long as they can keep distilling models, and get access to research other ways, they benefit. And if they can push western labs forward, they'll benefit from that themselves.
I tried to read this but it's just claude after a certain point.
It's unreadable.
I can't understand any serious writer essentially expecting other people to read stuff that they didn't write. Or if claude was actually good at formulating ideas and conducting information, fine. But the truth is that I'm just reading 80% empty prose that are forced in there by claude.
AI fundamentally insecure. Vulnerable to forcing hallucinations via search results. Vulnerable to invocation of commands in data stream. More AI means more vulnerabilities.
I can't even fathom the trend these days of "we don't review the code" from security team perspective.
Well, that's probably far from the main thing. The main thing is that they are not Americans. For one, that means that they value profits, but that there may be other items in their bag of wishes. The sort of thing an emotionally healthy human being may desire from super-intelligent AIs, say, curing diseases or new kinds of transgenic rice. Second, their government may be forcing the AI companies to release key IP in order to increase internal competition and prevent monopolies.
Unpopular, maybe, but what about the normal reasons? The researchers are looking to make a name for themselves, and/or they genuinely care about AI advancement.
It's just their usual national strategy like what they did to solar pane and EV. The solar panel industry is mostly dominated by China and their profit rate is basically ... negative. The EV industry in China is in similar condition, where the average profit rate is only 1.5%. Their upstream suppliers are also hold as hostages that most of them won't get their money back within 6 months.
The weird ideology here is to dominate the market at ANY COST, even it benefits the opponents.
Idk, but it's doing a lot for my view of China. Maybe that's a point? Maybe they just want their own innovation to go as fast as possible and they don't care that other countries also benefit? A rising tide lifts all boats? They are already know for the best manufacturing, they're just adding software dev to the list? Maybe they just want to undermine the US in a non-aggressive way?
Why did we (the west) ever start open sourcing anything? Maybe we just like sharing? Maybe humanity only grows on pre-competitive layers like Linux and clean water. Maybe, the chinese government is closer to their people, and does not let large companies influence them and just doesn't like closed private hyperscalers with a lot of power?
(Some points assume the government has a role in the openness, which I think is likely)
Yes - the Chinese labs serve a market that rely on open-weight models and managed deployments, and the labs gain competitive relevance by releasing those models. The cache optimization feature they came up with required new software to utilize on the inference-end, meaning that open source software would need to be specifically updated to work with these models. It wasn't the type of advancement that they could even theoretically keep secret.
Because it's entirely possible that Western labs already did this optimization but didn't publish it, and then the Chinese figured it out and decided to brag about it.
We don't know either way, so I find the whole thing silly to speculate on.
The chinese don't view AI as metaphysical, they view it as an engineering challenge they consider good for their state and want to dominate the global market like they do with batteries/EVs/photovoltaics. They want to proliferate them as much as possible and traditionally they don't care much for IP. They also want hardware makers to make optimized chips specifically for these models.
I think it’s fairly simple: they’re forced into this situation by being late and worse in terms of capabilities. They’re not far behind, but as long as they’re behind they’ve needed to give people some reason to try and use their models. Cost is one factor. But it probably wasn’t enough. Being open has given them a lot of attention. Free marketing. Good will.
Put another way: if they were not cheaper and open, they would simply not be competitive. They would already be dead.
I don’t think this ends well for the Chinese labs. This is going pretty much like I thought. Western labs is just copying their improvements (I don’t think publishing the techniques matter here.. they’d just hire to gain the knowledge or figure it out themselves), and they have access to more GPUs and have better branding, so in the end where can the Chinese labs compete? Even lower cost? Open weights? I’m not sure open is a sustainable way to compete either. Eventually there will be some fully open source AI models that cuts out that avenue of competition as well.
Western labs are creating the improvements, and Chinese labs are giving them away for free (presumably after examining the improvements). That's the jist of the article.
The post makes an assumption that US labs did not already possess similar optimization. It's also very possible that they did, but are simply not telling anyone in order to maintain obscene margins on cached read, similar to AWS' absurd pricing on bandwidth.
They get to make US labs look dumb and provide an open alternative that anyone can host themselves.
They're building bridges over the moats that companies with far too much US investment are trying to build, and if they do it continually it can help destabilize the US economy.
Yeah, would have been nice if the author put a bit more thought into this article to come up with something rather than to just give up once they've reached the point of their article lol
All the comments here are very US/Western-centric. Maybe they are a different culture, having a completely different economic model. Maybe they are not Capitalists and not thinking in pure Capitalistic terms, such as winning, growth, market domination, IPO, market value or competition. Maybe they are not obsessed with US labs. Maybe they are ideologically different than you. Maybe they have different priorities. Maybe they have a different playbook. Maybe they never thought of it as throwing a lifeline to American labs. Maybe they don't care. Maybe they're just different people.
I am talking about correlating Anthropic/OpenAI cache prices going down with Deepseek publication - neither of those labs have said that's what they used for example.
And the only data they are showing is that cache prices went down for new Claude/OpenAI models but that's proving nothing, IMO.
Deepseek's innovations are published as research. There's nothing 'assumed' about this. The common slur repeated in the West that Chinese labs are parasitic distillers is totally absurd when so many genuinely valuable advances and contributions to the field are published openly by China's labs.
Right now you are right -- even if I ran my local LLM all day, the quality is not nearly as great, and it runs slowly -- so I use the tier one AI subscription services as they are faster and smarter. But that might only be true for a limited amount of time, and a limited number of circumstances.
To borrow your steam engine analogy, if local LLMs get as good as a Toyota Prius, even if OpenAI / Anthropic offer Ferraris, most people will be happy with their Prius as their daily driver.
Similarly, if the big labs start raising prices or cutting usage, you won't be able to use it as much as you want -- whereas a local LLM will run all day every day without costing you any extra money.
So right now you are right, but who knows how long that will last.
> For nearly all tasks, I want the fastest and smartest AI model.
This is not nearly true for everyone else in the world.
For example, think about the world in ~2021 pre-LLM. Would anyone say the sentence "I only want the fastest and smartest humans working on my project"?
No of course not. Most people don't want to pay $10 million dollar salary to the best programmers in the world. They prefer to pay $200k salary to a median programmer and that's good enough for their ecommerce website.
I think a scooter or a bus is a more apt comparison than a steam engine. The scooter and bus can both get the job done with some acceptable trade-offs, depending on your circumstances and what you’re willing to accept.
However, there are many use cases where they aren’t the right tool for the job.
This will change when out of control billing gets noticed. Then us devs will, I presume, get token rations.
I can see the new Thursday afternoon "oh chit" moment being that I didn't complete my weekly task because i torched all of those tokens M-W doing task/ticket grooming using the hot hot model :D
I would argue performance optimizations help with local/open models but hurt openai and anthropic - because open and or cheap/alternative models are threat to those companies. There is a fundamental contradiction/conflict between the prevelance of open models and the financial success of openai and anthropic. That is why they are doing everything they can to kill any open/cheap/efficient/chinese models (take a look at this thread - it was top of HN 40 mins ago, with very high engagement... now it is buried in page 5... totally normal and legit).
The first sentence is: "If you read the news headlines these days, you would be forgiven for thinking that the Western labs are getting spawn-camped by Chinese labs en masse."
Since I have no idea what "spawn-camped" means I gave up reading the rest.
I'm also grateful to the Chinese labs for providing workarounds for the walled gardens that the US based AI companies are attempting to create.
Does anyone know if there are any distillation datasets available? I'd love to see these distributed on BitTorrent. I think it's critical that AI be democratized and not isolated in the hands of a few private companies.
> I think it's critical that AI be democratized and not isolated in the hands of a few private companies.
While I'd like to agree with this, the fact is that pushing the frontier out has always taken (and folks expect to continue to take) hundreds of millions/billions of dollars. Open source and distilled models can follow on for much cheaper, but it's hard to imagine the frontier ever being "democratized" given the huge sums of money required. It was this realization that forced OpenAI to take tons of private investment in the first place.
That's ok. They took a few trillion worth of content and burned a couple of hundred billion of their own money. I really don't see the problem, if they didn't think about this long and hard before they went down that part it should not be on the rest of us to bail them out. The 'frontier' is less important than the democratization process.
The amount of progress that came out of academia and other public sources should not be underestimated either and without all that OpenAI and Anthropic wouldn't even exist.
This would be true if Chinese models would be strictly distillations of frontier models, but in fact they also train on Chinese data Western models don't have access to.
As far as I can tell, neither of the frontier US labs have referred to distillation as "stealing", but someone please provide a link if I'm wrong.
They do claim that it violates their ToS, which we can assume is simply correct, since they get to put whatever they want in their ToS.
Given all that, I don't know what the fuss is. Are they supposed to not use the advances that were openly published by Chinese labs? The entire industry is built on a discovery made at Google, which was published openly. Should Chinese labs therefore not use transformers? Should US labs not try to prevent distillation of their models?
I've been running Qwen 3.8 27b (an opus 4.6 tier model), locally on a 5090 for just over two weeks @ 170 tokens/s. That's a frontier model from 9 months ago running on consumer hardware. Who knows where distillation and pruning gets us in another year.
Weird, I kinda gave up on 3.8 as anything other than a planner. I had it try to write some basic unit tests for an admittedly complex bit of code and it ran out of context thinking about the problem and exploring random parts of the code base repeatedly before it even wrote a single line. Toning down the thinking helped some, but then it wasn't much better than qwen coder.
The only swift variant I'm aware of is from ukisai, which uses the swift open license, and is NOT open for commercial use for businesses over $1mil. I'm respecting their choice by not using it for work, which means I'm also not using it for my personal projects in case I forget to switch models when I switch projects.
My day job doesn't have a dedicated enterprise contract with any of the ai vendors so I might trial it to see if it is worth promoting at the company. Part of me is still holding out hope that qwen 4 dials back the overthinking on its own.
> The only swift variant I'm aware of is from ukisai, which uses the swift open license,
Ah, I overlooked that, since I have a claude sub at work and all my explorations are purely personal. There's another fast version of 3.8: ThinkingCap[1] by bottlecapai but they have the same $1M restriction from what I can see since it's distributed under a PolyForm Small Business 1.0.0 + BottleCap personal-use grant.
> If you read the news headlines these days, you would be forgiven for thinking that the Western labs are getting spawn-camped by Chinese labs en masse.
Watch any interview with the Chinese AI leaders and compare it to the unending doomer word salads from America's mightiest paper billionaires. We're losing because we have a loser late stage capitalist scarcity mindset. They're winning because they're sharing notes and one-upping each other just like we used to until 2015 or so. They have a healthy ecosystem of competing small AI startups. We have two bloated unprofitable pigs both striving to be too big to fail. My money's on China for the immediate future.
Why is it a problem that the Chinese labs are just distilling down Anthropic’s models? Aren’t Anthropic’s models not just distilling down other people’s work?
Feels like Anthropic crying do as I say not as I do.
What Anthropic is doing requires way more resources than what the Chinese labs are doing. So their complaint is that they do 95% of the work and the Chinese labs do the last 5% and call it their own.
An argument can be made that Anthropic is also only doing the last 5% of the work (because the content they are training on was the other 95%) but that's a bit more philosophical.
It is kind of ironic that they scraped the web for publicly available data and used it freely to train their models and now their freely available models are being used to train other models.
I'm overall pro-Anthropic and pro-banning open-weights AI, but I agree with the parent commenter; distilling Claude models is not that different from pretraining on web data. It's all basically the same sort of thing.
I think a good litmus test here would be if Anthropic were to not care about distilling their models when the distillers keep the resulting models closed-source and sell tokens via an API. If they cared only about security concerns and not about people profiting off of their work, then they should be publicly fine with this and only protest against it going into open-weights models.
But that’s not more philosophical. It’s a perfect parallel! Enormous amounts of work, vacuumed up and resold. What’s the difference? If it’s ok to vacuum up all the knowledge in the world, then that includes knowledge of how to use all that to power an LLM.
> What Anthropic is doing requires way more resources than what the Chinese labs are doing.
Oh that’s very sad.
Meanwhile Anthropic made a product from the work effort of millions of people without compensating them, sell that product on tap and unless I am mistaken do not even have their competitors’ cover of having released any sort of meaningful open weights model.
They have taken from culture (including very specifically their most direct customers’ specific culture — our culture), turned it into a machine to make themselves rich, appear likely to predicate their valuation on permanently removing people from the workforce, then want to dump themselves onto pensions funds and ordinary savers to carry the bag.
It is, I agree, philosophical, because karma is a philosophy as well as a bitch.
I'd split it at 99.98% original, 0.015% Anthropic, 0.005% Chinese, and that's being exceedingly generous to the AI companies, there should be several more 9s and 0s in there.
Distillation doesn't "grab 95% of lab's work", that's ridiculous. At best it's icing on top of the cake that's already there. It's not even necessarily done on a better model (e.g. GLM 4.7 distilled Gemini 2.5, a weaker model), I'm pretty sure A\ and OAI could do (or even do) the same with greater efficiency since they have access to logits, weights, and internal state of open models.
>An argument can be made that Anthropic is also only doing the last 5% of the work (because the content they are training on was the other 95%) but that's a bit more philosophical.
How is this philosophical? They should release the unsupervised pretrains, at the very least.
Yeah, you need a lot of resources in order to waste a lot of resources. Look at codex and Claude code - these trillion dollar companies 'with top talent' cannot build what open code and pi/oh-my-pi have built in the open for free? Both codex and Claude code, are slow, buggy pieces of shit (and I say that as someone who still heavily uses both for work, moved to open code and pi for personal stuff). The reality is these companies mostly focus on marketing and market capture through, non-competitive means - their services and software are unreliable, buggy trash.
> An argument can be made that Anthropic is also only doing the last 5% of the work (because the content they are training on was the other 95%) but that's a bit more philosophical.
Actually, it's an interesting argument to make. How many labour-hours went into creating the training data Anthropic has collected? Probably multiple billions of hours. How many labour-hours did it take them to setup the datacenters, scrapers, and training algorithms? A few thousands hours?
> What Anthropic is doing requires way more resources than what the Chinese labs are doing. So their complaint is that they do 95% of the work and the Chinese labs do the last 5% and call it their own.
Anthropic: The Chinese have broken the implicit ethical rule of honorable thievery. We did most of the heavy lifting in this global heist, and here they are taking our righteous loot. This is not right.
By that analogy, Anthropic is doing more like 0.000001% of the work of learning all we learned, and writing it down so that Anthropic at al can condense it into their models.
Because no one outside the AI scientists understand what distilling means. They probably all think about Mash and a vodka still, and a completely unrelated association.
Because cost of original training >> cost of distilling. It's the same thing that happens with Chinese knockoffs of physical products - it takes a lot of money and R&D time to design a new product, but it's basically free to buy the product, reverse engineer it, and resell it. All the data they originally trained on was available for free on the internet. If the original work was so valuable, it shouldn't be up on the internet for free in the first place imo.
It's different because you have to pay Anthropic and follow their TOS to get access to their model. If it was actually on the internet for free, anyone could do whatever they wanted with it.
Right, but the humans willingly released all those creations for free. I think that my issues with the "AI companies stole human creations" stance is that the information was freely available to everyone, and they put a lot of money and effort into transforming it into something useful.
The conversation here is mostly moral and ethical but the problem here seems to be financial.
Anthropic and OpenAi are spending a $$$$ to "distill" human output into an AI model, then others are spending $$ to distill their AI model into a near-equivalent model.
This is the same reason IP rights exist. On the surface, something like a patent feels ludicrious and even feels morally wrong. Some guy wrote down the recipe for arranging atoms or bits in a particular way, and now I can't!? However, it's designed to solve the same problem, figuring out and describing the process is much more costly than replicating it.
Model weights wouldn't be covered in a patent. You could patent a method of creating weights in a model, but you couldn't patent the weights themselves.
I wish people here could at least bother to inform themselves about the IP rights they are so quick to insist are abhorrent, when they seem to not even have a first clue as to what they actually cover.
AI training, if viewed through the capitalist mindset, is plain theft. Anthropic can't morally defend copying someone else's IP, but denouncing others copying Anthropic's stolen IP.
That doesn't mean they won't try, and that also doesn't mean they won't succeed.
“So the Chinese labs have thrown a lifeline to the Western loss-making labs, and I just have no clue as to why.”
I don’t think it is intentional but this is actually quite bad for the western labs.
The entire booster narrative has been “look at how their revenue is growing! $10bn to $100bn ARR in under a year! This’ll be a multi-trillion IPO!” and the extrapolated future growth from $100bn to $500bn and $500bn to $1tn justified future investment… but that revenue was just because inference was expensive.
The revenue growth story is all that matters pre-IPO. If revenue falls from $100bn to $50bn that’s very very bad optics for OpenAI and Anthropic even if they are now profitable, it completely destroys the growth narrative.
I don't see why revenue has to fall even if marginal costs drop off a cliff. As long as they have the best models (perceived or otherwise) and can make security and IP guarantees that satisfy enterprise, and no firm with similar guarantees undercuts them on price (why would they want a race to the bottom?) they can have high revenue and high margin.
It is funny that so many comments vascilate between "It is so expensive these companies can't make money and will go bankrupt in seconds" and "Inference is so cheap that these companies can't make money and will go bankrupt in seconds".
I never take them seriously, I just assume they are coming from countries that don't understand how capitalism works or are operating out of bad faith. The underlying reality of the market is always changing and needs are always changing. Some AI companies will fail, that is a given. Remember alta-vista? Yahoo? Did search go away? How about Microsoft phones? Nokia? Motorola?
OpenAI and Anthropic are not in the inference business. That is a commodity. They need to sell products and solutions.
They’re not contradictory positions. Inference is too expensive now to make money because the industry is immature and hasn’t yet optimized for financial success while customers don’t care much about price because they’re more concerned about not missing out.
Inference will be too cheap long term to make money because it is being commoditized and customers will start to care about results and not just be wowed by impressive technology.
And of course this technology will continue to exist but that is irrelevant to the business. OpenAI investors don’t care if LLMs exist in 10 years, they care if their investment in OpenAI has made money.
How much conclusive evidence people need to stop parroting that inference is expensive? Literally this article is another example showing it's cheap and just got massively cheaper.
I was about to say, one take I've heard is that the party ideology considers profit a kind of "rent" in a derogatory way, and consequently seeks to undermine the ability of western companies to collect large profit margins
> So the Chinese labs have thrown a lifeline to the Western loss-making labs, and I just have no clue as to why.
"Thus the expert in battle moves the enemy, and is not moved by him."
They figured out a clever method for avoiding excessive training costs via distillation. That forces the hand of frontier labs to move faster, produce better models, etc. (to avoid embarrassment and 'falling behind'—all the while shouldering most of the cost), which they can just keep distilling—or applying other techniques against—much to the dismay of said frontier labs.
Yeah, and Anthropic probably got inspired to this new fast read-in thing for making agentic stuff make more sense from the latest DeepSeek model. Maybe it was in the pipeline, but it clearly has the same effect and DeepSeek had published it by the point Anthropic dropped their prices for reading tokens in, so they may well have copied it.
The best explanation is that it's a goal of the CCP to generally commodotize LLMs, because LLMs will ultimately be a compliment to manufacturing (which China dominates), and you always want to "commodotize your compliments".
I think this explains why they are open sourcing broadly. It's not to be nice. It's a strategic play by the Chinese government to help ensure there are many players in this race and not too much power accumulates to American labs (even if American labs benefit in the process)
Totally agreed. People are so ready to praise China for their free models, but they aren't doing it because they believe in free open-source software. If China ever gets ahead, they're going closed source and weights immediately.
Please quote people you appear to be patronizing. China can't do anything about previous released self-hosted Chinese models. If you can show that local Chinese models funnel vast amounts us data home I'm sure you can move a lot of people to your side.
Comments like this also always fail to address why there aren't Western AI companies doing the same thing. Is it because they might get sued into oblivion by Big AI in the US?
It might be better for all of us if you solve that first instead of repeating something the government has been repeating for the last decade or more. It does this, mind you, while sabotaging itself in countless high-tech fields and leaving it all to China for the taking.
> If China ever gets ahead, they're going closed source and weights immediately.
Anthropic itself admits that Chinese models are merely months behind. Your argument does not make sense, because Chinese labs are contributing massive optimizations like the one this post is about.
You do realize that the East has a different view of intellectual property than the West? And it's not just some arbitrary or capricious choice; it's rooted deep.
It's also possible that China has decided that this is ultimately going to be a race to the bottom, anyway, and values the soft power more highly than potential monetary profits.
Or perhaps they've looked at history and concluded that this historically hasn't been where the value is, anyway. It wouldn't be unprecedented - FAANG companies have a long tradition of publishing their algorithms and releasing open weight models. Because they saw the real value as being the training data and in proprietary special-purpose models. For example Google published the transformer architecture and released BERT as an open weight model, but doesn't really even talk in public about the (presumanbly) specialized internal models behind revenue-generating products.
> For example Google published the transformer architecture and released BERT as an open weight model, but doesn't really even talk in public about the (presumanbly) specialized internal models behind revenue-generating products.
That's giving a lot of credit to Google's organizational ability to productize Google's research...
It's a bad explanation because it assumes decision making about LLM releases is centralized in the CCP even though every AI lab has a different strategy. Some publish LLM research with small models but seem to be staying out of the race to the frontier (Weibo), some train large models but publish few research details and no weights (Bytedance, iFlyTek), some decide on a case-by-case basis what they publish or not (Alibaba, Baidu), ...
There is no rule that every Chinese LLM company must open-source their models, and many don't.
I'm aware of Article 17 of the Company Law of the People's Republic of China https://english.court.gov.cn/2016-04/14/c_761425_2.htm "The grass-root organizations of the Communist Party of China in companies shall carry out their activities in accordance with the Constitution of the Communist Party of China." but not any rule requiring them to be in leadership positions specifically. Maybe you could point me to it?
Even if there's a party member at the C-level of some company making day-to-day business decisions, that doesn't necessarily mean they're carrying out detailed instructions from higher-ups in the party as opposed to following their own best judgment. The CCP generally doesn't micro-manage everything, but operates more in a "mission tactics"-like way, where the top level gives out a broad goal that gets condensed down into a slogan like "new productive forces," and interpreting and elaborating on the details is left to lower levels who seek to align their initiatives with the top-level goals (e.g. by describing AI as a "new productive force") and initiatives that appear successful will get blessed as official policy.
That different companies follow different strategies when it comes to model releases is clear evidence that there is no unified party line on this question yet.
That the CCP doesn't have input in this, which is the most important tech in history, which could be more dangerous than even nukes, is naive.
Maybe because there is enough given away that there's a lot of competition they may let other companies do whatever they want but to think the CCP isn't managing this is insane to me
The CCP could have input, if they chose to. Currently they choose not to. There is no party consensus that AI is the most important tech (Xi Jinping cares more about the "real economy," i.e. manufacturing) and there definitely is no consensus on whether models should be open-weight or not. If consensus ever emerges, companies that bet on the wrong horse will get cracked down on, just like AI companion products got cracked down on recently.
Do you think there was a model a company wanted to keep closed, but the government ordered them to release weights? Conversely, of the closed-weight models, do you think there were any models the company wanted to make open, but the government ordered them to only offer API access?
But doesn't that justification work for any government, not just Chinese? In fact, it works for any easy-to-copy products, which act as positive externalities. It's just some Americans have this mindset that AI is a scarce good because everything must be.
Why always CCP this, CCP that. The parsimonious answer is this shit started when Deepseek founder decided to open source, because at the time the Whale was just hedgefund side project that serendipitously set the tone before they were even on CCP radar. Then other PRC domestic players have to copy / involute to free to be competitive, but due to sanctions / they are compute constrained and can "afford" to, because sections ironically means PRC AI can never dig themselves in the bottomless debt pit of western labs.
TLDR of timeline/chain of events: some billionaire hedgefund manager with AI as hobby, said fuck economics, let's opensource AI... and all the other players have to now compete with that market force. Of course it does not mean equilibrium will last forever, we already seeing defections from model, but imo thats basically what happened... singular hedgefund bro said fuck profits... other PRC players also had to fuck profits, which ends up tripping western labs so deep in debt who can not say fuck profits.
One explanation could be that they know Western consumers will never use the llms directly, so they can open weight the models and collect licensing fees from hosting providers in US who run it. Moonshot seems to have such terms in their open weights releases.
But it does not explain publishing techniques like the ones referenced here in this article.
It's funny how the western world seems to believe every single decision in China is made by the government. You guys really give the CCP too much credit. China also has plenty of closed models just FYI. You never heard of them because they are only available to the domestic market.
I'm beyond thankful that Chinese AI models are so good. I desperately want us to cure the myriad of maladies that humans suffer needlessly with on a daily basis. We're going to need more powerful models than we have now if we're gonna do that and the Chinese are providing the competition needed to push this thing as fast as we can.
I realize "going as fast as we can" is not the most popular position atm. But I'm far more interested in what good we can do than 10% apocalypse scenarios. I volunteer with a charity for childhood brain cancer and I do not want to see another 4 year old die. I'm willing to risk anything to stop this.
I don’t actually know. But it’s logical to me that a system designed to find answers in context is well suited for these healthcare problems. We know the answer is somewhere in all the data we have. It’s just too difficult and complex for us to get through quickly.
This really is a place where the guns-don't-kill-people argument applies, even moreso than guns themselves. The U.S. government massacred school children in Iran. Why does it matter how they targetted them?
If the knife was deliberately marketed and sold specifically to someone who the company knew would use it to stab someone, then, yes, the knife company shares part of the guilt.
How co-designed are these optimizations with the model itself? I'd imagine you can't just stick post-training adapters onto existing architectures for these things, or am I wrong?
I really want to explore the inference space, but it seems like many of the inference optimizations are coming from model-hardware codesign. I don't seem to recall many generic "inference engine" optimizations since prefill/decode disagg a year ago.
This matters for me since I want to break in but the bar seems to be understanding the actual theory of the training process now too given the codesign happening, and I'm not the richest guy on the block lol
The doublethink required to simultaneously believe "our safeguards prevent our models from doing unsanctioned cybersecurity tasks" and "distillation is why Chinese models are getting better at cybersecurity tasks" is genuinely quite funny.
> So the Chinese labs have thrown a lifeline to the Western loss-making labs, and I just have no clue as to why.
Because contrarily to the author's assumption, all labs, Western or not, have sufficient skills to discover the optimizations anyway, and publishing or not is not actually that important?
I’m incredibly skeptical that OpenAI is spinning up custom ASICs for improved inference performance, but they never thought of optimizing KV cache until a tiny Chinese lab did it? Give me a break.
193 comments
[ 3.8 ms ] story [ 95.6 ms ] threadAny ideas?
Basically, western labs are in it for the money / commercial monopoly. Chinese labs are in it for the tech? As long as they can keep distilling models, and get access to research other ways, they benefit. And if they can push western labs forward, they'll benefit from that themselves.
It's unreadable.
I can't understand any serious writer essentially expecting other people to read stuff that they didn't write. Or if claude was actually good at formulating ideas and conducting information, fine. But the truth is that I'm just reading 80% empty prose that are forced in there by claude.
I can't even fathom the trend these days of "we don't review the code" from security team perspective.
Just my guess though.
Well, that's probably far from the main thing. The main thing is that they are not Americans. For one, that means that they value profits, but that there may be other items in their bag of wishes. The sort of thing an emotionally healthy human being may desire from super-intelligent AIs, say, curing diseases or new kinds of transgenic rice. Second, their government may be forcing the AI companies to release key IP in order to increase internal competition and prevent monopolies.
Unpopular, maybe, but what about the normal reasons? The researchers are looking to make a name for themselves, and/or they genuinely care about AI advancement.
The weird ideology here is to dominate the market at ANY COST, even it benefits the opponents.
Why did we (the west) ever start open sourcing anything? Maybe we just like sharing? Maybe humanity only grows on pre-competitive layers like Linux and clean water. Maybe, the chinese government is closer to their people, and does not let large companies influence them and just doesn't like closed private hyperscalers with a lot of power?
(Some points assume the government has a role in the openness, which I think is likely)
We don't know either way, so I find the whole thing silly to speculate on.
Put another way: if they were not cheaper and open, they would simply not be competitive. They would already be dead.
I don’t think this ends well for the Chinese labs. This is going pretty much like I thought. Western labs is just copying their improvements (I don’t think publishing the techniques matter here.. they’d just hire to gain the knowledge or figure it out themselves), and they have access to more GPUs and have better branding, so in the end where can the Chinese labs compete? Even lower cost? Open weights? I’m not sure open is a sustainable way to compete either. Eventually there will be some fully open source AI models that cuts out that avenue of competition as well.
They're building bridges over the moats that companies with far too much US investment are trying to build, and if they do it continually it can help destabilize the US economy.
Edit: Apt domain.
And the only data they are showing is that cache prices went down for new Claude/OpenAI models but that's proving nothing, IMO.
If local LLMs get "good" enough, people will soon paying for subscriptions to ChatGPT and Claude, which hurts their revenue.
Think you missed a word there.
It is vanishingly rare I ask an older model to do any task. Newer bigger and smarter models will just do the task better.
Therefore, I believe we are nowhere near 'good enough'.
I never drive my steam engine to work these days. It isn't good enough.
To borrow your steam engine analogy, if local LLMs get as good as a Toyota Prius, even if OpenAI / Anthropic offer Ferraris, most people will be happy with their Prius as their daily driver.
Similarly, if the big labs start raising prices or cutting usage, you won't be able to use it as much as you want -- whereas a local LLM will run all day every day without costing you any extra money.
So right now you are right, but who knows how long that will last.
This is not nearly true for everyone else in the world.
For example, think about the world in ~2021 pre-LLM. Would anyone say the sentence "I only want the fastest and smartest humans working on my project"?
No of course not. Most people don't want to pay $10 million dollar salary to the best programmers in the world. They prefer to pay $200k salary to a median programmer and that's good enough for their ecommerce website.
But so many teams said they wanted to Raise the Bar to infinity and hire a World Class Team.
However, there are many use cases where they aren’t the right tool for the job.
I can see the new Thursday afternoon "oh chit" moment being that I didn't complete my weekly task because i torched all of those tokens M-W doing task/ticket grooming using the hot hot model :D
Since I have no idea what "spawn-camped" means I gave up reading the rest.
Do you have many mini tantrums like this per day? Probably makes you very difficult to work with Mr I was a CTO.
Does anyone know if there are any distillation datasets available? I'd love to see these distributed on BitTorrent. I think it's critical that AI be democratized and not isolated in the hands of a few private companies.
While I'd like to agree with this, the fact is that pushing the frontier out has always taken (and folks expect to continue to take) hundreds of millions/billions of dollars. Open source and distilled models can follow on for much cheaper, but it's hard to imagine the frontier ever being "democratized" given the huge sums of money required. It was this realization that forced OpenAI to take tons of private investment in the first place.
The amount of progress that came out of academia and other public sources should not be underestimated either and without all that OpenAI and Anthropic wouldn't even exist.
They do claim that it violates their ToS, which we can assume is simply correct, since they get to put whatever they want in their ToS.
Given all that, I don't know what the fuss is. Are they supposed to not use the advances that were openly published by Chinese labs? The entire industry is built on a discovery made at Google, which was published openly. Should Chinese labs therefore not use transformers? Should US labs not try to prevent distillation of their models?
My day job doesn't have a dedicated enterprise contract with any of the ai vendors so I might trial it to see if it is worth promoting at the company. Part of me is still holding out hope that qwen 4 dials back the overthinking on its own.
Ah, I overlooked that, since I have a claude sub at work and all my explorations are purely personal. There's another fast version of 3.8: ThinkingCap[1] by bottlecapai but they have the same $1M restriction from what I can see since it's distributed under a PolyForm Small Business 1.0.0 + BottleCap personal-use grant.
[1]: https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.8-27B
Ah brings back Halo 2 memories
Feels like Anthropic crying do as I say not as I do.
it's not complex. there's hundreds of billions of investor dollars counting on vendor lock in and walled gardens
An argument can be made that Anthropic is also only doing the last 5% of the work (because the content they are training on was the other 95%) but that's a bit more philosophical.
I think a good litmus test here would be if Anthropic were to not care about distilling their models when the distillers keep the resulting models closed-source and sell tokens via an API. If they cared only about security concerns and not about people profiting off of their work, then they should be publicly fine with this and only protest against it going into open-weights models.
I see absolutely no distinction between the two, aside from minor technical approaches to gathering the content.
And writing a book requires many more resources than what anthropic does
It sounds exactly the same, not more philosophical to me, except one is more inconvenient.
Oh that’s very sad.
Meanwhile Anthropic made a product from the work effort of millions of people without compensating them, sell that product on tap and unless I am mistaken do not even have their competitors’ cover of having released any sort of meaningful open weights model.
They have taken from culture (including very specifically their most direct customers’ specific culture — our culture), turned it into a machine to make themselves rich, appear likely to predicate their valuation on permanently removing people from the workforce, then want to dump themselves onto pensions funds and ordinary savers to carry the bag.
It is, I agree, philosophical, because karma is a philosophy as well as a bitch.
>An argument can be made that Anthropic is also only doing the last 5% of the work (because the content they are training on was the other 95%) but that's a bit more philosophical.
How is this philosophical? They should release the unsupervised pretrains, at the very least.
Actually, it's an interesting argument to make. How many labour-hours went into creating the training data Anthropic has collected? Probably multiple billions of hours. How many labour-hours did it take them to setup the datacenters, scrapers, and training algorithms? A few thousands hours?
Anthropic: The Chinese have broken the implicit ethical rule of honorable thievery. We did most of the heavy lifting in this global heist, and here they are taking our righteous loot. This is not right.
But yeah, ultimately it is a matter of framing where that line gets drawn.
The word itself is the pivot, not anything else.
You could say Anthropic distilled human knowledge and art.
How can one answer this statement in good faith? AI companies literally violated IP by massively pirating works instead of legally licensing them.
Can you elaborate on that? I mean my direct answer would be no, of course not. But what is your reasoning?
Anthropic and OpenAi are spending a $$$$ to "distill" human output into an AI model, then others are spending $$ to distill their AI model into a near-equivalent model.
This is the same reason IP rights exist. On the surface, something like a patent feels ludicrious and even feels morally wrong. Some guy wrote down the recipe for arranging atoms or bits in a particular way, and now I can't!? However, it's designed to solve the same problem, figuring out and describing the process is much more costly than replicating it.
I wish people here could at least bother to inform themselves about the IP rights they are so quick to insist are abhorrent, when they seem to not even have a first clue as to what they actually cover.
That doesn't mean they won't try, and that also doesn't mean they won't succeed.
I don’t think it is intentional but this is actually quite bad for the western labs.
The entire booster narrative has been “look at how their revenue is growing! $10bn to $100bn ARR in under a year! This’ll be a multi-trillion IPO!” and the extrapolated future growth from $100bn to $500bn and $500bn to $1tn justified future investment… but that revenue was just because inference was expensive.
The revenue growth story is all that matters pre-IPO. If revenue falls from $100bn to $50bn that’s very very bad optics for OpenAI and Anthropic even if they are now profitable, it completely destroys the growth narrative.
I never take them seriously, I just assume they are coming from countries that don't understand how capitalism works or are operating out of bad faith. The underlying reality of the market is always changing and needs are always changing. Some AI companies will fail, that is a given. Remember alta-vista? Yahoo? Did search go away? How about Microsoft phones? Nokia? Motorola?
OpenAI and Anthropic are not in the inference business. That is a commodity. They need to sell products and solutions.
Inference will be too cheap long term to make money because it is being commoditized and customers will start to care about results and not just be wowed by impressive technology.
And of course this technology will continue to exist but that is irrelevant to the business. OpenAI investors don’t care if LLMs exist in 10 years, they care if their investment in OpenAI has made money.
Western labs: Not like that!
"Thus the expert in battle moves the enemy, and is not moved by him."
They figured out a clever method for avoiding excessive training costs via distillation. That forces the hand of frontier labs to move faster, produce better models, etc. (to avoid embarrassment and 'falling behind'—all the while shouldering most of the cost), which they can just keep distilling—or applying other techniques against—much to the dismay of said frontier labs.
Checkmate.
I think this explains why they are open sourcing broadly. It's not to be nice. It's a strategic play by the Chinese government to help ensure there are many players in this race and not too much power accumulates to American labs (even if American labs benefit in the process)
Please quote people you appear to be patronizing. China can't do anything about previous released self-hosted Chinese models. If you can show that local Chinese models funnel vast amounts us data home I'm sure you can move a lot of people to your side.
Comments like this also always fail to address why there aren't Western AI companies doing the same thing. Is it because they might get sued into oblivion by Big AI in the US?
It might be better for all of us if you solve that first instead of repeating something the government has been repeating for the last decade or more. It does this, mind you, while sabotaging itself in countless high-tech fields and leaving it all to China for the taking.
> If China ever gets ahead, they're going closed source and weights immediately.
Anthropic itself admits that Chinese models are merely months behind. Your argument does not make sense, because Chinese labs are contributing massive optimizations like the one this post is about.
You do realize that the East has a different view of intellectual property than the West? And it's not just some arbitrary or capricious choice; it's rooted deep.
Commoditizing one’s compliments is a different strategy.
Or perhaps they've looked at history and concluded that this historically hasn't been where the value is, anyway. It wouldn't be unprecedented - FAANG companies have a long tradition of publishing their algorithms and releasing open weight models. Because they saw the real value as being the training data and in proprietary special-purpose models. For example Google published the transformer architecture and released BERT as an open weight model, but doesn't really even talk in public about the (presumanbly) specialized internal models behind revenue-generating products.
That's giving a lot of credit to Google's organizational ability to productize Google's research...
There is no rule that every Chinese LLM company must open-source their models, and many don't.
(Similar to how U.S. labs have NSA leadership)
Even if there's a party member at the C-level of some company making day-to-day business decisions, that doesn't necessarily mean they're carrying out detailed instructions from higher-ups in the party as opposed to following their own best judgment. The CCP generally doesn't micro-manage everything, but operates more in a "mission tactics"-like way, where the top level gives out a broad goal that gets condensed down into a slogan like "new productive forces," and interpreting and elaborating on the details is left to lower levels who seek to align their initiatives with the top-level goals (e.g. by describing AI as a "new productive force") and initiatives that appear successful will get blessed as official policy.
That different companies follow different strategies when it comes to model releases is clear evidence that there is no unified party line on this question yet.
Maybe because there is enough given away that there's a lot of competition they may let other companies do whatever they want but to think the CCP isn't managing this is insane to me
I hope the source of this information was well cleaned this morning.
I find it hard to believe the decision to open source any chinese model wasn't approved or heavily considered by the CCP.
TLDR of timeline/chain of events: some billionaire hedgefund manager with AI as hobby, said fuck economics, let's opensource AI... and all the other players have to now compete with that market force. Of course it does not mean equilibrium will last forever, we already seeing defections from model, but imo thats basically what happened... singular hedgefund bro said fuck profits... other PRC players also had to fuck profits, which ends up tripping western labs so deep in debt who can not say fuck profits.
But it does not explain publishing techniques like the ones referenced here in this article.
I realize "going as fast as we can" is not the most popular position atm. But I'm far more interested in what good we can do than 10% apocalypse scenarios. I volunteer with a charity for childhood brain cancer and I do not want to see another 4 year old die. I'm willing to risk anything to stop this.
How co-designed are these optimizations with the model itself? I'd imagine you can't just stick post-training adapters onto existing architectures for these things, or am I wrong?
I really want to explore the inference space, but it seems like many of the inference optimizations are coming from model-hardware codesign. I don't seem to recall many generic "inference engine" optimizations since prefill/decode disagg a year ago.
This matters for me since I want to break in but the bar seems to be understanding the actual theory of the training process now too given the codesign happening, and I'm not the richest guy on the block lol
Because contrarily to the author's assumption, all labs, Western or not, have sufficient skills to discover the optimizations anyway, and publishing or not is not actually that important?
I'm glad people are saying this out loud, because that is what they want. Not for the good of the world, but for the good of their pockets.
I’m incredibly skeptical that OpenAI is spinning up custom ASICs for improved inference performance, but they never thought of optimizing KV cache until a tiny Chinese lab did it? Give me a break.