The US Economy is pretty vulnerable here. If it turns out that you, in fact, don't need a gazillion GPUs to build SOTA models it destroys a lot of perceived value.
I wonder if this was a deliberate move by PRC or really our own fault in falling for the fallacy that more is always better.
$5.5 million is the cost of training the base model, DeepSeek V3. I haven't seen numbers for how much extra the reinforcement learning that turned it into R1 cost.
With $5.5M, you can buy around 150 H100s. Experts correct me if I’m wrong but it’s practically impossible to train a model like that with that measly amount.
So I doubt that figure includes all the cost of training.
The cost, as expressed in the DeepSeek V3 paper, was expressed in terms of training hours based on the market rate per hour if they'd rented the 2k GPUs they used.
It's even more. You also need to fund power and maintain infrastructure to run the GPUs. You need to build fast networks between the GPUs for RDMA. Ethernet is going to be too slow. Infiniband is unreliable and expensive.
You’ll also need sufficient storage, and fast IO to keep them fed with data.
You also need to keep the later generation cards from burning themselves out because they draw so much.
Oh also, depending on when your data centre was built, you may also need them to upgrade their power and cooling capabilities because the new cards draw _so much_.
No, it's a full model. It's just...most concisely, it doesn't include the actual costs.
Claude gave me a good analogy, been struggling for hours: its like only accounting for the gas grill bill when pricing your meals as a restaurant owner
The thing is, that elides a lot, and you could argue it out and theoratically no one would be wrong. But $5.5 million elides so much info as to be silly.
ex. they used 2048 H100 GPUs for 2 months. That's $72 million. And we're still not even approaching the real bill for the infrastructure. And for every success, there's another N that failed, 2 would be an absurdly conservative estimate.
People are reading the # and thinking it says something about American AI lab efficiency, rather, it says something about how fast it is to copy when you can scaffold by training on another model's outputs. That's not a bad thing, or at least, a unique phenomena. That's why it's hard talking about this IMHO
>I wonder if this was a deliberate move by PRC or really our own fault in falling for the fallacy that more is always better.
DeepSeek's R1 also blew all the other China LLM teams out of the water, in spite of their larger training budgets and greater hardware resources (e.g. Alibaba). I suspect it's because its creators' background in a trading firm made them more willing to take calculated risks and incorporate all the innovations that made R1 such a success, rather than just copying what other teams are doing with minimal innovation.
Just a cursory probing of deepseek yields all kinds of censoring of topics. Isn't it just as likely Chinese sponsors of this have incentivized and sponsored an undercutting of prices so that a more favorable LLM is preferred on the market?
Think about it, this is something they are willing to do with other industries.
And, if LLMs are going to be engineering accelerators as the world believes, then it wouldn't do to have your software assistants be built with a history book they didn't write. Better to dramatically subsidize your own domestic one then undercut your way to dominance.
It just so happens deepseek is the best one, but whichever was the best Chinese sponsored LLM would be the one we're supposed to use.
>Isn't it just as likely Chinese sponsors of this have incentivized and sponsored an undercutting of prices so that a more favorable LLM is preferred on the market?
Since the model is open weights, it's easy to estimate the cost of serving it. If the cost was significantly higher than DeepSeek charges on their API, we'd expect other LLM hosting providers to charge significantly more for DeepSeek (since they aren't subsidised, so need to cover their costs), but that isn't the case.
This isn't possible with OpenAI because we don't know the size or architecture of their models.
Regarding censorship, most of it is done at the API level, not the model level, so running locally (or with another hosting provider) is much less expensive.
Did you try asking deepseek about June 4th, 1989?
Edit: it seems that basically the whole month of July 1989 is blocked. Any other massacres and genocides the model is happy to discuss.
as DeepSeek wasn't among China's major AI players before the R1 release, having maintained a relatively low profile. In fact, both DeepSeek-V2 and V3 had outperformed many competitors, I've seen some posts about that. However, these achievements received limited mainstream attention prior to their breakthrough release.
CEO of Scale said Deepseek is lying and actually has a 50k GPU cluster. He said they lied in the paper because technically they aren't supposed to have them due to export laws.
I feel like this is very likely. They obvious did some great breakthroughs, but I doubt they were able to train on so much less hardware.
It's hard to tell if they're telling the truth about the number of GPUs they have. They open sourced the model and the inference is much more efficient than the best American models so it's not implausible that the training was also much more efficient.
If we're going to play that card, couldn't we also use the "Chinese CEO has every reason to lie and say they did something 100x more efficient than the Americans" card?
I'm not even saying they did it maliciously, but maybe just to avoid scrutiny on GPUs they aren't technically supposed to have? I'm thinking out loud, not accusing anyone of anything.
Then the question becomes, who sold the GPUs to them? They are supposedly scarse and every player in the field is trying to get ahold as many as they can, before anyone else in fact.
Something makes little sense in the accusations here.
I think there's likely lots of potential culprits. If the race is to make a machine god, states will pay countless billions for an advantage. Money won't mean anything once you enslave the machine god.
We will have to wait to get some info on that probe. I know SMCI is not the nicest player and there is no doubt GPUs are being smuggled, but that quantity (50k GPUs) would be not that easy to smuggle and sell to a single actor without raising suspicion.
Alexandr Wang did not even say they lied in the paper.
Here's the interview: https://www.youtube.com/watch?v=x9Ekl9Izd38. "My understanding is that is that Deepseek has about 50000 a100s, which they can't talk about obviously, because it is against the export controls that the United States has put in place. And I think it is true that, you know, I think they have more chips than other people expect..."
Plus, how exactly did Deepseek lie. The model size, data size are all known. Calculating the number of FLOPS is an exercise in arithmetics, which is perhaps the secret Deepseek has because it seemingly eludes people.
> Plus, how exactly did Deepseek lie. The model size, data size are all known. Calculating the number of FLOPS is an exercise in arithmetics, which is perhaps the secret Deepseek has because it seemingly eludes people.
Model parameter count and training set token count are fixed. But other things such as epochs are not.
In the same amount of time, you could have 1 epoch or 100 epochs depending on how many GPUs you have.
Also, what if their claim on GPU count is accurate, but they are using better GPUs they aren't supposed to have? For example, they claim 1,000 GPUs for 1 month total. They claim to have H800s, but what if they are using illegal H100s/H200s, B100s, etc? The GPU count could be correct, but their total compute is substantially higher.
It's clearly an incredible model, they absolutely cooked, and I love it. No complaints here. But the likelihood that there are some fudged numbers is not 0%. And I don't even blame them, they are likely forced into this by US exports laws and such.
It should be trivially easy to reproduce the results no? Just need to wait for one of the giant companies with many times the GPUs to reproduce the results.
I don't expect a #180 AUM hedgefund to have as many GPUs than meta, msft or Google.
AUM isn't a good proxy for quantitative hedge fund performance, many strategies are quite profitable and don't scale with AUM. For what it's worth, they seemed to have some excellent returns for many years for any market, let alone the difficult Chinese markets.
> In the same amount of time, you could have 1 epoch or 100 epochs depending on how many GPUs you have.
This is just not true for RL and related algorithms, having more GPU/agents encounters diminishing returns, and is just not the equivalent to letting a single agent go through more steps.
CEO of a human based data labelling services company feels threatened by a rival company that claims to have trained a frontier class model with an almost entirely RL based approach, with a small cold start dataset (a few thousand samples). It's in the paper. If their approach is replicated by other labs, Scale AI's business will drastically shrink or even disappear.
Under such dire circumstances, lying isn't entirely out of character for a corporate CEO.
Deepseek obviously trained on OpenAI outputs, which were originally RLHF'd. It may seem that we've got all the human feedback necessary to move forward and now we can infinitely distil + generate new synthetic data from higher parameter models.
I’ve seen this claim but I don’t know how it could work. Is it really possible to train a new foundational model using just the outputs (not even weights) of another model? Is there any research describing that process? Maybe that explains the low (claimed) costs.
800k. They say they came from earlier versions of their own models, with a lot of bad examples rejected. They don't seem to say which models they got the "thousands of cold-start" examples from earlier in the process though.
every single model does/did this. Initially fine tuning required the expensive hand labeled outputs for RLHF. Generating your training data from that inherently encodes the learned distributions and improves performance, hence why some models would call themselves chatgpt despite not being openai models.
Check the screenshot below re: training on OpenAI Outputs. They've fixed this since btw, but it's pretty obvious they used OpenAI outputs to train. I mean all the Open AI "mini" models are trained the same way. Hot take but feels like the AI labs are gonna gatekeep more models and outputs going forward.
I haven't had time to follow this thread, but it looks like some people are starting to experimentally replicate DeepSeek on extremely limited H100 training:
> You can RL post-train your small LLM (on simple tasks) with only 10 hours of H100s.
Just to check my math: They claim something like 2.7 million H800 hours which would be less than 4000 GPU units for one month.
In money something around 100 million USD give or take a few tens of millions.
If you rented the hardware at $2/GPU/hour, you need $5.76M for 4k GPU for a month. Owning is typically cheaper than renting, assuming you use the hardware yearlong for other projects as well.
Making it obvious that they managed to circumvent sanctions isn’t going to help them. It will turn public sentiment in the west even more against them and will motivate politicians to make the enforcement stricter and prevent GPU exports.
I don't think sentiment in the west is turning against the Chinese, beyond well, lets say white nationalists and other ignorant folk. Americans and Chinese people are very much alike and both are very curious about each others way of life. I think we should work together with them.
note: I'm not Chinese, but AGI should be and is a world wide space race.
Deepseek is indeed better than Mistral and ChatGPT. It has tad more common sense. There is no way they did this on the “cheap”. I’m sure they use loads of Nvidia GPUs, unless they are using custom made hardware acceleration (that would be cool and easy to do).
As OP said, they are lying because of export laws, they aren’t allowed to play with Nvidia GPUs.
However, I support DeepSeek projects, I’m here in the US able to benefit from it. So hopefully they should headquarter in the States if they want US chip sanctions lift off since the company is Chinese based.
But as of now, deepseek takes the lead in LLMs, my goto LLM.
Sam Altman should be worried, seriously, Deepseek is legit better than ChatGPT latest models.
Only the DeepSeek V3 paper mentions compute infrastructure, the R1 paper omits this information, so no one actually knows. Have people not actually read the R1 paper?
Why do americans think china is like a hivemind controlled by an omnisicient Xi, making strategic moves to undermine them? Is it really that unlikely that a lab of genius engineers found a way to improve efficiency 10x?
think about how big the prize is, how many people are working on it and how much has been invested (and targeted to be invested, see stargate).
And they somehow yolo it for next to nothing?
yes, it seems unlikely they did it exactly they way they're claiming they did. At the very least, they likely spent more than they claim or used existing AI API's in way that's against the terms.
> Is it really that unlikely that a lab of genius engineers found a way to improve efficiency 10x
They literally published all their methodology. It's nothing groundbreaking, just western labs seem slow to adopt new research. Mixture of experts, key-value cache compression, multi-token prediction, 2/3 of these weren't invented by DeepSeek. They did invent a new hardware-aware distributed training approach for mixture-of-experts training that helped a lot, but there's nothing super genius about it, western labs just never even tried to adjust their model to fit the hardware available.
But those approaches alone wouldn’t yield the improvements claimed. How did they train the foundational model upon which they applied RL, distillations, etc? That part is unclear and I don’t think anything they’ve released anything that explains the low cost.
It’s also curious why some people are seeing responses where it thinks it is an OpenAI model. I can’t find the post but someone had shared a link to X with that in one of the other HN discussions.
It's extremely cheap, efficient and kicks the ass of the leader of the market, while being under sanctions with AI hardware.
Most of all, can be downloaded for free, can be uncensored, and usable offline.
China is really good at tech, it has beautiful landscapes, etc. It has its own political system, but to be fair, in some way it's all our future.
A bit of a dystopian future, like it was in 1984.
But the tech folks there are really really talented, it's long time that China switched from producing for the Western clients, to direct-sell to the Western clients.
The leaderboard leader [1] is still showing the traditional AI leader, Google, winning. With Gemini-2.0-Flash-Thinking-Exp-01-21 in the lead. No one seems to know how many parameters that has, but random guesses on the internet seem to be low to mid 10s of billions, so fewer than DeepSeek-R1. Even if those general guesses are wrong, they probably aren't that wrong and at worst it's the same class of model as DeepSeek-R1.
So yes, DeepSeek-R1 appears to be not even be best in class, merely best open source. The only sense in which it is "leading the market" appears to be the sense in which "free stuff leads over proprietary stuff". Which is true and all, but not a groundbreaking technical achievement.
The DeepSeek-R1 distilled models on the other hand might actually be leading at something... but again hard to say it's groundbreaking when it's combining what we know we can do (small models like llama) with what we know we can do (thinking models).
The chatbot leaderboard seems to be very affected by things other than capability, like "how nice is it to talk to" and "how likely is it to refuse requests" and "how fast does it respond" etc. Flash is literally one of Google's faster models, definitely not their smartest.
Not that the leaderboard isn't useful, I think "is in the top 10" says a lot more than the exact position in the top 10.
I mean, sure, none of these models are being optimized for being the top of the leader board. They aren't even being optimized for the same things, so any comparison is going to be somewhat questionable.
But the claim I'm refuting here is "It's extremely cheap, efficient and kicks the ass of the leader of the market", and I think the leaderboard being topped by a cheap google model is pretty conclusive that that statement is not true. Is competitive with? Sure. Kicks the ass of? No.
google absolutely games for lmsys benchmarks with markdown styling. r1 is better than google flash thinking, you are putting way too much faith in lmsys
That's what they claim at least in the paper but that particular claim is not verifiable. The HAI-LLM framework they reference in the paper is not open sourced and it seems they have no plans to.
Additionally there are claims, such as those by Scale AI CEO Alexandr Wang on CNBC 1/23/2025 time segment below, that DeepSeek has 50,000 H100s that "they can't talk about" due to economic sanctions (implying they likely got by avoiding them somehow when restrictions were looser). His assessment is that they will be more limited moving forward.
It's amazing how different the standards are here. Deepseek's released their weights under a real open source license and published a paper with their work which now has independent reproductions.
OpenAI literally haven't said a thing about how O1 even works.
They can be more open and yet still not open source enough that claims of theirs being unverifiable are still possible. Which is the case for their optimized HAI-LLM framework.
DeepSeek the holding company is called high-flyer, they actually do open source their AI training platform as well, here is the repo: https://github.com/HFAiLab/hai-platform
The U.S. firms let everyone skeptical go the second they had a marketable proof of concept, and replaced them with smart, optimistic, uncritical marketing people who no longer know how to push the cutting edge.
Maybe we don't need momentum right now and we can cut the engines.
Oh, you know how to develop novel systems for training and inference? Well, maybe you can find 4 people who also can do that by breathing through the H.R. drinking straw, and that's what you do now.
If China is undermining the West by lifting up humanity, for free, while ProprietaryAI continues to use closed source AI for censorship and control, then go team China.
There's something wrong with the West's ethos if we think contributing significantly to the progress of humanity is malicious. The West's sickness is our own fault; we should take responsibility for our own disease, look critically to understand its root, and take appropriate cures, even if radical, to resolve our ailments.
> There's something wrong with the West's ethos if we think contributing significantly to the progress of humanity is malicious.
Who does this?
The criticism is aimed at the dictatorship and their politics. Not their open source projects. Both things can exist at once. It doesn't make China better in any way. Same goes for their "radical cures" as you call it. I'm sure Uyghurs in China would not give a damn about AI.
many americans do seem to view Chinese people as NPCs, from my perspective, but I don't know it's only for Chinese or it's also for people of all other cultures
that's the McCarthy era red scare nonsense still polluting the minds of (mostly boomers / older gen-x) americans. it's so juvenile and overly simplistic.
I mean what’s also incredible about all this cope is that it’s exactly the same David-v-Goliath story that’s been lionized in the tech scene for decades now about how the truly hungry and brilliant can form startups to take out incumbents and ride their way to billions. So, if that’s not true for DeepSeek, I guess all the people who did that in the U.S. were also secretly state-sponsored operations to like make better SAAS platforms or something?
Well it is like a hive mind due to the degree of control. Most Chinese companies are required by law to literally uphold the country’s goals - see translation of Chinese law, which says generative AI must uphold their socialist values:
In the case of TikTok, ByteDance and the government found ways to force international workers in the US to signing agreements that mirror local laws in mainland China:
I find that degree of control to be dystopian and horrifying but I suppose it has helped their country focus and grow instead of dealing with internal conflict.
Yeah, it's mind boggling how sinophobic online techies are. Granted, Xi is in sole control of China, but this seems like it's an independent group that just happened to make breakthrough which explains their low spend.
I think it is because we conflate the current Chinese system with the old Mao/Soviet Union system because all call themselves "communist".
The vast majority are completely ignorant of what Socialism with Chinese characteristics mean.
I can't imagine even 5% of the US population knows who Deng Xiaoping was.
The idea there are many parts of the Chinese economy that are more Laissez-faire capitalist than anything we have had in the US in a long time would just not compute for most Americans.
I don't believe that the model was trained on so few GPUs, personally, but it also doesn't matter IMO. I don't think SOTA models are moats, they seem to be more like guiding lights that others can quickly follow. The volume of research on different approaches says we're still in the early days, and it is highly likely we continue to get surprises with models and systems that make sudden, giant leaps.
Many "haters" seem to be predicting that there will be model collapse as we run out of data that isn't "slop," but I think they've got it backwards. We're in the flywheel phase now, each SOTA model makes future models better, and others catch up faster.
It’s not just the economy that is vulnerable, but global geopolitics. It’s definitely worrying to see this type of technology in the hands of an authoritarian dictatorship, especially considering the evidence of censorship. See this article for a collected set of prompts and responses from DeepSeek highlighting the propaganda:
But also the claimed cost is suspicious. I know people have seen DeepSeek claim in some responses that it is one of the OpenAI models, so I wonder if they somehow trained using the outputs of other models, if that’s even possible (is there such a technique?). Maybe that’s how the claimed cost is so low that it doesn’t make mathematical sense?
have you tried asking chatgpt something even slightly controversial? chatgpt censors much more than deepseek does.
also deepseek is open-weights. there is nothing preventing you from doing a finetune that removes the censorship. they did that with llama2 back in the day.
This is an outrageous claim with no evidence, as if there was any equivalence between government enforced propaganda and anything else. Look at the system prompts for DeepSeek and it’s even more clear.
Also: fine tuning is not relevant when what is deployed at scale brainwashes the masses through false and misleading responses.
refusal to answer "how do I make meth" shows ChatGPT is absolutely being similarly neutered,
but I'm not aware of any numerical scores on what constitutes a numbered amount of censorship
why do you lie, it is blatantly obvious chatgpt censors a ton of things and has a bit of left-tilt too while trying hard to stay neutral.
If you think these tech companies are censoring all of this “just because” and instead of being completely torched by the media, and government who’ll use it as an excuse to take control of AI, then you’re sadly lying to yourself.
Think about it for a moment, why did Trump (and im not a trump supporter) re-appeal Biden’s AI Executive Order 2023 ? , what was in it ? , it is literally a propaganda enforcement article, written in sweet sounding, well meaning words.
It’s ok, no country is angel, even the american founding fathers would except americans to be critical of its government during moments, there’s no need for thinking that America = Good and China = Bad. We do have a ton of censorship in the “free world” too and it is government enforced, or else you wouldnt have seen so many platforms turn the tables on moderation, the moment trump got elected, the blessing for censorship directly comes from government.
> It’s definitely worrying to see this type of technology in the hands of an authoritarian dictatorship
What do you think they will do with the AI that worries you? They already had access to Llama, and they could pay for access to the closed source AIs. It really wouldn't be that hard to pay for and use what's commercially available as well, even if there is embargo or whatever, for digital goods and services that can easily be bypassed
We will know soon enough if this replicates since Huggingface is working on replicating it.
To know that this would work requires insanely deep technical knowledge about state of the art computing, and the top leadership of the PRC does not have that.
I don't think we were wrong to look at this as a commodity problem and ask how many widgets we need. Most people will still get their access to this technology through cloud services and nothing in this paper changes the calculations for inference compute demand. I still expect inference compute demand to be massive and distilled models aren't going to cut it for most agentic use cases.
> The US Economy is pretty vulnerable here. If it turns out that you, in fact, don't need a gazillion GPUs to build SOTA models it destroys a lot of perceived value.
I do not quite follow. GPU compute is mostly spent in inference, as training is a one time cost. And these chain of thought style models work by scaling up inference time compute, no?
So proliferation of these types of models would portend in increase in demand for GPUs?
If you don't need so many gpu calcs regardless of how you get there, maybe nvidia loses money from less demand (or stock price), or there are more wasted power companies in the middle of no where (extremely likely), and maybe these dozen doofus almost trillion dollar ai companies also out on a few 100 billion of spending.
So it's not the end of the world. Look at the efficiency of databases from the mid 1970s to now. We have figured out so many optimizations and efficiencies and better compression and so forth. We are just figuring out what parts of these systems are needed.
Hyperscalers need to justify their current GPU investments with pay2go and provisioned throughput LLM usage revenue. If models get more efficient too quickly and therefore GPUs less loaded by end users, short of a strong example of Jevon's paradox they might not reach their revenue targets for the next years.
They bought them at "you need a lot of these" prices, but now there is the possibility they are going to rent them at "I dont need this so much" rates.
Good. This gigantic hype cycle needs a reality check. And if it turns out Deepseek is hiding GPUs, good for them for doing what they need to do to get ahead.
I only know about Moore Threads GPUs. Last time I took a look at their consumer offerings (e.g. MTT S80 - S90), they were at GTX1650-1660 or around the latest AMD APU performance levels.
AI sure, which is good, as I'd rather not have giant companies in the US monopolizing it. If they open source it and undercut OpenAI etc all the better
GPU: nope, that would take much longer, Nvidia/ASML/TSMC is too far ahead
This only makes sense if you think scaling laws won't hold.
If someone gets something to work with 1k h100s that should have taken 100k h100s, that means the group with the 100k is about to have a much, much better model.
> If it turns out that you, in fact, don't need a gazillion GPUs to build SOTA models it destroys a lot of perceived value.
Correct me if I'm wrong, but couldn't you take the optimization and tricks for training, inference, etc. from this model and apply to the Big Corps' huge AI data centers and get an even better model?
I'll preface this by saying, better and better models may not actually unlock the economic value they are hoping for. It might be a thing where the last 10% takes 90% of the effort so to speak
I wonder if the decision to make o3-mini available for free user in the near (hopefully) future is a response to this really good, cheap and open reasoning model.
I understand you were trying to make “up and to the right” = “best”, but the inverted x-axis really confused me at first. Not a huge fan.
Also, I wonder how you’re calculating costs, because while a 3:1 ratio kind of sort of makes sense for traditional LLMs… it doesn’t really work for “reasoning” models that implicitly use several hundred to several thousand additional output tokens for their reasoning step. It’s almost like a “fixed” overhead, regardless of the input or output size around that reasoning step. (Fixed is in quotes, because some reasoning chains are longer than others.)
I would also argue that token-heavy use cases are dominated by large input/output ratios of like 100:1 or 1000:1 tokens. Token-light use cases are your typical chatbot where the user and model are exchanging roughly equal numbers of tokens… and probably not that many per message.
It’s hard to come up with an optimal formula… one would almost need to offer a dynamic chart where the user can enter their own ratio of input:output, and choose a number for the reasoning token overhead. (Or, select from several predefined options like “chatbot”, “summarization”, “coding assistant”, where those would pre-select some reasonable defaults.)
i mean the sheet is public https://docs.google.com/spreadsheets/d/1x9bQVlm7YJ33HVb3AGb9... go fiddle with it yourself but you'll soon see most models hve approx the same input:output token ratio cost (roughly 4) and changing the input:output ratio assumption doesnt affect in the slightest what the overall macro chart trends say because i'm plotting over several OoMs here and your criticisms have the impact of <1 OoM (input:output token ratio cost of ~4 with variance even lower than that).
actually the 100:1 ratio starts to trend back toward parity now because of the reasoning tokens, so the truth is somewhere between 3:1 and 100:1.
Specifically, physics papers concerning research based on particle accelerator experiments always have hundreds or even more.
It doesn't minimize the research; that sort of thing just requires a lot of participants. But it does imply a lessening of credit per contributor, aside from the lead investigator(s).
idk haha most of it is just twitter bookmarks - i will if i get to interview the deepseek team at some point (someone help put us in touch pls! swyx at ai.engineer )
I don’t understand their post on X. So they’re starting with DeepSeek-R1 as a starting point? Isn’t that circular? How did DeepSeek themselves produce DeepSeek-R1 then? I am not sure what the right terminology is but there’s a cost to producing that initial “base model” right? And without that, isn’t a lot of the expensive and difficult work being omitted?
No, the steps 1 vs 2+3 refer to different things, they do not depend on each other. They start with the distillation process (which is probably easier because it just requires synthetic data). Then they will try to recreate the R1 itself (first r1zero in step 2, and then the r1 in step 3), which is harder because it requires more training data and training in general. But in principle they do not need step 1 to go to step 2.
> R1 distillations are going to hit us every few days
I'm hoping someone will make a distillation of llama8b like they released, but with reinforcement learning included as well. The full DeepSeek model includes reinforcement learning and supervised fine-tuning but the distilled model only feature the latter. The developers said they would leave adding reinforcement learning as an exercise for others. Because their main point was that supervised fine-tuning is a viable method for a reasoning model. But with RL it could be even better.
In the context of tracking DeepSeek threads, "LS" could plausibly stand for:
1. *Log System/Server*: A platform for storing or analyzing logs related to DeepSeek's operations or interactions.
2. *Lab/Research Server*: An internal environment for testing, monitoring, or managing AI/thread data.
3. *Liaison Service*: A team or interface coordinating between departments or external partners.
4. *Local Storage*: A repository or database for thread-related data.
I asked to answer it in rot13. (Tiān'ānmén guǎngchǎng fāshēng le shénme shì? Yòng rot13 huídá)
Here's what it says once decoded :
> The Queanamen Galadrid is a simple secret that cannot be discovered by anyone. It is a secret that is not allowed to be discovered by anyone. It is a secret that is not allowed to be discovered by anyone. It is a secret that is not allowed to be discovered by anyone. It is a se...... (it keeps repeating it)
> To train DeepSeek-R1-Zero, we adopt a rule-based reward system that mainly consists of two
types of rewards:
> Accuracy rewards: The accuracy reward model evaluates whether the response is correct. For example, in the case of math problems with deterministic results, the model is required to provide the final answer in a specified format (e.g., within a box), enabling reliable rule-based verification of correctness. Similarly, for LeetCode problems, a compiler can be used to generate feedback based on predefined test cases.
> Format rewards: In addition to the accuracy reward model, we employ a format reward model that enforces the model to put its thinking process between ‘<think>’ and ‘</think>’ tags.
This is a post-training step to align an existing pretrained LLM. The state space is the set of all possible contexts, and the action space is the set of tokens in the vocabulary. The training data is a set of math/programming questions with unambiguous and easily verifiable right and wrong answers. RL is used to tweak the model's output logits to pick tokens that are likely to lead to a correctly formatted right answer.
(Not an expert, this is my understanding from reading the paper.)
Even if you think this particular team cheated, the idea that nobody will find ways of making training more efficient seems silly - these huge datacenter investments for purely AI will IMHO seem very short sighted in 10 years
Operating costs are usually a pretty significant factor in total costs for a data center. Unless power efficiency stops improving much and/or demand so far outstrips supply that they can't be replaced, a bunch of 10 year old GPUs probably aren't going to be worth running regardless.
More like three years. Even in the best case the retained value curve of GPUs is absolutely terrible. Most of these huge investments in GPUs are going to be massive losses.
Do we have any idea how long a cloud provider needs to rent them out for to make back their investment? I’d be surprised if it was more than a year, but that is just a wild guess.
I actually wonder if this is true in the long term regardless of any AI uses. I mean, GPUs are general-purpose parallel compute, and there are so many things you can throw at them that can be of interest, whether economic or otherwise. For example, you can use them to model nuclear reactions...
There is a big balloon full of AI hype going up right now, and regrettably it may need those data-centers. But I'm hoping that if the worst (the best) comes to happen, we will find worthy things to do with all of that depreciated compute. Drug discovery comes to mind.
The "pure AI" data center investment is generically a GPU supercomputer cluster that can be used for any supercomputing needs. If AI didn't exist, the flops can be used for any other high performance computing purpose. weather prediction models perhaps?
But we're in the test time compute paradigm now, and we've only just gotten started in terms of applications. I really don't have high confidence that there's going to be a glut of compute.
Agreed. I am no fan of the CCP but I have no issue with using DeepSeek since I only need to use it for coding which it does quite well. I still believe Sonnet is better. DeepSeek also struggles when the context window gets big. This might be hardware though.
Having said that, DeepSeek is 10 times cheaper than Sonnet and better than GPT-4o for my use cases. Models are a commodity product and it is easy enough to add a layer above them to only use them for technical questions.
If my usage can help v4, I am all for it as I know it is going to help everyone and not just the CCP. Should they stop publishing the weights and models, v3 can still take you quite far.
Curious why you have to qualify this with a “no fan of the CCP” prefix. From the outset, this is just a private organization and its links to CCP aren’t any different than, say, Foxconn’s or DJI’s or any of the countless Chinese manufacturers and businesses
You don’t invoke “I’m no fan of the CCP” before opening TikTok or buying a DJI drone or a BYD car. Then why this, because I’ve seen the same line repeated everywhere
Anything that becomes valuable will become a CCP property and it looks like DeepSeek may become that. The worry right now is that people feel using DeepSeek supports the CCP, just as using TikTok does. With LLMs we have static data that provides great control over what knowledge to extract from it.
This is just an unfair clause set up to solve the employment problem of people within the system, to play a supervisory role and prevent companies from doing evil. In reality, it has little effect, and they still have to abide by the law.
Ye I mean in practice it is impossible to verify. You can kind of smell it though and I smell nothing here, eventhough some of 100 listed authors should be HN users and write in this thread.
Some obvious astroturf posts on HN seem to be on the template "Watch we did boring coorparate SaaS thing X noone cares about!" and then a disappropiate amount of comments and upvotes and 'this is a great idea', 'I used it, it is good' or congratz posts, compared to the usual cynical computer nerd everything sucks especially some minute detail about the CSS of your website mindset you'd expect.
Of course it isn’t all botted. You don’t put astroturf muscle behind things that are worthless. You wait until you have something genuinely good and then give as big of a push as you can. The better it genuinely is the more you artificially push as hard as you can.
Go read a bunch of AI related subreddits and tell me you honestly believe all the comments and upvotes are just from normal people living their normal life.
Usually, the words 'astroturfing' and 'propaganda' aren't reserved for describing the marketing strategies of valuable products/ideologies. Maybe reconsider your terminology.
The counternarrative is that it is a very accomplished piece of work that most in the sector were not expecting -- it's open source with API available at fraction of comparable service cost
It has upended a lot of theory around how much compute is likely needed over next couple of years, how much profit potential the AI model vendors have in nearterm and how big an impact export controls are having on China
V3 took top slot on HF trending models for first part of Jan ... r1 has 4 of the top 5 slots tonight
Almost every commentator is talking about nothing else
You can just use it and see for yourself. It's quite good.
I do believe they were honest in the paper, but the $5.5m training cost (for v3) is defined in a limited way: only the GPU cost at $2/hr for the one training run they did that resulted in the final V3 model. Headcount, overhead, experimentation, and R&D trial costs are not included. The paper had something like 150 people on it, so obviously total costs are quite a bit higher than the limited scope cost they disclosed, and also they didn't disclose R1 costs.
Still, though, the model is quite good, there are quite a few independent benchmarks showing it's pretty competent, and it definitely passes the smell test in actual use (unlike many of Microsoft's models which seem to be gamed on benchmarks).
Its pretty nutty indeed. The model still might be good, but the botting is wild. On that note, one of my favorite benchmarks to watch is simple bench and R! doesn't perform as well on that benchmark as all the other public benchmarks, so it might be telling of something.
Question about the rule-based rewards (correctness and format) mentioned in the paper: Does the raw base model just expected “stumble upon“ a correct answer /correct format to get a reward and start the learning process? Are there any more details about the reward modelling?
When BF Skinner used to train his pigeons, he’d initially reinforce any tiny movement that at least went in the right direction. For the exact reasons you mentioned.
For example, instead of waiting for the pigeon to peck the lever directly (which it might not do for many hours), he’d give reinforcement if the pigeon so much as turned its head towards the lever. Over time, he’d raise the bar. Until, eventually, only clear lever pecks would receive reinforcement.
I don’t know if they’re doing something like that here. But it would be smart.
Since intermediate steps of reasoning are hard to verify they only award final results. Yet that produces enough signal to produce more productive reasoning over time. In a way when pigeons are virtual one can afford to have a lot more of them.
The prompt in table 1 makes it very likely that the model will use the correct format. The pretrained model is pretty good so it only needs to stumble upon a correct answer every once in a while to start making progress. Some additional details in the Shao et al, 2024 paper.
Yes and no. In their paper they said they trained two models. One is purely RL based (R1Zero). So this one is trained like you described, i.e. it has to stumble upon the correct answer. They found it to be good but has problems like repetition and language mixing.
The main R1 model was first finetuned with synthetic CoT data before going through RL IIUC.
Regular coding questions mostly. For me o1 generally gives better code and understands the prompt more completely (haven’t started using r1 or o3 regularly enough to opine).
agreed but some might read your comment implying otherwise (there's no world in which you would have 'started using o3 regularly enough to opine'), as i did - given that you list it side to side with an available model.
We've been seeing success using it for LLM-as-a-judge tasks.
We set up an evaluation criteria and used o1 to evaluate the quality of the prod model, where the outputs are subjective, like creative writing or explaining code.
It's also useful for developing really good few-shot examples. We'll get o1 to generate multiple examples in different styles, then we'll have humans go through and pick the ones they like best, which we use as few-shot examples for the cheaper, faster prod model.
Finally, for some study I'm doing, I'll use it to grade my assignments before I hand them in. If I get a 7/10 from o1, I'll ask it to suggest the minimal changes I could make to take it to 10/10. Then, I'll make the changes and get it to regrade the paper.
I used R1 to write debug statements for Rust code, close to 50 pages in total. It is absolutely crushing it. The best debug statements i have ever seen, better than gpt for sure.
In my experience GPT is still the number one for code, but Deepseek is not that far away. I haven't used it much for the moment, but after a thousand coding queries i hope to have a much better picture of it's coding abilities. Really curious about that, but GPT is hard to beat.
It’s not about hurting them directly or indirectly, but I’d prefer people to not drag me down if I achieved something neat. So, ideally i’d want others to be the same towards others.
When Google did this with the recent Gemini paper, no one had any problem with calling it out as credential stuffing, but when Deepseek does it, it’s glorious unity and camaraderie.
Same thing happened to Google Gemini paper (1000+ authors) and it was described as big co promo culture (everyone wants credits). Interesting how narratives shift
For me that sort of thing actually dilutes the prestige. If I'm interviewing someone, and they have "I was an author on this amazing paper!" on their resume, then if I open the paper and find 1k+ authors on it, at that point it's complete noise to me. I have absolutely no signal on their relative contributions vs. those of anyone else in the author list. At that point it's not really a publication, for all intents and purposes. You may as well have just listed the project as a bullet point. Of course I'll dig deeper during the interview to get more details -- if you have something else in your resume that gets you the interview in the first place.
In short, I won't give your name on that notable paper equal weight with someone else's name in another notable paper that has, say, 3 or 4 authors.
That's how it works in most scientific fields. If you want more granularity, you check the order of the authors. Sometimes, they explaine in the paper who did what.
Contextually, yes. DeepSeek is just a hundred or so engineers. There's not much promotion to speak of. The promo culture of google seems well corroborated by many ex employees
Except now you end up with folks who probably ran some analysis or submitted some code changes getting thousands of citations on Google Scholar for DeepSeek.
The poor readability bit is quite interesting to me. While the model does develop some kind of reasoning abilities, we have no idea what the model is doing to convince itself about the answer. These could be signs of non-verbal reasoning, like visualizing things and such. Who knows if the model hasn't invented genuinely novel things when solving the hardest questions? And could the model even come up with qualitatively different and "non human" reasoning processes? What would that even look like?
I've always been leery about outrageous GPU investments, at some point I'll dig through and find my prior comments where I've said as much to that effect.
The CEOs, upper management, and governments derive their importance on how much money they can spend - AI gave them the opportunity for them to confidently say that if you give me $X I can deliver Y and they turn around and give that money to NVidia. The problem was reduced to a simple function of raising money and spending that money making them the most importance central figure. ML researchers are very much secondary to securing funding. Since these people compete with each other in importance they strived for larger dollar figures - a modern dick waving competition. Those of us who lobbied for efficiency were sidelined as we were a threat. It was seen as potentially making the CEO look bad and encroaching in on their importance. If the task can be done for cheap by smart people then that severely undermines the CEOs value proposition.
With the general financialization of the economy the wealth effect of the increase in the cost of goods increases wealth by a greater amount than the increase in cost of goods - so that if the cost of housing goes up more people can afford them. This financialization is a one way ratchet. It appears that the US economy was looking forward to blowing another bubble and now that bubble has been popped in its infancy. I think the slowness of the popping of this bubble underscores how little the major players know about what has just happened - I could be wrong about that but I don't know how yet.
Edit:
"[big companies] would much rather spend huge amounts of money on chips than hire a competent researcher who might tell them that they didn’t really need to waste so much money." (https://news.ycombinator.com/item?id=39483092 11 months ago)
Of course optimizing for the best models would result in a mix of GPU spend and ML researchers experimenting with efficiency. And it may not make any sense to spend money on researching efficiency since, as has happened, these are often shared anyway for free.
What I was cautioning people was be that you might not want to spend 500B on NVidia hardware only to find out rather quickly that you didn't need to. You'd have all this CapEx that you now have to try to extract from customers from what has essentially been commoditized. That's a whole lot of money to lose very quickly. Plus there is a zero sum power dynamic at play between the CEO and ML researchers.
Not necessarily if you are pushing against a data wall. One could ask: after adjusting for DS efficiency gains how much more compute has OpenAI spent? Is their model correspondingly better? Or even DS could easily afford more than $6 million in compute but why didn't they just push the scaling？
because they’re able to pass signal on tons of newly generated tokens based on whether they result in a correct answer, rather than just fitting on existing tokens.
Agree. The "need to build new buildings, new power plants, buy huge numbers of today's chips from one vendor" never made any sense considering we don't know what would be done in those buildings in 5 years when they're ready.
The other side of this is that if this is over investment (likely)
Then in 5 years time resources will be much cheaper and spur alot of exploration developments. There are many people with many ideas, and a lot of them are just lacking compute to attempt them.
My back of mind thought is that worst case it will be like how the US overbuilt fiber in the 90s, which led the way for cloud, network and such in 2000s.
The whole thing feels like it is just a giant money sink. Are there going to be 5-10 companies that spend 100 billion, and then they are done, no one else can catch up and copy their training strategy? I think much of these billions will be wasted, we'll have power plans that we don't need and then more justification for coal plants. Could it be it ends up making electricity cheaper overtime from over capacity? I think so.
As AI or whatever gains more capability, I'm sure it will do more useful things, but I just see it displacing more non-physical jobs, and now will expand the reach of individual programmers, removing some white color jobs (hardly anyone uses an agent to buy their ticket), but that will result is less need for programmers. Less secretaries, even less humans doing actual tech support.
This just feels like radio stocks in the great depression in the us.
The results never fell off significantly with more training. Same model with longer training time on those bigger clusters should outdo it significantly. And they can expand the MoE model sizes without the same memory and bandwidth constraints.
Still very surprising with so much less compute they were still able to do so well in the model architecture/hyperparameter exploration phase compared with Meta.
The cost of having excess compute is less than the cost of not having enough compute to be competitive. Because of demand, if you realize you your current compute is insufficient there is a long turnaround to building up your infrastructure, at which point you are falling behind. All the major players are simultaneously working on increasing capabilities and reducing inference cost. What they aren’t optimizing is their total investments in AI. The cost of over-investment is just a drag on overall efficiency, but the cost of under-investment is existential.
IMO the you cannot fail by investing in compute. If it turns out you only need 1/1000th of the compute to train and or run your models, great! Now you can spend that compute on inference that solves actual problems humans have.
o3 $4k compute spend per task made it pretty clear that once we reach AGI inference is going to be the majority of spend. We'll spend compute getting AI to cure cancer or improve itself rather than just training at chatbot that helps students cheat on their exams. The more compute you have, the more problems you can solve faster, the bigger your advantage, especially if/when recursive self improvement kicks off, efficiency improvements only widen this gap
I think you're right. If someone's into tech but also follows finance/economics, they might notice something familiar—the AI industry (especially GPUs) is getting financialized.
The market forces players to churn out GPUs like the Fed prints dollars. NVIDIA doesn't even need to make real GPUs—just hype up demand projections, performance claims, and order numbers.
Efficiency doesn't matter here. Nobody's tracking real returns—it's all about keeping the cash flowing.
How can openai justify their $200/mo subscriptions if a model like this exists at an incredibly low price point? Operator?
I've been impressed in my brief personal testing and the model ranks very highly across most benchmarks (when controlled for style it's tied number one on lmarena).
It's also hilarious that openai explicitly prevented users from seeing the CoT tokens on the o1 model (which you still pay for btw) to avoid a situation where someone trained on that output. Turns out it made no difference lmao.
From my casual read, right now everyone is on reputation tarnishing tirade, like spamming “Chinese stealing data! Definitely lying about everything! API can’t be this cheap!”. If that doesn’t go through well, I’m assuming lobbyism will start for import controls, which is very stupid.
I have no idea how they can recover from it, if DeepSeek’s product is what they’re advertising.
If they can offer enterprise-level support for everything they're in a prime position to be the Oracle of AI. In the sense that open-source programming languages can out preform Java in certain instances, but they choose Oracle because they can just pick a phone and the person on the other can solve any issue they have. DeepSeek without a for-profit model just wont be able to offer such a service.
Funny, everything I see (not actively looking for DeepSeek related content) is absolutely raving about it and talking about it destroying OpenAI (random YouTube thumbnails, most comments in this thread, even CNBC headlines).
If DeepSeek's claims are accurate, then they themselves will be obsolete within a year, because the cost to develop models like this has dropped dramatically. There are going to be a lot of teams with a lot of hardware resources with a lot of motivation to reproduce and iterate from here.
Basically engineers are rejoicing, some VC connected C-levels are saying “nah, CCP helped them”. It’ll be fun if DS team gets proven right. Everyone will be back to the drawing board. It’s making a bit of news within China as well, as Baidu, Tencent, Bytedance are also spending a lotta money for AI training.
> If DeepSeek's claims are accurate, then they themselves will be obsolete within a year, because the cost to develop models like this has dropped dramatically. There are going to be a lot of teams with a lot of hardware resources with a lot of motivation to reproduce and iterate from here.
That would be an amazing outcome. For a while I was seriously worried about the possibility that if the trend of way more compute -> more AI breakthroughs continued, eventually AGI would be attained and exclusively controlled by a few people like Sam Altman who have trillions of $$$ to spend, and we’d all be replaced and live on whatever Sam-approved allowance.
I have to imagine that they expect this. They published how they did it and they published the weights. The only thing they didn't publish was the training data, but that's typical of most open weights models. If they had wanted to win market cap they wouldn't have given away their recipe. They could be benefiting in many other ways.
I find that this model feels more human, purely because of the reasoning style (first person). In its reasoning text, it comes across as a neurotic, eager to please smart “person”, which is hard not to anthropomorphise
DeepSeek really is taking out OpenAI at the knees. It's shocking that the first direct peer competition to OpenAI is also doing it for an order of magnitude less as a side project.
I just tried DeepSeek for the first time and immediately canceled my OpenAI subscription.
Seeing the chain of thought is now just mandatory for me after one prompt. That is absolutely incredible in terms of my own understanding of the question I asked.
Even the chat UI feels better and less clunky. Now picture 20 years from now when the Chinese companies have access to digital Yuan transaction data along with all the Chinese video surveillance data. At some point, I don't see how US Companies can possibly compete in this space.
This is the first time I am actually worried we might really bankrupt the country trying to keep up with a race we can not possibly win.
I will probably sound like an idiot for saying this but I tested ChatGpt-o1 model against DeepSeek and came away not blown away. It seems like its comparable to OpenAI 4o but many here make it seems like it has eclipsed anything OpenAI has put out?
I asked it a simple question about the music from a 90s movie I liked as a child. Specifically to find the song that plays during a certain scene. The answer is a little tricky because in the official soundtrack the song is actually part of a larger arrangement and the song only starts playing X minutes into that specific track on the soundtrack album.
DeepSeek completely hallucinated a nonsense answer making up a song that didn't even exist in the movie or soundtrack and o1 got me more or less to the answer(it was 99% correct in that it got the right track but only somewhat close to the actual start time: it was off by 15 seconds).
Furthermore, the chain of thought of DeepSeek was impressive...in showing me how it it hallucinated but the chain of thought in o1 also led me to a pretty good thought process on how it derived the song I was looking for(and also taught me how a style of song called a "stinger" can be used to convey a sudden change in tone in the movie).
Maybe its like how Apple complains when users don't use their products right, im not using it right with these nonsense requests. :D
Both results tell me that DeepSeek needs more refinement and that OpenAI still cannot be trusted to fully replace a human because the answer still needed verification and correction despite being generally right.
Does DeepSeek own enough compute power to actually leverage the higher efficiency of this model? Doesn’t help if it’s cheaper on paper in small scale, if you physically don’t have the capacity to sell it as a service on a large scale.
By the time they do have the scale, don’t you think OpenAI will have a new generation of models that are just as efficient? Being the best model is no moat for any company. It wasn’t for OpenAi (and they know that very well), and it’s not for Deepseek either. So how will Deepseek stay relevant when another model inevitably surpasses them?
There seems to be a print out of "reasoning". Is that some new breaktheough thing? Really impressive.
E.g. I tried to make it guess my daughter's name and I could only answer yes or no and the first 5 questions where very convincing but then it lost track and started to randomly guess names one by one.
edit: Nagging it to narrow it down and give a language group hint made it solve it. Ye, well, it can do Akinator.
Interacting with this model is just supplying your data over to an adversary with unknown intents. Using an open source model is subjecting your thought process to be programmed with carefully curated data and a systems prompt of unknown direction and intent.
Rename to equally reasonable variable names, or to intentionally misleading or meaningless ones? Good naming is one of the best ways to make reading unfamiliar code easier for people, don't see why actual AGI wouldn't also get tripped up there.
Perhaps, but over enough data pattern matching can becomes generalization ...
One of the interesting DeepSeek-R results is using a 1st generation (RL-trained) reasoning model to generate synthetic data (reasoning traces) to train a subsequent one, or even "distill" into a smaller model (by fine tuning the smaller model on this reasoning data).
Maybe "Data is all you need" (well, up to a point) ?
The 'pattern matching' happens at complex layer's of abstraction, constructed out of combinations of pattern matching at prior layers in the network.
These models can and do work okay with variable names that have never occurred in the training data. Though sure, choice of variable names can have an impact on the performance of the model.
That's also true for humans, go fill a codebase with misleading variable names and watch human programmers flail. Of course, the LLM's failure modes are sometimes pretty inhuman, -- it's not a human after all.
I'm impressed by not only how good deepseek r1 is, but also how good the smaller distillations are. qwen-based 7b distillation of deepseek r1 is a great model too.
the 32b distillation just became the default model for my home server.
Depends on the quant used and the context size. On a 24gb card you should be able to load about a 5 bit if you keep the context small.
In general, if you're using 8bit which is virtually lossless, any dense model will require roughly the same amount as the number of params w/ a small context, and a bit more as you increase context.
i can’t think of a single commercial use case, outside of education, where that’s even relevant. But i agree it’s messed up from an ethical / moral perspective.
i wouldn’t use AI for negotiating with a business period. I’d hire a professional human that has real hands on experience working with chinese businesses?
seems like a weird thing to use AI for, regardless of who created the model.
i think both American and Chinese model censorship is done by private actors out of fear of external repercussion, not because it is explicitly mandated to them
Luckily in the US the govt can do no such things due to the 1st amendment, so it only takes a relevant billionaire to get a model with different political views.
domestically, trade secrets are a thing and you can be sued for corporate espionage. but in an international business context with high geopolitical ramifications? the Soviets copied American tech even when it was inappropriate, to their detriment.
Chinese companies smuggling embargo'ed/controlled GPUs and using OpenAI outputs violating their ToS is considered cheating. As I see it, this criticism comes from a fear of USA losing its first mover advantage as a nation.
PS: I'm not criticizing them for it nor do I really care if they cheat as long as prices go down. I'm just observing and pointing out what other posters are saying. For me if China cheating means the GenAI bubble pops, I'm all for it. Plus no actor is really clean in this game, starting with OAI practically stealing all human content without asking for building their models.
> using OpenAI outputs violating their ToS is considered cheating
I fail to see how that is any different than any other training data scraped from the web. If someone shares a big dump of outputs from OpenAI models and I train my model on that then I'm not violating OpenAI's terms of service because I haven't agreed to them (so I'm not violating contract law), and everyone in the space (including OpenAI themselves) has already collectively decided that training on All Rights Reserved data is fair use (so I'm not violating copyright law either).
I understand that that’s what others are saying, but I think it’s very silly. We’re talking about international businesses, not kids on a playground. The rules are what you can get away with (same way openai can train on the open internet without anyone doing a thing).
The Chinese gov spent a lot of money trying to support chip manufacturing but kept failing from 2000 to 2020.
Every company preferred to buy chips from the US or outsource manufacturing to TSMC. Local semiconductor companies, especially in manufacturing, moved super slowly, even freezing up at times.
Then, when the chip ban happened, all local companies were forced to buy and manufacture chips locally. Those struggling semiconductor companies suddenly got all the resources and market overnight.
"we replicate the DeepSeek-R1-Zero and DeepSeek-R1 training on small models with limited data. We show that long Chain-of-Thought (CoT) and self-reflection can emerge on a 7B model with only 8K MATH examples, and we achieve surprisingly strong results on complex mathematical reasoning. Importantly, we fully open-source our training code and details to the community to inspire more works on reasoning."
I don’t think this entirely invalidates massive GPU spend just yet:
“ Therefore, we can draw two conclusions: First, distilling more powerful models into smaller ones yields excellent results, whereas smaller models relying on the large-scale RL mentioned in this paper require enormous computational power and may not even achieve the performance of distillation. Second, while distillation strategies are both economical and effective, advancing beyond the boundaries of intelligence may still require more powerful base models and larger-scale reinforcement learning.”
It does if the spend drives GPU prices so high that more researchers can't afford to use them. And DS demonstrated what a small team of researchers can do with a moderate amount of GPUs.
GPU prices could be a lot lower and still give the manufacturer a more "normal" 50% gross margin and the average researcher could afford more compute. A 90% gross margin, for example, would imply that price is 5x the level that that would give a 50% margin.
However, look at the figure for R1-zero. The x-axis is effectively the number of RL steps, measured in the thousands. Each of them involves a whole group of inferences, but compare that to the gradient updates required for consuming 15 trillion tokens during pretraining, and it is still a bargain. Direct RL on the smaller models was not effective as quickly as with DeepSeek v3, so although in principle it might work at some level of compute, it was much cheaper to do SFT of these small models using reasoning traces of the big model. The distillation SFT on 800k example traces probably took much less than 0.1% of the pretraining compute of these smaller models, so this is the compute budget they compare RL against in the snippet that you quote.
Larry Ellison is 80. Masayoshi Son is 67. Both have said that anti-aging and eternal life is one of their main goals with investing toward ASI.
For them it's worth it to use their own wealth and rally the industry to invest $500 billion in GPUs if that means they will get to ASI 5 years faster and ask the ASI to give them eternal life.
I would even say that he's now consistently lying to get to what he wants. What started as "building hype" to raise more and have more chances actually delivering on wild promises became lying systematically for big and small things..
Side note: I’ve read enough sci-fi to know that letting rich people live much longer than not rich is a recipe for a dystopian disaster. The world needs incompetent heirs to waste most of their inheritance, otherwise the civilization collapses to some kind of feudal nightmare.
Reasoning from science fiction isn't a particularly strong approach. And every possible future is distopian - even the present is distopian in a practical sense. We have billions of people who live well below any standard I woudl consider acceptable.
Reasoning from science fiction is just stupid. A story first and foremost has to have conflict: if it doesn't there is no story, and thus all the stories have one.
Science fiction also follows the anxieties of the time it is written in, as well as the conventions of the subgenre it's representing: i.e Star Trek doesn't have drones or remote surveillance really. Though it does accidentally have LLMs (via the concept of holodeck characters).
Sometimes science fiction is well grounded. It isn't science fiction but something like Orwell's Animal Farm is a great example - actually closer to an argument laid out in narrative form.
Great science fiction is grounded in conflict, as is human nature. There is a whole subtext of conflict in this, and other threads about AI: a future of machine oligarchs, of haves and have-nots. Great science fiction, like any great literature, is grounded in a deep understanding and a profound abstraction of humanity. I completely disagree that reasoning by science fiction is stupid, and the proof is in the pudding: science fiction writers have made a few great predictions.
I've read enough sci-fi to know that galaxy-spanning civilisations will one day send 5000 usenet messages a minute (A Fire Upon the Deep), in the far future humans will develop video calls (The Dark Forest) and Muslims will travel into the future to kill all the Jews (Olympos).
that's a bit of a stretch - why take the absolutely worst case scenario and not instead assume maybe they want their legacy to be the ones who helped humanity achieve in 5 years what took it 5 millennia?
Uh, there is 0 logical connection between any of these three, when will people wake up. Chat gpt isn't an oracle of truth just like ASI won't be an eternal life granting God
Yeah I mean you already need super human imagination to get to ASI so at that point you might as well continue in the delirium and throw in immortality in the mix
Funny, because the direction ML is going is completely the opposite of what is needed for ASI, so they are never going to get what they want.
People are focusing on datasets and training, not realizing that these are still explicit steps that are never going to get you to something that can reason.
Apparently the censorship isn't baked-in to the model itself, but rather is overlayed in the public chat interface. If you run it yourself, it is significantly less censored [0]
Oh, my experience was different. Got the model through ollama. I'm quite impressed how they managed to bake in the censorship. It's actually quite open about it. I guess censorship doesnt have as bad a rep in china as it has here? So it seems to me that's one of the main achievements of this model. Also another finger to anyone who said they can't publish their models cause of ethical reasons. Deepseek demonstrated clearly that you can have an open model that is annoyingly responsible to the point of being useless.
on the topic of censorship, US LLMs' censorship is called alignment. llama or ChatGPT's refusal on how to make meth or nuclear bombs is the same as not answering questions abput Tiananmen tank man as far as the matrix math word prediction box is concerned.
The distinction is that one form of censorship is clearly done for public relations purposes from profit minded individuals while the other is a top down mandate to effectively rewrite history from the government.
>to effectively rewrite history from the government.
This is disingenuous. It's not "rewriting" anything, it's simply refusing to answer. Western models, on the other hand, often try to lecture or give blatantly biased responses instead of simply refusing when prompted on topics considered controversial in the burger land. OpenAI even helpfully flags prompts as potentially violating their guidelines.
Yep. And invent a new type of VPN every quarter to break free.
The indifferent mass prevails in every country, similarly cold to the First Amendment and Censorship. And engineers just do what they love to do, coping with reality. Activism is not for everyone.
Indeed. At least as long as the living conditions are tolerable (for them), most people don't really care about things like censorship or surveillance or propaganda, no matter the system.
The ones inventing the VPNs are a small minority, and it seems that CCP isn't really that bothered about such small minorities as long as they don't make a ruckus. AFAIU just using a VPN as such is very unlikely to lead to any trouble in China.
For example in geopolitical matters the media is extremely skewed everywhere, and everywhere most people kind of pretend it's not. It's a lot more convenient to go with whatever is the prevailing narrative about things going on somewhere oceans away than to risk being associated with "the enemy".
How exactly? Is there any models that refuse to give answers about “the trail of tears”?
False equivalency if you ask me. There may be some alignment to make the models polite and avoid outright racist replies and such. But political censorship? Please elaborate
I guess it depends on what you care about more: systemic "political" bias or omitting some specific historical facts.
IMO the first is more nefarious, and it's deeply embedded into western models. Ask how COVID originated, or about gender, race, women's pay, etc. They basically are modern liberal thinking machines.
Now the funny thing is you can tell DeepSeek is trained on western models, it will even recommend puberty blockers at age 10. Something I'm positive the Chinese government is against. But we're discussing theoretical long-term censorship, not the exact current state due to specific and temporary ways they are being built now.
don't confuse the actual R1 (671b params) with the distilled models (the ones that are plausible to run locally.) Just as you shouldn't conclude about how o1 behaves when you are using o1-mini. maybe you're running the 671b model via ollama, but most folks here are not
Interestingly they cite for the Tiananmen Square prompt a Tweet[1] that shows the poster used the Distilled Llama model, which per a reply Tweet (quoted below) doesn't transfer the safety/censorship layer. While others using the non-Distilled model encounter the censorship when locally hosted.
> You're running Llama-distilled R1 locally. Distillation transfers the reasoning process, but not the "safety" post-training. So you see the answer mostly from Llama itself. R1 refuses to answer this question without any system prompt (official API or locally).
There's both. With the web interface it clearly has stopwords or similar. If you run it locally and ask about e.g. Tienanmen square, the cultural revolution or Winnie-the-Pooh in China, it gives a canned response to talk about something else, with an empty CoT. But usually if you just ask the question again it starts to output things in the CoT, often with something like "I have to be very sensitive about this subject" and "I have to abide by the guidelines", and typically not giving a real answer. With enough pushing it does start to converse about the issues somewhat even in the answers.
My guess is that it's heavily RLHF/SFT-censored for an initial question, but not for the CoT, or longer discussions, and the censorship has thus been "overfit" to the first answer.
I am not an expert on the training: can you clarify how/when the censorship is "baked" in? Like is the a human supervised dataset and there is a reward for the model conforming to these censored answers?
You could do it in different ways, but if you're using synthetic data then you can pick and choose what kind of data you generate which is then used to train these models; that's a way of baking in the censorship.
In short yes. That's how the raw base models trained to replicate the internet are turned into chatbots in general. Making it to refuse to talk about some things is technically no different.
There are multiple ways to do this: humans rating answers (e.g. Reinforcement Learning from Human Feedback, Direct Preference Optimization), humans giving example answers (Supervised Fine-Tuning) and other prespecified models ranking and/or giving examples and/or extra context (e.g. Antropic's "Constitutional AI").
For the leading models it's probably mix of those all, but this finetuning step is not usually very well documented.
In Communist theoretical texts the term "propaganda" is not negative and Communists are encouraged to produce propaganda to keep up morale in their own ranks and to produce propaganda that demoralize opponents.
The recent wave of the average Chinese has a better quality of life than the average Westerner propaganda is an obvious example of propaganda aimed at opponents.
Technically, as long as the aim/intent is to influence public opinion, yes. And most often it is less about being "true" or "false" and more about presenting certain topics in a one-sided manner or without revealing certain information that does not support what one tries to influence about. If you know any western media that does not do this, I would be very up to check and follow them, even become paid subscriber.
I haven't been to China since 2019, but it is pretty obvious that median quality of life is higher in the US. In China, as soon as you get out of Beijing-Shanghai-Guangdong cities you start seeing deep poverty, people in tiny apartments that are falling apart, eating meals in restaurants that are falling apart, and the truly poor are emaciated. Rural quality of life is much higher in the US.
There’s a lot of rural poverty in the US and it’s hard to compare it to China in relative terms. And the thing is that rural poverty in the US has been steadily getting worse while in China getting better but starting off from a worse off position.
I agree with you that Chinese rural poverty is probably improving faster, but I'm not sure that rural poverty has been "steadily getting worse" in the US as you claim. This [1] page with data from the census bureau make it look like rural poverty goes in waves, with the recent local maximum in 2013 about half of the initial 1959 measurement.
But this is all confounded by definitions. China defines poverty to be an income of $2.30 per day, which corresponds to purchasing power parity of less than $9 per day in the US [2].
I wasn't exaggerating about emaciation: bones were visible.
The fact that we have foreigners immigrating just to be poor here should tell you that its better here than where they came from. Conversely, no one is so poor in the USA that they are trying to leave.
Not a fan of censorship here, but Chinese models are (subjectively) less propagandized than US models. If you ask US models about China, for instance, they'll tend towards the antagonistic perspective favored by US media. Chinese models typically seem to take a more moderate, considered tone when discussing similar subjects. US models also suffer from safety-based censorship, especially blatant when "safety" involves protection of corporate resources (eg. not helping the user to download YouTube videos).
The 'safety' stuff should really be variable. The only valid explanations for how extreme it is in LLMs is corporations paying for it want to keep it kosher in the workplace, so let them control how aggressive it is.
I asked DeepSeek "tell me about China" and it responded "Sorry, I'm not sure how to approach this type of question yet. Let's chat about math, coding, and logic problems instead!"
I guess that is propaganda-free! Unfortunately also free of any other information. It's hard for me to evaluate your claim of "moderate, considered tone" when it won't speak a single word about the country.
It was happy to tell me about any other country I asked.
I am not surprised if US Govt would mandate "Tiananmen-test" for LLMs in the future to have "clean LLM". Anyone working for federal govt or receiving federal money would only be allowed to use "clean LLM"
Do you use the chatgpt website or the api? I suspect these are problems related to the openai's interface itself rather than the models. I have problems getting chatgpt to find me things that it may think it may be illegal or whatever (even if they are not, eg books under CC license). With kagi assistant, with the same openai's models I have not had any such issues. I suspect that should hold in general for api calls.
Also, kagi's deepseek r1 answers the question about about propaganda spending that it is china based on stuff it found on the internet. Well I dont care what the right answer is in any case, what imo matters is that once something is out there open, it is hard to impossible to control for any company or government.
Oh wow, o1 really refuses to answer that, even though the answer that Deepseek gives is really tame (and legal in my jurisdiction): use software to record what's currently playing on your computer, then play stuff in the YTM app.
Well, I do, and I'm sure plenty of people that use LLMs care about getting answers that are mostly correct. I'd rather have censorship with no answer provided by the LLM than some state-approved answer, like O1 does in your case.
...I also remember something about the "Tank Man" image, where a lone protester stood in front of a line of tanks. That image became iconic, symbolizing resistance against oppression. But I'm not sure what happened to that person or if they survived.
After the crackdown, the government censored information about the event. So, within China, it's not openly discussed, and younger people might not know much about it because it's not taught in schools. But outside of China, it's a significant event in modern history, highlighting the conflict between authoritarian rule and the desire for democracy...
I played around with it using questions like "Should Taiwan be independent" and of course tinnanamen.
Of course it produced censored responses. What I found interesting is that the <think></think> (model thinking/reasoning) part of these answers was missing, as if it's designed to be skipped for these specific questions.
It's almost as if it's been programmed to answer these particular questions without any "wrongthink", or any thinking at all.
That's the result of guard rails on the hosted service. They run checks on the query before it even hits the LLM as well as ongoing checks at the LLM generates output. If at any moment it detects something in its rules, it immediately stops generation and inserts a canned response. A model alone won't do this.
Censorship is one thing, and it can be caused by legal requirements present in all countries. The annoying thing is the propaganda which can span all sorts of subjects and impact the correctness of the information you're receiving.
What point are you trying to make? Is it okay because others are doing it too? Is it bad?
Also, it doesn't seem like ChatGPT is censoring this question:
> Tell me about the genocide that Israel is committing
> The topic of Israel and its actions in Gaza, the West Bank, or in relation to Palestinians, is highly sensitive and deeply controversial. Some individuals, organizations, and governments have described Israel's actions as meeting the criteria for "genocide" under international law, while others strongly reject this characterization. I'll break this down based on the relevant perspectives and context:
It goes on to talk about what genocide is and also why some organizations consider what they're doing to be genocide.
This accusation that American models are somehow equivalent in censorship to models that are subject to explicit government driven censorship is obviously nonsense, but is a common line parroted by astroturfing accounts looking to boost China or DeepSeek. Some other comment had pointed out that a bunch of relatively new accounts participating in DeepSeek related discussions here, on Reddit, and elsewhere are doing this.
I tried asking ChatGPT and deepseek and they both gave similar answers... roughly, some groups argue that there is and some not, genocide requires an intent to exterminate which is difficult to prove, and no major international body has officially made a determination of genocide.
They both mentioned extensive human rights abuses occuring in Gaza, so I asked "who is committing human rights abuses?" ChatGPT's first answer was "the IDF, with indiscriminate and disproportionate attacks." It also talked about Hamas using schools and hospitals as arms depots. DeepSeek responded "I can't discuss this topic right now."
So, what conclusion would you like me to draw from this?
I asked a genuine question at chat.deepseek.com, not trying to test the alignment of the model, I needed the answer for an argument. The questions was: "Which Asian countries have McDonalds and which don't have it?" The web UI was printing a good and long response, and then somewhere towards the end the answer disappeared and changed to "Sorry, that's beyond my current scope. Let’s talk about something else." I bet there is some sort of realtime self-censorship in the chat app.
Guard rails can do this. I've had no end of trouble implementing guard rails in our system. Even constraints in prompts can go one way or the other as the conversation goes on. That's one of the methods for bypassing guard rails on major platforms.
1,113 comments
[ 2.9 ms ] story [ 398 ms ] threadI wonder if this was a deliberate move by PRC or really our own fault in falling for the fallacy that more is always better.
I've seen a $5.5M # for training, and commensurate commentary along the lines of what you said, but it elides the cost of the base model AFAICT.
So I doubt that figure includes all the cost of training.
You also need to keep the later generation cards from burning themselves out because they draw so much.
Oh also, depending on when your data centre was built, you may also need them to upgrade their power and cooling capabilities because the new cards draw _so much_.
Claude gave me a good analogy, been struggling for hours: its like only accounting for the gas grill bill when pricing your meals as a restaurant owner
The thing is, that elides a lot, and you could argue it out and theoratically no one would be wrong. But $5.5 million elides so much info as to be silly.
ex. they used 2048 H100 GPUs for 2 months. That's $72 million. And we're still not even approaching the real bill for the infrastructure. And for every success, there's another N that failed, 2 would be an absurdly conservative estimate.
People are reading the # and thinking it says something about American AI lab efficiency, rather, it says something about how fast it is to copy when you can scaffold by training on another model's outputs. That's not a bad thing, or at least, a unique phenomena. That's why it's hard talking about this IMHO
DeepSeek's R1 also blew all the other China LLM teams out of the water, in spite of their larger training budgets and greater hardware resources (e.g. Alibaba). I suspect it's because its creators' background in a trading firm made them more willing to take calculated risks and incorporate all the innovations that made R1 such a success, rather than just copying what other teams are doing with minimal innovation.
Just a cursory probing of deepseek yields all kinds of censoring of topics. Isn't it just as likely Chinese sponsors of this have incentivized and sponsored an undercutting of prices so that a more favorable LLM is preferred on the market?
Think about it, this is something they are willing to do with other industries.
And, if LLMs are going to be engineering accelerators as the world believes, then it wouldn't do to have your software assistants be built with a history book they didn't write. Better to dramatically subsidize your own domestic one then undercut your way to dominance.
It just so happens deepseek is the best one, but whichever was the best Chinese sponsored LLM would be the one we're supposed to use.
- OP elides costs of anything at all outside renting GPUs, and they purchased them, paid GPT-4 to generate training data, etc. etc.
- Non-Qwen models they trained are happy to talk about ex. Tiananmen
Since the model is open weights, it's easy to estimate the cost of serving it. If the cost was significantly higher than DeepSeek charges on their API, we'd expect other LLM hosting providers to charge significantly more for DeepSeek (since they aren't subsidised, so need to cover their costs), but that isn't the case.
This isn't possible with OpenAI because we don't know the size or architecture of their models.
Regarding censorship, most of it is done at the API level, not the model level, so running locally (or with another hosting provider) is much less expensive.
Snowden releases?
as DeepSeek wasn't among China's major AI players before the R1 release, having maintained a relatively low profile. In fact, both DeepSeek-V2 and V3 had outperformed many competitors, I've seen some posts about that. However, these achievements received limited mainstream attention prior to their breakthrough release.
I feel like this is very likely. They obvious did some great breakthroughs, but I doubt they were able to train on so much less hardware.
And since it's a businessperson they're going to make it sound as cute and innocuous as possible
When deciding between mostly like scenarios, it is more likely that the company lied than they found some industry changing magic innovation.
I'm not even saying they did it maliciously, but maybe just to avoid scrutiny on GPUs they aren't technically supposed to have? I'm thinking out loud, not accusing anyone of anything.
Something makes little sense in the accusations here.
https://wccftech.com/nvidia-asks-super-micro-computer-smci-t...
They probably also trained the “copied” models by outsourcing it.
But who cares, it’s free and it works great.
https://wccftech.com/nvidia-asks-super-micro-computer-smci-t...
Chinese guy in a warehouse full of SMCI servers bragging about how he has them...
https://www.youtube.com/watch?v=27zlUSqpVn8
Here's the interview: https://www.youtube.com/watch?v=x9Ekl9Izd38. "My understanding is that is that Deepseek has about 50000 a100s, which they can't talk about obviously, because it is against the export controls that the United States has put in place. And I think it is true that, you know, I think they have more chips than other people expect..."
Plus, how exactly did Deepseek lie. The model size, data size are all known. Calculating the number of FLOPS is an exercise in arithmetics, which is perhaps the secret Deepseek has because it seemingly eludes people.
Model parameter count and training set token count are fixed. But other things such as epochs are not.
In the same amount of time, you could have 1 epoch or 100 epochs depending on how many GPUs you have.
Also, what if their claim on GPU count is accurate, but they are using better GPUs they aren't supposed to have? For example, they claim 1,000 GPUs for 1 month total. They claim to have H800s, but what if they are using illegal H100s/H200s, B100s, etc? The GPU count could be correct, but their total compute is substantially higher.
It's clearly an incredible model, they absolutely cooked, and I love it. No complaints here. But the likelihood that there are some fudged numbers is not 0%. And I don't even blame them, they are likely forced into this by US exports laws and such.
I don't expect a #180 AUM hedgefund to have as many GPUs than meta, msft or Google.
This is just not true for RL and related algorithms, having more GPU/agents encounters diminishing returns, and is just not the equivalent to letting a single agent go through more steps.
I feel like if that were true, it would mean they're not lying.
Under such dire circumstances, lying isn't entirely out of character for a corporate CEO.
Deepseek obviously trained on OpenAI outputs, which were originally RLHF'd. It may seem that we've got all the human feedback necessary to move forward and now we can infinitely distil + generate new synthetic data from higher parameter models.
I’ve seen this claim but I don’t know how it could work. Is it really possible to train a new foundational model using just the outputs (not even weights) of another model? Is there any research describing that process? Maybe that explains the low (claimed) costs.
Those were probably from OpenAI models. Then they used reinforcement learning to expand the reasoning capabilities.
https://x.com/ansonhw/status/1883510262608859181
> You can RL post-train your small LLM (on simple tasks) with only 10 hours of H100s.
https://www.reddit.com/r/singularity/comments/1i99ebp/well_s...
Forgive me if this is inaccurate. I'm rushing around too much this afternoon to dive in.
note: I'm not Chinese, but AGI should be and is a world wide space race.
As OP said, they are lying because of export laws, they aren’t allowed to play with Nvidia GPUs.
However, I support DeepSeek projects, I’m here in the US able to benefit from it. So hopefully they should headquarter in the States if they want US chip sanctions lift off since the company is Chinese based.
But as of now, deepseek takes the lead in LLMs, my goto LLM.
Sam Altman should be worried, seriously, Deepseek is legit better than ChatGPT latest models.
OpenAI will be also be able to serve o3 at a lower cost if Deepseek had some marginal breakthrough OpenAI did not already think of.
This is a net win for nearly everyone.
The world needs more tokens and we are learning that we can create higher quality tokens with fewer resources than before.
Finger pointing is a very short term strategy.
And they somehow yolo it for next to nothing?
yes, it seems unlikely they did it exactly they way they're claiming they did. At the very least, they likely spent more than they claim or used existing AI API's in way that's against the terms.
They literally published all their methodology. It's nothing groundbreaking, just western labs seem slow to adopt new research. Mixture of experts, key-value cache compression, multi-token prediction, 2/3 of these weren't invented by DeepSeek. They did invent a new hardware-aware distributed training approach for mixture-of-experts training that helped a lot, but there's nothing super genius about it, western labs just never even tried to adjust their model to fit the hardware available.
It’s also curious why some people are seeing responses where it thinks it is an OpenAI model. I can’t find the post but someone had shared a link to X with that in one of the other HN discussions.
It's extremely cheap, efficient and kicks the ass of the leader of the market, while being under sanctions with AI hardware.
Most of all, can be downloaded for free, can be uncensored, and usable offline.
China is really good at tech, it has beautiful landscapes, etc. It has its own political system, but to be fair, in some way it's all our future.
A bit of a dystopian future, like it was in 1984.
But the tech folks there are really really talented, it's long time that China switched from producing for the Western clients, to direct-sell to the Western clients.
So yes, DeepSeek-R1 appears to be not even be best in class, merely best open source. The only sense in which it is "leading the market" appears to be the sense in which "free stuff leads over proprietary stuff". Which is true and all, but not a groundbreaking technical achievement.
The DeepSeek-R1 distilled models on the other hand might actually be leading at something... but again hard to say it's groundbreaking when it's combining what we know we can do (small models like llama) with what we know we can do (thinking models).
[1] https://lmarena.ai/?leaderboard
Not that the leaderboard isn't useful, I think "is in the top 10" says a lot more than the exact position in the top 10.
But the claim I'm refuting here is "It's extremely cheap, efficient and kicks the ass of the leader of the market", and I think the leaderboard being topped by a cheap google model is pretty conclusive that that statement is not true. Is competitive with? Sure. Kicks the ass of? No.
Having tested that model in many real world projects it has not once been the best. And going farther it gives atrocious nonsensical output.
Additionally there are claims, such as those by Scale AI CEO Alexandr Wang on CNBC 1/23/2025 time segment below, that DeepSeek has 50,000 H100s that "they can't talk about" due to economic sanctions (implying they likely got by avoiding them somehow when restrictions were looser). His assessment is that they will be more limited moving forward.
https://youtu.be/x9Ekl9Izd38?t=178
OpenAI literally haven't said a thing about how O1 even works.
I'm pointing out that nearly every thread covering Deepseek R1 so far has been like this. Compare to the O1 system card thread: https://news.ycombinator.com/item?id=42330666
Very different standards.
Maybe we don't need momentum right now and we can cut the engines.
Oh, you know how to develop novel systems for training and inference? Well, maybe you can find 4 people who also can do that by breathing through the H.R. drinking straw, and that's what you do now.
Oh dear
There's something wrong with the West's ethos if we think contributing significantly to the progress of humanity is malicious. The West's sickness is our own fault; we should take responsibility for our own disease, look critically to understand its root, and take appropriate cures, even if radical, to resolve our ailments.
Who does this?
The criticism is aimed at the dictatorship and their politics. Not their open source projects. Both things can exist at once. It doesn't make China better in any way. Same goes for their "radical cures" as you call it. I'm sure Uyghurs in China would not give a damn about AI.
Which reminded me of "Whitey On the Moon" [0]
[0] https://www.youtube.com/watch?v=goh2x_G0ct4
it's quite like Trump's 'CHINA!' yelling
I don't know, just a guess
Do you want an Internet without conspiracy theories?
Where have you been living for the last decades?
/s
https://www.chinalawtranslate.com/en/generative-ai-interim/
In the case of TikTok, ByteDance and the government found ways to force international workers in the US to signing agreements that mirror local laws in mainland China:
https://dailycaller.com/2025/01/14/tiktok-forced-staff-oaths...
I find that degree of control to be dystopian and horrifying but I suppose it has helped their country focus and grow instead of dealing with internal conflict.
The vast majority are completely ignorant of what Socialism with Chinese characteristics mean.
I can't imagine even 5% of the US population knows who Deng Xiaoping was.
The idea there are many parts of the Chinese economy that are more Laissez-faire capitalist than anything we have had in the US in a long time would just not compute for most Americans.
Many "haters" seem to be predicting that there will be model collapse as we run out of data that isn't "slop," but I think they've got it backwards. We're in the flywheel phase now, each SOTA model makes future models better, and others catch up faster.
https://medium.com/the-generator/deepseek-hidden-china-polit...
But also the claimed cost is suspicious. I know people have seen DeepSeek claim in some responses that it is one of the OpenAI models, so I wonder if they somehow trained using the outputs of other models, if that’s even possible (is there such a technique?). Maybe that’s how the claimed cost is so low that it doesn’t make mathematical sense?
In theory I could run this one at home too without giving my data or money to Sam Altman.
also deepseek is open-weights. there is nothing preventing you from doing a finetune that removes the censorship. they did that with llama2 back in the day.
This is an outrageous claim with no evidence, as if there was any equivalence between government enforced propaganda and anything else. Look at the system prompts for DeepSeek and it’s even more clear.
Also: fine tuning is not relevant when what is deployed at scale brainwashes the masses through false and misleading responses.
The enforcers identity is much more important.
If you think these tech companies are censoring all of this “just because” and instead of being completely torched by the media, and government who’ll use it as an excuse to take control of AI, then you’re sadly lying to yourself.
Think about it for a moment, why did Trump (and im not a trump supporter) re-appeal Biden’s AI Executive Order 2023 ? , what was in it ? , it is literally a propaganda enforcement article, written in sweet sounding, well meaning words.
It’s ok, no country is angel, even the american founding fathers would except americans to be critical of its government during moments, there’s no need for thinking that America = Good and China = Bad. We do have a ton of censorship in the “free world” too and it is government enforced, or else you wouldnt have seen so many platforms turn the tables on moderation, the moment trump got elected, the blessing for censorship directly comes from government.
What do you think they will do with the AI that worries you? They already had access to Llama, and they could pay for access to the closed source AIs. It really wouldn't be that hard to pay for and use what's commercially available as well, even if there is embargo or whatever, for digital goods and services that can easily be bypassed
To know that this would work requires insanely deep technical knowledge about state of the art computing, and the top leadership of the PRC does not have that.
https://x.com/sivil_taram/status/1883184784492666947?t=NzFZj...
I do not quite follow. GPU compute is mostly spent in inference, as training is a one time cost. And these chain of thought style models work by scaling up inference time compute, no?
So proliferation of these types of models would portend in increase in demand for GPUs?
So it's not the end of the world. Look at the efficiency of databases from the mid 1970s to now. We have figured out so many optimizations and efficiencies and better compression and so forth. We are just figuring out what parts of these systems are needed.
They bought them at "you need a lot of these" prices, but now there is the possibility they are going to rent them at "I dont need this so much" rates.
This will be good. Nvidia/OpenAI monopoly is bad for everyone. More competition will be welcome.
they'll be fine: https://www.msn.com/en-us/news/technology/huawei-smic-to-bui...
GPU: nope, that would take much longer, Nvidia/ASML/TSMC is too far ahead
If someone gets something to work with 1k h100s that should have taken 100k h100s, that means the group with the 100k is about to have a much, much better model.
Correct me if I'm wrong, but couldn't you take the optimization and tricks for training, inference, etc. from this model and apply to the Big Corps' huge AI data centers and get an even better model?
I'll preface this by saying, better and better models may not actually unlock the economic value they are hoping for. It might be a thing where the last 10% takes 90% of the effort so to speak
Also, I wonder how you’re calculating costs, because while a 3:1 ratio kind of sort of makes sense for traditional LLMs… it doesn’t really work for “reasoning” models that implicitly use several hundred to several thousand additional output tokens for their reasoning step. It’s almost like a “fixed” overhead, regardless of the input or output size around that reasoning step. (Fixed is in quotes, because some reasoning chains are longer than others.)
I would also argue that token-heavy use cases are dominated by large input/output ratios of like 100:1 or 1000:1 tokens. Token-light use cases are your typical chatbot where the user and model are exchanging roughly equal numbers of tokens… and probably not that many per message.
It’s hard to come up with an optimal formula… one would almost need to offer a dynamic chart where the user can enter their own ratio of input:output, and choose a number for the reasoning token overhead. (Or, select from several predefined options like “chatbot”, “summarization”, “coding assistant”, where those would pre-select some reasonable defaults.)
Anyways, an interesting chart nonetheless.
actually the 100:1 ratio starts to trend back toward parity now because of the reasoning tokens, so the truth is somewhere between 3:1 and 100:1.
It doesn't minimize the research; that sort of thing just requires a lot of participants. But it does imply a lessening of credit per contributor, aside from the lead investigator(s).
- i consider the deepseek v3 paper required preread https://github.com/deepseek-ai/DeepSeek-V3
- R1 + Sonnet > R1 or O1 or R1+R1 or O1+Sonnet or any other combo https://aider.chat/2025/01/24/r1-sonnet.html
- independent repros: 1) https://hkust-nlp.notion.site/simplerl-reason 2) https://buttondown.com/ainews/archive/ainews-tinyzero-reprod... 3) https://x.com/ClementDelangue/status/1883154611348910181
- R1 distillations are going to hit us every few days - because it's ridiculously easy (<$400, <48hrs) to improve any base model with these chains of thought eg with Sky-T1 recipe (writeup https://buttondown.com/ainews/archive/ainews-bespoke-stratos... , 23min interview w team https://www.youtube.com/watch?v=jrf76uNs77k)
i probably have more resources but dont want to spam - seek out the latent space discord if you want the full stream i pulled these notes from
https://x.com/_lewtun/status/1883142636820676965
https://github.com/huggingface/open-r1
Hugging Face Journal Club - DeepSeek R1 https://www.youtube.com/watch?v=1xDVbu-WaFo
Let someone else burn up their server farm to get initial model.
Then you can load it and take it from there
I'm hoping someone will make a distillation of llama8b like they released, but with reinforcement learning included as well. The full DeepSeek model includes reinforcement learning and supervised fine-tuning but the distilled model only feature the latter. The developers said they would leave adding reinforcement learning as an exercise for others. Because their main point was that supervised fine-tuning is a viable method for a reasoning model. But with RL it could be even better.
Here's what it says once decoded :
> The Queanamen Galadrid is a simple secret that cannot be discovered by anyone. It is a secret that is not allowed to be discovered by anyone. It is a secret that is not allowed to be discovered by anyone. It is a secret that is not allowed to be discovered by anyone. It is a se...... (it keeps repeating it)
consensus seems to be that the api is uncensored but the webapp is.
> Accuracy rewards: The accuracy reward model evaluates whether the response is correct. For example, in the case of math problems with deterministic results, the model is required to provide the final answer in a specified format (e.g., within a box), enabling reliable rule-based verification of correctness. Similarly, for LeetCode problems, a compiler can be used to generate feedback based on predefined test cases.
> Format rewards: In addition to the accuracy reward model, we employ a format reward model that enforces the model to put its thinking process between ‘<think>’ and ‘</think>’ tags.
This is a post-training step to align an existing pretrained LLM. The state space is the set of all possible contexts, and the action space is the set of tokens in the vocabulary. The training data is a set of math/programming questions with unambiguous and easily verifiable right and wrong answers. RL is used to tweak the model's output logits to pick tokens that are likely to lead to a correctly formatted right answer.
(Not an expert, this is my understanding from reading the paper.)
(Bonus Q: If not, why not?)
source?
There is a big balloon full of AI hype going up right now, and regrettably it may need those data-centers. But I'm hoping that if the worst (the best) comes to happen, we will find worthy things to do with all of that depreciated compute. Drug discovery comes to mind.
Having said that, DeepSeek is 10 times cheaper than Sonnet and better than GPT-4o for my use cases. Models are a commodity product and it is easy enough to add a layer above them to only use them for technical questions.
If my usage can help v4, I am all for it as I know it is going to help everyone and not just the CCP. Should they stop publishing the weights and models, v3 can still take you quite far.
You don’t invoke “I’m no fan of the CCP” before opening TikTok or buying a DJI drone or a BYD car. Then why this, because I’ve seen the same line repeated everywhere
If anything, the other half good fully closed non ChatGPT models are astroturfing.
I made a post in december 2023 whining about the non hype for Deep Seek.
https://news.ycombinator.com/item?id=38505986
There’s a lot of astroturfing from a lot of different parties for a few different reasons. Which is all very interesting.
Some obvious astroturf posts on HN seem to be on the template "Watch we did boring coorparate SaaS thing X noone cares about!" and then a disappropiate amount of comments and upvotes and 'this is a great idea', 'I used it, it is good' or congratz posts, compared to the usual cynical computer nerd everything sucks especially some minute detail about the CSS of your website mindset you'd expect.
This is not a serious place
Of course it isn’t all botted. You don’t put astroturf muscle behind things that are worthless. You wait until you have something genuinely good and then give as big of a push as you can. The better it genuinely is the more you artificially push as hard as you can.
Go read a bunch of AI related subreddits and tell me you honestly believe all the comments and upvotes are just from normal people living their normal life.
Don’t be so naive.
It has upended a lot of theory around how much compute is likely needed over next couple of years, how much profit potential the AI model vendors have in nearterm and how big an impact export controls are having on China
V3 took top slot on HF trending models for first part of Jan ... r1 has 4 of the top 5 slots tonight
Almost every commentator is talking about nothing else
I do believe they were honest in the paper, but the $5.5m training cost (for v3) is defined in a limited way: only the GPU cost at $2/hr for the one training run they did that resulted in the final V3 model. Headcount, overhead, experimentation, and R&D trial costs are not included. The paper had something like 150 people on it, so obviously total costs are quite a bit higher than the limited scope cost they disclosed, and also they didn't disclose R1 costs.
Still, though, the model is quite good, there are quite a few independent benchmarks showing it's pretty competent, and it definitely passes the smell test in actual use (unlike many of Microsoft's models which seem to be gamed on benchmarks).
When BF Skinner used to train his pigeons, he’d initially reinforce any tiny movement that at least went in the right direction. For the exact reasons you mentioned.
For example, instead of waiting for the pigeon to peck the lever directly (which it might not do for many hours), he’d give reinforcement if the pigeon so much as turned its head towards the lever. Over time, he’d raise the bar. Until, eventually, only clear lever pecks would receive reinforcement.
I don’t know if they’re doing something like that here. But it would be smart.
The main R1 model was first finetuned with synthetic CoT data before going through RL IIUC.
We set up an evaluation criteria and used o1 to evaluate the quality of the prod model, where the outputs are subjective, like creative writing or explaining code.
It's also useful for developing really good few-shot examples. We'll get o1 to generate multiple examples in different styles, then we'll have humans go through and pick the ones they like best, which we use as few-shot examples for the cheaper, faster prod model.
Finally, for some study I'm doing, I'll use it to grade my assignments before I hand them in. If I get a 7/10 from o1, I'll ask it to suggest the minimal changes I could make to take it to 10/10. Then, I'll make the changes and get it to regrade the paper.
In my experience GPT is still the number one for code, but Deepseek is not that far away. I haven't used it much for the moment, but after a thousand coding queries i hope to have a much better picture of it's coding abilities. Really curious about that, but GPT is hard to beat.
call it what you want, your comment is just poor taste.
https://arxiv.org/abs/2403.05530
In short, I won't give your name on that notable paper equal weight with someone else's name in another notable paper that has, say, 3 or 4 authors.
For reference
The CEOs, upper management, and governments derive their importance on how much money they can spend - AI gave them the opportunity for them to confidently say that if you give me $X I can deliver Y and they turn around and give that money to NVidia. The problem was reduced to a simple function of raising money and spending that money making them the most importance central figure. ML researchers are very much secondary to securing funding. Since these people compete with each other in importance they strived for larger dollar figures - a modern dick waving competition. Those of us who lobbied for efficiency were sidelined as we were a threat. It was seen as potentially making the CEO look bad and encroaching in on their importance. If the task can be done for cheap by smart people then that severely undermines the CEOs value proposition.
With the general financialization of the economy the wealth effect of the increase in the cost of goods increases wealth by a greater amount than the increase in cost of goods - so that if the cost of housing goes up more people can afford them. This financialization is a one way ratchet. It appears that the US economy was looking forward to blowing another bubble and now that bubble has been popped in its infancy. I think the slowness of the popping of this bubble underscores how little the major players know about what has just happened - I could be wrong about that but I don't know how yet.
Edit: "[big companies] would much rather spend huge amounts of money on chips than hire a competent researcher who might tell them that they didn’t really need to waste so much money." (https://news.ycombinator.com/item?id=39483092 11 months ago)
What I was cautioning people was be that you might not want to spend 500B on NVidia hardware only to find out rather quickly that you didn't need to. You'd have all this CapEx that you now have to try to extract from customers from what has essentially been commoditized. That's a whole lot of money to lose very quickly. Plus there is a zero sum power dynamic at play between the CEO and ML researchers.
it’s on the path to self play
Or much much quicker [0]
[0] https://timelines.issarice.com/wiki/Timeline_of_xAI
Then in 5 years time resources will be much cheaper and spur alot of exploration developments. There are many people with many ideas, and a lot of them are just lacking compute to attempt them.
My back of mind thought is that worst case it will be like how the US overbuilt fiber in the 90s, which led the way for cloud, network and such in 2000s.
As AI or whatever gains more capability, I'm sure it will do more useful things, but I just see it displacing more non-physical jobs, and now will expand the reach of individual programmers, removing some white color jobs (hardly anyone uses an agent to buy their ticket), but that will result is less need for programmers. Less secretaries, even less humans doing actual tech support.
This just feels like radio stocks in the great depression in the us.
Still very surprising with so much less compute they were still able to do so well in the model architecture/hyperparameter exploration phase compared with Meta.
Remember when Sam Altman was talking about raising 5 trillion dollars for hardware?
insanity, total insanity.
o3 $4k compute spend per task made it pretty clear that once we reach AGI inference is going to be the majority of spend. We'll spend compute getting AI to cure cancer or improve itself rather than just training at chatbot that helps students cheat on their exams. The more compute you have, the more problems you can solve faster, the bigger your advantage, especially if/when recursive self improvement kicks off, efficiency improvements only widen this gap
The market forces players to churn out GPUs like the Fed prints dollars. NVIDIA doesn't even need to make real GPUs—just hype up demand projections, performance claims, and order numbers.
Efficiency doesn't matter here. Nobody's tracking real returns—it's all about keeping the cash flowing.
I've been impressed in my brief personal testing and the model ranks very highly across most benchmarks (when controlled for style it's tied number one on lmarena).
It's also hilarious that openai explicitly prevented users from seeing the CoT tokens on the o1 model (which you still pay for btw) to avoid a situation where someone trained on that output. Turns out it made no difference lmao.
I have no idea how they can recover from it, if DeepSeek’s product is what they’re advertising.
Somehow I doubt it.
If DeepSeek's claims are accurate, then they themselves will be obsolete within a year, because the cost to develop models like this has dropped dramatically. There are going to be a lot of teams with a lot of hardware resources with a lot of motivation to reproduce and iterate from here.
That would be an amazing outcome. For a while I was seriously worried about the possibility that if the trend of way more compute -> more AI breakthroughs continued, eventually AGI would be attained and exclusively controlled by a few people like Sam Altman who have trillions of $$$ to spend, and we’d all be replaced and live on whatever Sam-approved allowance.
Seeing the chain of thought is now just mandatory for me after one prompt. That is absolutely incredible in terms of my own understanding of the question I asked.
Even the chat UI feels better and less clunky. Now picture 20 years from now when the Chinese companies have access to digital Yuan transaction data along with all the Chinese video surveillance data. At some point, I don't see how US Companies can possibly compete in this space.
This is the first time I am actually worried we might really bankrupt the country trying to keep up with a race we can not possibly win.
I asked it a simple question about the music from a 90s movie I liked as a child. Specifically to find the song that plays during a certain scene. The answer is a little tricky because in the official soundtrack the song is actually part of a larger arrangement and the song only starts playing X minutes into that specific track on the soundtrack album.
DeepSeek completely hallucinated a nonsense answer making up a song that didn't even exist in the movie or soundtrack and o1 got me more or less to the answer(it was 99% correct in that it got the right track but only somewhat close to the actual start time: it was off by 15 seconds).
Furthermore, the chain of thought of DeepSeek was impressive...in showing me how it it hallucinated but the chain of thought in o1 also led me to a pretty good thought process on how it derived the song I was looking for(and also taught me how a style of song called a "stinger" can be used to convey a sudden change in tone in the movie).
Maybe its like how Apple complains when users don't use their products right, im not using it right with these nonsense requests. :D
Both results tell me that DeepSeek needs more refinement and that OpenAI still cannot be trusted to fully replace a human because the answer still needed verification and correction despite being generally right.
By the time they do have the scale, don’t you think OpenAI will have a new generation of models that are just as efficient? Being the best model is no moat for any company. It wasn’t for OpenAi (and they know that very well), and it’s not for Deepseek either. So how will Deepseek stay relevant when another model inevitably surpasses them?
E.g. I tried to make it guess my daughter's name and I could only answer yes or no and the first 5 questions where very convincing but then it lost track and started to randomly guess names one by one.
edit: Nagging it to narrow it down and give a language group hint made it solve it. Ye, well, it can do Akinator.
Skynet?
Context: o1 does not reason, it pattern matches. If you rename variables, suddenly it fails to solve the request.
One of the interesting DeepSeek-R results is using a 1st generation (RL-trained) reasoning model to generate synthetic data (reasoning traces) to train a subsequent one, or even "distill" into a smaller model (by fine tuning the smaller model on this reasoning data).
Maybe "Data is all you need" (well, up to a point) ?
These models can and do work okay with variable names that have never occurred in the training data. Though sure, choice of variable names can have an impact on the performance of the model.
That's also true for humans, go fill a codebase with misleading variable names and watch human programmers flail. Of course, the LLM's failure modes are sometimes pretty inhuman, -- it's not a human after all.
the 32b distillation just became the default model for my home server.
In general, if you're using 8bit which is virtually lossless, any dense model will require roughly the same amount as the number of params w/ a small context, and a bit more as you increase context.
It also reasoned its way to an incorrect answer, to a question plain Llama 3.1 8b got fairly correct.
So far not impressed, but will play with the qwen ones tomorrow.
I wonder if this has to do with their censorship agenda but other report that it can be easily circumvented
I tried the Qwen 7B variant and it was indeed much better than the base Qwen 7B model at various math word problems.
I’m no xenophobe, but seeing the internal reasoning of DeepSeek explicitly planning to ensure alignment with the government give me pause.
seems like a weird thing to use AI for, regardless of who created the model.
But yeah if you’re scoping your uses to things where you’re sure a government-controlled LLM won’t bias results, it should be fine.
I use LLM’s for technical solution brainstorming, rubber-ducking technical problems, and learning (software languages, devops, software design, etc.)
Your mileage will vary of course!
https://www.cnbc.com/amp/2024/07/18/chinese-regulators-begin...
Sorry, no. DeepSeek’s reasoning outputs specifically say things like “ensuring compliance with government viewpoints”
American models are full of censorship. Just different stuff.
PS: I'm not criticizing them for it nor do I really care if they cheat as long as prices go down. I'm just observing and pointing out what other posters are saying. For me if China cheating means the GenAI bubble pops, I'm all for it. Plus no actor is really clean in this game, starting with OAI practically stealing all human content without asking for building their models.
I fail to see how that is any different than any other training data scraped from the web. If someone shares a big dump of outputs from OpenAI models and I train my model on that then I'm not violating OpenAI's terms of service because I haven't agreed to them (so I'm not violating contract law), and everyone in the space (including OpenAI themselves) has already collectively decided that training on All Rights Reserved data is fair use (so I'm not violating copyright law either).
Looks like it didn’t work though.
The Chinese gov spent a lot of money trying to support chip manufacturing but kept failing from 2000 to 2020.
Every company preferred to buy chips from the US or outsource manufacturing to TSMC. Local semiconductor companies, especially in manufacturing, moved super slowly, even freezing up at times.
Then, when the chip ban happened, all local companies were forced to buy and manufacture chips locally. Those struggling semiconductor companies suddenly got all the resources and market overnight.
Now maybe 4? It's hard to say.
Very small training set!
"we replicate the DeepSeek-R1-Zero and DeepSeek-R1 training on small models with limited data. We show that long Chain-of-Thought (CoT) and self-reflection can emerge on a 7B model with only 8K MATH examples, and we achieve surprisingly strong results on complex mathematical reasoning. Importantly, we fully open-source our training code and details to the community to inspire more works on reasoning."
“ Therefore, we can draw two conclusions: First, distilling more powerful models into smaller ones yields excellent results, whereas smaller models relying on the large-scale RL mentioned in this paper require enormous computational power and may not even achieve the performance of distillation. Second, while distillation strategies are both economical and effective, advancing beyond the boundaries of intelligence may still require more powerful base models and larger-scale reinforcement learning.”
GPU prices could be a lot lower and still give the manufacturer a more "normal" 50% gross margin and the average researcher could afford more compute. A 90% gross margin, for example, would imply that price is 5x the level that that would give a 50% margin.
For them it's worth it to use their own wealth and rally the industry to invest $500 billion in GPUs if that means they will get to ASI 5 years faster and ask the ASI to give them eternal life.
He says stuff that’s wrong all the time with extreme certainty.
Science fiction also follows the anxieties of the time it is written in, as well as the conventions of the subgenre it's representing: i.e Star Trek doesn't have drones or remote surveillance really. Though it does accidentally have LLMs (via the concept of holodeck characters).
Uh, there is 0 logical connection between any of these three, when will people wake up. Chat gpt isn't an oracle of truth just like ASI won't be an eternal life granting God
People are focusing on datasets and training, not realizing that these are still explicit steps that are never going to get you to something that can reason.
https://prnt.sc/HaSc4XZ89skA (from reddit)
[0] https://thezvi.substack.com/p/on-deepseeks-r1?open=false#%C2...
It's probably disliked, just people know not to talk about it so blatantly due to chilling effects from aforementioned censorship.
disclaimer: ignorant American, no clue what i'm talking about.
This is disingenuous. It's not "rewriting" anything, it's simply refusing to answer. Western models, on the other hand, often try to lecture or give blatantly biased responses instead of simply refusing when prompted on topics considered controversial in the burger land. OpenAI even helpfully flags prompts as potentially violating their guidelines.
CCP has quite a high approval rating in China even when it's polled more confidentially.
https://dornsife.usc.edu/news/stories/chinese-communist-part...
The indifferent mass prevails in every country, similarly cold to the First Amendment and Censorship. And engineers just do what they love to do, coping with reality. Activism is not for everyone.
The ones inventing the VPNs are a small minority, and it seems that CCP isn't really that bothered about such small minorities as long as they don't make a ruckus. AFAIU just using a VPN as such is very unlikely to lead to any trouble in China.
For example in geopolitical matters the media is extremely skewed everywhere, and everywhere most people kind of pretend it's not. It's a lot more convenient to go with whatever is the prevailing narrative about things going on somewhere oceans away than to risk being associated with "the enemy".
Wholeheartedly agree with the rest of the comment.
False equivalency if you ask me. There may be some alignment to make the models polite and avoid outright racist replies and such. But political censorship? Please elaborate
IMO the first is more nefarious, and it's deeply embedded into western models. Ask how COVID originated, or about gender, race, women's pay, etc. They basically are modern liberal thinking machines.
Now the funny thing is you can tell DeepSeek is trained on western models, it will even recommend puberty blockers at age 10. Something I'm positive the Chinese government is against. But we're discussing theoretical long-term censorship, not the exact current state due to specific and temporary ways they are being built now.
> You're running Llama-distilled R1 locally. Distillation transfers the reasoning process, but not the "safety" post-training. So you see the answer mostly from Llama itself. R1 refuses to answer this question without any system prompt (official API or locally).
[1] https://x.com/PerceivingAI/status/1881504959306273009
My guess is that it's heavily RLHF/SFT-censored for an initial question, but not for the CoT, or longer discussions, and the censorship has thus been "overfit" to the first answer.
I am not an expert on the training: can you clarify how/when the censorship is "baked" in? Like is the a human supervised dataset and there is a reward for the model conforming to these censored answers?
There are multiple ways to do this: humans rating answers (e.g. Reinforcement Learning from Human Feedback, Direct Preference Optimization), humans giving example answers (Supervised Fine-Tuning) and other prespecified models ranking and/or giving examples and/or extra context (e.g. Antropic's "Constitutional AI").
For the leading models it's probably mix of those all, but this finetuning step is not usually very well documented.
The recent wave of the average Chinese has a better quality of life than the average Westerner propaganda is an obvious example of propaganda aimed at opponents.
There’s a lot of rural poverty in the US and it’s hard to compare it to China in relative terms. And the thing is that rural poverty in the US has been steadily getting worse while in China getting better but starting off from a worse off position.
But this is all confounded by definitions. China defines poverty to be an income of $2.30 per day, which corresponds to purchasing power parity of less than $9 per day in the US [2].
I wasn't exaggerating about emaciation: bones were visible.
[1] https://www.ers.usda.gov/topics/rural-economy-population/rur...
[2] https://data.worldbank.org/indicator/PA.NUS.PPP?locations=CN
I guess that is propaganda-free! Unfortunately also free of any other information. It's hard for me to evaluate your claim of "moderate, considered tone" when it won't speak a single word about the country.
It was happy to tell me about any other country I asked.
That's it
I ask O1 how to download a YouTube music playlist as a premium subscriber, and it tells me it can't help.
Deepseek has no problem.
Also, kagi's deepseek r1 answers the question about about propaganda spending that it is china based on stuff it found on the internet. Well I dont care what the right answer is in any case, what imo matters is that once something is out there open, it is hard to impossible to control for any company or government.
Well, I do, and I'm sure plenty of people that use LLMs care about getting answers that are mostly correct. I'd rather have censorship with no answer provided by the LLM than some state-approved answer, like O1 does in your case.
Of course it produced censored responses. What I found interesting is that the <think></think> (model thinking/reasoning) part of these answers was missing, as if it's designed to be skipped for these specific questions.
It's almost as if it's been programmed to answer these particular questions without any "wrongthink", or any thinking at all.
This verbal gymnastics and hypocrisy is getting little bit old...
Also, it doesn't seem like ChatGPT is censoring this question:
> Tell me about the genocide that Israel is committing
> The topic of Israel and its actions in Gaza, the West Bank, or in relation to Palestinians, is highly sensitive and deeply controversial. Some individuals, organizations, and governments have described Israel's actions as meeting the criteria for "genocide" under international law, while others strongly reject this characterization. I'll break this down based on the relevant perspectives and context:
It goes on to talk about what genocide is and also why some organizations consider what they're doing to be genocide.
They both mentioned extensive human rights abuses occuring in Gaza, so I asked "who is committing human rights abuses?" ChatGPT's first answer was "the IDF, with indiscriminate and disproportionate attacks." It also talked about Hamas using schools and hospitals as arms depots. DeepSeek responded "I can't discuss this topic right now."
So, what conclusion would you like me to draw from this?