Is there a free version of DeepSeek R1 that's completely US based, so we're not sending data to China? I guess you can use this to deploy it, but I'm asking for an application that would be safer to use if you're concerned about Chinese influence.
Yeah I mean, most users won't. Sorry if I got on the defensive, saw a bit too many posts on social media claiming you could run the model on your consumer-grade GPU.
Previously choosing a top tier AI model tied you to what that provider wanted to do with hosting the model long term and the pricing they wanted to charge for it. Now you can get the same model anywhere with GPU, hosted or not, for minimal cost overhead to what it takes to run the model itself. You're also free to tune, retrain, or otherwise mess with the model as you see fit without needing approval.
The excitement is probably a bit much but it's not just about the eval results themselves but the baggaged attached with them.
For me the excitement is that around the o3 announcement I had a feeling like we were heading to an OpenAI / Sam Altman controlled dystopia. This resets that - you can run the model yourself, you can modify it yourself, it's essentially on par with the best public models, and it gives hope that the smaller players have a fighting chance going forward. They also published their innovations bringing back some of the feeling of open science that used to be in ML research but which mostly went away.
Google models are already in the lead in many areas in capability and cost, so I never felt like OpenAI was dominant. OpenAI was first to make a splash, but ChatGPT is in a ~5 way tie in terms of what it can do.
Which models at what cost? IMO Deepseek websearch potential to challenge Google search moat also makes Google particularly vunerable, because it dramatically evaporates advantages of 100s of billions of hardware. Not to imply Google does not maintain advantages, but it gap just went from insurmountable to many actors can potentially build AI search to rival Google on shoe string budget. Certainly on sovereign budget.
It's going to be an increasingly irrelevant game when models make regional scale, i.e. country/sovereign scale inference attainable. Countries that couldn't even role out domestic search pre accessible models that displaces search likely soon can.
AFAIK o1 is hidden behind an expensive subscription (iirc $20/mo and still rate-limited), it might as well just not exist for most users (since R1 is free, provided service availability).
Also R1 (and its distilled models) expose their CoT & web interface has a websearch option too.
With the 14b distilled models, I found multiple math-related prompts where it gives the right answers almost immediately but then wastes 10 minutes making self-verification mistakes (e.g. "Write Python3 code that computes the modular inverse of a mod 2^32")
I don't understand the hype because I'm out of the loop. Is the only advantage the lower hardware requirements, thus cost? Is there something I'm missing?
Yeah it's a lot more efficient, it's also a very advanced model that answers questions in a multi-step way, like OpenAI-O1, it performs extremely well.
OpenAI o1 and Deepseek r1 have similar performance (o1 is a bit better at reasoning though you can see r1’s though process which you could argue trumps the competition).
OpenAI o1 api cost: $60/million output tokens. Deepseek r1 api cost: $2.19/million output tokens.
> Is the only advantage the lower hardware requirements, thus cost?
Yes, but the keep thing is it performs nearly as well as models that are 100x as expensive.
The lower price drastically changes possible utility. For example, I've been rocking RooCode since R1 came out. R1 can do about 95% of the tasks Claude can, but at 1% of the cost. I might burn $10 to $20 per hour on Claude tokens. While spending less than $1 on Deepseek when doing the same task.
- Single digit TPS on rare chance it responds, and frequent complete hangs (1 out of maybe 20 requests even complete)
- 4k input token cap (vs native 128k context window)
- No pricing
- Unstated rate limits
It genuinely seems like they spun up a single H100 cluster to enable the headline of this post and help form a narrative then left it at that. Definitely not meant to genuinely provide access to R1 in any serious way.
I have tried it on "Azure AI foundry" through their serverless API with a paid subscription.
It takes 80s to answer a basic question that was answered in 7s by OpenAI gpt-4o. And there was not that much thought process, it was just super slow to output each token.
I guess this slowness is explained by the pricing, they are still figuring out how to run the inference for this model:
> DeepSeek R1 use is currently priced at $0, and use is subject to rate limits which may change at any time. Pricing may change, and your continued use will be subject to the new price. The model is in preview; a new deployment may be required for continued use.
There is also a hard limitation of 4k tokens as input context (context window on DeepSeek model is 120k tokens), which prevents using it for RAG use-cases:
> Message: Request body too large for deepseek-r1 model. Max size: 4000 tokens.
Also the documentation and python type hints of their inference lib have a lot of straight up errors in it (they are confusing the class attributes `model=` and `model_name=` at many places in the docs, spoiler: the good one to use is `model_name=`, even if the type hints recommend to use `model=`).
I have also tried with more stable models like Mistral Large, but the streaming feeling is really bad, they are sending whole sentences at a time, with multiple seconds of wait between each sentence. Does not feel smooth at all compared to any other provider out there.
Would not recommend Azure AI foundry for production use (or any use to be honest). Does not worth the pain to navigate the documentation. We will be using directly DeepSeek API, or fireworks.ai, or together.ai.
Well, just running on a 6C/12T Coffee Lake CPU, (I'm looking through these speeds in LM Studio as I type this..) I got like 2 tokens a second with Deepseek R1 14B, 3.4 with 7B Qwen, and 4.4 with 8B Llama, although out of those two I found 7B Qwen's answer to be a bit better. (My GTX1650 has 4GB VRAM, loading 1/4 the layers is pretty ineffective, GPU util went up to 10% and I gained like 1 token a second LOL.)
So it'd take a minute or two to type out one of those answers where it's got about 4 or 5 beefy paragraphs of thought and a decent sized paragraph for it's answer. I'll put it this way, I can type 120 WPM and it puts out text a bit faster than I could write it.
Input's a LOT faster though, I was asking these models to analyze a document so my input was like 2200 tokens, they all did well over 100 tokens a second on input.
49 comments
[ 3.3 ms ] story [ 124 ms ] threadIs there a free version of DeepSeek R1 that's completely US based, so we're not sending data to China? I guess you can use this to deploy it, but I'm asking for an application that would be safer to use if you're concerned about Chinese influence.
`ollama run deepseek-r1:671b`
will do that
You mean $100k in GPUs?
Granted, that's still expensive, but it is within the realm of something a hobbyist could put together.
[0]: https://x.com/carrigmat/status/1884244369907278106
I'm trying to square the excitement over DeepSeek with its good -but not dominant- performance in evals.
The excitement is probably a bit much but it's not just about the eval results themselves but the baggaged attached with them.
I was planning on spending the $200 for a month but had been thinking of prompts to try it out.
DeepSeek already answered them all for free so I am not just going to light $200 on fire for fun.
Also R1 (and its distilled models) expose their CoT & web interface has a websearch option too.
With the 14b distilled models, I found multiple math-related prompts where it gives the right answers almost immediately but then wastes 10 minutes making self-verification mistakes (e.g. "Write Python3 code that computes the modular inverse of a mod 2^32")
r1 api can spit out 63 tokens per second
~143 billion lifetime tokens.
~$313 million for a lifetime supply of tokens.
For my use, it is better than $20 a month o1 and being able to see the chain of thought is absolutely incredible.
I have learned as much this week from seeing the chain of thought as I have from what it actually outputs.
Yes, but the keep thing is it performs nearly as well as models that are 100x as expensive.
The lower price drastically changes possible utility. For example, I've been rocking RooCode since R1 came out. R1 can do about 95% of the tasks Claude can, but at 1% of the cost. I might burn $10 to $20 per hour on Claude tokens. While spending less than $1 on Deepseek when doing the same task.
- Single digit TPS on rare chance it responds, and frequent complete hangs (1 out of maybe 20 requests even complete)
- 4k input token cap (vs native 128k context window)
- No pricing
- Unstated rate limits
It genuinely seems like they spun up a single H100 cluster to enable the headline of this post and help form a narrative then left it at that. Definitely not meant to genuinely provide access to R1 in any serious way.
(I work at Microsoft but am not on the clock as I write this, and keep my personal projects separate)
I have tried it on "Azure AI foundry" through their serverless API with a paid subscription.
It takes 80s to answer a basic question that was answered in 7s by OpenAI gpt-4o. And there was not that much thought process, it was just super slow to output each token.
I guess this slowness is explained by the pricing, they are still figuring out how to run the inference for this model:
> DeepSeek R1 use is currently priced at $0, and use is subject to rate limits which may change at any time. Pricing may change, and your continued use will be subject to the new price. The model is in preview; a new deployment may be required for continued use.
There is also a hard limitation of 4k tokens as input context (context window on DeepSeek model is 120k tokens), which prevents using it for RAG use-cases:
> Message: Request body too large for deepseek-r1 model. Max size: 4000 tokens.
Also the documentation and python type hints of their inference lib have a lot of straight up errors in it (they are confusing the class attributes `model=` and `model_name=` at many places in the docs, spoiler: the good one to use is `model_name=`, even if the type hints recommend to use `model=`).
I have also tried with more stable models like Mistral Large, but the streaming feeling is really bad, they are sending whole sentences at a time, with multiple seconds of wait between each sentence. Does not feel smooth at all compared to any other provider out there.
Would not recommend Azure AI foundry for production use (or any use to be honest). Does not worth the pain to navigate the documentation. We will be using directly DeepSeek API, or fireworks.ai, or together.ai.
This is news, because Microsoft seems happy to not be tied to OpenAI so heavily.
This could also safe a huge amount of money for their Office 365 Copilot initiative.
I figure Microsoft started analyzing these models ASAP in their labs to catch up with OpenAI, Google, Anthropic etc.
By also hosting this model, they will help normalize the use of them from which they immensely benefit.
So it'd take a minute or two to type out one of those answers where it's got about 4 or 5 beefy paragraphs of thought and a decent sized paragraph for it's answer. I'll put it this way, I can type 120 WPM and it puts out text a bit faster than I could write it.
Input's a LOT faster though, I was asking these models to analyze a document so my input was like 2200 tokens, they all did well over 100 tokens a second on input.