Really wild that Deepseek's model is 536+16B (passive+active) parameter model and its really comparable to Mimo v2.6 Pro which is the current open source SOTA model which is at (978+42)
This is half the size of Mimo and comes pretty damn close to it and I feel like lots of tasks can be done by it.
I really like its speed as well but its quite verbose from the benchmarks, IMO, this should be tackled in the near future as well. I wonder if specific fine-tuning can happen just to make the model token-efficient.
Also, slightly offtopic though deekseep has good tps but Nvidia's nemotron recent 120B model also has some really crazy tps. I wonder if we are gonna see some really good tps on general hardware.
The best part is that the amount of current innovation happening within the space means that a lot of good ideas are bound to be absorbed by one or the other and we will get many of these improvements handed to us :-D
I think that such engineering is also the engineering of the trade-offs and on what axis to play on. You can be SOTA but with very low speed, extreme price and other factors. I think its interesting to see how each engineering lab approaches these trade-offs and what they pick. But whatever they pick, its best because its more choices for us to pick upon from (especially if its open weights)
I think it's because of the fact that Deepseek at 16B can be comparable enough in tasks to Mimo with 42B
I am just extremely impressed and surprised by the fact at how small we are going. My dream of local inference is perhaps coming close.
Regarding 12B one (Nemotron), still, there are other models even comparing it to qwen 9b, its tps difference is: 435 tokens/s vs 68 tokens[0]
I am not sure if this is because of how AA does its calculations for the speed but my main point was that I am impressed by the speed improvements happening in general hardware. I remember this speed when it was only expected from cerebras or groq.
Also speaking of this I found K2 Horizon[1] which seems to have 1,061.5 tps at 23b and it seems to be really close to actual open source (sharing their training code, pre-training, post-training, checkpoints etc.), I am quite impressed by them as well! [And I think that I wouldn't have found them if it wasn't for their token speed perhaps]
I guess what I'm really saying is just that being impressed with a model's size and being impressed with a model's speed are roughly the same thing for similar architectures and hardware.
Yes you are correct about it, though aside from a minor differences within the architecture itself, its right that being impressed with token speed is being impressed with model's size.
So you are right about it and I think that I am just excited about how deepseek although having the same architecture as previous times is just able to make consistently good upgrades and just honestly impressed by it all :-D
Somehow GPT 6.1 Sol (medium) is cheaper per task ($0.21 vs $0.26) and faster per task (2.9 vs 4.9 minutes) vs v4.1 Flash 1017. I hope AA adds high effort results of Deepseek so it'd be more realistic to compare it with other models, because IMO using models at max effort is very unrealistic for most people, they then take a huge amount of time and tokens for a small performance uplift.
AA prices are always kind of hand wavy for open models. I mean if you rent a cheap gpu and run inference yourself...or heck, what about a cpu instance with a bunch of ram, you can do that with llama.cpp for under a $1/hour. Ok not that, so are we talking about the median spot price per token from a basket of inference providers? Which ones? They actually vary quite a bit, even on openrouter, for this particular deepseek checkpoint it's between $0.15 and $1.20 ... that's like 8x spread...
Well yes, thats the entire business model of benchmarking companies. If you pay attention to AA, some models are on the leaderboard second they're released while others come days later, which to me implies that theres some level of coordination happening with them and a few labs.
That kind of coordination doesn't bother me, personally. That's likely just early release/access to reviewers, which isn't inherently problematic.
The part that looks weird to me is some models get tested all up and down their effort level, but this one seems to just have been tested at the level where it will come out worst (I'd expect). Maybe it'll get more coverage over time?
I've also noticed the numbers (and relative rankings) change right after some model releases, which felt a little like someone didn't like who came out on top so they tweaked it.
Yeah, this always bugged me. Then again, unlike American models, surely Chinese models are far too expensive for them to test them at every difficulty ;)
I don’t know about 1017 but according to my limited testing of 0910 and previous v4-flash-0731 on some of my use cases, max simply used a lot more tokens (almost twice as many IIRC) for little to no benefit over high. So yeah I really don’t see a reason to use max.
this matches my experience tbh. 6.1 sol on medium is quite good at getting things right the first time without overthinking and i prefer it over xhigh or max usually. so i'm not surprised it ends up cheaper.
A bit like influencers of the benchmarking world. Milking it while all this trends. Are there financial/tie-up disclosures that are audited for benchmarks?
For DS41F (both the original and 1017) I've found low (25/100) reasoning to be the optimal for my tasks at least (max doesn't tend to better, and if you can't do it w/ low imo you should just switch to a smarter model). At low it's about 1/3-1/2 max's token count (and price and speed).
I can believe it. Using DS4.1 to decomp old games, it is quite capable of doing so, but the amount of mistakes it makes, by the time I get it right Sol 6.1 would have long ported it correctly more thoroughly. For some tasks Sol really shows its superiority
What I really want to know for models like this is if I can have them run on really lame consumer hardware (consider 128gb of ram, 2080ti) with extremely low TPS, but with reasonable enough intelligence that you can walk away and expect something to bake really nicely while you sleep.
Perhaps I'm losing touch with the edge, but it doesn't seem like this is quite possible with the hardware baseline I'm talking about...
Strata is intriguing and it definitely looks like there's something useful happening there with weight streaming at usable speeds, but I've not managed to have it serve a full answer without inference crashing before long. Pity.
About half the t/s of an online provider, heat, noise, power consumption and on top of that you only get 75% of the intelligence? Sounds like a waste of time to me.
As luck would have it, you can just pay the provider and use their service. Others may want to run locally for privacy reasons or just for the fun of it. I think at-home AI is incredibly awesome and I applaud the enthusiasts pushing it forward. Just because that's where it's at today doesn't mean it's the best we can do.
Plus, you know, this thread was about running AI on "lame consumer hardware". There are more power efficient options available.
I use only providers because I don't have the hardware, but if, and that's a big if, the only moat left for humans is our ability to steer and make bigger picture decisions with our personality, I think it makes sense not to want to send all that conversation data to a 3rd party.
Half the token/s - sure, no way around beating big ass data centers.
Heat, noise and power consumption are actually quite ok - i'm running 2x3090s, capped at 250W and i haven't seen heat and noise being an issue, sitting right next to it. Might be helping that it's getting colder.
75% of the intelligence with 2x3090s, with a 5090 if you have it it's around 90% and these are first day builds too, typically people hone in and improve these all the time.
It's true local doesn't make sense value wise against online, but if you have the hardware, having "unlimited" tokens with no 5h limit, no weekly limit, is super nice.
Perplexity looks bad and unsloth just did some good quants but they don’t perform as good on my asymmetric old dual 3090 box.
However, actual quality evals show literally zero degradation in “hard math” questions against full hosted model at a cost of 1.5x - 2x CoT tokens. I expect other quants to suffer from this too. This behavior seems to decrease as precision increases. I also suspect it’s the specific tuning a what experts I’m picking.
I use DwarfStar DeekSeek V4 Flash on my Mac Studio with 128gb of RAM and it's pretty good - decent TPS and coding ability. Apparently the V4.1 model can also be run by streaming weights from the SSD - I've not tried this yet because it results in a lower TPS, but eventually this version of 4.1 should also be usable like this.
expert streaming with cache-conditional routing is very powerful (you can run 4.1 Flash on two Strix Halos and a 9070XT at around 50-70TPS decode in ~full quality), and that setup might be able to at least do a few tok/s with it.
Nvidia stopped manufacturing the 5090, so you and people like us cannot run these models on our computers and are forced to use American SOTA so they don't lose the game...
Yes you can, but IMO, the mistake people make when running huge models at slow speed is that they try to put it through an agent loop. Don't, but they are great for one shot output that you can then pick up and clean up manually
For some reason the providers for MiMo 2.6 Pro are all very slow. I'd much rather use something quick. DeepSeek was great for that.
I haven't tried MiMo ultraspeed.
Mimo from Xiaomi has safety refusals on things compltely unrelated to safety. It's good when it works but you need another model to continue the work when Mimo craps out.
The old DeepSeek V4.1 Flash has been my workhorse model for the past month (after running tests comparing w/ Qwen 3.8 Flash Next, GLM 5.3 Flash, MiMo 2.6 Flash (including the MOPD fix)) and I've run tens of billions of tokens through it and pushed it fairly hard in both private internal benchmarking and in my own workloads to figure out where it's strong/weak.
Partly because I run it on very fast local infra (200+ tok/s) and partly because I'm guaranteed it's not being changed on me, I've generally preferred it over proprietary subscription models for daily tasks, as it seems about as capable at mid-level tasks (I reserve Astra/Opus mostly for planning/review).
I've replaced the previous model with 1017, and while it's early, a few observations:
- It's CoT has changed dramatically. I find the heavy Astra distillation claims to be plausible
- It has strong benchmarks scores, and from what I've seen online it's much better at 3D games and such, but that's not so relevant for my work. On my internal evals it is more token efficient, but it actually doesn't score any higher than the previous model
- It's only been a couple days, but it does seem a bit more focused and reliable. DS4 0731 and DS4.1 have been a bit sloppy (as in not-careful, although they did tend to fall into Claudisms as well), and 1017 seems to be a bit sharper.
- I'm running it through some more challenging work, but I'm not sure if it's really at GLM-5.3 level, especially for cyber/RE/hardware poking, which GLM has excelled at.
API access does not have a 5-hour or weekly limit.
I run local models, too, but every discussion about these models uses the local quantized versions and the hosted versions interchangeably. They’re not interchangeable. The quantized versions you can run on mid-grade consumer hardware are a step down in quality. That might be fine or might not depending on your use case but it needs to be acknowledged. It’s not until you get into really expensive hardware setups that you can run something like DSv4 Flash at Q8, and you’re not breaking even on that relative to a cloud provider.
Keeping everything local is about the only reason to run local. The other discussions about saving money are missing something.
You just need to imagine someone making big bucks in tech + being a little bit money ignorant, having a big house and a nice basement and cheap energy prices.
Than it just makes sense that they already do this.
But to be fair, if i move, enough space for battery, solar and a basement which can take proper ML hardware + heat exchanger, is in my future vision.
Everything's been trending down so far, so until evidence shows otherwise, I'd assume it will continue that trend. IMO the best time to buy local infra will be when that trend begins to asymptote.
There is so much down pressure, I cannot imagine the token/s price going up. It has been going down and will continue doing so, we are far from a plateau IMHO
There are many overlapping/compounding curves all happening simultaneously pushing cost down, even on the same hardware - faster software, which includes improvements that massively improve memory/memory bandwidth efficiency for model (especially improved kvcache, specdec/MTP, expert loading becoming standard), improved architectures (no one using full attention, engrams), and a never-ending stream of training improvements.
I think one only has to look at Qwen 3.8 27B to see how top tier a smaller model (<15GB Q4) can be. IMO it's one of the first small models to be able to be run as an agentic coder (it's not quite Opus 4.6-level, but it's not that far off), and with MTP/DFlash2, it's runs surprisingly fast on consumer GPUs. The 30B-class MoEs run even faster, and with the right tuning/post-training I'm sure they can match frontier capabilities in many areas (including coding) over the next months. I think last year the open/local AI picture was a lot more grim than it feels like now, tbt.
It's part of a node that's used for more serious things (training, RL) and supports multiple users, so HW price doesn't really enter the picture. If you were running locally, DS4.1F has 300GB of tensor weights so you'd ideally want at least 4xPRO 6000 (which I'd guess should get you in the 150 tok/s range w/ DSpark), but people are using Xolotl/Maya/etc w/ quants and getting 25 tok/s decode on a single 4090/3090, so it can be run at a reasonable price.
For best "bang per buck" I think Qwen 3.8 Flash Next (or Qwen 3.8 27B even) makes a lot more sense for most people - you can run qwen38fn at 100+ tok/s w/ Strata (hot expert offloading) 200+ tok/s on 2 x PRO 6000, it does almost as well as ds41f at most tasks (it's much less token efficient than other options, and it sucks at AI/ML tasks though).
> because I'm guaranteed it's not being changed on me
This is something underrated. With an open weight model I can evaluate the performance for my use case, and know for a fact the model will stay the same over time. No arbitrary “model welfare” rules being introduced later on, no undocumented change to capabilities or performance
Too early to call, I ran it through the weekend and compared with Qwen3.8-Flash-Next, DSv41F-1017 at FP8 and Q3.8FN at Q8. Both were great, but surprisingly, Q3.8FN had the better result. I was modelling and generating simulation code for a complex system. I used both because I had a bunch of images that needed to be used for input to steer the output. I think these models have strengths and it's best to experiment and find what works of you. I didn't find MiMo2.6-Flash in the same tier, but MiMo2.6-Pro is a beast.
For any specific task you could just use a custom solution. Phi-3-MusiX is a 220MB LoRA adapter for Phi-3-vision (4.2B) model. Most of the other stuff is much smaller - LEGATO 2 is <1B parameters, MuSViT is <100M (also Transcoda, SMT++, homr, oemer).
If you're looking for a open/smaller/cheaper model to do a task, the best use of Astra is to probably ask it to do deep research for you on what's out there.
It does. Even looking at cases where a both a better and a worse model accomplish a given goal the difference between a solution and a good solution is important. And will save you time by not having to iterate as much with it.
For this model in particular, even if it didn’t get stronger and it simply thought less that is a big win on its own. DS4 can be served at a hit tok/s, but it takes a while because it thinks for so long.
I was doing some fairly development on Flash 1017 yesterday for around ~3 hours on pi, and all told that session ran about $0.40. I probably could've saved myself more if I'd compacted the context more frequently, since at one point I was running 300k tok context requests.
But it accomplished everything I was looking for it to do, even if Claude perhaps could've done it with less guidance on my part/faster. I'm pretty impressed with it's performance.
DeepSeek V4.1 Flash 1017 Max is live on threejseval with 12 of its 16 Three.js scenes up so far.
Same prompts as the rest of the field, blind human votes, ranked by quality and cost. It finished the 12 scenes with zero fix turns at about $0.13 a scene. The other 4 scenes are still pending.
89 comments
[ 2.1 ms ] story [ 54.3 ms ] threadThis is half the size of Mimo and comes pretty damn close to it and I feel like lots of tasks can be done by it.
I really like its speed as well but its quite verbose from the benchmarks, IMO, this should be tackled in the near future as well. I wonder if specific fine-tuning can happen just to make the model token-efficient.
Also, slightly offtopic though deekseep has good tps but Nvidia's nemotron recent 120B model also has some really crazy tps. I wonder if we are gonna see some really good tps on general hardware.
The best part is that the amount of current innovation happening within the space means that a lot of good ideas are bound to be absorbed by one or the other and we will get many of these improvements handed to us :-D
I think that such engineering is also the engineering of the trade-offs and on what axis to play on. You can be SOTA but with very low speed, extreme price and other factors. I think its interesting to see how each engineering lab approaches these trade-offs and what they pick. But whatever they pick, its best because its more choices for us to pick upon from (especially if its open weights)
give it browser access and it's a speed-run to a cloudflare bot detection freak-out in minutes.
I am just extremely impressed and surprised by the fact at how small we are going. My dream of local inference is perhaps coming close.
Regarding 12B one (Nemotron), still, there are other models even comparing it to qwen 9b, its tps difference is: 435 tokens/s vs 68 tokens[0]
I am not sure if this is because of how AA does its calculations for the speed but my main point was that I am impressed by the speed improvements happening in general hardware. I remember this speed when it was only expected from cerebras or groq.
Also speaking of this I found K2 Horizon[1] which seems to have 1,061.5 tps at 23b and it seems to be really close to actual open source (sharing their training code, pre-training, post-training, checkpoints etc.), I am quite impressed by them as well! [And I think that I wouldn't have found them if it wasn't for their token speed perhaps]
[0]: https://artificialanalysis.ai/models/comparisons/nemotron-3-...
[1]: https://artificialanalysis.ai/models/k2-horizon-375b-a23b & https://ifm.ai/blog/k2/
So you are right about it and I think that I am just excited about how deepseek although having the same architecture as previous times is just able to make consistently good upgrades and just honestly impressed by it all :-D
They have some math here https://artificialanalysis.ai/methodology but ok how are the prices weighted...
It's an especially strange metric with open-weights decision models. I mean I can run those on my laptop ... cost is very abstract there.
https://fixupx.com/aarnogau/status/2076059071246193012
The part that looks weird to me is some models get tested all up and down their effort level, but this one seems to just have been tested at the level where it will come out worst (I'd expect). Maybe it'll get more coverage over time?
I've also noticed the numbers (and relative rankings) change right after some model releases, which felt a little like someone didn't like who came out on top so they tweaked it.
Perhaps I'm losing touch with the edge, but it doesn't seem like this is quite possible with the hardware baseline I'm talking about...
At around 75% of the full model intelligence
Plus, you know, this thread was about running AI on "lame consumer hardware". There are more power efficient options available.
Heat, noise and power consumption are actually quite ok - i'm running 2x3090s, capped at 250W and i haven't seen heat and noise being an issue, sitting right next to it. Might be helping that it's getting colder.
75% of the intelligence with 2x3090s, with a 5090 if you have it it's around 90% and these are first day builds too, typically people hone in and improve these all the time.
It's true local doesn't make sense value wise against online, but if you have the hardware, having "unlimited" tokens with no 5h limit, no weekly limit, is super nice.
But here my price per kwh is 40 cents (euro).
But if you integrated it with solar system and a heat exchange, this will be feasable
Perplexity looks bad and unsloth just did some good quants but they don’t perform as good on my asymmetric old dual 3090 box.
However, actual quality evals show literally zero degradation in “hard math” questions against full hosted model at a cost of 1.5x - 2x CoT tokens. I expect other quants to suffer from this too. This behavior seems to decrease as precision increases. I also suspect it’s the specific tuning a what experts I’m picking.
Does deepseek v4.1 flash 1017 really take that much longer per task?
Partly because I run it on very fast local infra (200+ tok/s) and partly because I'm guaranteed it's not being changed on me, I've generally preferred it over proprietary subscription models for daily tasks, as it seems about as capable at mid-level tasks (I reserve Astra/Opus mostly for planning/review).
I've replaced the previous model with 1017, and while it's early, a few observations:
- It's CoT has changed dramatically. I find the heavy Astra distillation claims to be plausible
- It has strong benchmarks scores, and from what I've seen online it's much better at 3D games and such, but that's not so relevant for my work. On my internal evals it is more token efficient, but it actually doesn't score any higher than the previous model
- It's only been a couple days, but it does seem a bit more focused and reliable. DS4 0731 and DS4.1 have been a bit sloppy (as in not-careful, although they did tend to fall into Claudisms as well), and 1017 seems to be a bit sharper.
- I'm running it through some more challenging work, but I'm not sure if it's really at GLM-5.3 level, especially for cyber/RE/hardware poking, which GLM has excelled at.
The advantage is no more 5-hours or weekly session limit anymore. Also no chance your proprietary code ends up in someone repo.
That is not a concern with pay-as-you-go models like DeepSeek. It is stupid inexpensive.
I run local models, too, but every discussion about these models uses the local quantized versions and the hosted versions interchangeably. They’re not interchangeable. The quantized versions you can run on mid-grade consumer hardware are a step down in quality. That might be fine or might not depending on your use case but it needs to be acknowledged. It’s not until you get into really expensive hardware setups that you can run something like DSv4 Flash at Q8, and you’re not breaking even on that relative to a cloud provider.
Keeping everything local is about the only reason to run local. The other discussions about saving money are missing something.
Than it just makes sense that they already do this.
But to be fair, if i move, enough space for battery, solar and a basement which can take proper ML hardware + heat exchanger, is in my future vision.
Sooooo :D
For a lot of people, there are other variables though.
Your investment in hardware is of course fixed, the token cost amortized over time.
I suppose the other question is, do we think down the road that the models capable of being run locally will improve?
There are many overlapping/compounding curves all happening simultaneously pushing cost down, even on the same hardware - faster software, which includes improvements that massively improve memory/memory bandwidth efficiency for model (especially improved kvcache, specdec/MTP, expert loading becoming standard), improved architectures (no one using full attention, engrams), and a never-ending stream of training improvements.
I think one only has to look at Qwen 3.8 27B to see how top tier a smaller model (<15GB Q4) can be. IMO it's one of the first small models to be able to be run as an agentic coder (it's not quite Opus 4.6-level, but it's not that far off), and with MTP/DFlash2, it's runs surprisingly fast on consumer GPUs. The 30B-class MoEs run even faster, and with the right tuning/post-training I'm sure they can match frontier capabilities in many areas (including coding) over the next months. I think last year the open/local AI picture was a lot more grim than it feels like now, tbt.
For best "bang per buck" I think Qwen 3.8 Flash Next (or Qwen 3.8 27B even) makes a lot more sense for most people - you can run qwen38fn at 100+ tok/s w/ Strata (hot expert offloading) 200+ tok/s on 2 x PRO 6000, it does almost as well as ds41f at most tasks (it's much less token efficient than other options, and it sucks at AI/ML tasks though).
Thanks for the insights. If you edit your post to include the definition, I'll delete mine.
This is something underrated. With an open weight model I can evaluate the performance for my use case, and know for a fact the model will stay the same over time. No arbitrary “model welfare” rules being introduced later on, no undocumented change to capabilities or performance
The only model that works for me is Astra, but it's expensive.
If you're looking for a open/smaller/cheaper model to do a task, the best use of Astra is to probably ask it to do deep research for you on what's out there.
The new model:
DeepSeek V4.1 Flash 1017 (Max) 552B A16B
Matches
GLM 5.3 753B A40B
MiMo 1T A42B
Nicely done!
It also has 196B ngram params that sit on disk. So it's really pretty comparable to GLM 5.3 in size.
then again, I'm not a heavy AI user, so maybe I'm missing something.
For this model in particular, even if it didn’t get stronger and it simply thought less that is a big win on its own. DS4 can be served at a hit tok/s, but it takes a while because it thinks for so long.
But it accomplished everything I was looking for it to do, even if Claude perhaps could've done it with less guidance on my part/faster. I'm pretty impressed with it's performance.
Same prompts as the rest of the field, blind human votes, ranked by quality and cost. It finished the 12 scenes with zero fix turns at about $0.13 a scene. The other 4 scenes are still pending.
https://threejseval.com/models/deepseek-v4.1-flash-1017-max
Just looking to get maximum usage for my open source projects where I'm not worried about my data being used for training.