I used their Coding Plan for a few months. It is genuinely difficult to keep up with the models. The output is so fast. Qwen 3.8 27B is likely one of the strongest models they've hosted so far.
Edit: it looks like this is only available on a API token pricing. Does anyone know if they have rolled out prompt caching yet? It used to get pretty expensive for agentic coding tasks with no prompt caching.
Strongest model that they host on the public endpoint. They do a super fast version of GPT 5.6 Sol for OpenAI and have bigger open models on dedicated endpoints.
Last time I got one, I had to log into a Discord server and wait for "the drop" and IIRC Daniel Kim was giving them out based on who was there at the time. They were gone in less than a minute. This was ~8 months ago.
Why do they only host small models rather than the 2.4T version? Is the I/O and interconnect between the wafers bad due to the limited beachfront relative to the massive size of the chip?
Noticed they are present in OpenRouter, but Qwen 3.8 is not there yet. Hopefully it'll get there soon.
For those who haven't noticed though, the context size they allow for Qwen is just 128k. Still interesting as a specialized sub-agent but not really well suited for long tasks.
128k is effectively useless on even trivial toy size "not for real business use" coding projects, by the time you reach 105k to 115k tokens with loading code into context and new research/tasks, and ask it to modify something, it'll be vacating older stuff out of context and forgetting the whole picture of what you're working on.
I have done the vast majority of my agentic coding work under 128k on a 256k context model and when I cross 128k, all I do is just ask for a handoff prompt to feed back into the next session. I do this because I was told the quality degrades as the context fills up and I fundamentally dislike random compaction, I don't even know what compaction does, it is completely intransparent.
It would be great if they made their inference capacity for this model available via OpenRouter; the fastest provider on OpenRouter right now is at ~80tps https://openrouter.ai/qwen/qwen3.8-27b#providers
Something a lot of model providers don't talk about: any time an engine uses speculative decoding the throughput will depend on how much your output token distribution matches what the draft model was trained on.
The DFlash2 draft model we're using was trained on a lot of code, so if you use it in a coding agent you'll probably notice it run a lot faster (we've seen it break 300 tok/s).
FYI, I might be missing something but I think your billing system might not be working well - I'm not seeing any indication in the UI that my usage is being deducted from the $5 of free credits.
Hey Daniel! It's a bit hidden, but at the bottom of the billing page there's a "Credits" section which should show usage of any active credits and the balance remaining. The usage/billing metrics are batched/handled async so it might take a minute or so for usage to be reflected. Let us know if it feels off.
I'm saddened that Gemma4 is replaced by Qwen 3.8 on PayGo plan. Gemma4 31B is not coding model but it is excellent at intent understanding and task execution used in agentic software. This just shows that real world dominant usage for llms so far is to code generate. And not to augment business products. They must had barely anyone using Gemma to remove it from that tier.
I have used their gemma 4 31b model through kagi and getting real instantaneous answers is absolutely crazy. A very different feeling and UX. Even if the model is smaller, there is definitely a use case for these. I was wondering if they would put the qwen 27b model, it sounds very interesting to try.
The thing I didn’t realize for a while is 27B is rather smart. As many (or more) activated parameters as the flash models of the universe that we know about. It reasons very well. It just doesn’t have a lot of knowledge.
They seem to have good enough general intelligence that missing knowledge is not that big thing. If you are able to have a proper [free search engine], they can do almost anything. Having own local search index about relevant stuff can help a lof if you don’t want to pay for search API.
But running that fast… with a local RAG? Yeah, it is a very interesting model. Maybe you don’t need a lot of parameters, just a really big local database :)
I believe. I run it on my mac M5 pro at like 30t/s with some RAGs and let it work on stuff overnight and it's great. It isn't the same as the big models where things can be more unbounded, but if local models keep progressing there is a universe where a 200-300B model is all most of us will need to stay out of the big tech moats.
Really an aside, but yesterday I got the Gemma-4-12b (128k context) to build it's first web app in the minimal Dark Software Factory I've been building for myself.
It has been used in kagi's "quick assistant", so basically summarisation tasks (gets context from a bunch of webpages and finds the answer to a question in them, spits all the relevant to the query information, or similar).
It is great UX when you are in a search results page, but I don't use it in the assistant directly because usually this kind of speed is less relevant there.
Funnily enough the pricing isn't that much worse than on openrouter, where the best price at the moment is $0.24 in / $2.55 out, vs $1 / $1.5 on Cerebras.
Sure, 4x input , but cheaper output.
Though Cerebras doesn't have prompt caching, so not great for agentic workloads. (they do, but it doesn't affect the price.
I have been their user for more than year even used coding plans, though for normal coding the quota will definitely be a blocker if you are using opencode because rpm are bit less. Good for products/api though.
Just a couple days ago I learned about ninfer (https://github.com/Neroued/ninfer) and on RTX 5090 I can now get ~200 tok/s and over 400 tok/s on concurrent requests which is plenty fast for a local model of this strength.
Even without ninfer I would get over 80 on LM studio with default settings, so it should be noticeably more on 6000. You might want to try different a different inference engine or settings.
Just tried it on a medium size coding/debug problem on an existing codebase, observations:
- Input doesn't look faster than other models, it spends a lot of time reading
Read about 5M tokens
- Output is awesome, super fast as you expect from the 1500t/sec I think that's correct
- Tool call is failing more than say DS4, which leads to time wasted on retries (complex tools like browser control for example)
- Shell commands are still somewhat of a bottleneck
The net effect is that I spend about the same time waiting, and I still need to read that output so, at least for coding, it actually reconciles me with the 100-200t/sec you can get on DS4 or the like. Maybe that's a good sweet spot after all and faster t/sec is not where the bottleneck is.
Also maybe my setup (OMP) doesn't do the cache correctly but that's a huge cost driver... so atm it's quite pricy
150k TPM limit on public endpoint means that it's likely unusable for many coding tasks. When we've tried Cerebras in the past, our problem has always been rates. We'd love to not deal with dedicated and to have access to a more flexible rate pool.
Even trying it out, it seems like our account has gotten moved to some limbo where we can no longer add billing information.
```
Billing access restricted
Self-serve billing is not available on Enterprise accounts. Please contact your team for further questions.
```
We have no team (they removed themself from our slack channel after we talked about rate limits). Perplexingly, none of this even shows up in the request, which gives:
```
{"message":"Model does not exist or you do not have access to it.","type":"not_found_error","param":"model","code":"model_not_found"}
```
When the error is really about billing.
I always want to like Cerebras, but I get the vibe that as a tokens in tokens out consumer you are not valued at all.
No, the asic could only ever run one model/set of weights, no updates possible, ever. These are general purpose processors that can have their models updated. But the chips are enormous, with a substantial amount of on-die memory alongside the execution units, for a relatively insane amount of memory bandwidth.
I thought from what I read about the Taalas approach, the model architecture and overall size couldn't be changed, but model weight values could be updated after for further tuning.
Not as flexible as Cerebras though. And I'd love for someone who knows more to clue me in to the truth.
Nah, Taalas was putting the weights into silicon as a mask ROM. Their demo chip was hardwired to serve Llama 3.1 8B, and could never be updated. New models, even new versions without any architectural/size changes meant new tape outs.
But in exchange, you get insane speed and great energy efficiency. I could see it being a great approach for basic "good enough" models.
To qualify "insane speed" for anyone unaware, think a 10x improvement over even Cerebras. On the order of ~15,000 tokens/s. Not saying their approach scales well enough to keep pace with the various frontiers, but using their demo alone feels like a paradigm shift.
This seems like it would be an altogether different experience than the common experience of using an LLM, which is characterized by the person spending a lot of time waiting on the machine.
Yeah, I'm guessing this isn't unique, but I remember the first time I used ChatJimmy, I missed the fact that it had responded because I was still hitting the enter key, and getting ready to see tokens stream in, but they were already all sitting there, and I'd missed registering the visual diff somehow.
Yes and no. A single chip cannot have it's weights adjusted, once it's out, it is what it is.
But also, the model weights are in a single mask rom layer, high up in the metal stack. They could manufacture the die specialized for a given geometry of a model up to that layer, wait for updated weights, and then get the final product out in weeks after they got the weights, instead of many months which is what it would take to redesign the whole chip for the new weights.
Yeah, their public service isn't a serious/competitive offering. They don't have the capacity to serve all the customers who might want to use them at that speed. The public service exists so they get some users on OpenRouter, and that shows them as #1 on speed, which proves their tech is very fast, which gets them billions in hardware sales/licensing. If you have big enough pockets they can probably dedicate capacity to you. But for reliably fast small models you might want to rent some GPUs.
MiMo-V2.5-Pro-UltraSpeed gets pretty close with over 1000 TPS on 8x B200. It has 1.02T total parameters and 42B active, compared to 27B total/active for Qwen3.8-27B. Also, B300 are out now. I think 1500 TPS for Qwen3.8-27B should be doable.
It seems you forgot to account for the fact that cerebras uses a baker's minute which is 144 seconds instead of 60. (Seriously though what's the supposed issue here?)
I was using it quite a while back, different model, different quotas, but for coding tasks it routinely hit quotas which made it quite difficult to actually use.
100s/min seems pretty poor actually with sub-agents etc.
What kind of coding tasks would you expect to hit that limit? In my setup, on a very large codebase, it takes each agent 3-4 minutes at minimum to go past 100k tokens.
(note it's 150k uncached tokens, the total limit is 450k/min)
> 150k TPM limit on public endpoint means that it's likely unusable for many coding tasks.
I don't understand. How does that make it unusable? Is the limit shared by an entire team at once?
150,000 tokens per minute is a lot. You could start hitting that with a lot of concurrent requests in your session, but even throttled to 150k TPM it's still going to be faster than anything else you find.
I think the 128K context limit is the real ceiling. These models aren't amazing at long context, but once you account for a short input prompt, the input files, and headroom for a compaction summary, there isn't a lot left for the problem.
Exactly, it burns the tokens 3000x faster, which means the budget ($$$$$$) runs out so faster it will stop super quick, not able to perform long-duration work. At 27B parameter size, the intelligence is not able to accomplish work within a short amount time. Consequently, it become not usable.
I (we) run Qwen3.8-27B-FP8 on a DGX Spark box - that's roughly £4000 of hardware.
I did benchmark it in various ways and it runs quite well but it is a quantised jobbie and 1.5k t/s is also rather faster than anything I can possibly hope to achieve.
To run that model at those sorts of speeds is going to need some serious investment and you are going to have to pay for it.
Thank you, always nice to see real world performance figures.
We run a pretty large rig, 10 GPUs right now (this goes up and down with various experiments, getting this many GPUs to play nice at x16 GEN4 with any motherboard is a challenge), 240G VRAM in total. 256G RAM and a TR PRO. For small models the comms overhead is larger than the gains so there I have to reduce the number of active GPUs. On this machine I'm getting between 150 and 200 tg/s with FP8, but it took a lot of time and tweaking to get to that, and not all of the improvements held up when combined with other improvements. I've been playing with this stuff for a while now and it is interesting how fast the frontier is moving and how much you can now do on your own hardware. For larger models the communications overhead is low enough that we can run them on bigger groups of GPUs, and using hacked drivers to give us p2p capabilities on some of our GPUs also boosts performance considerably once you start to hit communications limits. Typically we get 50G/second in p2p mode (full duplex, half that one way).
From a cost perspective running locally is not interesting, but it allows us to do experiments that model providers would likely balk at, gives us censorship free access and allows us to work with data that we would not want to share with model providers (or can't share due to NDAs).
I will look into running ninfer, I was aware of them but had not yet gotten around to using it.
The problem is most providers hit tok/sec limits really fast. 1m/min is the default and the only place I can get 10m+ is from first party providers without a lot of upfront cash.
It's a limit on input tokens. So that's 3 50k requests per minute. At Cerebras speeds, that's about 5 seconds of usage per minute.
I was very excited last year for their coding plan but seeing a burst of requests pulse and then sitting there watching the cooldown reset is really not a great time.
Even though each individual request was fast, the sessions were only maybe 10% faster on wall clock time since there was so much waiting time.
They made the coding plan a bit better toward the end, but it was pretty tough to use throughout.
Seems like an Amdahl’s law of inference economics? there’s so much compute relative to SRAM on the chip and shoreline bandwidth onto the chip that caching buys ~nothing? The contended resource is SRAM and a given token of context needs just as much as another.
That’s not what caching is for. Caching lets you resume with a pre computed KV cache saving you from having to ingest everything in the chat history as input on every single round trip. You still need caching regardless of SRAM or not as it saves a huge amount (and ever growing) of compute ingesting the preceding history every time you want a completion.
all the above. They just simply do not care about non enterprise customers. Today they announced qwen, guess what - it's also the same day they pulled Gemma off their shared tier. No migration notice and all developers are scrambling as we speak trying to migrate. They gave a soft head-ups on discord a week ago and when folks complained about zero-day migration they started saying 'you aren't suppose to build production app on shared tier'.
I was wondering whether this was any good for programming, but it is too fast for its own good. There is a limit of 450,000 tokens per minute. I hit this limit in about 90 seconds and burned through $1.10 while doing so. This is because cached tokens count towards the token limit.
For comparison, I ran the same task with DeepSeek-V4-Flash, which finished in 172 seconds and cost $0.024 with a final context window size of 55217 tokens, while Qwen3.8-27B was not even close to being done with a 64178 context window.
This is a very efficient way to burn your money, but I would not recommend it for programming.
On the positive side, I got a $5 signup bonus, so it wasn't my own money.
The point of speed is to increase throughput. What the point of all this speed, if overall throughput is still so low?
This doesn't work for my use case at all (code generation).
These bursts of speed might work well for workflows that need bursts of quick decisions, followed by silence. But these workflows have needed provable determinism to som extent, so I haven't been using llms for those use cases. And I don't see myself using llms for them in the future too.
I don’t think I understand. Why would faster token generation burn more tokens? The LLM should not be generating anything in between tool calls so the only difference should be that the human waits less between turns.
To get a good coding agentic system you need to use big context (Specs and conversation context can't be condensed every minute), so you need to use prefix caching, and the price for the hit cache tokens can't be the same that miss cache or the final price could be insane.
I just did a little anecdotal test. Had pi + cerebras review a recent commit and asked a few quick followups on it. Worked great.
The Cerebras session cost me $1.60 and took a total of 5.1 mins. I did get a few brief 429 rate limit errors in there. The p50 speed was 890 tok/s and 0.64s TTFT.
Using OpenRouter averages, that would've cost $0.29 (no cache discount at Cerebras!) and would've taken about 14.4 minutes.
So on this one short session, cerebras was 5.6x more expensive in exchange for being 2.8x faster. Or, another way, $1.32 buys back about 9 minutes of your time. Not a bad trade IMHO but the cache situation is a real bummer. The longer your session the more relatively expensive Cerebras gets. The "good" news is you're also limited by its short context window.
(Also, I used to be on the Cerebras coding plan and the support is pretty bad for end users. My guess is these public endpoints are really just product demos for potential enterprise customers.)
Qwen 3.8 27B is an exceptional model for coding and ranks as one of the best local models for coding....BUT in my head I am confused why a company that's IPO'd doesn't invest in RL'd super specialized, super-damn-fast models for very specific tasks - instead of giving us the OSS GPT model from what feels like 200 years ago
192 comments
[ 0.28 ms ] story [ 12.4 ms ] threadEdit: it looks like this is only available on a API token pricing. Does anyone know if they have rolled out prompt caching yet? It used to get pretty expensive for agentic coding tasks with no prompt caching.
Hope they add such models to Code too :)
Psychopaths: tok/SEC
For those who haven't noticed though, the context size they allow for Qwen is just 128k. Still interesting as a specialized sub-agent but not really well suited for long tasks.
When I put Qwen3.8 27B xhigh towards adding scope proxying to the Guice library, it one shotted a great impl using 250k context before stopping.
Part of the greatness of the model is that it just keeps going until it gets a great result. 128k context is disappointing.
They do appear to host other models on OpenRouter so maybe Qwen3.8 will be there soon: https://openrouter.ai/provider/cerebras
https://mixlayer.com, LAUNCH-Q38-27B gets you $5 in credits if you want to kick the tires.
The DFlash2 draft model we're using was trained on a lot of code, so if you use it in a coding agent you'll probably notice it run a lot faster (we've seen it break 300 tok/s).
Still, 1500tokens/s is indeed wild
Having the choice is good as you can make a trade-off between speed, perf, and quality.
But I burn my 5$ allowance in 10 minutes ... and only because I was hitting rate limits, without it would probably be less than a minute.
https://joeldare.com/a-local-open-weight-model-builds-its-fi...
It is great UX when you are in a search results page, but I don't use it in the assistant directly because usually this kind of speed is less relevant there.
Sure, 4x input , but cheaper output. Though Cerebras doesn't have prompt caching, so not great for agentic workloads. (they do, but it doesn't affect the price.
Most of the cost for agentic coding is input tokens, you pay for the whole context at each tool call or message. Output tokens is just a small rate
The net effect is that I spend about the same time waiting, and I still need to read that output so, at least for coding, it actually reconciles me with the 100-200t/sec you can get on DS4 or the like. Maybe that's a good sweet spot after all and faster t/sec is not where the bottleneck is.
Also maybe my setup (OMP) doesn't do the cache correctly but that's a huge cost driver... so atm it's quite pricy
Even trying it out, it seems like our account has gotten moved to some limbo where we can no longer add billing information.
``` Billing access restricted Self-serve billing is not available on Enterprise accounts. Please contact your team for further questions. ```
We have no team (they removed themself from our slack channel after we talked about rate limits). Perplexingly, none of this even shows up in the request, which gives:
``` {"message":"Model does not exist or you do not have access to it.","type":"not_found_error","param":"model","code":"model_not_found"} ```
When the error is really about billing.
I always want to like Cerebras, but I get the vibe that as a tokens in tokens out consumer you are not valued at all.
Not as flexible as Cerebras though. And I'd love for someone who knows more to clue me in to the truth.
But in exchange, you get insane speed and great energy efficiency. I could see it being a great approach for basic "good enough" models.
But also, the model weights are in a single mask rom layer, high up in the metal stack. They could manufacture the die specialized for a given geometry of a model up to that layer, wait for updated weights, and then get the final product out in weeks after they got the weights, instead of many months which is what it would take to redesign the whole chip for the new weights.
Just the math there... 150k TPM... and 15k TPS means... you can run for 10 seconds every minute?
The basic math boggles the mind.
I was using it quite a while back, different model, different quotas, but for coding tasks it routinely hit quotas which made it quite difficult to actually use.
100s/min seems pretty poor actually with sub-agents etc.
It’s one order of magnitude less TPS, but still, that’s the limit with just one user…
(note it's 150k uncached tokens, the total limit is 450k/min)
I don't understand. How does that make it unusable? Is the limit shared by an entire team at once?
150,000 tokens per minute is a lot. You could start hitting that with a lot of concurrent requests in your session, but even throttled to 150k TPM it's still going to be faster than anything else you find.
I think the 128K context limit is the real ceiling. These models aren't amazing at long context, but once you account for a short input prompt, the input files, and headroom for a compaction summary, there isn't a lot left for the problem.
I did benchmark it in various ways and it runs quite well but it is a quantised jobbie and 1.5k t/s is also rather faster than anything I can possibly hope to achieve.
To run that model at those sorts of speeds is going to need some serious investment and you are going to have to pay for it.
We run a pretty large rig, 10 GPUs right now (this goes up and down with various experiments, getting this many GPUs to play nice at x16 GEN4 with any motherboard is a challenge), 240G VRAM in total. 256G RAM and a TR PRO. For small models the comms overhead is larger than the gains so there I have to reduce the number of active GPUs. On this machine I'm getting between 150 and 200 tg/s with FP8, but it took a lot of time and tweaking to get to that, and not all of the improvements held up when combined with other improvements. I've been playing with this stuff for a while now and it is interesting how fast the frontier is moving and how much you can now do on your own hardware. For larger models the communications overhead is low enough that we can run them on bigger groups of GPUs, and using hacked drivers to give us p2p capabilities on some of our GPUs also boosts performance considerably once you start to hit communications limits. Typically we get 50G/second in p2p mode (full duplex, half that one way).
From a cost perspective running locally is not interesting, but it allows us to do experiments that model providers would likely balk at, gives us censorship free access and allows us to work with data that we would not want to share with model providers (or can't share due to NDAs).
I will look into running ninfer, I was aware of them but had not yet gotten around to using it.
I was very excited last year for their coding plan but seeing a burst of requests pulse and then sitting there watching the cooldown reset is really not a great time.
Even though each individual request was fast, the sessions were only maybe 10% faster on wall clock time since there was so much waiting time.
Seems like an Amdahl’s law of inference economics? there’s so much compute relative to SRAM on the chip and shoreline bandwidth onto the chip that caching buys ~nothing? The contended resource is SRAM and a given token of context needs just as much as another.
I am unsure if it is incapable, but it sounds hard.
They have tons of cores with 64k of SRAM each, and relatively slow paths in/out.
On a GPU you can leave it in SRAM. On Cerebras, you have to send it out of the system which is a giant bottleneck.
"Context Length: 262,144 natively and extensible up to 1,000,000 tokens."
https://huggingface.co/Qwen/Qwen3.8-27B
It's in the link.
For comparison, I ran the same task with DeepSeek-V4-Flash, which finished in 172 seconds and cost $0.024 with a final context window size of 55217 tokens, while Qwen3.8-27B was not even close to being done with a 64178 context window.
This is a very efficient way to burn your money, but I would not recommend it for programming.
On the positive side, I got a $5 signup bonus, so it wasn't my own money.
This doesn't work for my use case at all (code generation).
These bursts of speed might work well for workflows that need bursts of quick decisions, followed by silence. But these workflows have needed provable determinism to som extent, so I haven't been using llms for those use cases. And I don't see myself using llms for them in the future too.
Also make sure you use the instruct temperatures/etc for implementation.
The Cerebras session cost me $1.60 and took a total of 5.1 mins. I did get a few brief 429 rate limit errors in there. The p50 speed was 890 tok/s and 0.64s TTFT.
Using OpenRouter averages, that would've cost $0.29 (no cache discount at Cerebras!) and would've taken about 14.4 minutes.
So on this one short session, cerebras was 5.6x more expensive in exchange for being 2.8x faster. Or, another way, $1.32 buys back about 9 minutes of your time. Not a bad trade IMHO but the cache situation is a real bummer. The longer your session the more relatively expensive Cerebras gets. The "good" news is you're also limited by its short context window.
(Also, I used to be on the Cerebras coding plan and the support is pretty bad for end users. My guess is these public endpoints are really just product demos for potential enterprise customers.)