I've not tried it out, but in the past I've been pretty disappointed in the performance of Kimi models. Deepseek, Qwen, and GLM all appear to perform much better even though Kimi often has higher benchmark scores. Ditto for Mimo.
The older Kimi K models (2.5 and 2.6) weren't bad for the time, but all of their models have had problems with overthinking. This also makes them significantly more expensive in practice than their pricing suggests, since they produce way more output (reasoning) tokens than most other models. Anecdotally, I've also had problems with all the pre-K3 models thinking themselves in circles trying to solve very simple problems that other models didn't struggle with. (I haven't played with K3 much because it's just too expensive.)
I have the opposite impression. I find K3 to be extremely capable at full-stack work; its UI design skills are great. I'd love to use it more, but the quota on Kimi's plans is abysmal. I wonder if these new models improve that.
You either include all labs which can reasonably produce models which are capable enough to be considered generally on-par with the best available models, in which case there has been way more than 2 frontier labs for a while now, or you take a more literal definition and only include labs which are capable of producing models which are considered "the best" out there, in which case there are basically just 2.
I think the former definition is a bit more meaningful since it avoids issues with determining what actually constitutes "the best" model out there. In that case, Google has been a frontier lab for a long time, Meta arguably as well, and you're also ignoring xAI which has been competitive for a while now. And, obviously, there's a bunch of Chinese labs which would deserve the title as well.
But if one is being strict (and willing to accept generally accepted definitions/measures/etc.), then there has only ever been two frontier labs: OpenAI and Anthropic. They're the only labs to have had "the best" models at any point in time since LLMs became a thing.
Moonshot has been frontier since this summer. At this point whenever they release a model it's on par with whatever the latest OpenAI/Anthropic model release was.
Not sure I'd put Google there in practice. And while Muse is an impressive jump from behind it's still not quite at the frontier either IMO.
I'm not sure what counts anymore. You've got OpenAI and Anthropic; then Meta, Google, and X; then over to Alibaba/Qwen, Z/GLM, Moonshot/Kimi, Deepseek; and Mistral in Europe. How many people out there in the real world can actually come up with tasks that differentiate those 10?
If you're gonna rank them: Anthropic > OpenAI > Google > Meta > (everyone else).
Gotta think Grok's cognitive dissonance due to the need to rewrite reality for Musk's political and moral biases is always going to keep them behind the frontier, but this is just speculation. The amount of money and power they have should put them at the top and I can't otherwise explain why they're consistently behind. Maybe Musk is simply too cheap/control freaky to throw enough money/benefits at good employees.
right now it looks like only anthropic since openai made the staggering business decision to release their midrange model (terra) rebranded as sol 6.1 and quantized astra into the ground.
everyone else mostly seems to have just decided to cheat on benchmarks to make their models look better than they actually are, and none seem to want to actually try to make better models.
I thought it was 6 Sol which was a rebadged Terra? You think 6.1 Sol is too? It does seem to benchmark better than 5.6 Terra? Regardless agree both models have been very bad for openAI's Sol brand. To be fair it's at least cheap compared to other OpenAI models.
It's still just 2. Google hasn't been at the top in a long while, and even when it was, it was just a tiny improvement over what other labs had previously released. Meta has never been at the top. Only two labs release LLMs that really push the limits out.
As long as LLM's can be distilled, and all these "labs" (what a weird name) can distill one another, there are going to be as many as capital is available to be burnt on the topic.
I mean, define frontier and in which sector, but for me (sample size n=1) programming I'd say 1 or 2.
Opus 5.5 is the developer frontier. 6 Astra might be alongside it, haven't tried it. Everyone else is at least slightly behind them. (I have tried Argon and it's better at graphic design and slightly worse at programming.)
Can someone please tell me what benchmarks to look for that measure world knowledge? For sspitballing, critical analysis, business projects, etc. Is the model size the only way to measure it?
> For the open-weight release of K3.1 Pro, we identified and removed cyber-related data from both training stages, while retaining the general-purpose coding tasks needed for model usefulness. We will continue to apply appropriate controls at every training stage, of which cyber-related data identification and restriction is one example.
I like it. If the increase in unrestricted cyber capabilities can be slowed down, please be it. The world needs to get ready, and that takes time. Way more time than even a slow progress would allow.
Unless you want to see more of what happened the past few days.
I think the argument that "cyber capabilities should be democratized, thus keep producing them now, fast, and for everybody" is a very naive one.
I am glad that at least one Chinese company seems to be listening to AI cybersecurity concerns and did concrete steps that actually reduce the problem. This was probably at least somewhat costly to them in terms of general model capabilities as well. They'd probably have gotten slightly better coding capabilities if not for removing the cyber training data and tasks.
I am glad about it as well. Don't get me wrong, I love OSS, I love personally being able to get my hands on the frontier stuff and frontier cybersec capabilities, and I dislike a few entities having exclusivity on it. Everybody does. But that absolutely doesn't mean it's the correct approach. I think people are blinded a little bit by general principles and by personal feelings, rather than operating from a point of pragmatism, consequences, and the big picture.
Also, to note: I don't claim to have the answers either. I don't even hold a strong position on this issue yet. I'm just sharing some of my intuitions, and reacting to people who, as far as I'm concerned, are making this issue all too simple.
The Pandora box can expand even further, and it matters. If it didn't matter, like you seem to be suggesting, then by the same logic there isn't a reason to push for further open-source development.
> Kneecapping cyber vulns only serve governments.
Uhh, no? Kneecapping cyber vulns server most of us. There are negatives to kneecapping it, but let's not pretend like it's perfectly okay and safe for everybody in the world to have a magic box that can hack anything and cause havoc all over the world. The more powerful, cheap, and accessible this magic box is, the worse the consequences.
This is a hard problem and there's real tension - like you said, the playing field not being leveled is a problem (major companies / entities / govts having exclusivity on the best cyber abilities etc), but everybody having access to it is a problem as well.
So let's not pretend like this is a simple problem, and don't say that it only serves the govt. That's very silly.
How does "the world" get ready, when these limits are so hard core that Opus regularly stops working when I'm just trying to find and fix security exploits in my apps?
Currently available, unhindered models can already find a bunch of exploits, not because they're smart, but because they're cheap and smart enough to just keep going for hours or days until it finds something.
The defenders, especially in the FOSS space, only have one big advantage: Unified scale. If defender A finds a vuln, then defender B doesn't need to find it again. Attackers generally act alone, so they need to do all the work themselves.
If we remove a bunch of defenders, the attackers are back at it again.
The problem with Grok 4.7 is that it's not amazing at anything, and didn't even retain the blazing fast performance of Grok 4.5 which made it awesome for highly interactive workflows.
But despite all the negative coverage, I can say with confidence that Grok 4.7 is a great code reviewer. You know who says that? Every other model. A lot of my time is spent orchestrating code reviews. I will give the same prompt to three models, then give each model the responses from the other two models to assess and rank them. Finally I give the rankings and meta-rankings to a fourth model to synthesise a final report.
Right now, for the code I'm dealing with (embedded c++, python, swift, and CFML) I'm fairly consistently seeing every other model rate Opus 5.5 first, with Grok 4.7 a very close second, often finding real issues other models miss.
It's a bit fascinating to me how all of these labs have rolled their own (it sounds like) completely unique RL environments.
I wonder how much the industry will centralize around a few key patterns, and if we'll start to see some RL framework projects gain traction and be used by the bigger labs
> First, we aim not to release open weights that would extend the absolute frontier of cyber capability. K3.1 Pro's offensive cyber capability is, by design, below what our pipeline can produce, and remains behind that of frontier proprietary models. Second, we intend to progressively raise the cyber capability available in open weights.
So they admit that they could do more, but they won't?
It's not actively nerfed like the Western frontier models, it's just downweighting offensive "cyber" as a domain for post-training/fine tuning. Which, if anything, ought to improve model capabilities wrt. other domains.
> So they admit that they could do more, but they won't?
Yeah, there's another paragraph:
> Identifying and removing cyber-related data. During development, we observed vulnerability research performance improving steadily as post-training scaled. Tracing the source of this trend, we found that cyber-related content was embedded within the general mid-training corpus, and cyber-related tasks were present in the RL environment mix — a natural byproduct of training on broad data. In preliminary experiments on internal models, we found that without data removal at either the mid-training or the RL stage, downstream cyber capability would increase substantially. These findings gave us a concrete lever for control. For the open-weight release of K3.1 Pro, we identified and removed cyber-related data from both training stages, while retaining the general-purpose coding tasks needed for model usefulness. We will continue to apply appropriate controls at every training stage, of which cyber-related data identification and restriction is one example.
(emphasis mine)
It's clear that cyber is a touchy subject right now. But it's also clear that RL works on it, because it's highly verifiable. So smaller models are going to become very interesting, because they can be fine-tuned later to cover this field.
It’s not much of a deterrent to a motivated organization.
They are basically saying that models themselves are already capable. If you fine tune them with some cyber security tasks, that should make them potent. :-/
There is no deterrent to a motivated and well resources organization. Post training a model in general, let alone just for offensive cybersecurity, is well within the technical and financial capabilities of many cyber criminals. There's just no way out of that.
I think this is misguided. Considering how much code is AI written these days do we really want LLMs writing insecure code because we lobotomised them?
>I hope the Chinese companies (and I say this as a patriotic American) slay these two companies and destroy their moats.
China cares about about stability way more than the US even, they aren't going to be big on AI models upsetting the governments power, and mass cyber attacks could do that.
At the end of the day we are going to have to realize AI is both dangerous and governments are going to control it. Worse if you start making and releasing your own dangerous models the 4chan party van is going to visit you.
> At the end of the day we are going to have to realize AI is both dangerous and governments are going to control it.
Your thinking is dangerous. Do you work for Anthropic?
AI is dangerous when these powers are being hobbled up by the select few. These systems threaten to put us all out of work. I refuse to be a slave to a company or a government that is taking away our livelihoods, scraping up our earnings potentials, then seeking to muzzle us and put us in jails with no rights.
I would rather burn down all of tech than have AI be owned by the elites.
Your thinking is a threat to all of our liberties.
>I would rather burn down all of tech than have AI be owned by the elites.
Then you better get your torches out now.
>I refuse to be a slave to a company or a government that is taking away our livelihoods
Don't worry, governments are fervently working on AI drones to take away your life when you get really upset about this. Oh, and they are also working at AI to detect when you write all this online, and track you down then monitor you on things like flock cameras. As they say, you'll be the first with your back against the wall when the revolution comes, I likely will be beside you for being too subversive.
>AI is dangerous when
Incorrect, AI is just dangerous. It doesn't matter who has it. The more powerful it becomes the more dangerous it gets. So, as you can tell I am an AI doomer.
----
Now, don't get me wrong, I want freedom for me, I want freedom for other people. I do not believe the technologies we're developing are conductive to having any freedom at all. Hell, if we're lucky we'll be slaves instead of all of us dead, including the select few.
>Your thinking is a threat to all of our liberties.
Lol, now this is kind of funny. "I want freedom to think whatever I want, but you, you shouldn't think like that!". Careful or you'll max out your irony meter. Anyway, "My kind of thinking" is just next ____ prediction. I see what is happening in the world around us, and based on the shape of that probability I see what is going to happen. No, you and I are not getting unlimited AI to do whatever we want. Yes, governments are going to lock down the shit out of AI for end users. Yes, governments are going to declare advanced LLMs munitions.
Balls in your court, better think of what your next move is.
Which is a great example of what makes him dangerous: with the exception of Sam Altman he's arguably the single person most responsible for the current capabilities arms race.
Anthropic has pushed the frontier harder and faster than OpenAI in the last few years, culminating with Mythos/Fable. He's always talked big Anthropic prioritising safety and "Responsible Scaling" but his words just don't align with reality while they keep going bigger until OpenAI can only half heartedly claim near parity at the top.
Whether it's wilful cognitive dissonance, cynical opportunism, financial greed or something else, that kind of self-assured moral superiority paired with his real world record of self-interested decisionmaking for Anthropic's market position is a big red flag.
Sam, I'll grant you, is probably among the top 10 most dangerous, but Dario? The only thing he's done to piss people off is to predict, correctly, that releasing too-powerful models too-quickly would cause havoc and chaos. This prediction actively hurts his company! It actively harms his chances of a huge exit! It actively raises the probability that Anthropic and OpenAI get nationalized and treated in the same way as, for instance, the nuclear program!
By comparison, do you think the Chinese models are going to be trained in such a manner that you enjoy the society that results? What makes you think that?
> It's clear that cyber is a touchy subject right now.
And that's why we need models that understand "cyber" or application and data security. The offensive models are powerful, and they are in the hands of script kiddies now. The models writing our software and designing our cloud infrastructure had better understand these issues as well, just to avoid coding in vulnerabilities.
> Prohibited: high-performance detection technologies used to counter cross-border data transmission security risks, including deep-synthesis forgery detection (266401J); AI cyber offensive capability technologies, such as malicious code generation and mutation, automated mass generation and distributed large-scale deployment of anti-detection variants (266402J); large models trained using China’s Core Data (266501J); and high-performance AI agents designed for highly sensitive scenarios, such as critical information infrastructure, national defense and core government affairs (266502J).
It's also interesting how the field is still at the "we poked this button then looked at what happened" phase and they can't predict what will cause what.
I mean they say they will release the weights by November 4, so presumably after that it won’t be hard for people to fine-tune it to be better at cyber right?
Presumably if it knows how to find and fix bugs it can be told how to exploit them. But who knows what it will actually be capable of doing in the real world as opposed to Kiki’s benchmarks
Moonshot (Kimi) seems to be unashamedly into American culture[0], I wonder if they're paying closer attention to and giving more credence to cultural aspects of American AI discourse including aspects of AI caution.
All postwar english language culture that isn't much more explicitly non-American than that is American culture. All good rock music is American culture by default, unless it is near-constantly screaming otherwise.
Regardless, expand both sides of the point from "American" to whatever you need to, the idea is the same either way.
With frontier competitors and the U.S. govt actively pushing "concerns" about Chinese models, Kimi including this statement on "offensive cyber capability" seems like smart defense. Models are opaque and training details are trade secrets, so whatever they did might be negligible or significant and no one can tell. This bit is also artfully done:
> "below what our pipeline can produce, and remains behind that of frontier proprietary models."
Like Ant and OAI, they've realized vaguely claiming to have nerfed their latest model to an unspecified degree not only looks 'good guy' prudent, it provides a built-in excuse when a model fails.
We really need some breakthrough like DiLoCo does, but for heterogeneous unstable consumer hardware to make volunteer crowdsourced AI training a reality. Yeah, I understand from current technical angle how much of a mountain that is. Call it AI@Home or whatever. I never thought it would bother me personally or even become a problem, but these companies acting like gatekeepers and virtue signaling while also in bed with governments etc.. this leads nowhere good.
I wish Opus 5.5 had an alternative just for the sake of not having a monopoly, but I have yet to see a model live up to its benchmark scores except for Fable and Opus; there is something about Anthropic that no other lab has.
Yeah, but it's not good enough. Ask Opus and Astra to make a landing page about lemon pies or whatever. Opus will get you a better result almost every time.
What kind of coding/tasks?
I find all of the main ones, including the Chinese ones, to be basically on par.
Enterprise programming (CRUD all the way down, not simple one though, full ERP)
Just ask Opus and Kimi to build a game about something random, like "make a Mario Kart-inspired game." Opus gives better results most of the time. That doesn't mean you can't get your particular job done with either of these models.
When the frontier is pacing itself in its speed in terms of adding new capabilities (notice how fast opus 5.5 got released after Astra), then that also means that the open weights models which were just behind are accelerating in their speed just as much as well.
Good news for us because the models which used to feel 6 months behind now feel like they are lesser.
Double threat to Anthropic and OAI because if open weights models become good enough at the current level of Sota while being cheaper (which time will tell if this model fits the bill or not), then people will use those models instead of OAI/Anthropic provided it is "good enough"
Though it has to be said that only real world use case can be trusted but I remain optimistic and hope that this turns out to be good in real world use cases as well.
But in my opinion, all of this is really good for us end users!
looks extremely impressive, the full benchmark table isn't opening for some reason though so do we really have agi if small important things like that keep breaking?
It would be super useful to have more specific analyses of what areas the model is good at. I'd like to know when I should reach for this model instead of Anthropic's or OpenAI's.
For example, Opus 5.5 is better at UI design than Sol. Anyone who has used both models knows that, but it's not something you can look up anywhere.
I'm sure that there's a need for this kind of benchmarking, not just for coding.
Not sure if this is benchmarkable besides for the own person deciding which one they like more. Models difference is more about taste, preferences and prompt style.
I had a friend yesterday tell me he likes the way Claude writes and finds it easier to understand compared to Gemini. But he mostly doesn't use Claude for coding. Whereas I find Claude insufferable in writing often and prefer Gemini for writing. But I don't use Gemini for coding.
Though to be fair Opus 5.5 is better than Opus 5 was in that regard. I still prefer the OpenAI models to either of them when it comes to regular prose, though.
Furthermore, "UI design" is not a narrow enough category, I think. Sure, statistically significant differences in performance between models do manifest when benchmarking against "UI design", but there's still just so much variance within the category. Opus 5.5 can have excellent one-shot results, but totally shit the bed when it's placed in an existing codebase and should uphold the UI/UX conventions, design language, and existing component primitives already established in an app.
But that's just the obvious. There are task by task differences (component by component, page by page, context by context) that the model is worse and better at, even though they are all within the general category of UI design.
I honestly don't have the clarity of mind to explain this coherently, but the crux of my point is that there's a huge variation in it's performance in different contexts, often ones that seem near similar, and in result, even Sol can beat Opus 5.5 in some variations of UI design.
And this is my problem in benchmarking and comparing model performance in general. Presently, we try to create broad categories to benchmark against, create a set of tasks, and then average out the results. That works generally, but it's kind of misleading and it's partly what drives people to feel a disconnect between benchmarks numbers and real world usage on their specific tasks.
It's such a messy field that it's kind of maddening.
One things Kimi is useful for is accessing/querying websites that have paywalls or try to prevent use by AI agents. It is aggressive and persistent for that, though not always successful.
So, the US economy spends trillions of dollars to be at the "frontier" and China puts out a model that basically does the same thing, for free, and for a tiny fraction of the development cost. How is this sustainable?
It's not like US companies spend trillions on training itself.
Mostly it's datacenter build out and circular financing magic when the same dollars being commited multiple times by multiple companies passing hot potato to each other. They basically commit capacities 2-3 years ahead.
There would be no Chinese model without US models to distill though. So, in a way, the Chinese models also cost a trillion dollars, they just don’t cost China that much.
It's 'sustainable' because there's enough regulatory risk to using Chinese models that many will avoid them, and a 3T parameter model is still a bit beyond even many enthusiasts for running locally.
That regulatory risk argument won't work forever. It's the same with how the US bans chinese EVs which is why Tesla has high market share in the US, but in the rest of the world, chinese EVs are doing big numbers. The future is going to be full of surprises.
Maybe I'm conflating. I could be dead wrong about all this. But I'm trying to zoom out and look at the big picture. So, I'm letting the details sort of blend away and I'm just asking myself: "Is such a large financial sink in our economy, at the macro level, really going to offer the value that the investment warrants?". So, the particulars of who exactly is spending what where I guess doesn't concern me as much. What I'm thinking about is American companies, American people, American dollars going to something that seems like an iffy value proposition.
Now, if you tell me American businesses are turning cheap Chinese tokens into goods and services of high value, for Americans and people all over the world, then obviously I would have a different outlook on what's happening here.
You are conflating. The build out is almost entirely for running inference, not training. Open weight models will make that inference build out bet more likely to work as it adds diversity to the suppliers, all the data centers are not beholden to just OAI and Anthropic to buy their compute - they can host their own models (even Chinese ones, in America) if that becomes a popular option.
I see. So maybe the situation isn’t as bad as I thought. I was thinking if you took all the money and resources spent on building the biggest models, across the two big labs, the hyperscalers, the dozens of neo labs, and you reallocated that money to building value on cheaper Chinese models that would be a much better proposition than doing whatever it takes to be head of Chinese model performance by a few percentage points for a few months. And I wasn’t just referring to the cost of training one model, I was considering the total cost of maintaining an entire ecosystem with the ability to do this.
But what I’m hearing is this is a negligible cost compared to inference buildouts. Fair enough.
To be fair I have a feeling that if you added 'animated' to the SVG request, the output quality from many other models would improve, just because of the extra chain-of-thought work needed to fulfill the more complex request.
If you are expecting any sort of usable limits on their subscription plan I suggest adjusting your expectations. I subscribed to $99 a month plan and within 47 minutes with one session (not asking for swarms or anything) was limited by 5-hour quota.
Welp, that ruins this news for me. I'm glad open models are catching up but if I can't practically run them locally _or_ through a flat fee then it doesn't really help me. I fear API pricing.
This'll end up on opencode go soon enough. $40/month on there gives you a bit more than just "light use"
Don't use frontier models like this for problems that can be solved with flash models, and you'll see API pricing isn't that terrible. Still cheaper than anything with remotely this level of capability 2.5 months ago.
Yeah, the plan is far too expensive for what it provides. I wonder if the quota with these new models will be better, but I'll wait for others to try this out before wasting another hundred bucks.
Had a similar experience here. Their limits are very, very low, and ultimately make it not worth subscribing. Can't leverage a good modal if it exhausts its 5-hour budget in 30 minutes.
Paying API prices for Deepseek 4.1 Flash feels like a much better deal at the moment.
Same experience. I had the $19 monthly plan for experimenting and running benchmarks, but it was a uselessly small amount of usage, a single task would blast through the whole five hour limit and a big chunk of the weekly limit. All of the US provider plans are remarkably more generous (though OpenAI plans have gotten a lot stingier lately, where they used to be the most generous).
I have been using DS Flash 4.1 pretty heavily. It runs for hours and hours and is pretty cheap.
I realized the other day that I had a $30 a month Claude subscription. So, I fired it up all excited and tried Opus 5.5 on a codebase I was already in with DS. After about 30 minutes I hit a 5 hour limit. The next day I tried again and the same thing happened. And it felt much slower to boot. I was super disappointed.
DS 4.1 is basically endless, even if you paid for APIs you will hardly touch some prices.
But it's slow.
I don't think I can any longer appreciate any major difference in "intelligence" between models for most tasks but Opus is consistently much faster than competitors at finishing tasks and keeping me in the loop.
Of course excluding ultra code, the you will wait 50 minutes for a minor refactor.
If you have a very large (multi million lines of code) and have the prompt say
build a comprehensive knowledge base of this codebase starting from high level user features all the way to how each file and method functions and do not return or stop until you have built and organized this knowledge base into an accessible and interactive web application.
Yeah, the $20 Claude plan is stingier than one might like, but not as stingy as Kimi. I think the best $20 subscription deal these days is probably from Google, now that Gemini 4 Argon exists and is an acceptably high quality model for coding. It's got quite generous usage and includes some other Google account upgrades (though I know some people don't use their regular account for Google AI for fear of account suspension or whatever, so the extra storage and other account upgrades are not a useful thing). I can use it pretty heavily without ever running into caps. The $20 ChatGPT plan used to be similarly generous, but no more.
When I've most done the same tasks across multiple providers I have a subscription to, I found the order of generosity of them was something like Gemini->Claude->ChatGPT->Kimi. OpenAI used to be a lot more generous than all of them, but that ended recently, and now it's quite stingy in the $20 plan. What competitive model has a more generous subscription than Gemini?
I have a DeepSeek account and almost always use it when I need something with API access. I consider it Good Enough™, if not competitive. I also have a MiMo token plan, though that's weird in that it's not a monthly subscription, it's just a bulk buy of tokens that expire after a year, but still seems to be a pretty good deal on a pretty good model.
It is a LOT more efficient on claude code. Takes forever to run out of credits, and a whole lot of useful code gets generated. The web Ui reconstructs code while claude code just tweaks it.
How long ago was this? The amount of usage they give you for a given plan seems to change over time, and the Claude limits seem to be more favourable recently.
That's surprising, I had the exact same experience as GP. I dropped my Kimi sub this month, but I'm keeping claude because I can use Opus 4.8 basically endlessly. Even Opus 5.5 doesn't drain the usage that much, though I rarely use it because it refuses to do anything cyber-related.
Honestly, use Pi, start the session with K3, design it, store the design + chat history + todo with phases + why + readme then when you are at the time to build do `/model` select deepseek and let it rip!!
You can even make a Pi extension to let it switch models on its own, then tell it to switch to one or the other for a given task. It's not perfect, but it is pretty reliable once you have the instructions dialed in.
I have mine configured to use a "thinker" model for planning and decision making, and a "doer" model for the easy stuff (digging around the repo, writing code based on the thinker's instructions, etc.).
I run a similar setup, just with different models. I also use herdr, which lets me have the same project worked on side by side by different models, each in its own git worktree.
How? I've been running k3.1 Pro with K2.8 preview for codegen (subagents) and I've only used 22% of my weekly and it's been running for 8 hours on a RE project.
Are you using k3 with thinking set to the absolute highest and only using k3 for everything?
Interesting that they specifically nerf cyper-capabilities of open weight models:
"In preliminary experiments on internal models, we found that without data removal at either the mid-training or the RL stage, downstream cyber capability would increase substantially. These findings gave us a concrete lever for control. For the open-weight release of K3.1 Pro, we identified and removed cyber-related data from both training stages, while retaining the general-purpose coding tasks needed for model usefulness."
> Kimi K3.1 Pro is a post-trained version of Kimi K3, with substantially larger post-training compute.
It's very nice that we entered a time where models are large enough and we can push the frontier just by training them longer.
When the next generation of gardware comes the ones on the smaller size will be running on a computer that might cost as little as a good car.
So after a home and a car, next major family purchase might be personal AI. And when the robots get good, pehraps in a decade, a fourth major purchase might pop up that will pair nicely with the third one.
Why would a family need an AI? Maybe it's the same kind of question as why would it need a personal computer.
568 comments
[ 4.9 ms ] story [ 686 ms ] threadChinese models never agreed to the pacing anyway lol.
It appears to generally beat or match Opus 5.5.
Edit: or they lie... https://artificialanalysis.ai/models/kimi-k3-1-pro
A few weeks ago it was 2. Argon happened 23 days ago; Muse Wonder was 11 days ago.
I think the former definition is a bit more meaningful since it avoids issues with determining what actually constitutes "the best" model out there. In that case, Google has been a frontier lab for a long time, Meta arguably as well, and you're also ignoring xAI which has been competitive for a while now. And, obviously, there's a bunch of Chinese labs which would deserve the title as well.
But if one is being strict (and willing to accept generally accepted definitions/measures/etc.), then there has only ever been two frontier labs: OpenAI and Anthropic. They're the only labs to have had "the best" models at any point in time since LLMs became a thing.
Not sure I'd put Google there in practice. And while Muse is an impressive jump from behind it's still not quite at the frontier either IMO.
Gotta think Grok's cognitive dissonance due to the need to rewrite reality for Musk's political and moral biases is always going to keep them behind the frontier, but this is just speculation. The amount of money and power they have should put them at the top and I can't otherwise explain why they're consistently behind. Maybe Musk is simply too cheap/control freaky to throw enough money/benefits at good employees.
everyone else mostly seems to have just decided to cheat on benchmarks to make their models look better than they actually are, and none seem to want to actually try to make better models.
As long as LLM's can be distilled, and all these "labs" (what a weird name) can distill one another, there are going to be as many as capital is available to be burnt on the topic.
Opus 5.5 is the developer frontier. 6 Astra might be alongside it, haven't tried it. Everyone else is at least slightly behind them. (I have tried Argon and it's better at graphic design and slightly worse at programming.)
Lame
Unless you want to see more of what happened the past few days.
I think the argument that "cyber capabilities should be democratized, thus keep producing them now, fast, and for everybody" is a very naive one.
Also, to note: I don't claim to have the answers either. I don't even hold a strong position on this issue yet. I'm just sharing some of my intuitions, and reacting to people who, as far as I'm concerned, are making this issue all too simple.
Kneecapping cyber vulns only serve governments, Pandora's Box is already open for cyber-related stuff.
> Kneecapping cyber vulns only serve governments.
Uhh, no? Kneecapping cyber vulns server most of us. There are negatives to kneecapping it, but let's not pretend like it's perfectly okay and safe for everybody in the world to have a magic box that can hack anything and cause havoc all over the world. The more powerful, cheap, and accessible this magic box is, the worse the consequences.
This is a hard problem and there's real tension - like you said, the playing field not being leveled is a problem (major companies / entities / govts having exclusivity on the best cyber abilities etc), but everybody having access to it is a problem as well.
So let's not pretend like this is a simple problem, and don't say that it only serves the govt. That's very silly.
Currently available, unhindered models can already find a bunch of exploits, not because they're smart, but because they're cheap and smart enough to just keep going for hours or days until it finds something.
The defenders, especially in the FOSS space, only have one big advantage: Unified scale. If defender A finds a vuln, then defender B doesn't need to find it again. Attackers generally act alone, so they need to do all the work themselves.
If we remove a bunch of defenders, the attackers are back at it again.
MiniMax -> Qwen -> DeepSeek (pro) -> Gemini (Carbon) -> Fable (to beat Carbon) -> Astra (to be slightly worse than Fable but cheaper).
I doubt we'll see any major US release besides Grok in the next 3 weeks.
Would be nice if we got a Terra that's competitive with Gemini Flash and DeepSeek Flash.
The fun part is, we're already at the point where we are seeing something frontier-ish almost every single week.
The really fun part will be when they all start adding System 1-3 natively (IMO) - which I think will be the next release cycle.
But despite all the negative coverage, I can say with confidence that Grok 4.7 is a great code reviewer. You know who says that? Every other model. A lot of my time is spent orchestrating code reviews. I will give the same prompt to three models, then give each model the responses from the other two models to assess and rank them. Finally I give the rankings and meta-rankings to a fourth model to synthesise a final report.
Right now, for the code I'm dealing with (embedded c++, python, swift, and CFML) I'm fairly consistently seeing every other model rate Opus 5.5 first, with Grok 4.7 a very close second, often finding real issues other models miss.
I wonder how much the industry will centralize around a few key patterns, and if we'll start to see some RL framework projects gain traction and be used by the bigger labs
P.S. weights by November 4th
> First, we aim not to release open weights that would extend the absolute frontier of cyber capability. K3.1 Pro's offensive cyber capability is, by design, below what our pipeline can produce, and remains behind that of frontier proprietary models. Second, we intend to progressively raise the cyber capability available in open weights.
So they admit that they could do more, but they won't?
Yeah, there's another paragraph:
> Identifying and removing cyber-related data. During development, we observed vulnerability research performance improving steadily as post-training scaled. Tracing the source of this trend, we found that cyber-related content was embedded within the general mid-training corpus, and cyber-related tasks were present in the RL environment mix — a natural byproduct of training on broad data. In preliminary experiments on internal models, we found that without data removal at either the mid-training or the RL stage, downstream cyber capability would increase substantially. These findings gave us a concrete lever for control. For the open-weight release of K3.1 Pro, we identified and removed cyber-related data from both training stages, while retaining the general-purpose coding tasks needed for model usefulness. We will continue to apply appropriate controls at every training stage, of which cyber-related data identification and restriction is one example.
(emphasis mine)
It's clear that cyber is a touchy subject right now. But it's also clear that RL works on it, because it's highly verifiable. So smaller models are going to become very interesting, because they can be fine-tuned later to cover this field.
Agreed that this could be reinjected later. Makes me think of neo in the matrix:
“I know zero days.”
“Show me.”
Do you think that going around making cyber weapons isn't going to get you buried in a shallow grave?
They are basically saying that models themselves are already capable. If you fine tune them with some cyber security tasks, that should make them potent. :-/
If the "scary Chinese open weights models" can do "dangerous cyber", then we might see regulations spring up against open weights sooner than later.
I hate Anthropic and OpenAI. As companies and leadership, anyway.
Dario and Sam are 100% to blame for this. They're trying to turn us all into jobless slaves and sex workers.
The only good future is open weights. They're trying their damnedest to keep us under boot.
I hope the Chinese companies (and I say this as a patriotic American) slay these two companies and destroy their moats.
Speak for yourself, I don't think there is a market for many of us as the latter.
China cares about about stability way more than the US even, they aren't going to be big on AI models upsetting the governments power, and mass cyber attacks could do that.
At the end of the day we are going to have to realize AI is both dangerous and governments are going to control it. Worse if you start making and releasing your own dangerous models the 4chan party van is going to visit you.
Your thinking is dangerous. Do you work for Anthropic?
AI is dangerous when these powers are being hobbled up by the select few. These systems threaten to put us all out of work. I refuse to be a slave to a company or a government that is taking away our livelihoods, scraping up our earnings potentials, then seeking to muzzle us and put us in jails with no rights.
I would rather burn down all of tech than have AI be owned by the elites.
Your thinking is a threat to all of our liberties.
Then you better get your torches out now.
>I refuse to be a slave to a company or a government that is taking away our livelihoods
Don't worry, governments are fervently working on AI drones to take away your life when you get really upset about this. Oh, and they are also working at AI to detect when you write all this online, and track you down then monitor you on things like flock cameras. As they say, you'll be the first with your back against the wall when the revolution comes, I likely will be beside you for being too subversive.
>AI is dangerous when
Incorrect, AI is just dangerous. It doesn't matter who has it. The more powerful it becomes the more dangerous it gets. So, as you can tell I am an AI doomer.
----
Now, don't get me wrong, I want freedom for me, I want freedom for other people. I do not believe the technologies we're developing are conductive to having any freedom at all. Hell, if we're lucky we'll be slaves instead of all of us dead, including the select few.
>Your thinking is a threat to all of our liberties.
Lol, now this is kind of funny. "I want freedom to think whatever I want, but you, you shouldn't think like that!". Careful or you'll max out your irony meter. Anyway, "My kind of thinking" is just next ____ prediction. I see what is happening in the world around us, and based on the shape of that probability I see what is going to happen. No, you and I are not getting unlimited AI to do whatever we want. Yes, governments are going to lock down the shit out of AI for end users. Yes, governments are going to declare advanced LLMs munitions.
Balls in your court, better think of what your next move is.
If I don't get it, you don't either.
Anthropic has pushed the frontier harder and faster than OpenAI in the last few years, culminating with Mythos/Fable. He's always talked big Anthropic prioritising safety and "Responsible Scaling" but his words just don't align with reality while they keep going bigger until OpenAI can only half heartedly claim near parity at the top.
Whether it's wilful cognitive dissonance, cynical opportunism, financial greed or something else, that kind of self-assured moral superiority paired with his real world record of self-interested decisionmaking for Anthropic's market position is a big red flag.
By comparison, do you think the Chinese models are going to be trained in such a manner that you enjoy the society that results? What makes you think that?
These times are not ideal but what can we do other than just take it as it comes? All the hate and rage is not good for your health.
Kimi: We took out the worst of it so have at it for free.
Everyone else: Back to my ablated model on openrouter for 1/10th the price where I can brute force issues.
https://featherless.ai/blog/abliterated-models-for-cybersecu...
Is a decent primer, then off to open router you go.
> Prohibited: high-performance detection technologies used to counter cross-border data transmission security risks, including deep-synthesis forgery detection (266401J); AI cyber offensive capability technologies, such as malicious code generation and mutation, automated mass generation and distributed large-scale deployment of anti-detection variants (266402J); large models trained using China’s Core Data (266501J); and high-performance AI agents designed for highly sensitive scenarios, such as critical information infrastructure, national defense and core government affairs (266502J).
[0] https://x.com/crystalsssup/status/1944287779896328668
Regardless, expand both sides of the point from "American" to whatever you need to, the idea is the same either way.
Maybe GP meant to say that the model was westernized.
> "below what our pipeline can produce, and remains behind that of frontier proprietary models."
Like Ant and OAI, they've realized vaguely claiming to have nerfed their latest model to an unspecified degree not only looks 'good guy' prudent, it provides a built-in excuse when a model fails.
They're not post training it to lose capabilities/censor them – they just don't spend training time on this topic, which sounds like a great idea.
It doesn't mean model can't do it, it just means it's less good at it; -10% on benchmark feels like a great result and overall approach to pace it.
I concede that term can carry overly broad connotations.
The way to stop the vulnerability apocalypse is to fix the vulnerabilities, not to shoot the damn messengers.
I hope someone abliterates and FTs this model on vulnerability discovery so we can find more bugs in our testing, insecure tech stacks.
When the frontier is pacing itself in its speed in terms of adding new capabilities (notice how fast opus 5.5 got released after Astra), then that also means that the open weights models which were just behind are accelerating in their speed just as much as well.
Good news for us because the models which used to feel 6 months behind now feel like they are lesser.
Double threat to Anthropic and OAI because if open weights models become good enough at the current level of Sota while being cheaper (which time will tell if this model fits the bill or not), then people will use those models instead of OAI/Anthropic provided it is "good enough"
Though it has to be said that only real world use case can be trusted but I remain optimistic and hope that this turns out to be good in real world use cases as well.
But in my opinion, all of this is really good for us end users!
The only reason people aren't doing this is that hardware prices are high, thanks to, well, OpenAI and Anthropic :D
I wonder when will this madness end
--
Model K3.1 Pro K3.1 Pro Highspeed. K3
Input ¥13.00 ¥26.00 ¥20.00
Output ¥68.00 ¥136.00 ¥100.00
Cache hit ¥1.30 ¥1.30 ¥2.00
Cache write ¥13.00 ¥26.00 ¥20.00
For example, Opus 5.5 is better at UI design than Sol. Anyone who has used both models knows that, but it's not something you can look up anywhere.
I'm sure that there's a need for this kind of benchmarking, not just for coding.
Familiarity breeds contempt.
Though to be fair Opus 5.5 is better than Opus 5 was in that regard. I still prefer the OpenAI models to either of them when it comes to regular prose, though.
But that's just the obvious. There are task by task differences (component by component, page by page, context by context) that the model is worse and better at, even though they are all within the general category of UI design.
I honestly don't have the clarity of mind to explain this coherently, but the crux of my point is that there's a huge variation in it's performance in different contexts, often ones that seem near similar, and in result, even Sol can beat Opus 5.5 in some variations of UI design.
And this is my problem in benchmarking and comparing model performance in general. Presently, we try to create broad categories to benchmark against, create a set of tasks, and then average out the results. That works generally, but it's kind of misleading and it's partly what drives people to feel a disconnect between benchmarks numbers and real world usage on their specific tasks.
It's such a messy field that it's kind of maddening.
Mostly it's datacenter build out and circular financing magic when the same dollars being commited multiple times by multiple companies passing hot potato to each other. They basically commit capacities 2-3 years ahead.
a lot of those money is the commitment from US AI labs that's not yet spend until the data center is roll out.
you are conflating.
Now, if you tell me American businesses are turning cheap Chinese tokens into goods and services of high value, for Americans and people all over the world, then obviously I would have a different outlook on what's happening here.
But what I’m hearing is this is a negligible cost compared to inference buildouts. Fair enough.
kimi-k3.1-pro $2.00 / MTok $10.00 / MTok
cached: $0.20 / MTok $2.00 / MTok $4.00 / MTok
Here's Kimi K3 for comparison, though I didn't try that on different reasoning levels: https://tools.simonwillison.net/markdown-svg-renderer?url=ht...
Also the recent Geminis - https://simonwillison.net/2026/Oct/13/llm-gemini/ - and GPT-6 Astra on Max: https://static.simonwillison.net/static/2026/gpt-pelicans-gr... - and Claude Fable 5.1 https://static.simonwillison.net/static/2026/claude-pelicans...
To be fair I have a feeling that if you added 'animated' to the SVG request, the output quality from many other models would improve, just because of the extra chain-of-thought work needed to fulfill the more complex request.
If enough people are flagging it, we may be at a point where the pelican benchmark is truly overdone.
Don't use frontier models like this for problems that can be solved with flash models, and you'll see API pricing isn't that terrible. Still cheaper than anything with remotely this level of capability 2.5 months ago.
I think Kimi just doesn't have the means to subsidize plans.
Paying API prices for Deepseek 4.1 Flash feels like a much better deal at the moment.
I realized the other day that I had a $30 a month Claude subscription. So, I fired it up all excited and tried Opus 5.5 on a codebase I was already in with DS. After about 30 minutes I hit a 5 hour limit. The next day I tried again and the same thing happened. And it felt much slower to boot. I was super disappointed.
and how are costs like for your usual workflows/etc.?
But it's slow.
I don't think I can any longer appreciate any major difference in "intelligence" between models for most tasks but Opus is consistently much faster than competitors at finishing tasks and keeping me in the loop.
Of course excluding ultra code, the you will wait 50 minutes for a minor refactor.
build a comprehensive knowledge base of this codebase starting from high level user features all the way to how each file and method functions and do not return or stop until you have built and organized this knowledge base into an accessible and interactive web application.
It will easily run over a 5hr limit
And that was 6.1 sol xhigh, not astra max..
Here's mine: https://pi.dev/packages/@arcanemachine/pi-model-switcher
I have mine configured to use a "thinker" model for planning and decision making, and a "doer" model for the easy stuff (digging around the repo, writing code based on the thinker's instructions, etc.).
https://github.com/earendil-works/pi
https://github.com/herdrdev/herdr
Are you using k3 with thinking set to the absolute highest and only using k3 for everything?
At the beginning it was the best coding agent I'd used so far and I'd used the most popular ones and the more niche opencode models.
Within the last week, Usage has fired up at least x3 to x4 times on the same projects and tasks.
Just now, sonnet and opus doubled-down on a game bug even when I provided a video which clearly showed what was wrong.
Something is definitely up.
"In preliminary experiments on internal models, we found that without data removal at either the mid-training or the RL stage, downstream cyber capability would increase substantially. These findings gave us a concrete lever for control. For the open-weight release of K3.1 Pro, we identified and removed cyber-related data from both training stages, while retaining the general-purpose coding tasks needed for model usefulness."
There was law proposal (rumour?) last week about Chinese labs having to get approved for weights release now.
Still get the model out there.
It's very nice that we entered a time where models are large enough and we can push the frontier just by training them longer.
When the next generation of gardware comes the ones on the smaller size will be running on a computer that might cost as little as a good car.
So after a home and a car, next major family purchase might be personal AI. And when the robots get good, pehraps in a decade, a fourth major purchase might pop up that will pair nicely with the third one.
Why would a family need an AI? Maybe it's the same kind of question as why would it need a personal computer.