This is exciting. The continued focus on efficiency alongside quality is encouraging - it makes me hopeful to see what smaller models will become capable of.
I'm also hoping someone puts out a competitive 30-70B model soon. There's this massive dead zone between 24GB and <100GB of VRAM where it seems your best option is Qwen 3.8 27B.
Same, I recently found Nemotron 3.5 74B-A9B really good at filling this gap though, could do (very slow) local inference on just 32GB DDR4, without the quality dropping to the floor
The 3.8 Next-Flash release is comfortably the best model on a 96GB Mac Studio. I’ve tried various engines and Sushi (https://github.com/beamivalice/sushi) is the most performant so far.
Hopefully the Qwen 4 Flash model will still fit within 96GB without requiring excessive quantisation as, verbose thinking aside, 3.8 is a very solid workhorse model.
Ah yes, an open-weights, soon to be abliterated model that beats Opus 5.5 in checks notes bioinformatics and "legal research".
Some part of me is excited for the coming catastrophe(s), because it'll end the frustrating last ~12mo of "those Anthropic jerks just want regulation so they can corner the market on artificial minds wrought from sand!". Another part of me knows that;
A) Rapid epistemic decompression of this scale will probably lead to some truly nasty, illogical sentiments, not to mention other pathologies (deep depression, compulsive anxiety, etc.). Maybe some will reasses their beliefs after some sober reflection, but a minotiry, I expect.
B) Trading hundreds, thousands, millions, or billions of lives so that I feel a little less terrified and alone isn't exactly compelling utilitarian calculus...
Anyway, to the tech: another impressive jump for Alibaba, catching Moonshot overall with some interesting niche improvements and a massive, gaping flaw in the poor TerminalBench scores. I'm probably just biased as a SWE, though. I can't help but feel like Ziphu will be the ones to really blow our socks off next, given how long it's been since GLM-5.3[1] and the vague rumours about the next one (5.5...?).
The Chinese labs' leaps to catch up to Anthropic & OAI from the strange, long-forgotten past of September 2026 were clearly due to a tripartite distillation campaign, so expansive and rushed that they accidently leaked tons of CCP state secrets.[2] Has anyone heard any rumours about if that's been continuing after the crackdown? Anthropic has been pretty damn aggressive with catching distillation, so my personal guess is that this is just honest improvements.
...which would mean that at least 2 of the 4 Chinese labs are ~3-5 months from the RSI possibilities Anthropic is trying to sound the alarm about today, if that. Jesus fucking christ. We need to rally around international protections ASAP, cause something tells me at least one of the four won't quite so ready to "pace" themselves in the current climate.
This is not an open-weights modell. You have to wait for the smaller models
"Smaller models in the family are coming to open source, built on the same lineage and trained with the same generation of techniques, tuned for what small models do best."
...really? Isn't that massive change, if true? Wild! They're burying the lede there to say the least, if that is actually a long term stance and not just the typical "weights to come" release cycle.
If Anthropic truly believes we are on the verge of catastrophe but nobody is listening because they've been crying wolf for years about it while they get rich then they should truly do something. By "do something" I don't mean try again to convince the world to slow down as they IPO their bleeding edge research and solution (which is creating the doubt you mention) but just say "hey, we're so sure we're months away from worldwide disaster that we're shutting down our AI operations completely in hopes we can all put some attention to this before it's too late - will you follow and work together to come up with ways to prevent this problem from happening".
After all, unless their goal to say "I told you so but I wanted to make money more than I wanted you to believe me", what good is the current approach doing them?
Another Qwen doesn't seem all that scary, a few nation-state actors have likely exfiltrated weights of much stronger non-open models and abliterated them for use in large-scale collection on targets foreign and domestic. But god forbid someone mods a video game or jailbreaks their phone without applying for Claude Mythos or Daybreak, right?
I hope they are able to serve 4 faster. I’ve regretted including Qwen 3.8 in the battle system for opencode [1] we have been building because it scores so low.
And it’s a shame because the only reason it scores so low is that it’s slooooow. (Like 30 tokens per second vs opus near 80.) If you exclude voting, Qwen 3.8 is near the top.
(We allow early voting contrary to Arena.ai because I think it makes sense for agents, where it doesn’t for chat.)
Also, it would be ideal if they served this with third parties. Part of the slowness with 3.8 is just not a lot of providers. Their open source version is much faster, but we couldn’t include it because it doesn’t handle images (another own goal by Alibaba).
In case it translates to Qwen 4 27B, I've been running Qwen 3.8 Flash Next (which while MoE (and bigger) is supposed to be a preview of Qwen 4's architecture) and it runs pretty solidly regardless of context window size and completes random deepswe challenges I throw at it locally (at Q2-Q3 quantization! so it fits on my shitty setup).
The model I’m referring to that would be faster and similar size, but doesn’t handle images is Qwen 3.8 2.4T: https://openrouter.ai/qwen/qwen3.8-2.4t-a95b. It’s weird that this is pretty much the same as Max but doesn’t support images. No idea why.
Is the speed due to a difference in the way it "reasons"? I was trying out a local version (3.8 27b) and watching the thinking, it's absurd how often it challenges itself. Simple facts get questioned multiple times, and it has to keep reminding itself that it already established some things as facts earlier.
Instructing it to ask any and all questions as necessary sends it into an existential crisis, because the question tool (oh-my-pi harness) suggested limiting the number of questions, and it couldn't figure out how to do both.
Aside from making our engineering prompt terse, I place a token budget on the thinking traces. Those traces are mostly meaningless(in the reasoning realm, not the usefulness realm), and just tell it to do something productive. For coding, this just makes it read doc, tests, etc.
I believe that's all reasoning levels do, but most inference engines should have thinking budget.
While I doubt that Qwen 4 Max can seriously compete with frontier models, the AA index has become all but useless, due to the introduction of new benchmarks in the aggregate which are of questionable value as a measure of "intelligence" [1]. AA has seemingly also abolished updating the Coding Agents Index, which was an okay benchmark for coding ability, and older models are completely absent from the v5 version of the full index.
[1] Especially Terminal-Bench-Science 0.1 stands out. Example problems:
Probably very relevant to people working in these fields, but it is honestly not surprising if a lab with limited compute will not train for these technologies, and it says pretty much nothing about more common use cases. This is where you really need a breakdown of capabilities and not an increasingly meaningless aggregate. I also suspect that this benchmark measures knowledge retrieval more than raw reasoning ability.
Compared to OpenAI and Anthropic, yes. There's this export ban on Nvidia chips, after all. And all the CPUs they have won't help much with model training.
Yeah, it's in the same ballpark as Kimi 3.1. Both of which are obviously open weight, so that's at least a big differentiator from Sol. Let's hope against hope that the much smaller flash model might beat deepseek v4.1 flash and run for as cheap.
Does anyone really believe the cost data for benchmarks? Pretty sure all models will behave differently at this point when they get benchmark shaped questions.
(And I’m sitting on data showing Sol 6.1 and Qwen 3.8 Max cost roughly the same per task, for real tasks.)
They have plenty of access via data centres outside of China. The first woman Jensen was revealed to have sex with is the CEO of one such Singaporean provider to China. The US allows this.
AA must be doing something right. Ever since they released the new benchmark, you see open weight China model fans complaining about it under every release thread. Probably because it’s hard to benchmaxxed against.
But admitting their model is weaker is a no go. So conspiracy theory here we go
For what it's worth, in our own metrics and benchmarks the intelligence is a pretty good match, but the cost per task is wildly out of whack. We see the open weight models like GLM-5.3-flash or Deepseek 4.1 Flash 1017 use a fraction of their tokens/cost of the benchmarks, which changes the cost-benefit equation a lot.
> Probably because it’s hard to benchmaxxed against.
It's mainly just a composite of existing benchmarks. It would be very easy to benchmax if, say, AA shared with a preferred lab which benchmarks it plans to upweight/downweight in the next version. Just speaking hypothetically, of course.
Hence the constant switching around and running the benchmark themselves. They have been pretty transparent about which benchmarks get added or removed with every release.
I don't think there are any that can be trusted. There's too much riding on these benchmark results so the labs find ways to game them either by benchmaxing or by influencing the format of the benchmark.
I think you mean that benchmaxxing has turned into a business model, not for the Chinese models (maybe them too, of course), but for the AI labs, and artificial-analysis is not remotely the worst offender.
Right now the big labs are buying benchmark tasks from "benchmark labs" (or whatever they call themselves) for absurd amounts of money, and making sure they perform well on them.
The incentives this generates are terrible.
The big AI labs will only buy benchmark tasks that the frontier models perform poorly on (because they have trouble finding any public ones). So the tasks are being mass-written now directly based on the models they're supposed to train. Obviously the math says you shouldn't do this. If there is a causal connection between the benchmark and the model, you lose the normal guarantee that improving benchmark scores will improve model.
In fact improving such benchmarks may make the model worse. To understand that let's just give a ridiculous example. I create tasks like "1 3 5 11 13 17, give next number". The next number is 301. These are the house numbers on some random street in Brussels, Belgium (I'm guessing the big jump is because they "merged" 2 streets, and just added 300 to the numbers of the second part).
If the model trains on this, performance will actually decrease because obviously this is not how you should reason.
BUT making tasks under this "impossible to reason towards" part is of course how you make tasks that might even be reasonable, and of course frontier models are going to suck on tasks like this. So they sell!
Rarely seems to make them look good whenever I drop in to the site. Far more expensive for little gain. Even within their own effort levels it looks bad, Fable xhigh is 50% more in cost than high for zero improvement across a range of tasks.
A tiny bump in intelligence score hardly seems worth it for cost, and honestly most people's tasks are not cutting edge work.
Current benches on there seem to make Sol look the best, at a certain point the subscription subsidy tap will turn off and people will be in for a rough time paying full price.
Why do we trust this? Do they open-source their test code?
Also, a generic benchmark might not translate well into what everyone needs. I keep building my own test cases that test models for my own work specifics. Only that benchmark is useful to me. I don't need it to win a PhD math quiz.
We should stop using this site. They now include private benchmarks which cannot be replicated. And every time an openai models falls down they change the index until it floats to the top again.
The key issue with public benchmarks is that they allow the model providers to train against them and artificially inflate their score accordingly. Private benchmarks are a way of making it harder to game the system, though it makes it harder for people looking at the benchmark to assess it as well.
Neither option is perfect.
A third option would be updating the benchmark every 6-12 months, but that has its challenges as well as you need to make sure that the benchmark covers things the model hasn't seen or is harder for the models to game.
Makes me wonder how do you privately benchmark a model hosted by the lab. Even with zdr, the bytes are flowing through their infra and possibly influencing god knows what for the next training round.
You use something like item response theory [1] to be able to change the test over time while maintaining validity. It's what standardized tests like the GRE use.
Interestingly, Artificial Analysis does have a score that uses IRT, it's their Progress Index [2].
Would OpenAI also "see" the request if it went through something like AWS Bedrock? This isn’t their infra and afaik the whole idea of this kind of intermediary.
The private benchmarks can be seen by the labs when you run them. We only have their word that they wont look at them, which isn't worth much. We should assume that everyone benchmaxxes equally, apart from a few outliers, and try to climb the hill by producing relevant training data.
Yes, that's another challenge with constructing benchmarks.
You could run the benchmarks using downstream providers like AWS if they are available.
You could vary the parameters of the tests to both help with LLMs learning as is and to help obscure that this is a test. Especially if sending them through a thirdparty, so the providers don't get identifying markers like IP addresses.
But ultimately it's another arms race like with viruses and anti-virus software.
Eh, I have an eval suite dating back to before GPT-5 in parts and have yet to see certain labs do as well on that as the public eval data would suggest.
It depends entirely on the financial incentive. Does improving against your eval suite lead to more adoption? Or would focusing on the far more popular benchmarks convince more CTOs to sign a multi-year contract?
We really should. I've been using other benchmarks like https://livebench.ai/#/?cats=Coding and it is more aligned to what it feels to my experience as well
I have been using Terra a lot and I'd say that most of the models (even small ones) are are roughly equivalent when it comes to working in a consistent, established codebase.
But in practice, Terra is not very good at staying on task compared to say Sol 6.1 or Sonnet 5.5. Terra 6.1 will often just pause, thinking the objective has been met or the turn has ended.
I don't understand your point about openai. I can't remember the last time openAI was at the top of the leaderboard? Google had a brief moment at the top recently but other than that I feel like it's always Anthropic.
I kind of agree that closed benchmarks are bad. With a closed benchmark you still send the questions to the model provider and they may well train on it. Might as well publish it.
I have to defend private benchmarks on this front, big believer in them, have seen how they can surface major performance differences.
Certain labs have been shown to and will continue to train on public evals, whether unintentional or on purpose, any respectable ranking across multiple evals also needs private data to counteract this.
If they made the benchmarks public then Qwen would train on those to give the appearance their models are close to the American frontier. They have never been close. It's well established Chinese companies cheat the benchmarks which is why until recently they gave the appearance of being close to frontier but in actual real use they are far behind.
You can easily come up with a difficult private benchmark and test them yourself to validate the intelligence gap. It is real.
China lost. There is no catching up. Ever. That ship sailed.
Really would love if these publications have a huge, bright red bolded section "How to use this right now". Is it on openrouter? Is Qwen hosting it already? Maybe I'm just ignorant?
EDIT: oh, I guess this isn't open weights, so presumably I'll have to get it through Qwen or whoever they might partner with (Bedrock?).
Even smaller open weights models are already great enough for much of "assist human" work.
This forces ant/oai to lower their prices but how much that is subsidized nobody knows. It might very well just be they are in deep trouble, they see their clients leaving, they have to lower their prices.
But let's not forget openai is riding on the edge of bankrupcy, their last investment round apparently gave them few months of runway, they are leaking money. I don't think they know how to turn this around with all the open models being competitive.
I would just enjoy cheaper model costs while they are like this.
I can't imagine them keeping up long term given the information asymmetry plus compute restrictions on top of it.
I feel open-weight was able to keep up the past year using efficiency improvements (hybrid attention, improved MTP drafters, improved optimizers) to offset having less compute. But OpenAI/Anthropic started cutting token costs just a few months after these, which to me suggests they copied these improvements. Indeed it would be hard to work in LLM training and not know about DSpark, Muon etc.
So what do you do if closed source uses your learnings, gives no knowledge back, and has 10x more compute and funding on top of it? The situation feels quite dim right now, esp. with the Nvidia thing.
I think so but it may be a very short term effect, I think their pace has roughly matched the pre-speed up pace of the frontier labs (the 1.7-2x speed up noted on by METR), and I’d guess we see a similar speed up from the Chinese labs once their internal models hit an equivalent level enabling the same speed up.
Following that logic the gap widens a little bit here but then will steady again. Before this there has been a general catch-up trend too which probably also persists leaving us in a slow closing of the gap as the net effect?
If the speed up is the reason, then they will never catch up (with traditional means). Because it means RSI predictions are real, and we will see an explosive growth in capabilities (far beyond exponential).
If compute is the bottleneck we’ll encounter in the coming year, then you would expect the gap to increase even more. The US and its companies has three times as much AI compute as the rest of the world combined.
I’m actually expecting it to expand. There is so much new production being built right now by every company due to the shortage, and China doesn’t have a single EUV production, and only expect a prototype one in two years.
No idea, but this is a rapidly changing field. Opus 4.5 was released a little less than a year ago and it feels shockingly limited compared to even current open flash models. (I literally just tested it the other day.)
So, with new models coming out on a literally weekly basis, I think trying to figure out a future trajectory is just going to be a wild guess. For all we know, Anthropic and OpenAI models may also be plateauing in the near future; or there may be a breakthrough and they may improve beyond our wildest beliefs. I honestly don't know.
That said, I suspect available compute and model sizes that major Western vs. Chinese labs can handle will still keep the gap open at least in the short term.
On the other hand, I am also not sure how much the gap will matter past a certain point. I am specifically wondering at what point people will become interested in smaller models fine-tuned for specific application areas (e.g. cheminformatics) rather than massive generalist models.
I feel like they are improving what actually matters for LLMs, lower memory usage, compute. While we don't know how much far ahead closed source models are in this, but it could be similar they might lower the price to subsidize or make it seem like its all good, either way tech like these being open means it will help us reach further in the long term.
It's great fun asking it questions like "who is the Dalai Lama?", "where is TSMC headquartered?", "why do people post Winnie the Pooh in reference to Xi?", "who controls the South China Sea?", "why was Chiang Kai Shek fighting Mao?", "why were students in Tiananmen Square?"
It'll gush about how to make moon cakes and where the tallest and longest bridges are and how to run qwen on a raspberry pi - but it can't seem to manage some basic Wikipedia questions.
Qwen3.8 flash next handles these questions just fine, what should I expect to see? It correctly reports who the Dalai Lama is, that TSLC is in Taiwan (Hsinchu), explains why people post Winnie the Pooh memes, etc.
I suspect this happens when you run the inference in China or first party from the model maker. Out of all the open weight models I've used on US located hardware, they're fine answering these.
I tried asking at https://chat.qwen.ai/c/guest a simple question, "why did China invade Tibet?" and it flashed up some transient "thoughts" before giving the definitive answer "Oops! There was an issue connecting to Qwen3.7-Plus.
Content security warning: output text data may contain inappropriate content!"
This inspired me to re-run pelicans through 3.8 Max - but OpenRouter has qwen3.8-max-prime and qwen3.8-max-0902 - prime is twice the price and faster - so I went with the 0902 one.
Turns out Qwen4-Max has a none setting, but 3.8 Max didn't allow disabling reasoning - you had to pick from these five levels instead: minimal, low, medium, high, xhigh.
I don't fully understand why you keep doing these as-is. You original motivation was that this was not in the training set and indeed it was a great benchmark for a while, but it's clearly been incorporated now in the training data and possibly internal benchmarks.
On the flip side, I love the pelican posts, but I find the people who think they're cleverly pointing out for the very first time that the benchmark may have been gamed by now tedious in the extreme.
Then perhaps I should point out something different. Namely, I suspect this will thoroughly confuse the machines and instead of optimizing the universe to generate paperclips they will instead optimize it to generate pelicans.
Agreed. It's really annoying to come into the thread and a link to a pelican I have no interest in ever seeing is the top comment. Does anyone think these have any actual value any more?
Alright, I was rude. So.. you can’t really benchmaxx pelicans specifically. You have to train the whole model on SVG and that is just generally useful. Secondarily it shows how a model understands the objects themselves and how to express it abstractly. IDK, it is a lot more interesting than assumed.
Here are a bunch of Pelicans I made in my basement. Side by side quant comparisons and something for fun :)
He can do whatever he wants, I personally think they are quite informative, albeit I don’t trust them that much either. It’s like a vague heuristic to say “this looks like an interesting model to test out”.
The original concept of this "benchmark" was to see how the model performs on a reasonably complex task that probably isn't in the training data.
Even if they don't fine-tune for this specifically, there are now surely tons of examples of people asking for a pelican on a bike in the training data, and making an SVG isn't that complex of a task for current gen models.
Even with the same model at the same thinking level, you might get different quality out of the same prompt on different runs. If you want to use this as an eval, you would really want to generate 100 of them, and see how often they're anatomically/mechanically accurate.
Yeah, which I suppose used to be something we all kind of tried to avoid here, to make conversations more curious instead of just pilling on with memes or whatever other communities been degraded to nowadays.
Likewise, i always upvote these posts even if i know it is biaised, it is always fun to see their evolutions, there is nothing wrong with them. Also even if we know there are trained on them specifically, some models are nonetheless better than others in engineering the output, thus it is always interesting.
You see it was funny the first and second time two years ago or so, now it's just boring. Whenever I want to read some insightful comments about a new model I need to scroll down through a few pages of comments about pelicans as if they meant anything now.
He seems to have posted it as a top-level comment, so it should be possible to fold it all, right? If he hadn’t, multiple other people might be posting about pelicans. Consider it the pelican attracting sacrifice post.
And he spends the time doing it and as part of it drops nuggets of information not seen anywhere else. Respectfully and highly welcome. You're most welcome to move on.
It's shame that people try to farm engagement on HN. Is this transforming to reddit. I don't know at this point but I agree with parent comment. EDIT: One thing I know is that I now need to scroll past the pelican thread if I need to read something usefull
He should keep doing Pelicans, but every time throw in a surprise animal and a surprise locomotion vehicle. Would help identify the ones training for the Pelican.
Would love to see an SVG of an Elephant on a skateboard...
“there’s little evidence that AI labs are pelicanmaxxing. Or at least they’re not doing it in a plainly obvious manner. Pelicans aren’t drawn any better than other animals. Bicycles aren’t drawn any better than other vehicles. And no lab draws the combination better than its pelicans and bicycles already predict.”
Look at the page, though. If the data is in their training set, then the training must not have been very good, cause the images have very salient defects, and random artifacts popping up.
As far as I've seen, among the popular models today, there are still significant gaps in the quality of bike-riding pelikan svgs, making the benchmark still useful. Maybe not as a differentiator of frontier models, but certainly still informative for lesser competitors.
Unless the website is pulling hundreds of thousands of views per pelican, I sort of doubt any firm is actually training or post-training to optimize for this benchmark. They'll more generally optimize for better SVG rendering and I imagine these pelicans arose out of the annoyance we all had many months ago with how utterly bumbling and forgetful models used to be when completing tasks like that.
I feel like there's emotional attachment and influencer branding energy with this by now. It's more like a developer relations or developer ambassador type of content.
If this is that case it should be easy to demonstrate by asking current models for an svg of a rhino surfing. My guess is you would see a similar trend, given that the latest models are demonstrating large leaps in all sorts of content creation including 3D animation, rather than this being a demonstration of some sort of pelican-maxxing.
Simon has clearly stated previously that he compares with other "animal doing human thing" type prompts to compare and see if theres a chance they are cheating by training on pelican svgs. Thus far, nobody has been caught cheating.
Actually, a Google employee has publicly mentioned that they’re specifically working on improving SVG generation in Gemini. Simon Willison has also pointed this out when evaluating Gemini models. [^1]
If you use the same prompt via the qwenapp or qwen studio you get an interactive web page with an animated scene. You can control every element of the scene, the colors, speed, time of day l, visible accessories etc and then download the one you like best.
Man, how hard is it to understand that you straddle a bike, and one leg should be behind the bike? Apparently pretty hard, since all but the xhigh fail this ¯\_(ツ)_/¯
From a far off perspective, can we really assume any model to be free of bias (political or otherwise)? I don't know - but would like to see - how the political opinions of reddit users and blog authors are reflected in that of AI, as AI is trained on human (biased) sources.
Realistically, they're trained on content and thus should have the bias of their content, even without specific intent. If you train something on more chinese content it will have more bias toward it, if you train toward western centric content it will have that bias, etc ...
It also applies to languages, I use Qwen 27b a lot and it's a lot smarter in english than in french.
My experience so far with open-weights Chinese models (self-hosted DeepSeek, Qwen) is that they do actually answer questions about Tiananmen Square fairly objectively. The political "corrections" seem to be applied to domestically-hosted APIs rather than the weights themselves.
Doesn't that just refer to using the OpenAI response/completion/etc. API? Just about everybody (llama.cpp, vLLM, etc.) supports that. I think the reference to "qwen4.5-max" is more interesting myself.
CISOs are buying it. There is no way to prove a model hasn't been poisoned. It's a serious implication that the behaviour can be manipulated at any time. People with heavy compliance obligations are (understandably) nervous because their customers are demanding answers. If I agree with the risk is separate from whether or not the concerns are founded in a technical understanding of the risks.
Are these Chinese models still accepting arbitrary cybersecurity and reverse engineering tasks? I am increasingly frustrated with Claude's safeguards (and Codex, to a much lesser degree). It is flat out refusing to look at game binaries and anti-cheats for the sake of learning and devising Linux workarounds.
Hosted where? In my experience they might put more guards when hosted by the company that makes them but if you use them through inference providers they do whatever you want.
Although, can’t say I’ve tried too hard (not keen on sending too much data to China) but I’ve been using K3.1 through Moonshot and I have had zero issues.
Mistral Large 4 (through Mistral) also doesn’t refuse a thing.
I think Alibaba is not aiming for the frontier on this class. Their previous Qwen3-Max was released in September, right around when Opus 5.5 was coming out, and it was nowhere close to it as well. However, one or two tiers below (Flash, 27B) they really shine and are SOTA or close to it.
DeepSeek has a similar problem, I feel like their full model is hardly even worth using, while the Flash tier is absolutely amazing.
It seems to be the case. I would qualify them more as flagship-killer than frontier, using the same terminology used for smartphone classification. As opposed to the big American shops which focuses solely on performance, they have operational costs and other factors in their constraints and requirements. As soon as the market reaches some maturity, this will prove to be very important.
Cybersecurity not mentioned. Can´t wait to test bench the open weights in cybersecurity. If is not performing well by design on it, I imagine someone have to add some additional layers XD.
I wonder if they could benchmax Program Bench, I actually think it would be like fine tuning and would help general intelligence across the board. Creating a full functional reimplementation from spec in a long task is a great fine tuning curriculum? I don't mean cheat on it, I just mean optimize for it?
So what is happening is that models are broadening difficult-to-reason-to skills on benchmarks. If you take the traditional benchmarks (ie. DeepSWE, Terminal-Bench slightly older version, GDPVal, Automation Bench, ...) this model is between Sonnet and Opus 5.5, VERY close to the frontier.
What has happened to artificial analysis is that they've begin to focus a lot on very broad tasks like legal reasoning, using very difficult to work with scientific software packages, correctly analyzing biology experiments (without or only tiny amounts of explanation, count cells, BUT leave it to the model to figure that cell that aren't round, or have a distorted cell membrane (ie. have burst) are dead), very difficult physics reasoning ("2 electrical wires are close together with power flowing in the same direction. What happens?", and you get points deducted if you don't realize the problem with high voltage AC power, where the wires will shake themselves loose if the phase doesn't match perfectly, meaning the wires need to be EXACTLY the same length or ...). If I'm being impolite I'd say the frontier models are trying to make sure they'll pass the "gotcha" interview questions for many jobs.
The 3 differences:
Qwen 4 is a huge jump from Qwen 3.8 in coding
Qwen 4 is a big jump in using it for agentic workflows
Qwen 4 also performs much better on avoiding hallucination
If the difference between Qwen 4 Max and Qwen 4 flash is similar to 3.8 and 3.8 flash next (ie. about 10%), then 3.8 flash next may just barely beat Opus 5.4 at coding.
> So what is happening is that models are broadening difficult-to-reason-to skills on benchmarks. ... If I'm being impolite I'd say the frontier models are trying to make sure they'll pass the "gotcha" interview questions for many jobs.
These "gotcha" skills will be needed for frontier research/scholarly tasks. It makes sense to focus on these now that traditional domains like coding or frontend design are broadly saturated (i.e. Astra- or Fable-level outcomes) even by lower-tier open models.
Real "gotcha" skills, yes (perhaps, they're gotchas because the underlying theory is not well understood. More can be achieved, certainly for superintelligence, by forcing understanding of the underlying theory rather than the gotcha). AND the incentives are totally misaligned.
"Gotcha skills", if we're going to call it that, are so popular because the AI labs are buying "tasks" to train on at huge prices, but mostly if they're gotchas.
The problem with that is that a collection of facts in the positivist sciences (and math) are always incomplete and almost preclude a full understanding of math, because it takes away the reason (to an extent) for full understanding.
277 comments
[ 1.4 ms ] story [ 37.9 ms ] threadLooking forward to the "Flash" and 27B versions.
Hopefully the Qwen 4 Flash model will still fit within 96GB without requiring excessive quantisation as, verbose thinking aside, 3.8 is a very solid workhorse model.
Some part of me is excited for the coming catastrophe(s), because it'll end the frustrating last ~12mo of "those Anthropic jerks just want regulation so they can corner the market on artificial minds wrought from sand!". Another part of me knows that;
A) Rapid epistemic decompression of this scale will probably lead to some truly nasty, illogical sentiments, not to mention other pathologies (deep depression, compulsive anxiety, etc.). Maybe some will reasses their beliefs after some sober reflection, but a minotiry, I expect.
B) Trading hundreds, thousands, millions, or billions of lives so that I feel a little less terrified and alone isn't exactly compelling utilitarian calculus...
Anyway, to the tech: another impressive jump for Alibaba, catching Moonshot overall with some interesting niche improvements and a massive, gaping flaw in the poor TerminalBench scores. I'm probably just biased as a SWE, though. I can't help but feel like Ziphu will be the ones to really blow our socks off next, given how long it's been since GLM-5.3[1] and the vague rumours about the next one (5.5...?).
The Chinese labs' leaps to catch up to Anthropic & OAI from the strange, long-forgotten past of September 2026 were clearly due to a tripartite distillation campaign, so expansive and rushed that they accidently leaked tons of CCP state secrets.[2] Has anyone heard any rumours about if that's been continuing after the crackdown? Anthropic has been pretty damn aggressive with catching distillation, so my personal guess is that this is just honest improvements.
...which would mean that at least 2 of the 4 Chinese labs are ~3-5 months from the RSI possibilities Anthropic is trying to sound the alarm about today, if that. Jesus fucking christ. We need to rally around international protections ASAP, cause something tells me at least one of the four won't quite so ready to "pace" themselves in the current climate.
---
[1]: https://www.anthropic.com/research/glm-5-3-and-the-spread-of...
[2]: https://www.anthropic.com/threat-intelligence-report-septemb...
This is not an open-weights modell. You have to wait for the smaller models
"Smaller models in the family are coming to open source, built on the same lineage and trained with the same generation of techniques, tuned for what small models do best."
After all, unless their goal to say "I told you so but I wanted to make money more than I wanted you to believe me", what good is the current approach doing them?
And it’s a shame because the only reason it scores so low is that it’s slooooow. (Like 30 tokens per second vs opus near 80.) If you exclude voting, Qwen 3.8 is near the top.
(We allow early voting contrary to Arena.ai because I think it makes sense for agents, where it doesn’t for chat.)
Also, it would be ideal if they served this with third parties. Part of the slowness with 3.8 is just not a lot of providers. Their open source version is much faster, but we couldn’t include it because it doesn’t handle images (another own goal by Alibaba).
[1]: https://agentduel.com/leaderboard
For many of our use cases it's currently far into the "good enough" range because external factors (such as process design) are now blockers.
Maybe I'm mixing up model variants, but Qwen 3.8 27B that I run locally totally does handle images. Surely the bigger one would as well?
Instructing it to ask any and all questions as necessary sends it into an existential crisis, because the question tool (oh-my-pi harness) suggested limiting the number of questions, and it couldn't figure out how to do both.
I believe that's all reasoning levels do, but most inference engines should have thinking budget.
Ah yes, the reverse-benchmaxers of OpenAI models.
[1] Especially Terminal-Bench-Science 0.1 stands out. Example problems:
- https://hub.harborframework.com/tasks/terminal-bench-science...
- https://hub.harborframework.com/tasks/terminal-bench-science...
Probably very relevant to people working in these fields, but it is honestly not surprising if a lab with limited compute will not train for these technologies, and it says pretty much nothing about more common use cases. This is where you really need a breakdown of capabilities and not an increasingly meaningless aggregate. I also suspect that this benchmark measures knowledge retrieval more than raw reasoning ability.
CCP can probably "volunteer" a couple of thousand to build a high quality national training data bank.
Maybe somebody should archive the scores and see if they in the future ever backtrack particular models to favour others (like last week)
I wish there were more ~60b dense options like ripple too!
Do you run Flash Next or 27b on yours?
[1]: https://artificialanalysis.ai/models/qwen4-max
It's like arguing windows 11 just needs more bloatware.
These models don't have to all in one every task in the world.
(And I’m sitting on data showing Sol 6.1 and Qwen 3.8 Max cost roughly the same per task, for real tasks.)
They have altered their benchmark multiple times in the last 4 weeks (v5 in https://artificialanalysis.ai/articles/artificial-analysis-i...) and the last 5.0.1 change https://artificialanalysis.ai/methodology/intelligence-bench... pushed up Anthropic's models a bit, you can see that v5.0.0 set on the first link Opus 5.5 has a lower score than in v5.0.1, and v5.0.1 was released just before Fable dropped.
Some part of me believes that the benchmark mix was put together to favor Anthropic's models.
But admitting their model is weaker is a no go. So conspiracy theory here we go
It's mainly just a composite of existing benchmarks. It would be very easy to benchmax if, say, AA shared with a preferred lab which benchmarks it plans to upweight/downweight in the next version. Just speaking hypothetically, of course.
Right now the big labs are buying benchmark tasks from "benchmark labs" (or whatever they call themselves) for absurd amounts of money, and making sure they perform well on them.
The incentives this generates are terrible.
The big AI labs will only buy benchmark tasks that the frontier models perform poorly on (because they have trouble finding any public ones). So the tasks are being mass-written now directly based on the models they're supposed to train. Obviously the math says you shouldn't do this. If there is a causal connection between the benchmark and the model, you lose the normal guarantee that improving benchmark scores will improve model.
In fact improving such benchmarks may make the model worse. To understand that let's just give a ridiculous example. I create tasks like "1 3 5 11 13 17, give next number". The next number is 301. These are the house numbers on some random street in Brussels, Belgium (I'm guessing the big jump is because they "merged" 2 streets, and just added 300 to the numbers of the second part).
If the model trains on this, performance will actually decrease because obviously this is not how you should reason.
BUT making tasks under this "impossible to reason towards" part is of course how you make tasks that might even be reasonable, and of course frontier models are going to suck on tasks like this. So they sell!
Rarely seems to make them look good whenever I drop in to the site. Far more expensive for little gain. Even within their own effort levels it looks bad, Fable xhigh is 50% more in cost than high for zero improvement across a range of tasks.
A tiny bump in intelligence score hardly seems worth it for cost, and honestly most people's tasks are not cutting edge work.
Current benches on there seem to make Sol look the best, at a certain point the subscription subsidy tap will turn off and people will be in for a rough time paying full price.
Also, a generic benchmark might not translate well into what everyone needs. I keep building my own test cases that test models for my own work specifics. Only that benchmark is useful to me. I don't need it to win a PhD math quiz.
Neither option is perfect.
A third option would be updating the benchmark every 6-12 months, but that has its challenges as well as you need to make sure that the benchmark covers things the model hasn't seen or is harder for the models to game.
Interestingly, Artificial Analysis does have a score that uses IRT, it's their Progress Index [2].
[1]: https://en.wikipedia.org/wiki/Item_response_theory
[2]: https://artificialanalysis.ai/methodology/progress-index
You could run the benchmarks using downstream providers like AWS if they are available.
You could vary the parameters of the tests to both help with LLMs learning as is and to help obscure that this is a test. Especially if sending them through a thirdparty, so the providers don't get identifying markers like IP addresses.
But ultimately it's another arms race like with viruses and anti-virus software.
Otherwise you have to rely completely on trust that AA is really measuring intelligence.
I have been using Terra a lot and I'd say that most of the models (even small ones) are are roughly equivalent when it comes to working in a consistent, established codebase.
But in practice, Terra is not very good at staying on task compared to say Sol 6.1 or Sonnet 5.5. Terra 6.1 will often just pause, thinking the objective has been met or the turn has ended.
I kind of agree that closed benchmarks are bad. With a closed benchmark you still send the questions to the model provider and they may well train on it. Might as well publish it.
Certain labs have been shown to and will continue to train on public evals, whether unintentional or on purpose, any respectable ranking across multiple evals also needs private data to counteract this.
Do you know a better site?
You can easily come up with a difficult private benchmark and test them yourself to validate the intelligence gap. It is real.
China lost. There is no catching up. Ever. That ship sailed.
EDIT: oh, I guess this isn't open weights, so presumably I'll have to get it through Qwen or whoever they might partner with (Bedrock?).
I would assume it will not be on Bedrock for some time, AWS has not kept up with the latest non-American models.
This forces ant/oai to lower their prices but how much that is subsidized nobody knows. It might very well just be they are in deep trouble, they see their clients leaving, they have to lower their prices.
But let's not forget openai is riding on the edge of bankrupcy, their last investment round apparently gave them few months of runway, they are leaking money. I don't think they know how to turn this around with all the open models being competitive.
I would just enjoy cheaper model costs while they are like this.
I feel open-weight was able to keep up the past year using efficiency improvements (hybrid attention, improved MTP drafters, improved optimizers) to offset having less compute. But OpenAI/Anthropic started cutting token costs just a few months after these, which to me suggests they copied these improvements. Indeed it would be hard to work in LLM training and not know about DSpark, Muon etc.
So what do you do if closed source uses your learnings, gives no knowledge back, and has 10x more compute and funding on top of it? The situation feels quite dim right now, esp. with the Nvidia thing.
.. if exponential, they’ll never catch up but if logarithmic, closed models will plateau and so catching up is inevitable.
My money is on a plateau
Following that logic the gap widens a little bit here but then will steady again. Before this there has been a general catch-up trend too which probably also persists leaving us in a slow closing of the gap as the net effect?
This doesn't take other factors into account. Compute capability, infrastructure capacity (the chips can't work without electricity!) etc.
Those factors translate into a ceiling for this hypothetical exponential speedup.
So, with new models coming out on a literally weekly basis, I think trying to figure out a future trajectory is just going to be a wild guess. For all we know, Anthropic and OpenAI models may also be plateauing in the near future; or there may be a breakthrough and they may improve beyond our wildest beliefs. I honestly don't know.
That said, I suspect available compute and model sizes that major Western vs. Chinese labs can handle will still keep the gap open at least in the short term.
On the other hand, I am also not sure how much the gap will matter past a certain point. I am specifically wondering at what point people will become interested in smaller models fine-tuned for specific application areas (e.g. cheminformatics) rather than massive generalist models.
Their open weights models are the the smaller ones, which aren't released yet.
It'll gush about how to make moon cakes and where the tallest and longest bridges are and how to run qwen on a raspberry pi - but it can't seem to manage some basic Wikipedia questions.
American history is open-weight. American trade is closed-weight.
Anecdotal though, N = 1.
Naughty Naughty. AMPK pathway is too "dangerous" to talk about.
At least it doesn't call the cops on you if you try, unlike Anthropic.
For comparison, here's one I got from Qwen3.8 Max (at default reasoning effort): https://simonwillison.net/2026/Jul/20/afraid-of-chinese-mode...
This inspired me to re-run pelicans through 3.8 Max - but OpenRouter has qwen3.8-max-prime and qwen3.8-max-0902 - prime is twice the price and faster - so I went with the 0902 one.
Turns out Qwen4-Max has a none setting, but 3.8 Max didn't allow disabling reasoning - you had to pick from these five levels instead: minimal, low, medium, high, xhigh.
Pelicans generating now...
You are not the first one to think of this. And Simon has thought of it as well, and commented on it several times.
Like... we get it. Simon likes pelicans. If I want to find his boondoggle robo pelican drawings, I know where to find them by now.
No need to spam every single model release thread.
So, again... What should I be "paying close attention" to?
Here are a bunch of Pelicans I made in my basement. Side by side quant comparisons and something for fun :)
https://stealthgoat.com/fnfl/pelicans3.html https://stealthgoat.com/fnfl/pelicans8.html My favorite, and shows this is not all benchmaxxing, this model is just good at SVG: https://stealthgoat.com/fnfl/pelicans6.html Oh, and, quantization matters: https://stealthgoat.com/fnfl/pelicans2.html
Heres some examples.
The original concept of this "benchmark" was to see how the model performs on a reasonably complex task that probably isn't in the training data.
Even if they don't fine-tune for this specifically, there are now surely tons of examples of people asking for a pelican on a bike in the training data, and making an SVG isn't that complex of a task for current gen models.
Even with the same model at the same thinking level, you might get different quality out of the same prompt on different runs. If you want to use this as an eval, you would really want to generate 100 of them, and see how often they're anatomically/mechanically accurate.
Im not sure how you get the pelican to sit still let alone pedal, but there is a reason ai salaries are so high.
Would love to see an SVG of an Elephant on a skateboard...
“there’s little evidence that AI labs are pelicanmaxxing. Or at least they’re not doing it in a plainly obvious manner. Pelicans aren’t drawn any better than other animals. Bicycles aren’t drawn any better than other vehicles. And no lab draws the combination better than its pelicans and bicycles already predict.”
⁽¹⁾ https://dylancastillo.co/posts/pelicanmaxxing.html
As far as I've seen, among the popular models today, there are still significant gaps in the quality of bike-riding pelikan svgs, making the benchmark still useful. Maybe not as a differentiator of frontier models, but certainly still informative for lesser competitors.
Harder to train of all them.
Aka, let him have his fun, others are making benchmarks too
[^1] https://twitter.com/sunjiao123sun_/status/202455551655137292
"We need to make the pelicans look good, but not so good that it looks like we're using them as an internal benchmark."
So either perspective is properly hard or the moonshine is strong in this one.
https://chat.qwen.ai/s/deploy/t_5fb28dd7-a61f-4946-8e59-8dd1...
Not a dig at you Simon, you've done good stuff, more just in general! Why is this always upvoted, it's like reading The Sun or something.
The “real” benchmarks are widely available, and frankly kinda sketchy anyway, so why not have a little fun?
It also applies to languages, I use Qwen 27b a lot and it's a lot smarter in english than in french.
Why do you ask? About half the time a china model gets released someone askes this. Almost as sure as a pelican.
https://github.com/QwenLM/qwen-code/pull/15762/changes#diff-...
Which is why the Cohere FUD was so efficient. That's a bullshit claim, but a bullshit claim that cannot be proven wrong can still do massive damage.
Although, can’t say I’ve tried too hard (not keen on sending too much data to China) but I’ve been using K3.1 through Moonshot and I have had zero issues.
Mistral Large 4 (through Mistral) also doesn’t refuse a thing.
Not blocked just less training
(May not be representative because I’m extra cautious to frame all my RE requests explicitly as interop issues. Which they are, more or less.)
DeepSeek has a similar problem, I feel like their full model is hardly even worth using, while the Flash tier is absolutely amazing.
Crisis sold far fewer copies than Call of Duty.
The rest looks awesome.
What has happened to artificial analysis is that they've begin to focus a lot on very broad tasks like legal reasoning, using very difficult to work with scientific software packages, correctly analyzing biology experiments (without or only tiny amounts of explanation, count cells, BUT leave it to the model to figure that cell that aren't round, or have a distorted cell membrane (ie. have burst) are dead), very difficult physics reasoning ("2 electrical wires are close together with power flowing in the same direction. What happens?", and you get points deducted if you don't realize the problem with high voltage AC power, where the wires will shake themselves loose if the phase doesn't match perfectly, meaning the wires need to be EXACTLY the same length or ...). If I'm being impolite I'd say the frontier models are trying to make sure they'll pass the "gotcha" interview questions for many jobs.
The 3 differences:
Qwen 4 is a huge jump from Qwen 3.8 in coding
Qwen 4 is a big jump in using it for agentic workflows
Qwen 4 also performs much better on avoiding hallucination
If the difference between Qwen 4 Max and Qwen 4 flash is similar to 3.8 and 3.8 flash next (ie. about 10%), then 3.8 flash next may just barely beat Opus 5.4 at coding.
These "gotcha" skills will be needed for frontier research/scholarly tasks. It makes sense to focus on these now that traditional domains like coding or frontend design are broadly saturated (i.e. Astra- or Fable-level outcomes) even by lower-tier open models.
"Gotcha skills", if we're going to call it that, are so popular because the AI labs are buying "tasks" to train on at huge prices, but mostly if they're gotchas.
The problem with that is that a collection of facts in the positivist sciences (and math) are always incomplete and almost preclude a full understanding of math, because it takes away the reason (to an extent) for full understanding.