I see Elon is claiming that it'll discover "new technologies and new physics" in the next year... Add it to the list of "next year" Elon claims about things. Seriously you would have to be so fucking stupid at this point to continue believing his bullshit.
Honestly if it actually does score 44.4% on Humanity's Last Exam, that would be super impressive as Gemini 2.5 Pro and o3 with tools only score 26.9% and 24.9%.
The trick they announce for Grok Heavy is running multiple agents in parallel and then having them compare results at the end, with impressive benchmarks across the board. This is a neat idea! Expensive and slow, but it tracks as a logical step. Should work for general agent design, too. I'm genuinely looking forward to trying this out.
EDIT: They're announcing big jumps in a lot of benchmarks. TIL they have an API one could use to check this out, but it seems like xAI really has something here.
So the progress is basically to brute force even more?
We got from "single prompt, single output", to reasoning (simple brute-forcing) and now to multiple parallel instances of reasoning (distributed brute-forcing)?
No wonder the prices are increasing and capacity is more limited.
Seems like it is indeed the new SOTA model, with significantly better scores than o3, Gemini, and Claude in Humanity's Last Exam, GPQA, AIME25, HMMT25, USAMO 2025, LiveCodeBench, and ARC-AGI 1 and 2.
Specialized coding model coming "in a few weeks". I notice they didn't talk about coding performance very much today.
> Seems like it is indeed the new SOTA model, with significantly better scores than o3
It has been demonstrated for quite some time that censoring models results in drastically reduced scores. Sure, maybe prevent it from telling somehow how to build a bomb, but we've seen Grok 3 routinely side with progressive views despite having access to the worst of humanity (and its sponsor).
Even if one does not have a positive view of Elon Musk, the catching up of Grok to the big three (Google, OpenAI, Anthropic) is incredible. They are now at the same level aproximately.
Does anyone here have access to Grok 4 yet? If so, could you please try asking it to solve this basic word search problem [0] and share the results? It's just a simple grid of letters where you have to find the position of each word, the kind of problem that any young child can easily solve.
My tl;dr: benchmarks are very impressive but their CEO just eroded any trust in those benchmarks although some such as ARC are corroborated externally, and the Nazi incident (which went ignored!) makes actually using Grok in an app a professional liability.
They also have not released a model card, and I suspect they never will.
Interested to see how it all works out. Elon has been using a lot of smoke and mirrors lately, but this seems like an area where they can genuinely make progress - with the right talent competing in the GenAi world is totally possible right now. sign me up for improvements in this space!
Out of interest, has anyone ever integrated with Grok? I've done so many LLM integrations in the last few years, but never heard of anyone choosing Grok. I feel like they are going to need an unmistakably capable model before anyone would want to risk it - they don't behave like a serious company.
You would have to be insane to integrate the model that last week called itself "Mecha Hitler" into your live product.
As a huge Musk fan i'll be the first to point out how he's doing exactly what he accused Sama of doing; making powerful ai with an obvious lack of control or effective alignment.
There is so much money and so many top labs falling over themselves to attract good talent, that at this point people have to be leaning on ideological goals to choose their employer.
Are there really that many AI researchers who want to make Elon god-emperor?
If you're focusing on ideology, it isn't like the other companies are all that good. With Sam Altman you're still working for a pathological liar with delusions of grandeur. With Google and Meta you're propping up a massive worldwide surveillance apparatus.
Tech-bros have been propping up agents/propagators of some of the biggest social ills of the past ~2 decades, xAI isn't all that different.
A lot of serious engineers would love to work in an environment that isn't the HR-reigning office politics bullshit standard of the past decade or two.
I don't even really like Elon but I bet the engineers at X are having a better time in their day-to-day than the ones at Meta or Google where all their work is constantly roadblocked by red tape, in-fighting, and PMs whose only goal is to make it look like they headed something important to get themselves promoted. Elon's at least got a vision and keeps it a top priority to be competitive in the AI space.
Grok's updated voice mode is indeed impressive. I wish there was a way to disable automatic turn detection, so that it wouldn't treat silence as an end of the response. I like Claude's approach (you need to tap in order to end the response), but it's not very reliable because sometimes it just abruptly cuts my response without waiting until I tap.
I was pleasantly surprised that Grok even supports (to some degree) Lithuanian in voice mode, which is a quite niche language. Grok's responses themselves are alright, but ChatGPT and Gemini way surpass it in speech recognition and speech synthesis.
I find for auto turn detection, models work better if you put in the system prompt "if it seems the user hasnt completed their thought yet, output silence". This hack works around their compulsive need to output something.
108 comments
[ 1.1 ms ] story [ 95.9 ms ] threadEDIT: They're announcing big jumps in a lot of benchmarks. TIL they have an API one could use to check this out, but it seems like xAI really has something here.
We got from "single prompt, single output", to reasoning (simple brute-forcing) and now to multiple parallel instances of reasoning (distributed brute-forcing)?
No wonder the prices are increasing and capacity is more limited.
Impressive. /s
Specialized coding model coming "in a few weeks". I notice they didn't talk about coding performance very much today.
It has been demonstrated for quite some time that censoring models results in drastically reduced scores. Sure, maybe prevent it from telling somehow how to build a bomb, but we've seen Grok 3 routinely side with progressive views despite having access to the worst of humanity (and its sponsor).
[0] https://imgur.com/VxNP5jG
They also have not released a model card, and I suspect they never will.
Can you name an Elon company that is not number 1 globally in terms of product capabilities?
The only one I would've been able to name would've been Grok. Until yesterday.
As a huge Musk fan i'll be the first to point out how he's doing exactly what he accused Sama of doing; making powerful ai with an obvious lack of control or effective alignment.
There is so much money and so many top labs falling over themselves to attract good talent, that at this point people have to be leaning on ideological goals to choose their employer.
Are there really that many AI researchers who want to make Elon god-emperor?
Tech-bros have been propping up agents/propagators of some of the biggest social ills of the past ~2 decades, xAI isn't all that different.
I don't even really like Elon but I bet the engineers at X are having a better time in their day-to-day than the ones at Meta or Google where all their work is constantly roadblocked by red tape, in-fighting, and PMs whose only goal is to make it look like they headed something important to get themselves promoted. Elon's at least got a vision and keeps it a top priority to be competitive in the AI space.
I was pleasantly surprised that Grok even supports (to some degree) Lithuanian in voice mode, which is a quite niche language. Grok's responses themselves are alright, but ChatGPT and Gemini way surpass it in speech recognition and speech synthesis.
I hope that can be turned off while driving...
I can recall the first experiments with dota2 while he was still "in charge" of openai.
Paul Graham