They had two big substantive flaws on top of the political stuff. Aside from a brief window last summer Grok has been behind the curve for coding, and before the Cursor acquisition they didn’t have a harness. Now they have an Opus tier model and a real harness they have at a minimum the opportunity to undercut the competition on price. And with the 5T and 10T models being trained on Colossus 2 they have the possibility to leap ahead.
Is there a reason the AI companies usually announce new products so close to each other. Like not just the same day but literally hours apart. GPT Live then an hour later Grok 4.5. As if they try to one up. I expect something new from Anhtropic as well today.
The joke is that McDonald's spends hundreds of thousands of dollars to identify new locations - traffic studies, visibility, demographics, nearby traffic generators, site characteristics, drive-thru feasibility, etc. They have one of the most rigorous processes in the industry. Burger King's process is to open a location across the street.
I'm guessing that they already have the model ready and the announcement blogposts locked and loaded, and then release them as soon as they see a competitor make the first move, trying to overshadow the first announcement or at least be swept up in the hype just as people start talking about new models again.
They get to compare their model to the old ones from the competition.
In this case, ChatGPT 5.6 Sol / Ultra releases tomorrow, so today is the last day Grok can compare Grok 4.5 to Codex 5.5. If they did it tomorrow people would point out they're comparing themselves against old models.
It seems to be extremely economical - 4x better reasoning efficiency compared to Opus while being priced at $2/$6. For comparison, GPT 5.4 is $2.5/$15, GPT 5.5/5.6 are $5/$30, Opus 4.8 is $5/$25, Fable is $10/$50.
Yeah but depends how you use it - with superpowers and it’s prevalence of splitting things into smaller focused subagents - this could seriously reduce costs …
I wish my company gave me more options than just using Claude to test these things out
The recent Databricks comparison has GLM 5.2 performing identically to Opus 4.8 on high effort, and some early Twitter reports (e.g. from the OpenCode developers) strongly favor GPT 5.6 Sol over Fable.
As always it depends on what you are using them for, and how you are using them.
Also, the cache hit pricing is 25% of the input pricing ($2 vs $0.50). Long agentic workflows are dominated by cached input. The US frontier labs typically have this at 10% of the input price, and DeepSeek/Xiaomi etc take it to the extreme 1% range (which is why those are cheap to run in real world agentic loops with dozens of toolcalls per run)
Just to add context (no pun intended), OpenAI also charge differently based on context usage with GPT 5.5 being $5/30 below 200K, and $10/45 above.
Anthropic have a fixed price regardless of context usage.
These per-token pricing schemes aren't directly comparable though since these models all use different numbers of tokens, even for input (Anthropic's recent tokenizer change generates 30% more tokens for exact same input), as well as for reasoning, and context/token usage also varies wildly by harness with Claude Code using 3x the context/tokens of Pi.
I guess it does increase their cost, or rather your share of their hardware depreciation.
AI serving cost is apparently mostly hardware depreciation rather than operating cost (electricity etc), and if your large context request is occupying VRAM for some fraction of a second then you are paying for the depreciation that occurs in that time!
e. H100 costs $20-40K to buy, with a lifetime of maybe 3 years, and will only consume maybe $2K in electricity if run 24x7 for those 3 years.
Think of the entirety of the context (the full thread of conversation, all tool call output, etc.) as one message that's been submitted to the LLM all at once just so it can generate the next token (which is just the next word or even syllable.) Once that token is generated it is appended to the context, and the entire context is once again used to generate the next token. Keep doing that until the entire response is generated. The larger the context, the more stuff the model has to pay attention to as it generates each next token.
To use an analogy: imagine your friend is the author of an unfinished book. They die with 19 chapters written, and on their death bed ask you to write the 20th chapter. Assuming you're up to the task, you can only do this well if you take the time to absorb the entirety of what's been written so far.
This is how LLMs work. Context caching is an optimization on top of this, but it has its limits.
I have a theory that xAI has one of the largest clusters but with far less traffic + tokens to process bc its less popular than its competition, and xAI can pass the savings on to the end user.
SpaceX, like Tesla, seems to have the same "portrayals over profits" mindset investors. So it doesn't even really matter whether or not xAI is making any money.
Profitability is never a constraint for Elon companies. He has always been able to be able to extract money from the middle east, government, banks, retail investors (or these same parties through his other companies) whenever they need more.
His net worth is orders of magnitude bigger than the cumulative profits his companies have ever produced (even if you only count the profitable quarters)
It's really easy to do this actually. You just create cars that drive themselves and rockets that land themselves and people start throwing money at you.
From your local Tesla dealer. Technically, you do still have to "supervise" it, which basically means making sure the little camera that watches your eyes doesn't catch you with your eyes off the road too much.
My sister-in-law's mother drove one from Florida to the northeast without touching the steering wheel or pedal/brake, right down to the parking at each end.
I've been a humongous fsd sceptic for a while, but had to lay that aside after I went for a test drive (test ride?) in one of these things.
What that technicality means is that you are liable when the car kills someone, not Tesla. Level 3 self-driving is a completely broken idea. People cannot closely supervise a process that never requires their input - when you suddenly are needed you will not be prepared to respond quickly enough. Either you are driving the car, or you aren't. If you are liable, you are driving it.
It doesn't matter what you call it. FSD is a system that can mostly drive itself, but it has not been certified to drive autonomously, and it cannot safely do so. Humans cannot supervise such a system. Has Tesla resumed reporting disengagement data publicly or to ANY government agency? Why not?
Wouldn't be bad if the product hadn't been paid for in advance. If it can't be delivered in working form for this car, that's not the end of the world. Tesla can just refund the undelivered product, with interest.
How does it compare to Chinese APIs? It doesn't seem like xAI is meaningfully more competent or any single bit more honest than Chinese labs anyway, so you might as well send tasks straight to China unless theirs is substantially cheaper.
But very expensive compared to Deepseek v4 Pro, which performs similarly.
Grok is stuck in a difficult place - not the best model at anything, and not the cheapest either. It's hard to make a case for using it on any dimension, even before you factor in the history (I'm not sure suggesting the company uses the model that refers to itself as "MechaHitler" is the way to a promotion).
Grok 4.5 is a huge step up from their next best model and now around the same performance as GLM 5.2, but it's not exactly at the frontier of the cost efficiency curve in our coding evaluations. That curve is defined by the 2 lighter GPT 5.6 models.
However, the fact that they finally have a strong post-training and RL setup bodes well for future releases. They certainly are not compute-constrained anymore.
Of the 3 models I tried, Grok did the best at making an iOS app I wanted for personal use (a bike computer with specific qualities). (Claude just gave up and did an HTML/CSS implementation but I insisted on native SwiftUI+Metal.) Grok definitely fumbles sometimes, but I have been surprised what it CAN intuit versus me having to micromanage it.
(I am not an iOS developer, so getting something specific that I needed in a few hours/days was really helpful instead of spending months/years learning the language, APIs, etc.)
(I am absolutely not "vibe-coding" Caddy btw, just tinkering with it for personal projects.)
> Claude just gave up and did an HTML/CSS implementation but I insisted on native SwiftUI+Metal.
That sounds very odd and very contrary to my experience. You don’t say which model you actually used, but I never had opus 4.8 (or sonnet for that matter) ignore which language/stack i wanted to use.
I stopped updating CLAUDE.md because I felt like Claude always ignored it. But over time I noticed there is still times (especially during planning and review) where it's good to maintain the official document as a reference. As opposed to memory.md or manual edits.
As someone also not happy with my bike computer (some truly horrific UI/UX decisions), could you share or explain what you made? I like your web server.
I wanted a phone app rather than yet another electronic device. Phones do not have great screens in bright sunlight, and they run hot, so it's not ideal for a bike computer in the first place. But I can't deny the convenience of the multipurpose tool that is my phone.
This app will have a few UI/UX modes. The default is the futuristic-looking HUD, but it has a low-power mode that's mostly monochrome on black, and an even lower-power "Cruise mode" that removes the map entirely and just shows you speed, approximate heading, and nav directions. Still very WIP and mostly for my own amusement!
I swear I have read either this exact or a very similar comment before. Same gist about a bike computer iOS app, and one of the models giving up.
As an aside, big thanks for Caddy! Really helped me get my greenfield project off the ground and it simply “just working” out of the box was one less source of errors I had to worry about when onboarding my team.
I am very curious what your Claude thread looks like. I have never had Claude swap languages, in fact my experience is the opposite, sometimes it holds on too much when working on a large code base.
> Grok 4.5 and Composer 2.5 are two different model weight classes, and we're excited to support both sizes and weights. Composer 2.5 will remain offered, and we will release new models of this size going forward.
Its remarkable how Anthropic is able to maintain their edge against all competition. Anyone have any idea what the secret sauce is that has Anthropic at the top of all leaderboards for the past few years?
From what I have read, their pre-training team is much better than anyone else. For OpenAI, their post-training team is better. And apparently OpenAI has consistently struggled at training a bigger model than GPT 4 level
I’m a VP Eng — the backend team I manage strongly prefers CC and Opus. The Android team I manage strongly prefers Codex and GPT 5. I’m personally not sure that the answer doesn’t just come down to stylistic differences in prompting and ergonomics in the harness. The folks that prefer Codex seem to get better one-shot results, whereas those that prefer CC are doing more iterative prompting. At any rate, I don’t think you should write OpenAI off when it comes to coding.
I think it's focus? Anthropic seemed to double down early on being more business/prosumer focused. While OAI, Gemini, Grok, etc were also doing various side quests like image generation, Anthropic seemed to only focus on 1 thing, and that seemed to pay off
My gut feel is Anthropic is very technical and pedantic which makes their models really technical and pedantic. They're top at code and technical benchmarks but anecdotally I've found OpenAI to be significantly farther ahead for general usage.
Opus 4.8 will burn 10k tokens trying to answer something 100% whereas GPT-5.5 will burn 2k getting it 90% which is good enough for many things.
The problem is that the remaining 10% can bite you in bad ways.
I was in Cotswolds, UK a couple of months ago. For those of you who don't know, it's a rural region known for its "chocolate-box" villages and honey-colored limestone architecture. Basically, you go from village to village, most commonly via bus, taking in the sights and doing touristy stuff.
When planning the trip, my sister used ChatGPT, which helpfully (and relatively quickly) found the bus schedules and times for each hop.
Midway through the day, though, we ran into a huge problem: it turns out bus schedules are different on Sundays, and more limited. Which meant we couldn't actually go to our primary destination (the Model Village), and had to cut the trip short.
Yes, ChatGPT was quick and pleasant to use, but missed a crucial detail.
Afterwards I tried it with Opus and it did not make the same mistake.
Arguably I'd call that the 90%. In my case, answering the restaurant question correctly with "Rishi" in my tests was the sole intent and 90% of the problem. All the models "helpfully" added extra junk about the closure, dates, quotes, etc and many of them got these details wrong--the 10% or extra crap not central to the question.
If the central question was "what is the bus schedule on `day`" and the model screws that up, it gets a fail in my book.
Also curious if Google Maps gets the timetables correct (assuming it has them).
Semi-related, I also discovered that the default web search/fetch tools are pretty primitive and Exa MCP annihilates them. I ended up doing some comparisons with Claude Code comparing built-in server-side to Exa and to a Python MCP that used SearXNG for search and Exa was a clear winner and Python+SearXNG ended up coming out roughly the same after a few cycles of letting Claude optimize the Python code and adjust SearXNG settings. Ultimately it landed on this (making some changes to optimize returning relevant context directly in the search results so the model didn't need an additional web fetch call) https://gist.github.com/nijave/604c43e3e0fdcd60f5280d3a6b109...
This likely comes down to how it accessed the bus schedules (i.e. web search tool) and not intelligence.
You need to add the actual bus schedule to context somehow (research agent, custom tool or just dump in prompt) and even the simpler modern models will be able to do the planning.
This isn't tool usage competency, it's tool quality and/or luck. Regular web search is not good for grounding if you want accurate results. You can ask the model to make a tool for getting bus schedules and then use it only then you are comparing apples to apples in this case.
If the model can't get the information it needs to accurately answer the question, it must surface that risk to you instead of guessing. This is part of model intelligence and tool use competency. Fable and to a lesser extent Opus is very good at this
It's both. Some models, given web search and web fetch may only run a search and assume the summary text is correct and blindly return it. Others will validate by running a web fetch and checking the whole page contents. Even better, the model will run multiple searches and fetches to cross check the information. The best models will attempt to verify whether a source is authoritative or not and try to only return authoritative results.
Tldr; all the Claude models had identical tools and some used them efficiently and verified data while others did a crap job and hallucinated responses. Additionally, Exa MCP tools generally worked better even on older/smaller model (Llama)
I think it's the talent, laser focus on single product set and being early so ahead, same with Open AI who are only a sliver behind. Google, XAI are the next level down but they have other concerns.
I think it is a mix of the sibling replies here. I'd add that the company has seemed to find ways to ~do more with less.
I have never liked the various nerfs Anthropic has used to balance GPU (slowing down responses, quota variance, model optimizations etc) and it definitely has burned a lot of good-will.
But it has seemed that being able to look beyond the short term pitchforks has worked quite well.
I think they have a better agent personality which pushes back and isn't sycophantic. It has been awhile since I've used the others but that's where it locked me in and I've stuck with it.
I think the "secret sauce" is not juicing the benchmarks. Claude models just feel like they are better than the benchmarks suggest, in terms of smarts and creativity, while models from every other company feel worse relative to what you'd think from the benchmarks. Only company to really internalize Goodhart's Law, IMO.
Yeah every model has great benchmarks. Claude is the only model I want to use when I'm not worried about the marginal cost of tokens (which is most of the time at work.)
I then use cheaper models like GLM for personal projects but they're noticeably much worse despite being similar in benchmarks.
Even the tiny Gemma 4 models seem to be better at "rule following" than Opus. Opus seems to just loosely follow constraints in prompts. This works fine in coding/free text responses, but pretty badly in API/pipeline style workflows.
I think most people here just comment primarily on coding capabilities.
You can try it out. Ask Opus to output a JSON in a specific schema without constraining the output via schema enforcement (alongside a decently sized prompt). It screws up a lot
One angle could be their interpretability research? They understand what's going on in LLMs probably much better than anyone else. This must somehow pay off.
I think it's not only an alignment/security tool but could perhaps be used for capabilities as well.
>Its remarkable how Anthropic is able to maintain their edge against all competition. Anyone have any idea what the secret sauce is that has Anthropic at the top of all leaderboards for the past few years?
It's self-reinforcing: they've got the best coding/research model, which helps them to improve their models better than the competition so they stay ahead.
Isn't this the same Twitter company that was supposed to go bankrupt a few years ago? Now it is somehow part of an Space company that has an AI division inside of it?
I think we are going to be waiting a long time for Twitter / X to go bankrupt as it was (erroneously) predicted a long time ago.
Twitter was supposed to go bankrupt if you only read news articles from journalists about it. If you looked at Musk's operating track record, you might have had a different opinion.
In the transaction announcement (xAI buying twitter) that twitter had $12b in debt on acquisition, roughly the amount originally sourced ($13b), so it apparently made good on its debt covenants during the operating period. I have no idea if it received additional capitalization from Musk to do that or not.
That said, the deal was classic Musk - anybody who went on the equity ride with him in Twitter just KILLLED it; xAI was valued at $80bn and twitter at $33bn, so the owners there became 30% owners of xAI. xAI was acquired for $250bn at a SpaceX valuation of $1 trillion, or 20% of the resulting entity, so the twitter stock was 6% of spaceX at about $2 trillion, or $120bn on an equity purchase price basis of $30bn. and that $120bn in value is on really good daily trading volumes; lots of depth.
The solar system diagram doesn't work for me. When I click on the planets, it will center on them. When I click on the sun, nothing happens. When I click on a planet next, it goes to the sun.
Low effort and uninformed comment. The team published a good followup on why this happened: the model pulled in people's own tweets as context to prompting so edge lords that wrote innocuous prompts got to see edge lord content.
- It doesn't seem available in EU (?)
- Using a VPN seems to sort of fix it, but it's way slower than I expected, when everyone was praising it, it feels like the speed is slowly ramping up
- Cost is $2/$6 for <200k context only, above that, cost is $4/$12
- GLM-5.2 still seems smarter, faster and much cheaper:
You have control over how the connections work and what they do, rather than just relying on the VPN exit server routing to get you to the right place.
With things like a fly machine or whatever, you might even be able to run it for free
I concur that GLM-5.2 still seems "better" in my experimentation I did with it tonight, although Grok 4.5 is cheap if you can tolerate the way a Cursor subscription works.
So basically since US stopped OpenAI and Anthropic for 4 weeks, it allowed all other AI Labs to almost catch up.
GLM 5.2 caught up, Cognition RL'ed Kimi 2.7, Grok 4.5 is out, DeepSeek v4 GA is out in a few days...
What is the moat? and why should we pay for the expensive tokens today instead of just waiting a few months/weeks and getting AI for significantly cheaper?
I must say, I feel like companies spending Millions on Anthropic tokens are just negative capex'ing and wasting money, even OpenAI is barely ok pricing...
This is the bind of an arms race. Any lab that tries to pump the breaks quickly becomes second rate. Regulatory capture doesn't work either because the technology crosses jurisdictions.
You can get fable-ish performance with gpt 5.5 watching over opus output. Although it fundementally cannot work as well because gpt 5.5 doesn't see the thinking process behind opus 4.8 unlike fable which presumeably self-steers and is natively trained for it.
See more: https://omp.sh - turn on advisor and set advisor role to gpt 5.5 xhigh thinking.
I have been using GPT 5.5 to review Opus implementation and vice versa.
This does not require any special tools, the skill creators in Claude Code or Codex can set this up for you in five minutes.
It is good for catching bugs, particularly edge cases, and it often suggests abstractions.
It does noting to make Opus deliver the more usable results Fable gives me for user facing features, where the UI typically looked and worked better out of the box with Fable. With Opus, I have to test it myself and give it my feedback first.
just because one model is stopped from being released publicly doesnt mean they completely stop. Anthropic has moved on to training the next gen model months ago.
Interesting. I experimented with Grok 4 for openclaw when they made clear they wanted to bring claw users in the fold. It was (as expected) more verbally fluid than 5.5, but had real trouble with agentic tool calling - the model felt like it hadn't been trained to think of tool calling as one of its primary modalities. I'll give this a try, the speed and the benchmarks look good. In my experience, Grok slightly punches above its weight in language fluidity, and seems to not benchmaxx on coding, so this is an encouraging release.
you can force any model to use structured outputs by just giving it instructions to do so and serializing the response. But yes, Gemini is the best at this, even better than fable.
for what it's worth, it's fairly popular among my non-technical coworkers here in Russia. we have access to all models and they still prefer Gemini over Opus and GPT 5.5. I never asked why, but I assume it's better at communicating in Russian.
Grok's latest model is objectively superior to any of the current Google models in the most relevant use cases. I don't like Elon musk but that does not change reality.
And Google came up with the Transformer architecture (2017 "Attention is all you need"). The Attention mechanism they based it on is from Bahdanau, Cho, and Bengio (2014, ICLR 2015). And there were many other self-attention variants by 2017. It was an amazing paper but let's not twist the story and give proper credit.
And not one of the people in that paper are still at Google, AFAIK.
Google has more compute, more data, and had the best 2 labs. And it seems they squandered it all. I'd blame their McKinsey CEO, the board, and management in general. It's a shadow of what it used to be. And it's a shame.
Still way ahead of xAI in anything meaningful, including market share, enterprise customer buy-in, existing office software moat, etc… Not saying consultant/business-idiot CEOs aren’t a problem, but have you seen Edolf Musk? Again, kind of a silly take…
Google wanted to release 3.5 Pro last month but because of the trouble Anthropic got with Fable they might have wanted to wait a bit for the dust to settle I could imagine. And now there is quite some competition. 3.5 Flash for me is a replacement to 3.1 Pro. It's more like a 3.2 Pro. It costs about the same (or more!) than 3.1 Pro, is a little bit smarter in many cases and a little bit faster.
3.5 Pro will be a lot more expensive and I expect it to juuuust be able to hang with Opus 4.8 and GPT-5.5.
I wish Google was able to actually push the industry further, either in terms of quality (intelligence) or quantity (price) but they've been playing catch up a lot.
They are playing the game a bit differently than all the others. The others have useable IDEs etc. while Google has a boatload of half-assed products.
Google better come out with a banger 3.5 Pro because who would have thought that Grok and GLM would be beating them?
I sometimes wonder if they are leaving their best models for purely internal use. They are after regular users, integration locking with their full stack lock-in...having the best AI public might not add much to that play...just good enough for most people...while their internal models can help their company achieve faster production.. idk
They don't even beat 3.5 flash.. I'm really not sure where you're getting this from.. vibes? 3.5 flash beats out even fable on tool calling, which is really all that matters.
I have paid personal subs to ChatGPT, Claude, Grok, Gemini
For science (primary biology/pharmacology) questions, Gemini 3.1 Flash Extended produces the answers I _personally_ find "best", in terms of content, phrasing, and formatting.
I concur. Generally I find Gemini answers to be the least biased and most factually accurate, without too many of the annoying AI writing quirks.
However, I find the Gemini web app to be by far the worst, and Gemini itself second only to Claude in terms of refusing legitimate requests. It used to be the worst for that, but Claude has really put up the guardrails since their run in with the US president.
Grok has no concept of safety, which means that it can do certain things that none of the other models are allowed do, especially when it comes to research, creative tasks, humour and games.
Lots of stuff involving cybersecurity or reverse engineering for one. At one point Gemini wouldn’t tell me the command line to disable approval on codex because YOLO mode is apparently a hacking technique. I’ve been using Grok as a backend for some experimental AI game storytelling, and obviously fiction isn’t all rainbows and unicorns, so often ChatGPT/Gemini bail out when the going gets rough.
Other things… well people are going to disagree on what’s “legitimate”, but perfectly legal queries about morally questionable topics are commonly refused. ChatGPT refuses to translate or OCR texts containing racial slurs, won’t help with information about building firearms, won’t give you information about how to stage a bank robbery, won’t talk about anything involving suicide, refuses questions around sex work and kink activities.
I can go to the library and rent a book on gunsmithing, sex work is legal, heist fiction is popular. I totally understand why they want to block that stuff off, but I also don’t really like my computer telling me what is acceptable for me to think about based on someone else’s moral code. Also a world where games don’t include anything immoral or “unsafe” is probably quite bland.
I learned that outside of tech, Gemini is the enterprise model.
E.g. in the insurance company where my SO works, the major tasks are writing Gemini "gems" (some kind of prompts I think) and NotebookLM is a killer product for e.g. collecting and summarizing new laws, cross checking documents and what internal regulations are.
I then learned it's used in a chemistry consultancy company of a friend of mine to process reports. Flash and Pro models are also wildly popular in another European bank I know people in to assist in customer care (pre processing tickets before handing them to humans).
Google suite is already at the core of many businesses and Google easily adds these offerings without new contracting being needed.
Don't confuse our bubble with the real world. You can have a disaster product like teams and still dominate enterprise because you were already there with excel, outlook and SharePoint.
I often use Gemini as my "chat" app to ask questions, etc.
I stopped using ChatGPT because of they're weird login system, where it keeps switching to my Workspace Codex account, which doesn't actually have the free/chat functionality.
I usually just switch between gemini/grok when asking questions or to research something online.
What are you talking about gemini 3.5 flash beats fable at tool calling and is 5x faster... I think it's very competitive and what most normal people are using.
> Training included trillions of tokens of Cursor data which capture a wide-range of user interactions with codebases and software tools. This dataset lets the model learn both from existing software as well as developer-agent interactions, capturing how developers work and how agents interact with their environments.
This is what the big money was for. Cursor is the first big player that had real-world data from real-world projects, before cc / codex were a thing.
> We used reinforcement learning on difficult problems in realistic environments spanning both software engineering and broader knowledge work. These environments teach the model to investigate problems, use tools, recover from mistakes, and verify results.
> Many of these problems had to be designed to be difficult enough that even frontier models fail at them. As models improve, existing tasks stop teaching them anything new, and problems that once required extensive reasoning become routine.
> We developed a distributed agent system to construct these environments at scale. Engineers specify a problem and how a solution is verified, and large groups of agents construct, test, and refine each environment.
This is where scale comes in. You use the previous gen model to prepare datasets for the next model iteration. The better the models, the better the data, the better the next models. (they also have a comparison with their composer2.5 training run, for people still thinking chinese models are "close to SotA"...)
Reports of xAIs demise (after giving a lot of compute to Anthropic) were slightly exaggerated, it seems.
> Grok 4.5 was trained across tens of thousands of NVIDIA GB300 GPUs
Cursor has had a good AI product tons of people used for real work for 2yrs (up until recently when the Claude gap widened significantly) while Microsoft/Github has just been pretending they do with Copilot and awful Github AI integrations nobody likes. Meanwhile Github's code has already been vacuumed up by all the models by now.
> You use the previous gen model to prepare datasets for the next model iteration
I've read multiple times that this approach is harmful in training.
You're essentially describing what many call distillation, but it's only useful in post training to guide behavior, it teaches how to behave, not how to think.
I might be wrong though and would be glad if someone more knowledgeable provided more insights.
> I've read multiple times that this approach is harmful in training.
There's a lot of nuance here. Note that I said "prepare" datasets and not just "generate" datasets.
First, the "model collapse" paper(s) were highly misunderstood and the "media" / content creators ran with it because negativity sells. In that initial paper the authors took things to the extreme, and presented as a given what happens in the literal worse case scenario. They used small models, they generated data w/ those models and indiscriminately trained on that data. It obviously led to model collapse. But that's not what you do in the real world.
The way you do this in the real world is different. For pre-training data you can do things to improve the quality of your inputs:
First, you can use the models to curate your datasets. And this is something that everyone has done since the days of "chug common crawl into the model and see what comes out". It turns out that quality of the data is very important and common crawl is really bad. So we've seen attempts at curating that data. The better the filtering models, the better the initial pre-training data.
Then you can have data augmentation, where you take some piece of content, and generate augmentations for it. Current models are good enough that you can take a piece of "authoritative" text (say a book on writing style) and a bunch of articles, and "improve" them. Or take a piece of content and "translate" it into simple / advanced explanations. Or take a piece of code and "explain" what it does, based on a paragraph from an authoritative book. And so on.
Then, for the mid-training / post-training with RL:
You need to find both good scenarios (i.e. problems) for your model to solve and a good verification schema. Like they say in the quote above, those problems need to be complicated enough for each new model. Here you can again use old models to prepare datasets for the new models.
One simple approach is to take a codebase, have your current model identify a set of features. Then instruct the model to remove code relating to feature "a" but keep its tests. Then verify that every other feature works in the code, bar the one you removed. Then, during RL, you train your new model on that task (you present it as a "prompt" / "situation") and you score the model based on the new feature passing the original tests.
Then there are more advanced ways of using prev gen models for "open ended" problems. You can't really apply RL if the task is not easily verifiable (like above, with tests). But you can use something like RLAIF (reinforcement learning w/ AI feedback) where you grade responses with the previous gen models. Now, in general this is lower quality / lower signal than RLVR (verifiable rewards) but you can still do smart things. Instead of rating an answer good / bad, or ask it one-shot what answer is better, you can use a method based on rubrics. You can first ask the preparing model to select tasks, and a list of rubrics on how that task should be scored (like they generally do on open ended exam questions). Then while doing RL you grade each response by asking the prev gen model to generate said rubrics. Does the answer touch on subject a / b / c? Does the answer mention x y z? Is this mathematically sound? And so on. You still get better results than nothing, even if the task is "open ended". And, again, as models improve so does your pipeline.
There was one highly discussed paper. And about a month after publication much much stronger models were released. The models got better because they used such synthetic data.
> You use the previous gen model to prepare datasets for the next model iteration.
you can also use a previous gen model to literally generate data for the next gen model. people used to believe that this is a bad idea but it turns out if you create a scaffold which sinks a lot of compute into generating and grading the data the quality turns out great.
Props to them for including three benchmarks that actually seem to say something, instead of focusing on totally gamed benchmarks like regular SWE-Bench. That could mean this model is actually pretty close to the SOTA as the benchmarks indicate.
Most labs - including OpenAI and Anthropic, but also Google and Chinese labs - highlight their scores in benchmarks that have fixed, widely available answers. Those answers end up in the training data and so models can just regurgitate training data instead of actually doing the benchmark. As a result, most benchmarks often quoted are essentially meaningless for gauging model performance.
Terminal-Bench still publishes answers, but neither DeepSWE and SWE-Bench Pro do. Especially for DeepSWE it's been difficult for models to fake good results so far. SWE-Bench Pro does have weird outliers like good performance for e.g. the atrocious Muse Spark, but it also doesn't provide answers for the training data.
So either they're good, or they found a way to game DeepSWE. Given that the Cursor team previously published the well-received Composer 2.5 a good score here doesn't come out of nowhere, so this might hold up. Cursor has enormous amounts of training data to train good coding models with.
- Very fast, easily beats GPT 5.5/Opus 4.8/GLM 5.2 because of higher t/s (around 90?) and very high token efficiency
- Very good price, no contest vs GPT and Opus which are very overpriced if you pay API costs, and probably cheaper than GLM 5.2 when you take into account the token efficiency.
- Will take quite a while to get a feel for how smart it is, but it's definitely good, I'd say in the same tier as opus, occupying the lower end of that tier together with GLM 5.2.
Tried on a "this test suite is weaker than I'd like, too often depending on internal state rather than outcomes" problem via Cursor, asking it to "review and suggest solutions." It gave me a quality overview of the test approaches, strengths, weaknesses, and gaps then recommended a disciplined multi-prong approach based on a common, trusted testing library (https://hypothesis.readthedocs.io/en/latest/). It broke down the things we could do this improvement pass or leave to later (staged scoping), identified some very hard/possibly-out-of-scope cases and gave me the option of focusing on them or not, and organized new tests in a logical way. After one round of feedback and plan tuning, I put it in agent mode and let it work. A few minutes later I had a much better test suite.
Have not tried Grok before and didn't have much confidence, but it did great. Exactly the sort of complex, detailed, nuanced analysis and multi-step task I would previously only trusted to GPT or Opus.
My benchmark is ripping tailwind out of a few year old elixir Phoenix liveview app, and replacing it with component level scoped styles
It's a good and complex task, that requires touching the build system, most components, the stylesheets, and more. Opus 4.6 could barely do it. Sonnet 4 cannot (haven't tried 5 yet). MiniMax actually did fairly well
Grok aced it, rather quickly and cheaply, surprisingly
I run each through Oh my pi, with dexter providing the LSP for elixir
hmm interesting. maybe im doing something wrong. this model feels borderline unusable to me. it fumbles the most basic asks that require very little to no context consistently (inline these helper functions - re-rewrote half of the modules involved instead of making a 10 line change)
I think there is some india specific pricing with supergrok.
It's just 541 INR/mo here... I have signed up for it will check out the session limits and see if I should switch.
Claude and codex basic 20usd plans are 1800/mo here so this is ridiculously cheaper.
All my friends on opencode go (900/mo - gives u access to glm, ds4pro, etc) will of course switch to this provided it has decent session limits, which I have to test.
The fact that it is more token efficient will itself lead it to be smarter since the context will be smaller for the same task. However, in opus models, you'll have built internal correlations like "if it did X it will usually do Y" which may not be true here, since grok 4.5 may have done X purely due to the smaller context size, but can't do Y cos it wasn't RLd on that pattern enough. So it will be a unique experience as far as opus tier models go.
Can someone breakdown to me how this makes any sort of economical sense? Spending billions and billions to have the 3rd best model while even the number 1 and 2 players already seem to struggle making a profit. What am I missing here? Not trying to go full Ed Zitron but this doesn’t make sense to me.
How is this any different than the browser wars? We use to have a diverse market full of choices, and now we have Chromium (almost all market share) and Firefox/Safari on the edges.
> A diverse market full of choices keeps it from becoming the browser wars all over again.
This is a great analogy but I worry you might be implying something I don't agree with but you didn't explicitly say what I'm worried about, so let me call it out:
Microsoft played a dirty game with I.E, but they are in the dirty game business. It wasn't only I.E, it was their OS, Office suite and everything else they do business in.
Google Chrome took advantage of that dirty game and now you have the Chromium engine that powers a lot of browserlike frameworks.
No one born in the LLM age even knows what I.E means or stands for, as it should be - a horribly designed, poorly working product foisted upon users via the Windows distribution system - a dishonorable product from an ethically corrupt company forever lost in history, right alongside Clippy and DCOM.
OTOH, I am glad that Microsoft played a dirty game with I.E and didn't just stop playing dirty there - they jacked up the price of Windows if an OEM even dared to bundle in Netscape Navigator instead - who knows, if they hadn't done that, there wouldn't have been a Google or Apple. We would all be using Windows and Windows Search and Windows Phone.
And without Google, we might not have had the modern LLM as we know it. We would have had some trashy Windows Autocomplete Copilot Clippy. Ugh!
"No one born in the LLM age even knows what I.E means or stands for, as it should be - a horribly designed, poorly working product"
As one of my first jobs involved getting a website to work with IE6 I surely hated it, but when it came out, it seemed to have pushed the web technologies in general.
The problem was not the browser technology, but microsoft abusing it's monopoly to don't give a shit about (open) web standards.
I still have no clue what you're implying that I was implying? Reading too much into someone's words is definitely a thing.
There used to be 4+ major browser engines. Now there's only two, and Google owns almost complete share & following standards? Google prefers their monoculture.
More diversity in browser engines was a good thing & standards were lovely.
Not everything has to be a slug-fest between #1 & #2.
& personally I'm glad there's Grok & Gemini to keep Anthropic & OpenAI on their toes even more than the competitive band of open weights models already do.
The corporate models tend to be defensive and sound like HR has approved every word of the script. Not a fan.
Google is playing a different game. I don't really know what game they're playing, but they're not trying to beat Claude Code. They have coding capabilities and Antigravity, but I'd be surprised if it's much more than an afterthought. They're focusing on efficiency, models at the edge, human interaction, image and video, etc. in ways Anthropic, in particular, is not.
Google wants its AI to be pervasive in everyone's daily life. Merely being the best at coding is not how you get there.
I am more bullish on Google in AI than most folks, I think, as they have been focused on efficiency in a way most US vendors have not. They've published a ton of papers on ways to make LLMs more efficient and capable on smaller devices..
Google wants to own the on-device market for AI, and I don't see many credible competitors in that space.
This is 1000% not their (Google's) aim. The position they are in is due much more to their organisational laziness and incompetence than it is any grand strategy.
> Google wants its AI to be pervasive in everyone's daily life.
Google wants nothing more than the world to remain stuck in 2000 - 2020 where search was king. Their organisational inertia will fight its AI progress every step of the way and this very well explains why they are not leading the AI pack despite inventing the technology.
I mean, I'm sure there are people within Google who are behaving as though they can keep the dream of the 00s alive in Mountain View, but there's also a whole bunch of people doing work at the frontier in AI. Google has a large lead in hardware, they have the smartest very small models, they have among the most efficient large models (I'd wager their margins on Gemini 3.5 Flash inference are absurd). They have among the best image, video, and audio models, going in every direction (generating, editing, understanding).
Viewed from a consumer lens, the AI the average person interacts with daily, Google seems like the clear leader, especially after locking in Apple as a customer for iPhones.
Why lol? Because you don't like Elon? XAI has continued to stay within a few months of leading edge, and now suddenly they'll just never do it again, despite doing it literally today?
Culture. XAI employs were bragging about spending all weekend at the office. Many tweet that they have been fired after 8 months. Some get fired right before their first big stock vest.
The team is mostly H1Bs. Do you think they don’t have every incentive to benchmax? Do you think top tier talent wants to work there?
> hence Google paying SpaceX hundreds of millions of dollars a month to SpaceX for capacity there.
I don't think that necessarily follows. It could just be old fashioned capacity issues, for example. If nothing else nvidia are able to charge an insane markup on their AI chips at the moment so even if google TPUs aren't competitive in a pure performance sense they are surely competitive from a pricing perspective.
Google seems to become a dead business very soon. Search traffic is being split between AI and social networks and google is bad on both fronts. Its AI proposition is more or less like the Google Plus. Nobody really wants it but they know about it because google pushes it everywhere it can.
AI is a blackhole and Google search along with Adsense and many other “web” publishing platforms is the big shiny star nearby. Can you feel the wrath of AI?
Is uncensored a selling point? What do people use uncensored Grok for (like, real use cases) that they can't or won't use other LLMs for? Literally the only thing I can think of is generating bad porn of unconsenting people.
I mean absolutely read any thread about Fabel and it's fill with people complaining about how it instantly downgrades or refuses if anything has CVE in the name.
Other then that there is the whole alignment issue. Models that are 'nerfed' in just about any manner tend to exhibit reduced performance is seemingly unrelated areas.
That said Grok doesn't appear to be close enough to the frontier for that to matter. Maybe if they catch up it will.
Thanks for the reply. I only use local/open models, and don't use them for security work, so I don't have much exposure to frontier alignment and Fable/Mythos stuff beyond what I read from others here on HN.
> What do people use uncensored Grok for (like, real use cases) that they can't or won't use other LLMs for? Literally the only thing I can think of is generating bad porn of unconsenting people
untrue. There's a full thread about it: https://news.ycombinator.com/item?id=48837162 - but as much as I love Claude products, nothing's more aggravating than it refusing to help me diagnose a stack trace because it "violates Anthopic policy".
I don't remember online discourses on filter avoidance for Grok to be any different from typical ones, except that it allegedly have tendency to take porn-biased interpretations of prompts, I think the "uncensored" pitch they had for a while was pure marketing in the end.
Was that actually benchmarked and compared, or was that some British activist group insisting? I thought it was just the latter.
There is/was the official bot on Twitter that you can tag with a prompt, like "@grok put this spacesuit on a horse on a moon", that's not equal to being uncensored.
They have the same dreams as their competitors - finding a breakthrough that gives them an edge over the others and makes them dominant. And also, having the word 'AI' anywhere near your company makes all the right numbers go up, so having an in-house AI division that Musk can bundle with the other companies to pump their valuations with is very helpful to him, even if the product itself loses some money.
I think it’s simply Musk cynicism winning through here, and you’re right about it being a side-show that lets him juice the stock price. I’m not even sure he’s wrong: if his lab is the one that gets a definitive break-away advantage, then every dollar investment in his stock will be paid back many times over.
Grok build already punched above its weight. I think Anthropic and OpenAI are both engaged in their own theatre, trying to define and redefine what game they are even playing, trying to shift to immeasurables like safety or security or exclusivity. There's definitely room at the top.
I had to scroll down so far to see someone who speaks my language. Thank you. If Grok was the last model on the planet, I would not use it. For the very reason mentioned above. And no, none of the other tech CEOs are that comically evil that they’d take it upon themselves to cut aid from the world’s most vulnerable children while also being the world’s richest man. The optics of that alone… Never letting it go.
Frontier is one thing, but low-cost really good models are another. All the chatbots and day-to-day corporate bots are likely to use models that offer the best performance at the lowest cost. I think Grok has an angle here if they can build customer trust.
Then don’t read my response and don’t reply to it, simple. This is a public thread on a social media platform. I don’t need to be asked to speak. I certainly don’t need your approval to say anything I wish to say. Who do you think you are? Lmao… Check your ego.
My comment was directly relevant to yours. So I’ll say what I want, without needing your permission, thank you. Kudos to the mods for not flagging my original comment. They do that sometimes if you say the thing ;)
My hypothesis is that all the top providers realize that, lacking vendor lock in, all SOTA models in a year or so's time will be similar in capability. Also, open weights models are continuing to catch up in a year's time, sometimes less.
So they are trying to lure you in with differentiating, superior capabilities into their proprietary, non-open, non-standard agent harness.
It's the Hotel California playbook: These amazing capabilities are to attract you like moths to a flame and keep you warm and alive around the flame but waterboard and shock you if you attempt to move away from it. Like AWS Egress charges.
Its less about the model; elon is trying to make SpaceXAI a hyper scaler that also happens to have a good model. Grok is just the cherry on top of a powerful AI cluster that can also rent compute to its competitors, like aws.
My guess is that the use here is similar to the reason AWS started as Amazon selling their excess capacity.
Between Tesla, SpaceX, X, Boring Co and Neuralink they probably want the capability internally for a lot of different applications.
If the whole data centers in space thing works out AND people keep protesting/blocking data center build outs on land SpaceX will eventually dominate the entire AI industry just based on escaping scarcity.
It's simple. Elon's top priority now is "killing the woke mind virus" at any cost, and his Nazi AI is a key tool for that. As long as twitter users take Grok at face value, and spread its talking points all over, it's worth it to him. It doesn't matter if it doesn't make economical sense, it only matters that Elon Musk personally wants to keep it going.
I don't want to go into it, because I agree that Elon is a very disturbing person, and there's clear evidence that Grok's harness attempts to bias towards his views.
However, Grok also seems to come out consistently as the most balanced of the chat-based LLMs...
So I'm not sure how to reconcile that.. maybe that's in line with "free speech absolutism", and if so, that's something I can get behind.
Anthropic is already profitable, economics is no longer an issue as they have found PMF in enterprise software market. You might need to update your views.
Having a profitable quarter in which they were given an undisclosed discount on compute only in that quarter does not necessarily mean “Anthropic is Profitable.” It doesn’t mean they’re not, either. Even the breathless article about their first profitable quarter, (which, frankly, read more like a press release,) mentioned in passing that it’s “not clear” if it’s sustainable because their compute expenses are likely to increase. I get the feeling that if they were sustainably profitable, they’d shout it from the rooftops.
But we don’t know.
If someone proudly announces they and their partner could afford to eat at a particular fancy restaurant every night last week, but for that specific week the restaurant had a BOGO deal, and they also didn’t disclose how they determined that they could afford it, you don’t really know if they could sustainably afford to eat there every night, right?
Previously they were a distant fourth. They're not going to single-shot catch up to OpenAI or Anthropic, but they moved up the ladder one rung.
In the short term labs are not profitable, although supposedly Anthropic is close. But Amazon was also famously unprofitable for many many years, and then won huge. Current profits or lack thereof are not necessarily important to investors: what's important is they believe in your future potential profits.
In this case, Elon clearly believes much of the economy will be run by AI in the future, and the economic value of a token will rise faster than the cost of generating the token — including the amortized cost of training the model to produce that token. Thus he is building a lab to train models and charge for inference of those models, and — he believes — it will eventually become profitable even if it isn't now.
You may or may not agree with him (and you may or may not agree he's capable of beating Ant/OAI), but current profits aren't a great indicator of whether he believes future profits are attainable. Tesla and SpaceX were also very unprofitable, until they weren't.
Personally I agree with him that there will be massive profits in the future, although I am not as confident in his ability to beat Ant/OAI, at least given his recent difficulties in retaining researchers.
That is the whole purpose of research. You could have put the same argument for any breakthrough technology, like why spend billions on something new when you have XYZ already.
By tying his rockets to AI with his vision of “orbital data centers”, Elon turned an $8 per share IPO (according to financial times and morgan stanley) into a $135 (1.8T) IPO.
To be clear: Morgan Stanley says that they are skeptical of this, but that’s their perception of where the value is supposed to come from based on SpaceX’s pitch
This market is far from mature or established to be making rankings. There's been plenty of tech markets where the early days didn't predict the later years.
I'm personally skeptical of Grok but maybe they can pull off a profitable niche with Cursor integration once Claude loses it's edge.
Cursor has a lot of proprietary data to take advantage of, so worth a try? And once you have something workable, even if it doesn't lead, you might as well release it.
They only struggle to make a profit because of investments into their future, basically: training, aggressive hiring of AI researchers. Anthropic seems to have 90% margin on their API pricing and all enterprise customers have to use this.
And the reason they can do this is because they can create a $1Tr company in 5 years, so they know the investment will pay off.
Why Elon wants his own model so much is a good question with many possible answers, but if Cursor/xAI can produce a truly good model at competitive pricing I don't see why many people won't jump on it.
Stock market / investor driven products do not make economical sense, yet here we are; just because they (may) not make a profit, doesn't mean they don't generate value. My house doesn't make a profit, but it does appreciate in value over time. AMD famously didn't make a profit for years.
960 comments
[ 0.20 ms ] story [ 164 ms ] threadThis is the first grok model that seems actually pretty competitive at SWE.
Like the reason that close to a McDonals there is usually a Burger King.
In this case, ChatGPT 5.6 Sol / Ultra releases tomorrow, so today is the last day Grok can compare Grok 4.5 to Codex 5.5. If they did it tomorrow people would point out they're comparing themselves against old models.
And by benchmarks (unless they gamed them), seems to be at around Opus 4.7 level, which is what Elon mentioned in https://x.com/elonmusk/status/2074911038286295049.
I guess the Cursor data was very useful.
I wonder how good their subscription discount is on both their subscription types.
Noam Brown (OpenAI) "Implications of Large-Scale Test-Time Compute" https://xcancel.com/i/article/2064210146558136827
Above that (max context is 500K) pricing doubles to $4/12.
https://docs.x.ai/developers/models/grok-4.5
No longer feels as inexpensive. Will likely just include this in the rolodex of <200k context tasks, like being one of my review agents.
I wish my company gave me more options than just using Claude to test these things out
As always it depends on what you are using them for, and how you are using them.
Anthropic have a fixed price regardless of context usage.
These per-token pricing schemes aren't directly comparable though since these models all use different numbers of tokens, even for input (Anthropic's recent tokenizer change generates 30% more tokens for exact same input), as well as for reasoning, and context/token usage also varies wildly by harness with Claude Code using 3x the context/tokens of Pi.
Does anyone know why they would charge more for higher context usage (other than they can)?
AI serving cost is apparently mostly hardware depreciation rather than operating cost (electricity etc), and if your large context request is occupying VRAM for some fraction of a second then you are paying for the depreciation that occurs in that time!
e. H100 costs $20-40K to buy, with a lifetime of maybe 3 years, and will only consume maybe $2K in electricity if run 24x7 for those 3 years.
To use an analogy: imagine your friend is the author of an unfinished book. They die with 19 chapters written, and on their death bed ask you to write the 20th chapter. Assuming you're up to the task, you can only do this well if you take the time to absorb the entirety of what's been written so far.
This is how LLMs work. Context caching is an optimization on top of this, but it has its limits.
really dont think they have a lot of idle power
His net worth is orders of magnitude bigger than the cumulative profits his companies have ever produced (even if you only count the profitable quarters)
My sister-in-law's mother drove one from Florida to the northeast without touching the steering wheel or pedal/brake, right down to the parking at each end.
I've been a humongous fsd sceptic for a while, but had to lay that aside after I went for a test drive (test ride?) in one of these things.
Not bad to get a product that underdeliver 8 years late ?
Grok is stuck in a difficult place - not the best model at anything, and not the cheapest either. It's hard to make a case for using it on any dimension, even before you factor in the history (I'm not sure suggesting the company uses the model that refers to itself as "MechaHitler" is the way to a promotion).
However, the fact that they finally have a strong post-training and RL setup bodes well for future releases. They certainly are not compute-constrained anymore.
Data at https://gertlabs.com/rankings?mode=oneshot_coding
(I am not an iOS developer, so getting something specific that I needed in a few hours/days was really helpful instead of spending months/years learning the language, APIs, etc.) (I am absolutely not "vibe-coding" Caddy btw, just tinkering with it for personal projects.)
That sounds very odd and very contrary to my experience. You don’t say which model you actually used, but I never had opus 4.8 (or sonnet for that matter) ignore which language/stack i wanted to use.
Sure. I'm not sure if I will actually publish this thing, but I can show you: https://x.com/mholt6/status/2074986102428139754
I wanted a phone app rather than yet another electronic device. Phones do not have great screens in bright sunlight, and they run hot, so it's not ideal for a bike computer in the first place. But I can't deny the convenience of the multipurpose tool that is my phone.
This app will have a few UI/UX modes. The default is the futuristic-looking HUD, but it has a low-power mode that's mostly monochrome on black, and an even lower-power "Cruise mode" that removes the map entirely and just shows you speed, approximate heading, and nav directions. Still very WIP and mostly for my own amusement!
As an aside, big thanks for Caddy! Really helped me get my greenfield project off the ground and it simply “just working” out of the box was one less source of errors I had to worry about when onboarding my team.
Notably:
> Grok 4.5 and Composer 2.5 are two different model weight classes, and we're excited to support both sizes and weights. Composer 2.5 will remain offered, and we will release new models of this size going forward.
The API cost difference is ~2.5x, probably because xAI has much higher costs to recoup.
They also target a cost-insensitive market (corporate/coding users) compared to Google/OpenAI which support massive amounts of free users.
Would be nice if an insider would drop some hints so that the open-source space could make some good progress.
Same as with rich person autobiographies: even when they tell you what they think it is, they can't see the path not travelled.
Opus 4.8 will burn 10k tokens trying to answer something 100% whereas GPT-5.5 will burn 2k getting it 90% which is good enough for many things.
Some personal testing on a "help me find that restaurant" prompt https://gist.github.com/nijave/2873b8b10d8c732e46264237b0755...
I was in Cotswolds, UK a couple of months ago. For those of you who don't know, it's a rural region known for its "chocolate-box" villages and honey-colored limestone architecture. Basically, you go from village to village, most commonly via bus, taking in the sights and doing touristy stuff.
When planning the trip, my sister used ChatGPT, which helpfully (and relatively quickly) found the bus schedules and times for each hop.
Midway through the day, though, we ran into a huge problem: it turns out bus schedules are different on Sundays, and more limited. Which meant we couldn't actually go to our primary destination (the Model Village), and had to cut the trip short.
Yes, ChatGPT was quick and pleasant to use, but missed a crucial detail.
Afterwards I tried it with Opus and it did not make the same mistake.
If the central question was "what is the bus schedule on `day`" and the model screws that up, it gets a fail in my book.
Also curious if Google Maps gets the timetables correct (assuming it has them).
Semi-related, I also discovered that the default web search/fetch tools are pretty primitive and Exa MCP annihilates them. I ended up doing some comparisons with Claude Code comparing built-in server-side to Exa and to a Python MCP that used SearXNG for search and Exa was a clear winner and Python+SearXNG ended up coming out roughly the same after a few cycles of letting Claude optimize the Python code and adjust SearXNG settings. Ultimately it landed on this (making some changes to optimize returning relevant context directly in the search results so the model didn't need an additional web fetch call) https://gist.github.com/nijave/604c43e3e0fdcd60f5280d3a6b109...
You need to add the actual bus schedule to context somehow (research agent, custom tool or just dump in prompt) and even the simpler modern models will be able to do the planning.
I have an example here: https://gist.github.com/nijave/2873b8b10d8c732e46264237b0755...
Tldr; all the Claude models had identical tools and some used them efficiently and verified data while others did a crap job and hallucinated responses. Additionally, Exa MCP tools generally worked better even on older/smaller model (Llama)
If I add "Research the question extensively" to your prompt at the end I get the correct answer from Haiku and Sonnet Med on first try.
I have never liked the various nerfs Anthropic has used to balance GPU (slowing down responses, quota variance, model optimizations etc) and it definitely has burned a lot of good-will.
But it has seemed that being able to look beyond the short term pitchforks has worked quite well.
I then use cheaper models like GLM for personal projects but they're noticeably much worse despite being similar in benchmarks.
Even the tiny Gemma 4 models seem to be better at "rule following" than Opus. Opus seems to just loosely follow constraints in prompts. This works fine in coding/free text responses, but pretty badly in API/pipeline style workflows.
I think most people here just comment primarily on coding capabilities.
You can try it out. Ask Opus to output a JSON in a specific schema without constraining the output via schema enforcement (alongside a decently sized prompt). It screws up a lot
I think it's not only an alignment/security tool but could perhaps be used for capabilities as well.
It's self-reinforcing: they've got the best coding/research model, which helps them to improve their models better than the competition so they stay ahead.
I think we are going to be waiting a long time for Twitter / X to go bankrupt as it was (erroneously) predicted a long time ago.
In the transaction announcement (xAI buying twitter) that twitter had $12b in debt on acquisition, roughly the amount originally sourced ($13b), so it apparently made good on its debt covenants during the operating period. I have no idea if it received additional capitalization from Musk to do that or not.
That said, the deal was classic Musk - anybody who went on the equity ride with him in Twitter just KILLLED it; xAI was valued at $80bn and twitter at $33bn, so the owners there became 30% owners of xAI. xAI was acquired for $250bn at a SpaceX valuation of $1 trillion, or 20% of the resulting entity, so the twitter stock was 6% of spaceX at about $2 trillion, or $120bn on an equity purchase price basis of $30bn. and that $120bn in value is on really good daily trading volumes; lots of depth.
I'll give this one a try with a grain of salt and lowering my levels of expectations
With things like a fly machine or whatever, you might even be able to run it for free
GLM 5.2 caught up, Cognition RL'ed Kimi 2.7, Grok 4.5 is out, DeepSeek v4 GA is out in a few days...
What is the moat? and why should we pay for the expensive tokens today instead of just waiting a few months/weeks and getting AI for significantly cheaper?
I must say, I feel like companies spending Millions on Anthropic tokens are just negative capex'ing and wasting money, even OpenAI is barely ok pricing...
See more: https://omp.sh - turn on advisor and set advisor role to gpt 5.5 xhigh thinking.
This does not require any special tools, the skill creators in Claude Code or Codex can set this up for you in five minutes.
It is good for catching bugs, particularly edge cases, and it often suggests abstractions.
It does noting to make Opus deliver the more usable results Fable gives me for user facing features, where the UI typically looked and worked better out of the box with Fable. With Opus, I have to test it myself and give it my feedback first.
It's pretty good for image/video inputs, though.
Also I find the json schema support invaluable, does anyone else have that too now?
Not sure it's a valid data point.
I did not have this one in my 2026 bingo card.
And Google came up with the Transformer architecture (2017 "Attention is all you need"). The Attention mechanism they based it on is from Bahdanau, Cho, and Bengio (2014, ICLR 2015). And there were many other self-attention variants by 2017. It was an amazing paper but let's not twist the story and give proper credit.
And not one of the people in that paper are still at Google, AFAIK.
Google has more compute, more data, and had the best 2 labs. And it seems they squandered it all. I'd blame their McKinsey CEO, the board, and management in general. It's a shadow of what it used to be. And it's a shame.
I wish Google was able to actually push the industry further, either in terms of quality (intelligence) or quantity (price) but they've been playing catch up a lot.
They are playing the game a bit differently than all the others. The others have useable IDEs etc. while Google has a boatload of half-assed products.
Google better come out with a banger 3.5 Pro because who would have thought that Grok and GLM would be beating them?
For science (primary biology/pharmacology) questions, Gemini 3.1 Flash Extended produces the answers I _personally_ find "best", in terms of content, phrasing, and formatting.
However, I find the Gemini web app to be by far the worst, and Gemini itself second only to Claude in terms of refusing legitimate requests. It used to be the worst for that, but Claude has really put up the guardrails since their run in with the US president.
Grok has no concept of safety, which means that it can do certain things that none of the other models are allowed do, especially when it comes to research, creative tasks, humour and games.
Other things… well people are going to disagree on what’s “legitimate”, but perfectly legal queries about morally questionable topics are commonly refused. ChatGPT refuses to translate or OCR texts containing racial slurs, won’t help with information about building firearms, won’t give you information about how to stage a bank robbery, won’t talk about anything involving suicide, refuses questions around sex work and kink activities.
I can go to the library and rent a book on gunsmithing, sex work is legal, heist fiction is popular. I totally understand why they want to block that stuff off, but I also don’t really like my computer telling me what is acceptable for me to think about based on someone else’s moral code. Also a world where games don’t include anything immoral or “unsafe” is probably quite bland.
I learned that outside of tech, Gemini is the enterprise model.
E.g. in the insurance company where my SO works, the major tasks are writing Gemini "gems" (some kind of prompts I think) and NotebookLM is a killer product for e.g. collecting and summarizing new laws, cross checking documents and what internal regulations are.
I then learned it's used in a chemistry consultancy company of a friend of mine to process reports. Flash and Pro models are also wildly popular in another European bank I know people in to assist in customer care (pre processing tickets before handing them to humans).
Google suite is already at the core of many businesses and Google easily adds these offerings without new contracting being needed.
Don't confuse our bubble with the real world. You can have a disaster product like teams and still dominate enterprise because you were already there with excel, outlook and SharePoint.
I stopped using ChatGPT because of they're weird login system, where it keeps switching to my Workspace Codex account, which doesn't actually have the free/chat functionality.
I usually just switch between gemini/grok when asking questions or to research something online.
terminal is nice but codex desktop app is very useful
> Training included trillions of tokens of Cursor data which capture a wide-range of user interactions with codebases and software tools. This dataset lets the model learn both from existing software as well as developer-agent interactions, capturing how developers work and how agents interact with their environments.
This is what the big money was for. Cursor is the first big player that had real-world data from real-world projects, before cc / codex were a thing.
> We used reinforcement learning on difficult problems in realistic environments spanning both software engineering and broader knowledge work. These environments teach the model to investigate problems, use tools, recover from mistakes, and verify results.
> Many of these problems had to be designed to be difficult enough that even frontier models fail at them. As models improve, existing tasks stop teaching them anything new, and problems that once required extensive reasoning become routine.
> We developed a distributed agent system to construct these environments at scale. Engineers specify a problem and how a solution is verified, and large groups of agents construct, test, and refine each environment.
This is where scale comes in. You use the previous gen model to prepare datasets for the next model iteration. The better the models, the better the data, the better the next models. (they also have a comparison with their composer2.5 training run, for people still thinking chinese models are "close to SotA"...)
Reports of xAIs demise (after giving a lot of compute to Anthropic) were slightly exaggerated, it seems.
> Grok 4.5 was trained across tens of thousands of NVIDIA GB300 GPUs
I've read multiple times that this approach is harmful in training.
You're essentially describing what many call distillation, but it's only useful in post training to guide behavior, it teaches how to behave, not how to think.
I might be wrong though and would be glad if someone more knowledgeable provided more insights.
There's a lot of nuance here. Note that I said "prepare" datasets and not just "generate" datasets.
First, the "model collapse" paper(s) were highly misunderstood and the "media" / content creators ran with it because negativity sells. In that initial paper the authors took things to the extreme, and presented as a given what happens in the literal worse case scenario. They used small models, they generated data w/ those models and indiscriminately trained on that data. It obviously led to model collapse. But that's not what you do in the real world.
The way you do this in the real world is different. For pre-training data you can do things to improve the quality of your inputs:
First, you can use the models to curate your datasets. And this is something that everyone has done since the days of "chug common crawl into the model and see what comes out". It turns out that quality of the data is very important and common crawl is really bad. So we've seen attempts at curating that data. The better the filtering models, the better the initial pre-training data.
Then you can have data augmentation, where you take some piece of content, and generate augmentations for it. Current models are good enough that you can take a piece of "authoritative" text (say a book on writing style) and a bunch of articles, and "improve" them. Or take a piece of content and "translate" it into simple / advanced explanations. Or take a piece of code and "explain" what it does, based on a paragraph from an authoritative book. And so on.
Then, for the mid-training / post-training with RL:
You need to find both good scenarios (i.e. problems) for your model to solve and a good verification schema. Like they say in the quote above, those problems need to be complicated enough for each new model. Here you can again use old models to prepare datasets for the new models.
One simple approach is to take a codebase, have your current model identify a set of features. Then instruct the model to remove code relating to feature "a" but keep its tests. Then verify that every other feature works in the code, bar the one you removed. Then, during RL, you train your new model on that task (you present it as a "prompt" / "situation") and you score the model based on the new feature passing the original tests.
Then there are more advanced ways of using prev gen models for "open ended" problems. You can't really apply RL if the task is not easily verifiable (like above, with tests). But you can use something like RLAIF (reinforcement learning w/ AI feedback) where you grade responses with the previous gen models. Now, in general this is lower quality / lower signal than RLVR (verifiable rewards) but you can still do smart things. Instead of rating an answer good / bad, or ask it one-shot what answer is better, you can use a method based on rubrics. You can first ask the preparing model to select tasks, and a list of rubrics on how that task should be scored (like they generally do on open ended exam questions). Then while doing RL you grade each response by asking the prev gen model to generate said rubrics. Does the answer touch on subject a / b / c? Does the answer mention x y z? Is this mathematically sound? And so on. You still get better results than nothing, even if the task is "open ended". And, again, as models improve so does your pipeline.
In the blog post, it is unclear whether Grok 4.5 is also a finetune on top of Kimi; they do imply it is also a finetune.
> Training included trillions of tokens of Cursor data… We used reinforcement learning on difficult problems
If xAI pivoted from a frontier base model company, to a finetuning company, it does mark a stark change to their relevance in the industry.
[0]: https://cursor.com/blog/composer-2-5
Most labs - including OpenAI and Anthropic, but also Google and Chinese labs - highlight their scores in benchmarks that have fixed, widely available answers. Those answers end up in the training data and so models can just regurgitate training data instead of actually doing the benchmark. As a result, most benchmarks often quoted are essentially meaningless for gauging model performance.
Terminal-Bench still publishes answers, but neither DeepSWE and SWE-Bench Pro do. Especially for DeepSWE it's been difficult for models to fake good results so far. SWE-Bench Pro does have weird outliers like good performance for e.g. the atrocious Muse Spark, but it also doesn't provide answers for the training data.
So either they're good, or they found a way to game DeepSWE. Given that the Cursor team previously published the well-received Composer 2.5 a good score here doesn't come out of nowhere, so this might hold up. Cursor has enormous amounts of training data to train good coding models with.
- Very fast, easily beats GPT 5.5/Opus 4.8/GLM 5.2 because of higher t/s (around 90?) and very high token efficiency
- Very good price, no contest vs GPT and Opus which are very overpriced if you pay API costs, and probably cheaper than GLM 5.2 when you take into account the token efficiency.
- Will take quite a while to get a feel for how smart it is, but it's definitely good, I'd say in the same tier as opus, occupying the lower end of that tier together with GLM 5.2.
Tried on a "this test suite is weaker than I'd like, too often depending on internal state rather than outcomes" problem via Cursor, asking it to "review and suggest solutions." It gave me a quality overview of the test approaches, strengths, weaknesses, and gaps then recommended a disciplined multi-prong approach based on a common, trusted testing library (https://hypothesis.readthedocs.io/en/latest/). It broke down the things we could do this improvement pass or leave to later (staged scoping), identified some very hard/possibly-out-of-scope cases and gave me the option of focusing on them or not, and organized new tests in a logical way. After one round of feedback and plan tuning, I put it in agent mode and let it work. A few minutes later I had a much better test suite.
Have not tried Grok before and didn't have much confidence, but it did great. Exactly the sort of complex, detailed, nuanced analysis and multi-step task I would previously only trusted to GPT or Opus.
It's a good and complex task, that requires touching the build system, most components, the stylesheets, and more. Opus 4.6 could barely do it. Sonnet 4 cannot (haven't tried 5 yet). MiniMax actually did fairly well
Grok aced it, rather quickly and cheaply, surprisingly
I run each through Oh my pi, with dexter providing the LSP for elixir
It's just 541 INR/mo here... I have signed up for it will check out the session limits and see if I should switch.
Claude and codex basic 20usd plans are 1800/mo here so this is ridiculously cheaper.
All my friends on opencode go (900/mo - gives u access to glm, ds4pro, etc) will of course switch to this provided it has decent session limits, which I have to test.
The fact that it is more token efficient will itself lead it to be smarter since the context will be smaller for the same task. However, in opus models, you'll have built internal correlations like "if it did X it will usually do Y" which may not be true here, since grok 4.5 may have done X purely due to the smaller context size, but can't do Y cos it wasn't RLd on that pattern enough. So it will be a unique experience as far as opus tier models go.
A diverse market full of choices keeps it from becoming the browser wars all over again.
This is a great analogy but I worry you might be implying something I don't agree with but you didn't explicitly say what I'm worried about, so let me call it out:
Microsoft played a dirty game with I.E, but they are in the dirty game business. It wasn't only I.E, it was their OS, Office suite and everything else they do business in.
Google Chrome took advantage of that dirty game and now you have the Chromium engine that powers a lot of browserlike frameworks.
No one born in the LLM age even knows what I.E means or stands for, as it should be - a horribly designed, poorly working product foisted upon users via the Windows distribution system - a dishonorable product from an ethically corrupt company forever lost in history, right alongside Clippy and DCOM.
OTOH, I am glad that Microsoft played a dirty game with I.E and didn't just stop playing dirty there - they jacked up the price of Windows if an OEM even dared to bundle in Netscape Navigator instead - who knows, if they hadn't done that, there wouldn't have been a Google or Apple. We would all be using Windows and Windows Search and Windows Phone.
And without Google, we might not have had the modern LLM as we know it. We would have had some trashy Windows Autocomplete Copilot Clippy. Ugh!
As one of my first jobs involved getting a website to work with IE6 I surely hated it, but when it came out, it seemed to have pushed the web technologies in general.
The problem was not the browser technology, but microsoft abusing it's monopoly to don't give a shit about (open) web standards.
There used to be 4+ major browser engines. Now there's only two, and Google owns almost complete share & following standards? Google prefers their monoculture.
More diversity in browser engines was a good thing & standards were lovely.
Not everything has to be a slug-fest between #1 & #2.
& personally I'm glad there's Grok & Gemini to keep Anthropic & OpenAI on their toes even more than the competitive band of open weights models already do.
The corporate models tend to be defensive and sound like HR has approved every word of the script. Not a fan.
Google wants its AI to be pervasive in everyone's daily life. Merely being the best at coding is not how you get there.
I am more bullish on Google in AI than most folks, I think, as they have been focused on efficiency in a way most US vendors have not. They've published a ton of papers on ways to make LLMs more efficient and capable on smaller devices.. Google wants to own the on-device market for AI, and I don't see many credible competitors in that space.
It's an unbelievable failure.
Google wants nothing more than the world to remain stuck in 2000 - 2020 where search was king. Their organisational inertia will fight its AI progress every step of the way and this very well explains why they are not leading the AI pack despite inventing the technology.
Viewed from a consumer lens, the AI the average person interacts with daily, Google seems like the clear leader, especially after locking in Apple as a customer for iPhones.
The team is mostly H1Bs. Do you think they don’t have every incentive to benchmax? Do you think top tier talent wants to work there?
I don't think that necessarily follows. It could just be old fashioned capacity issues, for example. If nothing else nvidia are able to charge an insane markup on their AI chips at the moment so even if google TPUs aren't competitive in a pure performance sense they are surely competitive from a pricing perspective.
Citation needed. The gemini app has 750 million MAU, hardly a dead business.
Terms like "Google it" have been completely replace by "Ask AI".
I personally mostly use google to find businesses close to me and to search reddit and wikipedia.
Other then that there is the whole alignment issue. Models that are 'nerfed' in just about any manner tend to exhibit reduced performance is seemingly unrelated areas.
That said Grok doesn't appear to be close enough to the frontier for that to matter. Maybe if they catch up it will.
untrue. There's a full thread about it: https://news.ycombinator.com/item?id=48837162 - but as much as I love Claude products, nothing's more aggravating than it refusing to help me diagnose a stack trace because it "violates Anthopic policy".
There is/was the official bot on Twitter that you can tag with a prompt, like "@grok put this spacesuit on a horse on a moon", that's not equal to being uncensored.
Not sure that is a good thing.
Elon's reaction to these kinds of statements is oddly predictable.
My hypothesis is that all the top providers realize that, lacking vendor lock in, all SOTA models in a year or so's time will be similar in capability. Also, open weights models are continuing to catch up in a year's time, sometimes less.
So they are trying to lure you in with differentiating, superior capabilities into their proprietary, non-open, non-standard agent harness.
It's the Hotel California playbook: These amazing capabilities are to attract you like moths to a flame and keep you warm and alive around the flame but waterboard and shock you if you attempt to move away from it. Like AWS Egress charges.
https://news.ycombinator.com/item?id=48828648
Also Elon has a grudge with Sam Altman and wants to beat him
Between Tesla, SpaceX, X, Boring Co and Neuralink they probably want the capability internally for a lot of different applications.
If the whole data centers in space thing works out AND people keep protesting/blocking data center build outs on land SpaceX will eventually dominate the entire AI industry just based on escaping scarcity.
However, Grok also seems to come out consistently as the most balanced of the chat-based LLMs...
So I'm not sure how to reconcile that.. maybe that's in line with "free speech absolutism", and if so, that's something I can get behind.
https://arxiv.org/pdf/2603.23841
https://www.washingtonpost.com/technology/interactive/2026/0... (X post about it: https://x.com/genejchan/status/2070099701547090340)
https://www.promptfoo.dev/blog/grok-4-political-bias/
https://www.wsj.com/tech/ai/mind-blowing-growth-is-about-to-...
But we don’t know.
If someone proudly announces they and their partner could afford to eat at a particular fancy restaurant every night last week, but for that specific week the restaurant had a BOGO deal, and they also didn’t disclose how they determined that they could afford it, you don’t really know if they could sustainably afford to eat there every night, right?
Anthropic is definitely profitable now, in fact, they’re crushing it.
In the short term labs are not profitable, although supposedly Anthropic is close. But Amazon was also famously unprofitable for many many years, and then won huge. Current profits or lack thereof are not necessarily important to investors: what's important is they believe in your future potential profits.
In this case, Elon clearly believes much of the economy will be run by AI in the future, and the economic value of a token will rise faster than the cost of generating the token — including the amortized cost of training the model to produce that token. Thus he is building a lab to train models and charge for inference of those models, and — he believes — it will eventually become profitable even if it isn't now.
You may or may not agree with him (and you may or may not agree he's capable of beating Ant/OAI), but current profits aren't a great indicator of whether he believes future profits are attainable. Tesla and SpaceX were also very unprofitable, until they weren't.
Personally I agree with him that there will be massive profits in the future, although I am not as confident in his ability to beat Ant/OAI, at least given his recent difficulties in retaining researchers.
Amazon was 'unit profitable' very early.
Yes - it's not unreasonable for Elon to bet long horizon ... there are after all many car companies, why not AI?
He's already winning gov. contracts, that could continue.
It's an odd bet but not entirely wrong or dubious.
And token costs vs. raw compute+electric cost is unit profitable too.
By tying his rockets to AI with his vision of “orbital data centers”, Elon turned an $8 per share IPO (according to financial times and morgan stanley) into a $135 (1.8T) IPO.
I'm personally skeptical of Grok but maybe they can pull off a profitable niche with Cursor integration once Claude loses it's edge.
And the reason they can do this is because they can create a $1Tr company in 5 years, so they know the investment will pay off.
Why Elon wants his own model so much is a good question with many possible answers, but if Cursor/xAI can produce a truly good model at competitive pricing I don't see why many people won't jump on it.
Did anthropic found their moat or we hit a Wall?