I'd love to! For real coding though, SOTA models barely get the job done. It wasn't until Opus 4.5 that you could really get decent results.
I'm sure this will change (and I can't wait for it!) but as of today, open models might be fine for summarizing and writing docs, but you need SOTA to work on code if you want to be competitive.
I do think there are now open weight models that are on par with (or beating) Opus 4.5 by now (e.g. Kimi K3, GLM5.3). But yeah obviously the frontier closed source models seem to have pulled away once again, so open weight seems to be a few months behind right now (which might be too long to wait for a lot of people!).
Every time I check in on this I hear a more recent model is the one where they started doing good work. I'm excited to here that Astra is where it got capable enough to work on code next year
You're not corporate America (and trust me, I mostly mean that as a plus).
I also work in software, and while I vaguely disagree that open models can't be used (they absolutely fit into productive niches here, and holy hell are the last generation [ex laguna s1, kimi k3, glm 5.3, etc] actually decent) - I will agree that SOTA are a better fit for software development, especially when used in conjunction with an already very expensive employee who's driving them.
But for "Corporate America"... no. You absolutely don't need SOTA. They're doing things like transcription, summarization, customer interaction, minor technical tasks like form creation in existing tools, report generation (ex - powerpoint, pdf, docs, etc) and other general "white collar tasks". Think about roles in business that are in the 60-85k compensation range.
It's mostly busy work that keeps existing processes flowing and the business on the rails. Important, but not research/novel.
And cheap ai... is a wonderful fit for a lot of this. No one wants to replace an employee making 80k with a less reliable AI that costs 45k a year in tokens (SOTA). But they're absolutely willing to drop 2-3k/year on AI (~100/month - right in the open model cost range) for that employee if they can get a 10% bump in productivity or happiness.
Yeah, I totally agree. I'm sure these open models are more than good enough for non-dev work. I'm also sure they'll be good enough for dev work soon enough (and some people are saying the latest already are). My point was that given the difficulty of writing great code and dealing with large systems, SOTA just recently emerged as a viable option. I expect open models to catch up soon.
"real coding" carries a lot of the weight in that comment.
Seems like if you ask 5 different people what "real coding" means you might get 5 different answers.
Not everyone is building the next framework or compiler.
Self-hosted Qwen 3.8 @Q4 on my RTX 3090 can produce beautiful functional CRUD pages and apps all day long. And that is 90% of the "real coding" being done in corporate settings.
The quote in the article about Mazda vs. Maserati captures this. Many might want the Maserati and drool over its specs and capabilities, but balk at the cost and how often are they really going to run it up to full performance limits on their daily commute to their cubicle?
Greg Kroah-Hartman from the Linux kernel team said recently in a talk (https://www.youtube.com/watch?v=_MwMLPmMccs) running open models locally is already good enough for finding Linux kernel bugs and generating patches to fix them. He said that the false positive rate he's encountering was only one third to one quarter IIRC. He doesn't provide any details but it didn't sound like he was running some kind of huge server or something either, just a desktop workstation of some sort.
I'm having no difficulty getting deepseek v4 to blast out good code. I guess devs want to outsource all of their thinking now? Yes open models maybe can't design the whole thing soup to nuts but why is that necessary?
> Some U.S. firms remain reluctant to use Chinese A.I. models because of concerns over regulation and data privacy. AT&T researches Chinese models but is not using them, Mr. Markus said. Instead, it is working with popular alternatives made by American companies such as the Gemma A.I. model from Google and the Llama A.I. models from Meta.
This makes sense since corporations require legal certainty, and using an open model from an American company (probably) provides them some level of indemnity, and also someone to sue.
The current U.S. regime is also replacing some amount of that legal certainty with regime fealty. Picking Chinese options over American ones probably runs a risk of upsetting their leader. I've got to imagine American companies are weighing this factor in their decisions.
That's certainly one factor, yes. But even before getting to that part I think the bigger issues for Big Corp legal teams is mostly around the legal ambiguity of the models themselves. What representations are made about the training data? What jurisdiction governs the license? If somebody alleges that the model infringes their IP, what rights does AT&T have?
Counterparty risk is a lot more straight-forward to evaluate when dealing entirely within the US, with US companies.
My knowledge is a few years outdated by now, but I remember digging into this and realizing that most of the chinese open-source libs were license-washing software. E.g. PaddleOCR is licensed under Apache 2.0, a very permissive license, however its models were often-times built on/fine-tunes of less permissively licensed foundation models such as Microsoft's LayoutXLM (Creative Commons Attribution Non Commercial Share Alike 4.0). (Which in my laymans understanding is also a kind of viral license in that changes need to be shared back under a similar license?)
The link is annoying enough to find that I can imagine "Mea Culpa" being an effective enough strategy for businesses moving into the ML/AI field, changing their tune after they get caught, but matured their own software to stand on its own feet.
Exactly. See TikTok trouble as example and quite honestly, try a local open source LLM and ask it to use profanity, paint nudes - the LLM doesn’t answer the question of it is from OpenAI or Google.
The thing is that needs more attention is reverse engineered a LLM which is highly fascinating. I tried it, but it seems I am not there yet to put it mildly. It requires serious effort.
I am just speculating but can LLMs be sleepers? You write software and it seeds traces here and there under certain conditions that pose a serious security risk.
Or a kill switch?
I don’t know. I distrust Chinese LLMs but even more due to training data.
It is after all not a Western model. Different biases and the might be subtle but nevertheless substantial.
In short: no open source LLM may be usable without additional Finetuning for certain valid use cases.
The real value is versioning and autonomy as well as lot more stable answering despite model rot.
Also testing and the supporting systems are easier to maintain.
Easy prediction: LLMs will get shrunk down further and further until GenAI is just something that ships on a chip as part of your hardware. In the future it will seem quaint that we needed a network connection to talk to our LLM.
Adoption of open-source models to my mind is a similar step in that direction. In all cases, the goal is to become untethered from a mercurial vendor.
Like taalas.com (very recently acquired by AMD), or cerebras.ai (whole wafer is a chip)? As you said, I also think that is one of the main direction many companies (and academia) is moving to.
I currently have a small TTS model running in the background on my machine through which my agent(s) speak to me as they work. If that can be baked into an ASIC along with a few thousand voices in every major language then it should just be a utility chip on your mobo for anything that needs it. And yes, I too, am looking forward to it.
We're gonna start baking in models like TTS with thousands of voices available in any language as a chip on device. They just need to hit 99% accuracy and then it's a done deal.
I swear, Qwen 3.8 27B @ Q8 is smarter than Sonnet 5 most of the time. Why wouldn’t corporate America self host at this point, especially with better options like Deepseek Flash and GLM 5.3 flash that’s a middle ground between Sonnet and Opus
Agreed. And conversely, American models can also just as easily be secretly influenced for bad things, or be more tightly controlled by the government, to corporate America's own detriment.
From whence shall we expect the approach of danger? Shall some trans-Atlantic military giant step the earth and crush us at a blow? Never. All the armies of Europe and Asia...could not by force take a drink from the Ohio River or make a track on the Blue Ridge in the trial of a thousand years. No, if destruction be our lot we must ourselves be its author and finisher. As a nation of free men we will live forever or die by suicide.
― Abraham Lincoln
It's wild that people are so lost in the sauce of social media that they have no qualms handing over all their data to an authoritarian ethno state with a single ruler who appointed themselves for life with unrestricted unilateral control over every aspect of the state. And here we are calling for collapse because mom might get recommended Gain instead of Tide.
For most people around the world America is more of a threat than China. Heck for most Americans the American government is more of a threat than the Chinese government.
Still a ton of non tech companies doing a digital / tech transformation out there too lol. Maybe some so far behind they still have the real estate and rack space to get ahead on this one
Every larger company I talk to these days has an active project on moving away from OpenAI and Anthropic to open models. And they’re actively shifting, as the article says, so the threat is far from theoretical.
Unless they both dramatically slash prices then they’re in big trouble. Neither of them can afford to do that and both desperately need to convince the street that the opposite will happen if they want any hope at a successful IPO.
However the cold reality for both is that there is zero moat to a model anymore. It’s a pure commodity. Those selling compute and access to open models are gearing up to wipe the floor with Open AI and Anthropic.
I knew that there was no real moat from the very start, I mean, these things were close enough from the very start, how could it not result in a race to the bottom, especially as you can't really prevent distillation reliably?
I support and use open models as much as possible, but I'm not totally convinced that OAI or Anthropic have no moat, even as open models catch up to the frontier. Serving and inference are still hard problems when you're talking about a 2 trillion parameter model. Fine-tuning, if that remains a realistic need for businesses, is also a difficult infra problem at that scale. In the most bearish case, where there is no competitive advantage to using their models, big labs still have an advantage in this area.
Maybe there is some threshold where the price/quality math for your standard business tips in favor of smaller models and self-hosting the entire stack. I'd certainly love that.
I think this is happening right now. It's beyond ridiculous and many seem to be praising 5.6 Sol and others, even Grok 4.6 seems to be a breath of fresh air
I bet many people who use Claude at their workplaces still call it "chatgpt" because chatgpt is getting the kleenex treatment and they have no idea what Claude is
OK, but somehow there won't be companies who will sell you appropriate hardware and a turnkey system to serve inference? Or companies that will help you fine tune popular models?
It's not about money, it's about control. Companies have lots of money and want control over their key technology.
There will be companies doing this. I'm saying the labs are well positioned to be those companies, as they effectively are those companies right now.
The same dynamics that define the public cloud ecosystem are at play here. What AWS sells you is access to appropriate hardware and turnkey infra for your needs. Looking at the cloud industry over the last 20 years, I find it hard to believe that it is impossible to build a moat or a huge business around this.
There are most definitely is a moat - but it works both ways. The railguards in the models create moats keeping customers out. And the cost to build a modern agentic model is in the 10 figure range and growing. This is an expensive arms race that is going to create moats.
But most commodities are the same way. It’s super expensive to drill for oil. I need oil and I’m in no position to mine my own because of the massive capital investment. But it doesn’t stop it from being a pure commodity.
I couldn’t care less which company drilled for the oil… it’s all the same to me. Models are increasingly no different.
OpenAI and Anthropic are a gas station saying “buy our gas for 10x the price!” When the world is looking at them saying it’s just gas, we’ll take the cheaper brand. We’ve tested your gas and it’s really no better than the stuff that’s 1/10th the price.
Even commodity markets recognize different grades of product. The oil market separately prices different grades of oil, different refined products. All that really matters is that when you go to the market to buy, you can say, "I need X amount of this grade of this product" and that is what you will get. If AI models can be sold that way, you basically have a commodity market.
I don't think most motorists would care if OpenAI's gas stations just released 106-Octane "Intersteller" gas, unless their cars specifically require it.
Of course. But that market won’t produce a $800 billion company, unless AI becomes ludicrously widespread — energy is used every day by virtually every person on the planet, and of course has plenty of mass consumption & “luxury” customers too.
Yes… and as the article says folks are still leaving some work to the big labs. But the big money is to be made at scale and those use cases don’t require OpenAI or Anthropic.
The crazy setup here is that even with that fraction of the pie these companies might be worth say $100 billion optimistically, which would be amazing in normal times. Problem is it’s a train wreck for their investors and the associated debt bubble if they can’t sustain a valuation of 1-2 trillion and the present setup does not put them on a course to that trajectory.
I don't see any difference where I put gas from one place to another. But there is definitely differences between one model and another or even plans themselves .
That's true for sure in most business endeavors. The goal is to fill the area under the demand curve and there are demands for F1 race cars and for scooters. The analogy breaks down somewhat with software in general and for sure with superintelligence. A superintelligence can provide those "low-level" (ie scooter) services perhaps just as effectively because it's super intelligent and knows how to do things efficiently - for example by spawning agents of different intelligence levels. It can thus fill the area under the demand curve. This is what the big AI firms are shooting for.
seems self evident if you read the new. You can Google it yourself, but here's the results from my googling - and this is just for the hardware. Double that to add personnel and corporate infrastructure
"To build or purchase the physical hardware required to store tens of petabytes of data and train a State-of-the-Art (SOTA) frontier AI model, you are looking at a capital expenditure (CapEx) ranging from $320 million to well over $1 billion."
I know what happens in many big companies and not a single one is moving away from Anthropic/OpenAI/SpaceXAI.
> Unless they both dramatically slash prices then they’re in big trouble
False, they have already done so many times.
> Neither of them can afford to do that and both desperately need to convince the street that the opposite will happen if they want any hope at a successful IPO.
False, margins are higher and I can have a formal bet that prices will go lower.
> However the cold reality for both is that there is zero moat to a model anymore
False, LLMs are not fungible and there exists a natural moat. I like the behaviour of Fable, not the behaviour of Opus - the fact that many people speak about this is evidence.
I don’t know what your sources are, but I work on AI at a Fortune 100 and open models now make up >90% of our internal token spend. Used to be 100% closed before this summer.
Same, but not a whiff of open source/chinese models for anything other than someone fucking around with local LLM demos.
Not a tech company though, so maybe that differs and we’re just behind what’s en vogue. However we did get in on everything pretty early, building our ChatGPT RAG clone right about when azure got gpt-3.5-turbo on api
> like the behaviour of Fable, not the behaviour of Opus - the fact that many people speak about this is evidence.
I'm wondering how much of that is the harness vs the model. Overall, the 'feel' of a model seems to be largely due to the harness than the model itself.
> I can have a formal bet that prices can go lower
Is this a typo? Have you actually made a formal bet on a prediction market or something to put your money where your mouth is, or are you just saying that you could? There's a lot of things I could plausibly make bets on, but that doesn't mean that they're likely to happen.
I can have a formal bet with anyone. No prediction market will ever have a stupid bet like this because anyone who has skin in game has studied it enough to know it is a 99.99% chance
Okay, I "can have" a formal bet with my friend that the sky will be green tomorrow, and I'm sure they'd agree to bet against that. I don't see why you think that's a useful statement in the context of arguing that it's a likely outcome though.
Not sure who you’re talking to but the NYTimes reporting clearly refutes your statements that nobody is doing it with clear facts. It’s been a tidal shift in attitudes over these last new months and the messaging back to OpenAI and Anthropic has been clear. Slash your prices by an order of magnitude or you’re done for most use cases.
They kind of are fungible, up to a certain level of task. And much like most software developers don't need to exercise deep comp sci skills, most sw engineering doesn't have tasks that require the best models.
If companies are really doing this, then we're saying they have no problems spending tens of millions to get somewhat decent TPS and then having their employees complain they are timesliced and getting lots of timeouts because their org has 500 employees?
This is all spitballing, but I'd wager it's a third the type of workloads, a third hedging against your business depending on a single external provider, and a third trust.
Not everybody is coding or doing work that lends itself to burning tokens for warmth. Reuters for example seems to be more interested in using it for research, editing and formatting citations and the like. There's only so much of that work that needs doing, it doesn't always need to be real-time, and they probably don't see it scaling exponentially. They also need to be very aware and in control of their model's biases, or they risk it compromising their work output.
It's widely expected that all of the major providers will need to - and surely want to - drastically raise prices to justify the ludicrous amount of capital they're burning. Multiple companies have already talked about how their AI costs have exploded, and from what I understand that scale of enterprise is paying API rates. I would be disappointed if big business wasn't having a think about what that liability could look like. It's one thing to be reliant on a relatively "stable" vendor like Microsoft for Windows and Office, another to get AWS sticker shock, and then this is promising to be an order of magnitude worse.
Then just plain trust. What if ChatGPT starts recommending your competitors products, or the USA bars export of Anthropic's latest model (again, but for real this time), or they stop serving a model your business now depends on, and so on... That's a lot of risk to leave outside of your control.
I'm not sure why someone hasn't developed a company offering services that distributes AI across all idle or under-utilized VM's and PC's for enterprises in order to serve open sourced models. Outside of the electricity bill, there's no additional expenditure and you get the AI.
We've all seen the office spaces where there's 200 empty computers on a floor. Combined, it's something like 500 cores at ~3 Ghz each and around 3 TB of RAM. The networking is already there and software like exo already exists.
Vibe code that app and business. It’s a great idea like a kind of SETI@Home for business but I’m not sure the compute and latency will really be good enough unless you’re talking about a business with literally thousands of machines sitting half idle and always on, or always mostly on. i imagine it would take maybe 50-100 MacBook Pros running such a service to get to Claude level performance for one or two people.
Is there some mock input one could calculate to actually verify that? That many modern Mac Pros is no small potatoes when it comes to compute power. Someone must have a pre made formula anyone can use to do closer estimates?
Really not sure, the numbers aren’t too hard to run. GPUs in particular are really not doing very much most of the time. But spinning them up does max out the machine and make it hot and increase the power consumption quite a lot so it would be obvious the machine is getting stressed out all the time compared to its quiet idle background which is where it’s at for most browsing and basic stuff.
I have not seen this. The one time I did it was after a major layoff. People use laptops and take them home. I'm not saying it's not viable but the scene you depict I think is outdated.
This is a very odd take. I burned something like 600 million tokens in a single week - a very highly productive week - but I can't imagine someone doing less than a million in a day unless they were truly sleeping.
Maybe open AI and Anthropic could just license their models to run on your own hardware. So a fixed cost instead of per token pricing or subscription with limits
This is the future, because the market will demand it.
I think there will be separate and huge markets for models, hardware and compute. That will maximize competition and innovation.
Why? Because even Blind Freddy can see the huge usefulness and power of these (and future non-LLM) models and no-one in their right mind is interested in becoming OpenAI's or Anthropic's bitch. Those companies have tickets on themselves.
Given the recent behavior of tech companies and the US administration, no one trusts either anymore.
This is exactly true, I get annoyed by Claude one day and switch to something else, and the only thing that's ever keeping me tied towards Claude is the ability to search my old chats easily.
But Claude also makes it really hard to do that, so what am I even really paying for? Time to extract all my data, put it into a sqlite with FTS5 and make sure I never rely on the overly-opinionated, low-thinking PMs from these giant orgs again.
Of course, that "easy" step has lots of partial solutions like CTK (Conversation Toolkit) or MyChatArchive and I haven't found the perfect one yet, ideally it'd be something that dumped everything into Obsidian or an Obsidian-alike, but surely somebody is working on that? I'd pay $5/month for somebody to solve that problem for me, as long as I still owned the data...
Claude by default deletes old chats after a few months. I installed a custom end-of-session hook that throws chat transcripts into a database so that any model can read any other model's chat history. Super easy
Same experience, all my other friends in the industry report the same; at work we went from a huge push for ChatGPT last year to switching to Claude, back to ChatGPT when it became cheaper than Claude, and in parallel a deployment of open models being trialed with mechanisms to route to other models when needed (and based on pricing).
Over time I can imagine us becoming mostly open models on our deployments when hardware is more accessible and the need for expensive frontier models is constrained to very few use-cases that might demand their capabilities.
ChatGPT or ChatGPT Work or Codex? Are folks at your company still just using a chatbot to do work? (If so, this is quite surprising, as I find harness-based agent usage to be much much better than chatbotbot agent usage.)
No, the full suite: ChatGPT Work, Codex, chat, Deep Research, etc., same for Claude with Code, Cowork, etc. And any new release of model or tool is trialed and assessed.
Just too many products to exhaustively list, it's a big tech company, we're trialing a lot under the sun to find workflows, tools, integrations, including a lot of bespoke internal research, there's a large ML department since almost the inception of the company.
The open models are good because of distillation, which the US labs are actively working against via not revealing CoT ever and now you can see with OpenAI Astra 6 not even having a lot of CoT equivalents being emitted as tokens. Once the anti-distillation stuff is in place the open distillation models will probably start having larger and larger gaps.
If the companies survive the next few years, which they probably will because they represent too much of US economic growth to allow them to fail, this gap will keep on expanding.
Starting from zero without distillation is a lot harder, a lot more expensive and a lot more work. OSS models is what a laggard does to get adoption. China's gov't might keep on sponsoring it as a counter GPU embargo thing, but when gov't get involved, usually the other side gets involved too.
As for people asking where is the evidence for half of this, you will never have public evidence for most of this, but deduce what the partly hidden parts reveal about the whole and it is fairly obvious, especially if you look at the past behaviors of the governments and other actors.
Yes, evidence is needed but especially for the claim that distillation is what makes these open models good. Serious citation needed.
Think about it: even if they distill the shit out of frontier models, the model still gotta learn, right?
If anything, as you can see from the K2 Horizon release, aggressive (self-proclaimed) reliance on distillation does not result in a model that has remotely any frontier capability. Try asking K2 Horizon to write iambic pentameter for instance, or even give it the car wash prompt. I tried both these on the Q8 quant for the 7B model and the results were depressing.
To whoever downvoted this, it would be helpful if you actually reply with something substantive. The post I'm responding to makes sweeping characterizations, and I challenge it with a relatively good heuristic and indirect evidence, and only get downvoted?
Because it's evident you didn't engage with the last sentence I wrote properly. "Citation needed" in more words is not a sufficient response.
> As for people asking where is the evidence for half of this, you will never have public evidence for most of this, but deduce what the partly hidden parts reveal about the whole and it is fairly obvious, especially if you look at the past behaviors of the governments and other actors.
My point is simply this: Chinese labs going to great lengths to distill Claude is evidence that distillation is useful but not that that is what makes them so good. That one doesn't even need them to admit it.
Nathan Lambert has made this distinction explicitly: he thinks Chinese labs likely innovate heavily on distillation, while saying he wouldn't call it a crucial factor in their post-training capabilities, in part because the RL work still has to be done by the lab itself -- just like what I said above.
And yet their models are all behind the US models. I don't think LLM progress is strictly correlated with the number of computer science degree holders in a specific country.
[There are many examples of distillation attacks.](https://cyberpress.org/anthropic-claude-ai-distillation-atta...) Of course, you could argue this is just a form of "learning" from other intelligent systems, and this is arguably what LLMs have been doing from the beginning. So I don't begrudge the Chinese labs doing it - they all do it. But let us not pretend that it's not happening.
If we accept the premise that the top Chinese labs are simply distilling and can't compete otherwise: why don't US labs simply do the same thing? Distill their own models and slash their costs by 99% while keeping the same quality output. It should be a piece of cake if even the open labs could figure it out, after all.
One way or another they're getting the same results as proprietary labs, with a fraction of the hardware for a fraction of the cost. OpenAI can't keep raising funding rounds of $100billions to subsidize their compute costs. And if they're having trouble keeping up with Chinese labs' efficiency, maybe they should stop worrying about distilling and instead hire some of the people responsible to improve the books before IPO.
Because distillation-only is a quick performance shortcut that only lets you get to the level of the thing your distilling for the most part or a little bit worse and does not allow you to actually progress past it. It's like only being able to make VHS copies of videos, and maybe do some basic video editing without being able to actually go out with cameras and make new movies.
To actually have something competitive and improved within the next 3 months and not be perpetually behind, you need your own independent model creation process. So to extend the metaphor, a complete movie studio with cameras, actors, staff, sets, budgets, etc. It's the right strategic move to do when you are GPU constrained, which the Chinese labs are, but it won't let you get past it.
A bunch of pedantic people will come out of the wood work citing a bunch of things saying that is not the case because of some detailed mechanics of how model training works and they will get fixated on some of the words I used, but zoom out to the level of what an AI lab is able to produce and this becomes evident.
That doesn't really answer the question. What they do internally for training the next model is a separate issue. I'm talking about the models they offer publicly.
Per the article, companies are dropping OpenAI+Anthropic (partly) because of costs. If distilling is so simple and easy, why doesn't OpenAI take this "quick shortcut" and serve a self-distilled model externally, so they can charge reasonable prices and stop bleeding customers? Wouldn't more customers and less opex look good for the IPO?
Anthropic, OpenAI, GDM, and Meta spend more on training than other labs by an order of magnitude. If they felt safe reducing this spend they would. These labs fear getting outcompeted.
If that was the case the major labs would have done that.
They’re burning cash like there’s no tomorrow. They desperately need to show that they have a real business and not just a giant burning pile of cash doing academically interesting things. If they could simply sell models that are 95% as good at 1/10th the price they’d do that. They’re losing the enterprise sector because they’ve not done that.
I'm not sure why this style of cocksure Zitronesque comment is so trendy. It seems strange to make this statement, as if one has teleported to September 2026 from the end of the universe. Commenters in this vein aren't looking back to see that Anthropic and OpenAI models were the only truly usable ones for software development at the beginning of the year. Even Deepseek, which was and is revolutionary, was not useful for independent code commits longer than a few dozen lines.
It certainly is possible that Anthropic and OpenAI are doomed to bankruptcy, but without a known drop in revenue it seems exceptionally confident to make such a certain claim.
Did you? There was a single data point noted in the article. AT&T is a large company. What fraction of total US commercial inference spend does it represent?
Basically your argument relies on two claims, both of which must be true.
1. Competitors to OpenAI and Anthropic are good because of distillation.
2. OpenAI and Anthropic will come up with some methods for preventing distillation in the future.
Both of these are dubious imo. For RLVR tasks like coding in particular, you definitely don’t need continuous distillation to improve, otherwise OpenAI and Anthropic themselves would not be able to improve because there is no better model to distill from.
The schadenfreude is that, at least for OpenAI, they were originally set up to make open models.
They were set up as a public benefit company and their returns were capped at 100x. They (well, Sam) went out of their way to put themselves in this death march to the IPO. If they had just done what Mark Zuckerberg did with Muse, they're not in this position.
Do you know how bad you have to be at the tech business to make Mark Zuckerberg look like a prudent-yet-visionary leader?
If he knew how to wear a suit and answer questions from politicians, there are several regulatory penalties over the last decade that Meta doesn't have to pay.
The job of a CEO is to have a vision that is compelling for the market and an ability to execute upon it, thus the "executive" in "Chief Executive Officer". Zuckerberg doesn't have this. What he does have is an absolute majority of voting control over Meta's shares, so his actual ability to do his job doesn't matter.
It's almost as if creating incentivizing systems that don't hold people to account is a bad idea.
> However the cold reality for both is that there is zero moat to a model anymore.
The moat right now is a) the hardware, b) the electricity, c) the intelligence, and d) scalability.
On hardware, it's very expensive to purchase anything which can provide a fraction of the performance of a subscription. Traditional accounting depreciation would imply that purchasing local hardware is a terrible financial decision.
On electricity, this is a surprising cost center depending on location. A system with just one 5090 can easily pull 1kW, and to achieve usable performance for a workplace is going to require dozens of machines. This can represent an extra $10-20k in electricity in cheap places. In California or Europe this could be $30-60k per year.
As for intelligence, the frontier models from OpenAI and Anthropic are still superior, and they have at least a 3-6 month head start. Distilled models are closing the gap on some metrics, but they still can't compete. That's why they cost so much less.
The last major moat is the ability for subscriptions to scale with need. This means easily adding and removing licenses. This is far easier than purchasing extremely expensive hardware (and managing it), and selling it if/when internal demand changes. It's the same reason companies use contractors. The ramp up/down costs are very high.
The only real moat that local LLMs have right now is privacy.
Privacy is non-negotiable for corporate. Even without considering costs or country of origin, we've seen from OpenAI that claims of AI safety are worth less than the (virtual) paper they're printed on.
All it takes is one incident, and all your company's internal data will start showing up in public users' chats. You can rely on a contract to prevent this, or you can guarantee it by using a locally hosted model you fully control.
When combined with the cost savings and good enough performance mentioned in the article, this can become a huge selling point.
Claude is different from S3. AWS doesn’t need to rifle through your files to stay ahead of the competition or to mine them for business ideas because the core business is overvalued and rapidly commoditizing. AI labs, on the other hand, have an incentive to exploit every last drop of data they can lay their ethically challenged hands on. And they’ve already demonstrated that they will do this, even when it involves blatant fiduciary violations (see eg, Anthropic / Figma board member scandal).
Corporate doesn't care at all about privacy. It's why the run outlook and windows and let Microsoft scoop up all of their company secrets. It's why they hand every scrap of data they have on their customers to salesforce and surrender their data to Atlassian and use Confluence and JIRA over countless alternatives.
All companies care about is that when data breaches happen publicly they can point the finger at someone else.
Ah, Jira, the urinal of corporate cyberspace. You would literally be better off divining with some bunches of reeds than trying to figure out what the hell is going on with some stupid upside down Austrialian trash software.
Your welcome to use the wonderful BMC Helix remedyforce platform we are forced to use at work... I'll take jira any day over this pile of expensive, Gartner recommended garbage of a product.
Some industries are forced to care about privacy. HIPAA does have some holes, but IT are extremely cautious. What that ultimately means is a contract that shifts liability to the vendor. And from what I understand legal is in danger of getting to a similar place because courts are ruling things shared with remote AI are not inherently protected by privilege (meaning they are discoverable).
You’re missing the point that 98+% of the use cases for AI don’t require the latest greatest model and are far better positioned to use the fast-follow distilled cheap models.
OpenAI and Anthropic are fighting to win a race (build the biggest baddest model) that has no prize. The prize is mass adoption at scale at the best price, which is why companies are rapidly shifting to open model. They don’t need to pay 10x for a model that’s provides no practical additional benefit.
Yea, they're really hoping that the 2% will help make up an outsized share of the revenue and that brand recognition will keep them going with the plebs.
Exactly, unless they solve reliability and jaggedness somehow and keep a moat with it, I don't see sudden brilliance with compunding errors leading to substantially more adoption. Most things that need doing in corporate america are quite simple but need reliable follow-through
> As for intelligence, the frontier models from OpenAI and Anthropic are still superior, and they have at least a 3-6 month head start. Distilled models are closing the gap on some metrics, but they still can't compete. That's why they cost so much less.
I would argue that the reason they cost so little is because anyone can run open models and offer them as a service, so there's actual competition and the price is closer to cost. i.e. if the open models were just as intelligent as frontier models but cost the same to run as they do right now, the price wouldn't be higher (unless demand went up so high that marginal cost to provide more of the service went up, due to scarcity of hardware and or electricicy).
On the other hand, if what you're saying is the frontier labs have some pricing power due to their models being better, and that is the reason they are able to charge more than the companies providing open models as a service, then I would agree.
Actually, the moat is regulatory. Expect these companies to behave themselves in progressively more grotesque and sycophantic ways to get the federal government to make open/foreign models (and their output) illegal. After all, their very survival depends on it.
>On hardware, it's very expensive to purchase anything which can provide a fraction of the performance of a subscription.
One of the basic questions/concerns here though is that it's not like the AI places are getting the GPUs for 10x less. It's true they have some economies of scale, but they also have some waste, and frankly in this particular case it's not clear they get that much gain over what a lot of businesses could achieve. The biggest traditional gain for central providers is that a lot of typical computing usage is burst-y, and in turn local kit might be underutilized. But with LLMs heavy users tend to use them all the time assuming their tokens allow it (and in the case of local hardware there's nothing stopping you, quite the contrary), they can use it directly interactively or leave them to go overnight on something too.
So it's reasonable to suspect that the reason subscriptions are only a fraction of the cost is that we're in a bubble seeing these companies losing money in an attempt to gain some sort of durable advantage. Just as every previous time, there is the chance that the music stops at some point, and they need to crank up pricing or pull other schemes to actually make money. Of course, it can be a good deal in the mean time, you basically get to suck down investor money for nothing, but it's also not unreasonable to at least be consider fallbacks. Even beyond questions of control and risk etc. I know at least a few places that are now genuinely considering questions like "what happens if a datacenter we depend on gets droned" that would have never had an iota of thought devoted to them even 5 years ago.
>On electricity, this is a surprising cost center depending on location. A system with just one 5090 can easily pull 1kW, and to achieve usable performance for a workplace is going to require dozens of machines. This can represent an extra $10-20k in electricity in cheap places.
I don't think that's "surprising" at all, everyone knows about power use. And this seems like it gets heavily into what you're defining as "usable" and is also more useful to define in terms of cost-per-employee vs total. Obviously a bigger business will have a higher line number total even if the cost per employee is identical, but simultaneously can be expected to be making more revenue to pay for it.
If we're defining an average of a dedicated 5090 pulling 1 kW for every single employee (presumably some people wouldn't use it all the time, but others would then pull the compute for other work), running 24/7 (to cover people running stuff when they're away), then that'd be 8760 kWh per year. At my not particularly cheap New England location that'd be about $1900 per employee per year at the generalized residential rate (~$0.22/kWh), or $156 per month. That doesn't seem radical if it really does boost productivity. However, there is a lot of room to go lower. I'd expect a business to run backup anyway, and these days there are a lot of incentives to do that at least partially with batteries. That also opens up rate shifting as another way to pay back the cost. If we change to time of day pricing, that's 8 hours of peak pricing with the rest off-peak. 8 kWh of battery can now be had for a few thousand. And the off-peak rate is only ~$0.14/kWh, cutting the cost per year by about $700 to $1200 per employee per year. Solar power is also usually far more valuable to use yourself then sell back to the grid, and also continues to plummet in price.
None of this is to say that it makes sense for every place at all, but it's close enough to the the line that the math is at least worth exploring, or could at least lower the cost enough to be worth it given other things. It really comes down to how much extra value the company (or individual) expects to come out of it per month.
>In California or Europe this could be $30-60k per year.
> As for intelligence, the frontier models from OpenAI and Anthropic are still superior
I'll grant they are superior at least right now. But also, they are too expensive.
We ($work) are finding that it is best to build engineering discipline around AI usage (who would've thought!) and use the cheaper models like Cursor Composer.
Using Opus we can blow through an entire month budget in an afternoon, so while more powerful, it is no longer practical except for rare very complex tasks.
OpenAI and Anthropic rent their compute from AWS & friends. When we say large enterprises are moving to open weight models it means they are cutting out the middleman and renting the compute directly from AWS instead of giving OpenAI and Anthropic a margin.
> they have at least a 3-6 month head start
This is a moat of nothing. Our company still hasn’t gotten access to Fable so switching to open weight models would mean getting access to similar quality models. In some orgs they are still on 2025 models.
"Traditional accounting depreciation would imply that purchasing local hardware is a terrible financial decision."
Depreciation is designed to _encourage_ purchasing of useful local tools, by incrementally matching fractions of the cost of the tool to the revenue it generates over its useful life. The fact that a graphics card might have a book value of $0 after five years of depreciation is a feature, not a bug.
Since the invention of corporation tax it has also had the benefit of offsetting tax over the same period, instead of just one big offset in the first year.
With all due respect, it sounds like you believe that when an asset is depreciated, it means the company gets to claim back the pro rata capex amount in tax. That's not remotely how it works. It's a business cost. Revenue minus costs equals profit, and that is taxable. Depreciation allows the business to declare lower profit (and thus pay less tax), but note that *the business is making less profit.* That's bad.
I'm not challenging the concept of depreciation. It's a necessary tax function. I'm explaining the business case for local LLMs is poor.
You don't need to self host to get the benefits of an open model. There are many hosted providers cheaper than OpenAI or Anthropic who can give you a SLA, ZDR, BAA and all the other three letter acronyms your compliance department needs.
The important part is if they break the contract or raise their prices you can always move to a different provider. You get lower cost and lower risk at the same time which is extremely rare in business. That's just not possible for closed models where your only options are the official branded API or Azure/Bedrock.
Hmm yes but the equipment cost moat is artificial. This scarcity was created by the big AIs by buying up all the future production capacity. That works for a while but it won't last forever.
It's the same with the subscriptions. Local models can't compete because they're simply giving too much value for money. They're effectively subsidised by Big AI. Again something that won't last.
Your annualized energy cost estimates are off by an order of magnitude. 1kwH @ $0.1 (Texas) is $2.40/day if 100% utilized 24x7, California is roughly twice that per my understanding.
I know a few Australian devs who work at places also moving, or already have, from Anthropic… and not because of cost but because of Trump’s edicts to ban non-nationals using AI.
Think about how crazy it is for non-US companies to use American AI providers - their marketing boasts that you can treat their models as co-workers, assign tasks, invite them to slack annd video calls, etc. Taken at face value, would you hire someone remote who lived in a country that commonly does random shit like deciding whether or not remote workers aren’t allowed to go to work?
The only moat lives at the Pareto frontier. If you are on the Pareto frontier you are good and can charge money. But the frontier is moving every week so it's super competitive. If you are the quickest innovator, I still believe there is a chance for a working business model for them
I'm not seeing it. Corporate America needs someone they can sue if anything goes sideways with AI given the rate of change and legal ambiguities. It took years/decades for actual, real, open source to be widely adopted in corporations for the same reasons.
The main indemnity that providers like GitHub Copilot provide is against being sued by the copyright holders who's works were "transformed" to make the models. They don't offer that for the Chinese models and I can't see any reason other than back-room deals making it so: they're all using stolen works. But if you're talking specifically about the fitness or otherwise of the actual software these models produce, that is generally handled with the use of sacrificial employees.
What IDE/extensions do you use for open-source LLMs? I tried VSCode with ollama and lm studio, and the experience is very subpar to the built-in copilot. It's not very usable.
Seconding the TUI usage. Opencode, codex, and pi are ones that I have / my friends have had success with using locally. I mostly use pi. It feels like the most "boring tool that just does its job" out of the big options.
I’ve been using Opencode, but recently migrated to oh-my-pi and have really enjoyed it.
I’ve also had luck with copilot-cli, but find that to be more limiting and it’s only really worth using if you are paying for GitHub Copilot (or your company pays for it, as is my case).
I actually think they’re hooked on models they can fine tune.
You can’t further train the closed models. The open models can be fine tuned for your company. Big companies fine tune models on all the internal systems and documentation, not just through .md files (you’d blow up the context trying it that way) but actual fine tuning of open weights models. A low tier but open weights model actually beats frontier models when you do this for a specific task.
I think the frontier providers need to have a way to isolate instances (bedrock style?) and allow fine tuning to compete. Big companies are absolutely fine tuning models right now and getting better results than even the best frontier models for their use cases.
I had a conversation with a large group of friends and we independently came to the conclusion that Openai/Claude does not deliver more than a open source model. It takes about the same and the quality is about the same, and this does not mean it is good
I'm just glad the dweebs who were parroting "OMG, Distillation attacks!!1!" have been empirically proven wrong, and discussion around open-weight models are more rational now.
Distilled data or no, the Chinese labs are bringing a lot of solid original work in training and inference efficiency. Deepseek's work has probably made everyone's AI cheaper to run by this point.
IMO this will be a blip. There’s a lot of talk in the wake of all the Uber handwringing about token spend. Legacy enterprises want to look innovative to Wall Street without spooking them, so it’s easy to hop on the narrative and “show” that they’re innovating in a cost responsible manner.
This feels reminiscent of the big push to RAG a few years ago. And, more broadly the skunkworks projects that big companies tout in the press before they end up killing, when the operational overhead becomes too much for their liking.
Ultimately, the narrative is good for the consumer and the enterprise. It’ll mean OpenAI and anthropic will have to keep prices low. But ultimately, in the course of the next 10 years, I don’t see enterprises wanting to do this themselves. It’ll just be simpler (and eventually safer in their eyes) to send traffic to the big labs.
I don't see this being a blip, for multiple reasons.
A) the uptime matrix between Github, OpenAI, and Anthropic means that we've faced multiple entire days of not being able to ship org-wide due to our reliance on automated code review and other tooling. Every cloud service baked into our CI pipeline becomes a point of failure. We can't live without AI anymore, but it too often either directly or indirectly gets interferes with our ability to ship, and I don't see this improving any time soon.
B) Locally hosted AI has serious advantages with regard to PII/sensitive data management, and there's not much the frontier models can do to overcome this. There are so many things I want to build and let loose in a sensitive data environment but can't due to data governance around frontier models.
C) Anthropic and OpenAI cannot keep prices low forever. They're still burning insane amounts of cash and at some point, they're going to have to transition from growth mode to profit mode. They're already juicing their sales pipelines to the max with introductory pricing and other things to get people in the door. But those are all short-term online marketing plays.
I think ultimately the folks with the purses won’t care enough about a for it to be taken seriously, even if it’s an engineering bottleneck.
B definitely has scope but still smaller than I’d expect. When I was an intern at yelp, I was migrating us off internal credit card management to Braintree/stripe. No one would have imagined outsourcing that in early 2000s. There’s a long tail of stuff that you can’t sent to a 3rd party; but for most use cases it’ll suffice.
For c, true; but this is expensive for everyone, including the Chinese model companies that are trying to undercut Open Ai / anthropic. In the nth degree, i think the field will bring the cost down to the place where it’s manageable (see the existence of the cheap Chinese models). The real question is the $$ spent on pushing the research forward at scale.
The thing is, for big companies (or even small ones owned by PE, which is MOST of them), it's not just about cost. The big thing is risk.
In my experience as a tech diligence assessor for PE firms for the last 7 years, investors really, really don't like companies being beholded to single entities that they don't control. Anthropic and OpenAI have demonstrated that they are not trustworthy, or predicatable, or finanically safe, or even capable of hitting three fucking nines. Investors know they need companies to be on the AI train, but they really don't like vendor lockin to the big AI companies. Every diligence I get asked "how easily can they change models?"
I think when open models reach 80% or 90% capability (or maybe even less!) a whole lot of companies are going to say "almost as good with way less risk is a better deal".
Well the big thing that stands out in the reliability front is that AWS and Google cloud are typically stable, certainly more than 3 nines. Meanwhile openai seems to roll dice on every request to see if they’re going to return a 500 or not.
Our clients do indeed see multi-cloud as a nice-to-have, but it costs so much more to do, and the big clouds are so trusted, that it's rare that I see it.
On the other hand, nobody trusts the big AI companies not to pull shenanigans or dramatically raise prices... or even be in business in five years.
I think openrouter is well on its way to being "big cloud". OpenAI/Anthropic not so much, at least as long as they bet the farm on incremental quality advances.
At our small company we are hooked on individual subs. But yeah a larger dev shop can't really pull that off and I get how they'd be dying by the token cost.
how much of performance comes from inference time tricks like scaling, topn ect .
maybe models providers are also in position to run their models vs running os models by a generic providerc
I think using open-source AI is no longer about API cost but about company survival.
Take Anthropic for an example. Anthropic has successfully destroyed customer trust, at least for me. DHH in a recent interview mentioned that Claude refused to translate an article about immigration. Not summarize. Not editorialize. Translate! I think this reveals an unacceptable level of paternalism: Anthropic fundamentally believes that it possesses a moral authority superior to the people actually paying for the API. If such basic and mechanical translation is already too sensitive to touch, the goalposts have moved from safety into outright censorship. What prevents them from quietly deciding tomorrow that your proprietary business logic, financial data, or legal documents cross their invisible moral line?
Let alone how Anthropic treats Cursor and Figma - not that they are wrong as companies are free to compete legally, but nonetheless it shows that companies can't outsource their intelligence to a potential competitor.
I get what you're saying and it's concerning how much power these big labs have amassed and how little transparency there is in what they do with it...
But I doubt this a major factor in the trend. I just don't think it's something most corporate users run into. My understanding is these guardrails are negotiable for enterprise customers anyway.
And, not for nothing, but if I owned a human-powered translation company I would've refused to translate it too.
Ah, yes, I'm sure the article that moral paragon DHH wished to translate was not at all harmful, and that this was a good-faith effort on his part /s
While I agree that Claude can be overly paternalistic at times, how should it respond to a request to translate, say, bomb-making instructions? It's reasonable to me that it might refuse this.
It was an incredibly racist post claiming “gypsies” are like invading wolves and that something more drastic must be done to get rid of them before they kill all the “sheep” in Copenhagen.
I like the Claude constitution overall - I hope it becomes something representatives vote on and amend, to avoid the centralized corporate censorship you describe. In the meantime, I am fine with it abstaining from doing DHH’s bidding, especially because there are so many AI alternatives.
I have recently come to the conclusion that thinking for 2 seconds and using a cheap model with a slightly more detailed prompt works just as well as zero-shotting an idea with a fancy model. I work in science, and instead of asking the model “write a topic extraction algorithm”, I just say “hey look at this matrix factorization script I found in a repo, now make it use plotly and duckdb”. Have others come to the same conclusion here?
It makes me skeptical that the flagship companies are sustainable. Every company is going to maximize “fuel efficiency” to save time and money.
> AT&T turned to artificial intelligence models from Anthropic and OpenAI in recent years to help with customer service, call transcription and coding. [...]
> By May, open models accounted for 20 percent of AT&T’s A.I. use. That has since risen to 40 percent and may jump to 60 percent in the coming months, Mr. Markus said in an interview.
This is missing a crucial detail. We know they "help with customer service, call transcription and coding", but which of those have been upgrade to open models?
Call transcription is trivial to do with open models. I can run Whisper or Parakeet on a low-spec laptop.
"Customer service" could mean a lot of things, but it sounds feasible for open models too.
"Coding" - they might go to open models for that, but I expect the costs involved in paying for closed models for software developers within AT&T are a fraction of the costs involved in transcribing all of their calls or handling aspects of custom service for millions of customers.
I'd assume the author is just getting confused because of ollama and llama.cpp and all the other ecosystem "llama" that are still in use. Llama really did kick off the open models thing
Some of the most insidious parts of AI infrastructure includes the embedding model. Corporations have already spent an outstanding amount of time and money creating embedding vectors that are closed source and not reproducible. This means that all their data is locked into whatever embedding model they chose initially.
I highly recommend utilizing an open sourced embedding model instead of paying for a closed source one. It's vastly more reasonable to run an open sourced embedding model as a first step. They're much, much smaller and, due to the overhead of network latency, and running it locally has almost the same speed as through an API even on slow computers.
I would even go so far as to say that closed source embedding models have a high risk of data hostage. If a team doesn't have access to the embedding model, the embeddings become useless. A corporation like OpenAI could, say, hike the prices to that model by 1000x and everyone would have to pay up or forfeit any utility of the data.
I envision a future where open source embedding models are shipped with relevant technologies and implemented by currently under-utilized chips like NPU's. A startup developing cheap microprocessors that can run them is an idea I would pay cash for. Or perhaps they will be bundled with security tokens.
While it might be impractical for all corporate teams to run language models, it is very realistic for everyone to operate an open sourced embedding model, at least in their private cloud. Better yet, utilize transfer learning on an open sourced one to train your own, that way the embedding vector is more secure against competitors and trade secrets.
247 comments
[ 0.20 ms ] story [ 12.9 ms ] threadI'm sure this will change (and I can't wait for it!) but as of today, open models might be fine for summarizing and writing docs, but you need SOTA to work on code if you want to be competitive.
I also work in software, and while I vaguely disagree that open models can't be used (they absolutely fit into productive niches here, and holy hell are the last generation [ex laguna s1, kimi k3, glm 5.3, etc] actually decent) - I will agree that SOTA are a better fit for software development, especially when used in conjunction with an already very expensive employee who's driving them.
But for "Corporate America"... no. You absolutely don't need SOTA. They're doing things like transcription, summarization, customer interaction, minor technical tasks like form creation in existing tools, report generation (ex - powerpoint, pdf, docs, etc) and other general "white collar tasks". Think about roles in business that are in the 60-85k compensation range.
It's mostly busy work that keeps existing processes flowing and the business on the rails. Important, but not research/novel.
And cheap ai... is a wonderful fit for a lot of this. No one wants to replace an employee making 80k with a less reliable AI that costs 45k a year in tokens (SOTA). But they're absolutely willing to drop 2-3k/year on AI (~100/month - right in the open model cost range) for that employee if they can get a 10% bump in productivity or happiness.
Seems like if you ask 5 different people what "real coding" means you might get 5 different answers.
Not everyone is building the next framework or compiler.
Self-hosted Qwen 3.8 @Q4 on my RTX 3090 can produce beautiful functional CRUD pages and apps all day long. And that is 90% of the "real coding" being done in corporate settings.
The quote in the article about Mazda vs. Maserati captures this. Many might want the Maserati and drool over its specs and capabilities, but balk at the cost and how often are they really going to run it up to full performance limits on their daily commute to their cubicle?
Have you been brought into line? Open source AI also violates copyrights.
This makes sense since corporations require legal certainty, and using an open model from an American company (probably) provides them some level of indemnity, and also someone to sue.
Counterparty risk is a lot more straight-forward to evaluate when dealing entirely within the US, with US companies.
The link is annoying enough to find that I can imagine "Mea Culpa" being an effective enough strategy for businesses moving into the ML/AI field, changing their tune after they get caught, but matured their own software to stand on its own feet.
Why play ball with a hostile government when you can host your own frontier models?
The thing is that needs more attention is reverse engineered a LLM which is highly fascinating. I tried it, but it seems I am not there yet to put it mildly. It requires serious effort.
I am just speculating but can LLMs be sleepers? You write software and it seeds traces here and there under certain conditions that pose a serious security risk.
Or a kill switch?
I don’t know. I distrust Chinese LLMs but even more due to training data.
It is after all not a Western model. Different biases and the might be subtle but nevertheless substantial.
In short: no open source LLM may be usable without additional Finetuning for certain valid use cases.
The real value is versioning and autonomy as well as lot more stable answering despite model rot.
Also testing and the supporting systems are easier to maintain.
It is mainly an infrastructure challenge.
Adoption of open-source models to my mind is a similar step in that direction. In all cases, the goal is to become untethered from a mercurial vendor.
https://chatjimmy.ai/ blew my mind at how fast etched model weights can be.
For on-device LLMs, there's a point of diminishing returns, meaning you don't need to have the latest frontier model for most operations.
The most insidious advertising in the world is about to be surfaced as people use LLMs to look for product recommendations.
but if there's roughly Sonnet 4.6 level capable open small models, then I'd be impressed
https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates
Because they've been trained to think "cloud-first" for a decade?
Unless they both dramatically slash prices then they’re in big trouble. Neither of them can afford to do that and both desperately need to convince the street that the opposite will happen if they want any hope at a successful IPO.
However the cold reality for both is that there is zero moat to a model anymore. It’s a pure commodity. Those selling compute and access to open models are gearing up to wipe the floor with Open AI and Anthropic.
Maybe there is some threshold where the price/quality math for your standard business tips in favor of smaller models and self-hosting the entire stack. I'd certainly love that.
But the "open" models are right there too, and I'll be using them more in the next weeks as I get better chat interfaces.
The problem is serving is a skill readily mastered by the hyperscalers. That's their MO.
All they need is weights to serve. And the open models provide that.
OpenAI is relatively well placed in that they have inference chips they've designed and they own compute.
It's not about money, it's about control. Companies have lots of money and want control over their key technology.
The same dynamics that define the public cloud ecosystem are at play here. What AWS sells you is access to appropriate hardware and turnkey infra for your needs. Looking at the cloud industry over the last 20 years, I find it hard to believe that it is impossible to build a moat or a huge business around this.
I couldn’t care less which company drilled for the oil… it’s all the same to me. Models are increasingly no different.
OpenAI and Anthropic are a gas station saying “buy our gas for 10x the price!” When the world is looking at them saying it’s just gas, we’ll take the cheaper brand. We’ve tested your gas and it’s really no better than the stuff that’s 1/10th the price.
Thats why their present business plan is screwed.
I agree 90% of the world can work with 87 gas, but there's always niche/luxury market where 93 can make small difference.
(edit: typo)
The crazy setup here is that even with that fraction of the pie these companies might be worth say $100 billion optimistically, which would be amazing in normal times. Problem is it’s a train wreck for their investors and the associated debt bubble if they can’t sustain a valuation of 1-2 trillion and the present setup does not put them on a course to that trajectory.
"To build or purchase the physical hardware required to store tens of petabytes of data and train a State-of-the-Art (SOTA) frontier AI model, you are looking at a capital expenditure (CapEx) ranging from $320 million to well over $1 billion."
> Unless they both dramatically slash prices then they’re in big trouble
False, they have already done so many times.
> Neither of them can afford to do that and both desperately need to convince the street that the opposite will happen if they want any hope at a successful IPO.
False, margins are higher and I can have a formal bet that prices will go lower.
> However the cold reality for both is that there is zero moat to a model anymore
False, LLMs are not fungible and there exists a natural moat. I like the behaviour of Fable, not the behaviour of Opus - the fact that many people speak about this is evidence.
You lost all credibility when you mentioned SpaceXAI.
Nobody uses GrokBot except lower-middle class influencers.
Not a tech company though, so maybe that differs and we’re just behind what’s en vogue. However we did get in on everything pretty early, building our ChatGPT RAG clone right about when azure got gpt-3.5-turbo on api
I'm wondering how much of that is the harness vs the model. Overall, the 'feel' of a model seems to be largely due to the harness than the model itself.
Is this a typo? Have you actually made a formal bet on a prediction market or something to put your money where your mouth is, or are you just saying that you could? There's a lot of things I could plausibly make bets on, but that doesn't mean that they're likely to happen.
Not everybody is coding or doing work that lends itself to burning tokens for warmth. Reuters for example seems to be more interested in using it for research, editing and formatting citations and the like. There's only so much of that work that needs doing, it doesn't always need to be real-time, and they probably don't see it scaling exponentially. They also need to be very aware and in control of their model's biases, or they risk it compromising their work output.
It's widely expected that all of the major providers will need to - and surely want to - drastically raise prices to justify the ludicrous amount of capital they're burning. Multiple companies have already talked about how their AI costs have exploded, and from what I understand that scale of enterprise is paying API rates. I would be disappointed if big business wasn't having a think about what that liability could look like. It's one thing to be reliant on a relatively "stable" vendor like Microsoft for Windows and Office, another to get AWS sticker shock, and then this is promising to be an order of magnitude worse.
Then just plain trust. What if ChatGPT starts recommending your competitors products, or the USA bars export of Anthropic's latest model (again, but for real this time), or they stop serving a model your business now depends on, and so on... That's a lot of risk to leave outside of your control.
We've all seen the office spaces where there's 200 empty computers on a floor. Combined, it's something like 500 cores at ~3 Ghz each and around 3 TB of RAM. The networking is already there and software like exo already exists.
Keep in mind that a vanishingly small number of workers are SWE's churning millions of tokens daily.
https://cloud.google.com/blog/topics/hybrid-cloud/gemini-is-...
I think there will be separate and huge markets for models, hardware and compute. That will maximize competition and innovation.
Why? Because even Blind Freddy can see the huge usefulness and power of these (and future non-LLM) models and no-one in their right mind is interested in becoming OpenAI's or Anthropic's bitch. Those companies have tickets on themselves.
Given the recent behavior of tech companies and the US administration, no one trusts either anymore.
But Claude also makes it really hard to do that, so what am I even really paying for? Time to extract all my data, put it into a sqlite with FTS5 and make sure I never rely on the overly-opinionated, low-thinking PMs from these giant orgs again.
Of course, that "easy" step has lots of partial solutions like CTK (Conversation Toolkit) or MyChatArchive and I haven't found the perfect one yet, ideally it'd be something that dumped everything into Obsidian or an Obsidian-alike, but surely somebody is working on that? I'd pay $5/month for somebody to solve that problem for me, as long as I still owned the data...
I’d try:
1 exporting my data (I imagine it’s common outside of GDPR?)
2 asking Claude to convert it to an easily digestible format :)
Don't rely on chat history. Have it write and maintain summary files that you can import into different sessions, at least for anything important.
Over time I can imagine us becoming mostly open models on our deployments when hardware is more accessible and the need for expensive frontier models is constrained to very few use-cases that might demand their capabilities.
Just too many products to exhaustively list, it's a big tech company, we're trialing a lot under the sun to find workflows, tools, integrations, including a lot of bespoke internal research, there's a large ML department since almost the inception of the company.
If the companies survive the next few years, which they probably will because they represent too much of US economic growth to allow them to fail, this gap will keep on expanding.
Starting from zero without distillation is a lot harder, a lot more expensive and a lot more work. OSS models is what a laggard does to get adoption. China's gov't might keep on sponsoring it as a counter GPU embargo thing, but when gov't get involved, usually the other side gets involved too.
As for people asking where is the evidence for half of this, you will never have public evidence for most of this, but deduce what the partly hidden parts reveal about the whole and it is fairly obvious, especially if you look at the past behaviors of the governments and other actors.
Think about it: even if they distill the shit out of frontier models, the model still gotta learn, right?
If anything, as you can see from the K2 Horizon release, aggressive (self-proclaimed) reliance on distillation does not result in a model that has remotely any frontier capability. Try asking K2 Horizon to write iambic pentameter for instance, or even give it the car wash prompt. I tried both these on the Q8 quant for the 7B model and the results were depressing.
> As for people asking where is the evidence for half of this, you will never have public evidence for most of this, but deduce what the partly hidden parts reveal about the whole and it is fairly obvious, especially if you look at the past behaviors of the governments and other actors.
Nathan Lambert has made this distinction explicitly: he thinks Chinese labs likely innovate heavily on distillation, while saying he wouldn't call it a crucial factor in their post-training capabilities, in part because the RL work still has to be done by the lab itself -- just like what I said above.
https://www.interconnects.ai/p/how-much-does-distillation-re...
Chinese AI labs do well because they have the best AI people coming out of a huge talent pool.
[There are many examples of distillation attacks.](https://cyberpress.org/anthropic-claude-ai-distillation-atta...) Of course, you could argue this is just a form of "learning" from other intelligent systems, and this is arguably what LLMs have been doing from the beginning. So I don't begrudge the Chinese labs doing it - they all do it. But let us not pretend that it's not happening.
One way or another they're getting the same results as proprietary labs, with a fraction of the hardware for a fraction of the cost. OpenAI can't keep raising funding rounds of $100billions to subsidize their compute costs. And if they're having trouble keeping up with Chinese labs' efficiency, maybe they should stop worrying about distilling and instead hire some of the people responsible to improve the books before IPO.
To actually have something competitive and improved within the next 3 months and not be perpetually behind, you need your own independent model creation process. So to extend the metaphor, a complete movie studio with cameras, actors, staff, sets, budgets, etc. It's the right strategic move to do when you are GPU constrained, which the Chinese labs are, but it won't let you get past it.
A bunch of pedantic people will come out of the wood work citing a bunch of things saying that is not the case because of some detailed mechanics of how model training works and they will get fixated on some of the words I used, but zoom out to the level of what an AI lab is able to produce and this becomes evident.
Per the article, companies are dropping OpenAI+Anthropic (partly) because of costs. If distilling is so simple and easy, why doesn't OpenAI take this "quick shortcut" and serve a self-distilled model externally, so they can charge reasonable prices and stop bleeding customers? Wouldn't more customers and less opex look good for the IPO?
They’re burning cash like there’s no tomorrow. They desperately need to show that they have a real business and not just a giant burning pile of cash doing academically interesting things. If they could simply sell models that are 95% as good at 1/10th the price they’d do that. They’re losing the enterprise sector because they’ve not done that.
It certainly is possible that Anthropic and OpenAI are doomed to bankruptcy, but without a known drop in revenue it seems exceptionally confident to make such a certain claim.
1. Competitors to OpenAI and Anthropic are good because of distillation.
2. OpenAI and Anthropic will come up with some methods for preventing distillation in the future.
Both of these are dubious imo. For RLVR tasks like coding in particular, you definitely don’t need continuous distillation to improve, otherwise OpenAI and Anthropic themselves would not be able to improve because there is no better model to distill from.
They were set up as a public benefit company and their returns were capped at 100x. They (well, Sam) went out of their way to put themselves in this death march to the IPO. If they had just done what Mark Zuckerberg did with Muse, they're not in this position.
Do you know how bad you have to be at the tech business to make Mark Zuckerberg look like a prudent-yet-visionary leader?
You're giving him too much credit if you think he figured anything on his own - this is the guy who thought 'metaverse' was a good idea.
The job of a CEO is to have a vision that is compelling for the market and an ability to execute upon it, thus the "executive" in "Chief Executive Officer". Zuckerberg doesn't have this. What he does have is an absolute majority of voting control over Meta's shares, so his actual ability to do his job doesn't matter.
It's almost as if creating incentivizing systems that don't hold people to account is a bad idea.
The moat right now is a) the hardware, b) the electricity, c) the intelligence, and d) scalability.
On hardware, it's very expensive to purchase anything which can provide a fraction of the performance of a subscription. Traditional accounting depreciation would imply that purchasing local hardware is a terrible financial decision.
On electricity, this is a surprising cost center depending on location. A system with just one 5090 can easily pull 1kW, and to achieve usable performance for a workplace is going to require dozens of machines. This can represent an extra $10-20k in electricity in cheap places. In California or Europe this could be $30-60k per year.
As for intelligence, the frontier models from OpenAI and Anthropic are still superior, and they have at least a 3-6 month head start. Distilled models are closing the gap on some metrics, but they still can't compete. That's why they cost so much less.
The last major moat is the ability for subscriptions to scale with need. This means easily adding and removing licenses. This is far easier than purchasing extremely expensive hardware (and managing it), and selling it if/when internal demand changes. It's the same reason companies use contractors. The ramp up/down costs are very high.
The only real moat that local LLMs have right now is privacy.
All it takes is one incident, and all your company's internal data will start showing up in public users' chats. You can rely on a contract to prevent this, or you can guarantee it by using a locally hosted model you fully control.
When combined with the cost savings and good enough performance mentioned in the article, this can become a huge selling point.
Corporate doesn't care at all about privacy. It's why the run outlook and windows and let Microsoft scoop up all of their company secrets. It's why they hand every scrap of data they have on their customers to salesforce and surrender their data to Atlassian and use Confluence and JIRA over countless alternatives.
All companies care about is that when data breaches happen publicly they can point the finger at someone else.
OpenAI and Anthropic are fighting to win a race (build the biggest baddest model) that has no prize. The prize is mass adoption at scale at the best price, which is why companies are rapidly shifting to open model. They don’t need to pay 10x for a model that’s provides no practical additional benefit.
> As for intelligence, the frontier models from OpenAI and Anthropic are still superior, and they have at least a 3-6 month head start. Distilled models are closing the gap on some metrics, but they still can't compete. That's why they cost so much less.
I would argue that the reason they cost so little is because anyone can run open models and offer them as a service, so there's actual competition and the price is closer to cost. i.e. if the open models were just as intelligent as frontier models but cost the same to run as they do right now, the price wouldn't be higher (unless demand went up so high that marginal cost to provide more of the service went up, due to scarcity of hardware and or electricicy).
On the other hand, if what you're saying is the frontier labs have some pricing power due to their models being better, and that is the reason they are able to charge more than the companies providing open models as a service, then I would agree.
One of the basic questions/concerns here though is that it's not like the AI places are getting the GPUs for 10x less. It's true they have some economies of scale, but they also have some waste, and frankly in this particular case it's not clear they get that much gain over what a lot of businesses could achieve. The biggest traditional gain for central providers is that a lot of typical computing usage is burst-y, and in turn local kit might be underutilized. But with LLMs heavy users tend to use them all the time assuming their tokens allow it (and in the case of local hardware there's nothing stopping you, quite the contrary), they can use it directly interactively or leave them to go overnight on something too.
So it's reasonable to suspect that the reason subscriptions are only a fraction of the cost is that we're in a bubble seeing these companies losing money in an attempt to gain some sort of durable advantage. Just as every previous time, there is the chance that the music stops at some point, and they need to crank up pricing or pull other schemes to actually make money. Of course, it can be a good deal in the mean time, you basically get to suck down investor money for nothing, but it's also not unreasonable to at least be consider fallbacks. Even beyond questions of control and risk etc. I know at least a few places that are now genuinely considering questions like "what happens if a datacenter we depend on gets droned" that would have never had an iota of thought devoted to them even 5 years ago.
>On electricity, this is a surprising cost center depending on location. A system with just one 5090 can easily pull 1kW, and to achieve usable performance for a workplace is going to require dozens of machines. This can represent an extra $10-20k in electricity in cheap places.
I don't think that's "surprising" at all, everyone knows about power use. And this seems like it gets heavily into what you're defining as "usable" and is also more useful to define in terms of cost-per-employee vs total. Obviously a bigger business will have a higher line number total even if the cost per employee is identical, but simultaneously can be expected to be making more revenue to pay for it.
If we're defining an average of a dedicated 5090 pulling 1 kW for every single employee (presumably some people wouldn't use it all the time, but others would then pull the compute for other work), running 24/7 (to cover people running stuff when they're away), then that'd be 8760 kWh per year. At my not particularly cheap New England location that'd be about $1900 per employee per year at the generalized residential rate (~$0.22/kWh), or $156 per month. That doesn't seem radical if it really does boost productivity. However, there is a lot of room to go lower. I'd expect a business to run backup anyway, and these days there are a lot of incentives to do that at least partially with batteries. That also opens up rate shifting as another way to pay back the cost. If we change to time of day pricing, that's 8 hours of peak pricing with the rest off-peak. 8 kWh of battery can now be had for a few thousand. And the off-peak rate is only ~$0.14/kWh, cutting the cost per year by about $700 to $1200 per employee per year. Solar power is also usually far more valuable to use yourself then sell back to the grid, and also continues to plummet in price.
None of this is to say that it makes sense for every place at all, but it's close enough to the the line that the math is at least worth exploring, or could at least lower the cost enough to be worth it given other things. It really comes down to how much extra value the company (or individual) expects to come out of it per month.
>In California or Europe this could be $30-60k per year.
I'll grant they are superior at least right now. But also, they are too expensive.
We ($work) are finding that it is best to build engineering discipline around AI usage (who would've thought!) and use the cheaper models like Cursor Composer.
Using Opus we can blow through an entire month budget in an afternoon, so while more powerful, it is no longer practical except for rare very complex tasks.
OpenAI and Anthropic rent their compute from AWS & friends. When we say large enterprises are moving to open weight models it means they are cutting out the middleman and renting the compute directly from AWS instead of giving OpenAI and Anthropic a margin.
> they have at least a 3-6 month head start
This is a moat of nothing. Our company still hasn’t gotten access to Fable so switching to open weight models would mean getting access to similar quality models. In some orgs they are still on 2025 models.
Depreciation is designed to _encourage_ purchasing of useful local tools, by incrementally matching fractions of the cost of the tool to the revenue it generates over its useful life. The fact that a graphics card might have a book value of $0 after five years of depreciation is a feature, not a bug.
Since the invention of corporation tax it has also had the benefit of offsetting tax over the same period, instead of just one big offset in the first year.
I'm not challenging the concept of depreciation. It's a necessary tax function. I'm explaining the business case for local LLMs is poor.
The important part is if they break the contract or raise their prices you can always move to a different provider. You get lower cost and lower risk at the same time which is extremely rare in business. That's just not possible for closed models where your only options are the official branded API or Azure/Bedrock.
It's the same with the subscriptions. Local models can't compete because they're simply giving too much value for money. They're effectively subsidised by Big AI. Again something that won't last.
Think about how crazy it is for non-US companies to use American AI providers - their marketing boasts that you can treat their models as co-workers, assign tasks, invite them to slack annd video calls, etc. Taken at face value, would you hire someone remote who lived in a country that commonly does random shit like deciding whether or not remote workers aren’t allowed to go to work?
No one I know uses the built in VSCode extensions anymore. It's all TUIs now. You can use Opencode as a TUI now for local.
I’ve also had luck with copilot-cli, but find that to be more limiting and it’s only really worth using if you are paying for GitHub Copilot (or your company pays for it, as is my case).
Which makes the Muse 1.3 launch this week particularly interesting, although to get the low cost version you do need to agree to share data with Meta.
You can’t further train the closed models. The open models can be fine tuned for your company. Big companies fine tune models on all the internal systems and documentation, not just through .md files (you’d blow up the context trying it that way) but actual fine tuning of open weights models. A low tier but open weights model actually beats frontier models when you do this for a specific task.
I think the frontier providers need to have a way to isolate instances (bedrock style?) and allow fine tuning to compete. Big companies are absolutely fine tuning models right now and getting better results than even the best frontier models for their use cases.
This feels reminiscent of the big push to RAG a few years ago. And, more broadly the skunkworks projects that big companies tout in the press before they end up killing, when the operational overhead becomes too much for their liking.
Ultimately, the narrative is good for the consumer and the enterprise. It’ll mean OpenAI and anthropic will have to keep prices low. But ultimately, in the course of the next 10 years, I don’t see enterprises wanting to do this themselves. It’ll just be simpler (and eventually safer in their eyes) to send traffic to the big labs.
A) the uptime matrix between Github, OpenAI, and Anthropic means that we've faced multiple entire days of not being able to ship org-wide due to our reliance on automated code review and other tooling. Every cloud service baked into our CI pipeline becomes a point of failure. We can't live without AI anymore, but it too often either directly or indirectly gets interferes with our ability to ship, and I don't see this improving any time soon.
B) Locally hosted AI has serious advantages with regard to PII/sensitive data management, and there's not much the frontier models can do to overcome this. There are so many things I want to build and let loose in a sensitive data environment but can't due to data governance around frontier models.
C) Anthropic and OpenAI cannot keep prices low forever. They're still burning insane amounts of cash and at some point, they're going to have to transition from growth mode to profit mode. They're already juicing their sales pipelines to the max with introductory pricing and other things to get people in the door. But those are all short-term online marketing plays.
I think ultimately the folks with the purses won’t care enough about a for it to be taken seriously, even if it’s an engineering bottleneck.
B definitely has scope but still smaller than I’d expect. When I was an intern at yelp, I was migrating us off internal credit card management to Braintree/stripe. No one would have imagined outsourcing that in early 2000s. There’s a long tail of stuff that you can’t sent to a 3rd party; but for most use cases it’ll suffice.
For c, true; but this is expensive for everyone, including the Chinese model companies that are trying to undercut Open Ai / anthropic. In the nth degree, i think the field will bring the cost down to the place where it’s manageable (see the existence of the cheap Chinese models). The real question is the $$ spent on pushing the research forward at scale.
In my experience as a tech diligence assessor for PE firms for the last 7 years, investors really, really don't like companies being beholded to single entities that they don't control. Anthropic and OpenAI have demonstrated that they are not trustworthy, or predicatable, or finanically safe, or even capable of hitting three fucking nines. Investors know they need companies to be on the AI train, but they really don't like vendor lockin to the big AI companies. Every diligence I get asked "how easily can they change models?"
I think when open models reach 80% or 90% capability (or maybe even less!) a whole lot of companies are going to say "almost as good with way less risk is a better deal".
Has there been a sea change in how investors view these things in general, or is it only AI?
But in terms of reliability - uptime, product, legal - they are in a different league.
There were exactly 0 instances waking up to a product decision at AWS completely breaking your product or workflows.
On the other hand, nobody trusts the big AI companies not to pull shenanigans or dramatically raise prices... or even be in business in five years.
Corporal, put him away.
Take Anthropic for an example. Anthropic has successfully destroyed customer trust, at least for me. DHH in a recent interview mentioned that Claude refused to translate an article about immigration. Not summarize. Not editorialize. Translate! I think this reveals an unacceptable level of paternalism: Anthropic fundamentally believes that it possesses a moral authority superior to the people actually paying for the API. If such basic and mechanical translation is already too sensitive to touch, the goalposts have moved from safety into outright censorship. What prevents them from quietly deciding tomorrow that your proprietary business logic, financial data, or legal documents cross their invisible moral line?
Let alone how Anthropic treats Cursor and Figma - not that they are wrong as companies are free to compete legally, but nonetheless it shows that companies can't outsource their intelligence to a potential competitor.
But I doubt this a major factor in the trend. I just don't think it's something most corporate users run into. My understanding is these guardrails are negotiable for enterprise customers anyway.
And, not for nothing, but if I owned a human-powered translation company I would've refused to translate it too.
While I agree that Claude can be overly paternalistic at times, how should it respond to a request to translate, say, bomb-making instructions? It's reasonable to me that it might refuse this.
Also, you see zero distinction between hearing opinions on political topics you might find objectionable, and building a bomb to kill people?
It was an incredibly racist post claiming “gypsies” are like invading wolves and that something more drastic must be done to get rid of them before they kill all the “sheep” in Copenhagen.
It makes me skeptical that the flagship companies are sustainable. Every company is going to maximize “fuel efficiency” to save time and money.
Who's going to be the new Bill Gates, with a vision for "a GPU cluster in every home?"
> By May, open models accounted for 20 percent of AT&T’s A.I. use. That has since risen to 40 percent and may jump to 60 percent in the coming months, Mr. Markus said in an interview.
This is missing a crucial detail. We know they "help with customer service, call transcription and coding", but which of those have been upgrade to open models?
Call transcription is trivial to do with open models. I can run Whisper or Parakeet on a low-spec laptop.
"Customer service" could mean a lot of things, but it sounds feasible for open models too.
"Coding" - they might go to open models for that, but I expect the costs involved in paying for closed models for software developers within AT&T are a fraction of the costs involved in transcribing all of their calls or handling aspects of custom service for millions of customers.
I'd assume the author is just getting confused because of ollama and llama.cpp and all the other ecosystem "llama" that are still in use. Llama really did kick off the open models thing
I highly recommend utilizing an open sourced embedding model instead of paying for a closed source one. It's vastly more reasonable to run an open sourced embedding model as a first step. They're much, much smaller and, due to the overhead of network latency, and running it locally has almost the same speed as through an API even on slow computers.
I would even go so far as to say that closed source embedding models have a high risk of data hostage. If a team doesn't have access to the embedding model, the embeddings become useless. A corporation like OpenAI could, say, hike the prices to that model by 1000x and everyone would have to pay up or forfeit any utility of the data.
I envision a future where open source embedding models are shipped with relevant technologies and implemented by currently under-utilized chips like NPU's. A startup developing cheap microprocessors that can run them is an idea I would pay cash for. Or perhaps they will be bundled with security tokens.
While it might be impractical for all corporate teams to run language models, it is very realistic for everyone to operate an open sourced embedding model, at least in their private cloud. Better yet, utilize transfer learning on an open sourced one to train your own, that way the embedding vector is more secure against competitors and trade secrets.