Yes but unironically? IBM should be what NVIDIA would be competing with their NVL72 and DGX products. It doesn't make sense that IBM is just watching NVIDIA take its lunch.
I didn't think that IBM had gear that mortals would want to run. If you're not running a mainframe, what do they have to sell you? They still sell Power servers and it looks like they have an AI card that can be used on both. But, if we're talking about the IBM CEO talking about on-premises, what are they really talking about hardware-wise? It's not something they would sell, is it?
This is more of a statement for their consulting customers, right?
I can picture that they’d have the software side sorted out with RedHat. But I don’t know what hardware they’d push to support local on-premises models. It looks like when you buy Fusion HCI, you can choose from Dell or Lenovo as the underlying hardware supplier.
Can we please stop giving weight to CEOs talking their book? Like of course IBM wants you to think this, it would be great for them if it’s true. And maybe it is true! But IBM’s CEO saying it’s true tells me just as much as if a frontier lab said it’s not: nothing.
While he thinks most businesses can build useful AI infrastructure in just two or three racks, he feels many will struggle to get it running.
“The question becomes: how do you do that?” he said. “The question that the enterprise struggles with is: ‘Where are my kernel engineers? Where are the people who worry about the underlying network? Who can make sure that the model is clustered on GPUs that are close to each other in network proximity? Who can make sure that you’re doing the right caching of results so you get an incredible response?’”
Has he heard of “agentic AI”? You simply fire up Claude or ChatGPT and ask it how to do all of the above.
That's based on the idea that humans can learn and adapt and solve problems, and that's not what companies want to think about when they're seeking a profitable transaction such as hiring.
Do not do that if you can't prompt it correctly. It is a potential disaster waiting to happen. LLMs are likely to pull things off the shelf without real defense in depth and that's a problem. Maybe slightly less if it's only in an internal network, but still.
It can be very dangerous to let people loose with such advice, FWIW. They don't necessarily know what "internal usage" actually means, or how important it is to keep it that way as long as you don't have a real expert to at least help direct the LLMs.
It's not a one time thing though - onprem inference requires continuous monitoring and tuning to get the most out of your GPUs. Looking at 25-50% performance benefits. Today, it does require dedicated Inference Engineers...
The optimal inference configuration varies according to the model, the specific traffic (which changes in time), and the specific hardware it's running to. The "knowledge" for it doesn't exist yet.
Usually in similar scenarios an agent will try to experiment with various configurations and get to a good result. However, you can't have an agent test out different inference serving configurations/architectures, since every experiment might involves 10-60 minutes of spinning up a model (on very expensive GPUs)...
It's absolutely true. I have three different setups here, a rig with 8 gpus that I use in either one of two modes (all 8, very heavy model or just one much lighter) and two DGX sparks. I haven't used a paid model in months. Sure, they use a few more tokens and the results may be slightly less good (but this doesn't really matter because I'm still in the loop so it is easy to correct). But the fact that I'm not sharing my work with the paid providers is much more to my liking than having big brother looking over my shoulder at every step and occasionally giving me the 'I can't let you do that, Jacques' treatment.
> Sure, they use a few more tokens and the results may be slightly less good (but this doesn't really matter because I'm still in the loop so it is easy to correct)
I can't wait to use models that are 'slightly less good' when I am at work instead of something like Fable that lets me focus on the problem at hand as much as possible rather than also spending additional effort correcting for the tool I am using...
Gaming used to be done in arcades for 25c a play*.
Over time NeoGeo realized they could sell you the hardware to play at home (“on-prem”) at arcade quality albeit at a price out of reach to most people. Alongside them, Nintendo and Atari gave you 80% of the fun at 20% of the cost. History will view games arcades as a flash in the pan compared to gaming at home.
Namco, Capcom, and Konami went into selling software instead of hardware but it feels hard to imagine buying a home copy of Fable** for $X,000 any time soon.
* That’s for video games. In fairground arcades you’d literally pay for tokens then use tokens for play. The analogy almost writes itself.
If you think the idea of buying Fable like model is hard, you should really try some of the latest open models like Qwen 3.8. They’re getting there very fast.
Qwen 3.8 and GLM 5.3 are my workhorses right now. Qwen for simpler stuff because it is wicked fast using 'ninfer', GLM 5.3 because it is very good a more complex stuff and less likely to get stuck so I can let it run for much longer without intervention.
Even 2 DGX sparks are limited to Q2 at best. As for GPUs, it of course depends, but anything under 50k just can't run it, not even Q2.
And at Q2, I think you're better of with Qwen3-coder-next on one DGX spark (or much better: on any apple silicon max, we are currently in the ridiculous situation that a second hand m1 max 64G will beat a DGX spark that's at least triple the price)
I have Qwen 3.8 running under ninfer or Strata sized to fill my 5090.
From a fair bit of coding with the agent, my experience with Qwen 3.8 (Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-0 on Strata, Qwen 3.8 27B on ninfer). In my particular case, I was trying to clone a recent HN post which allowed you to visualize how out of tune your singing was relative to a song, but as a web app or portable native app.
My experience is that both those models will generate an enormous amount of noise (literally: it was running audible unit tests on my machine to test the note-detection algorithms) while coming to a solution, if they do find a solution- while Claude (whatever the latest model is) or ChatGPT (ditto) will do 1/10th the amount of work to get the right solution. I suspect if I had the resources to run GLM 5.3, it would probably be significantly faster to get to a working solution than Qwen-3.8 27B, but likely still somewhat inferior to Claude or ChatGPT solutions, and also would generate a much larger energy bill on my part.
> Rent-seeking is the act of growing one's existing wealth by manipulating public policy or economic conditions without creating new wealth.
Game industry has first manipulated the economic conditions by restricting access to game content (skins, modding capabilities, always online drm, etc.) that previously was included in the price (buy game, you own that game outright).
Manipulation of public policy was a more recent thing with the laws surrounding underage gambling via 'loot-boxes'.
So I think the onus is on you to show us why these behaviors, which increase the wealth the companies are able to extract per game, are not rent seeking behavior.
Changing how you sell your own products is not "manipulating economic conditions" whatsoever. If it was, that would mean that every business in the history of commerce has engaged in it.
As for the laws on underage gambling, countries all over the world are outlawing that, suing gaming companies, etc. The public is not in favor of underage gambling, and it's unclear to me how banning it increases the wealth of these companies? So I think the onus is on you to explain how this is an example of rent seeking.
Modern gaming companies can be considered rent seekers not because they charge money for their products, but because many increasingly rely on extracting additional revenue from consumers by exploiting market power, consumer lock-in, and artificial scarcity rather than creating proportionately greater value. In economics, rent seeking is the pursuit of economic gains through the exploitation of existing advantages rather than through productive activity. Developing a great game is productive; leveraging an established franchise or player community to extract additional money without providing equivalent value is rent seeking.
Consider the use of microtransactions, battle passes, paid progression, and downloadable content. These systems often exploit the time, money, and emotional investment players have already made in a game. By deliberately slowing progression, withholding content, or introducing artificial scarcity, companies can encourage consumers to spend more money not because they are receiving a better product, but because they want to avoid frustration or access something they already value. The company is monetizing the player's existing commitment rather than necessarily creating new value.
This does not mean every gaming company or microtransaction is inherently rent seeking. Selling meaningful new content is a legitimate business activity. The criticism is that parts of the modern gaming industry increasingly prioritize extracting revenue from an existing player base over developing better games. When companies use their control over popular franchises and digital ecosystems to extract additional money from consumers who have few alternatives, they exhibit the very characteristics of rent seeking that economists criticize in other industries.
I'd say Nintendo gave you 10x the fun of neo geo. Yeah it looked cool, but mario and Zelda on the nes played like nothing else. And then we got street fighter 2 on snes which is better than any snk fighter I've played :)
(I'll run for cover now as I'm sure neogeo fans will disagree!)
I'll agree, since arcades could never give the experience a JRPG could. No shop would allow a player to occupy a machine all day unless they spent a lot of quarters.
It will take a while till prices drop to a level where affording 256GB card (even if it would be shared between few coworkers) will have any sense over buying tokens.
And the AI debt bubble price spike will probably stay there for a while.
That's going to be pretty pricey though. And I really want to see some independent benchmarks, that all looks very good on paper but there are some complications (notably: chip count, run length of the traces).
It reminds me that coin-op SF2 champion edition was like $6000 new, and you rented time with tokens. Eventually you could play a very good but more limited SNES version of SF2 at home for no tokens. And then PC hardware matured to the point where you can play the exact bits via emulator at better than original fidelity.
It will be very nice when AI gets to the emulator stage.
I'm already nostalgic for very early Dall-e LSD-like art. Ref: "Sleeping Horses - Nov 2021."
Those early GAN image generators were so weird that they produced art that felt genuinely creative. I don't know that whatever people are using to create modern Tung-Tung-Tung-Sahur instagram reel slop is even capable of producing similar things.
Arcades mostly failed to adapt to the advances of home consoles to provide experiences you couldn't get at home. That's on the dimwitted game suits of the 1990s who cared little about novel gameplay, networked gameplay, or better NPC AI and physics. Then mobile happened and there was a brief glorious Cambrian explosion of new ideas until the Zynga playbook and DLC. Evidence: lived through it as a small game studio developer trying to do original things. The fun experiment now and then is to break out your 1980s video game consoles and play through their libraries. Most of the games are objectively terrible but gems like Yar's Revenge are still gems today IMO.
You've stumbled on exactly why the big providers are lobbying for regulation and big claims that Ai is unsafe. They don't want you running these at home.
I mean IBM already has a track record of building gigantic amounts of garbage and over charging large corps and govts for it coasting on their brand name and networks to get deals. So using some cheap ass model fits their MO.
Having used their shit mountain called websphere, what an IBM CEO says is not something I take too seriously.
IBM has two faces. Their consultancy arm is the exact opposite of what I would call a good company, even so, they make a good bit of money. The hardware side of the business is still quite impressive, yes, it is too expensive but it is effectively a little mainframe and the performance is - I have to admit - impressive. That said I would never buy it because I can get much better value elsewhere. Even so I think they could do this if they really wanted to.
They do have big advantanges because many large corps have become totally dependent on them having outsourced everything and lost internal knowledge. That internal ops knowledge is very crucial/useful to take full advatange of what AI offers. In the coming years much of it will leak into frontier models. After which corps might not need IBM/TCS etc and just rebuild internal teams to handle ops like they once did.
Well... that's a tricky one. I bought a lot of these second hand. I can sell them for more than they originally cost new, so indeed, it is not 'cheap'.
Then again, I run a business and for the business it is absolutely manageable, it is roughly the price of a single job. Privately this would be a different matter but I assume that if you would do this privately you'd simply get a single (but powerful) GPU second hand and offload to RAM. Not as fast but with llama.cpp, 'ninfer', DwarfStar and various other clones of those two you can get some pretty decent performance out of single GPUs.
I've been buying powerful hardware since the 90's, starting with SGI and then ever larger Linux boxes. I make my living with them so it's never been a huge decision to invest some of the proceeds back into the tools. But on an absolute scale, yes, it is a lot of money.
Would it make more sense to just rent the infra from a Cloud provider just for amortization purposes? While...and this depending on the the Cloud provider...getting the same guarantees of governance and personal sovereignty?
You might unknowingly suffer from switcher's bias. Claude Opus is really much better than anything else I've seen on openrouter and Astra is also quite good.
It really depends on usecase i guess. I've tried a few open models, they don't hold a candle Argon/Opus/Sonnet when it comes to my current major use case which revolves around CAD and aero simulation development work.
What's your use case where the open models are working well?
You mean, the Chinese models, right? Like the ones that stole all the weights from ChatGPT and Claude, just like those companies stole all the IP from the rest of the world, right?
I’ve always wondered about those models. Are we just sending all of our data over to China, instead of ChatGPT or Claude? Can we trust them any more than we do ChatGPT or Claude?
Considering the disclosed numbers for DeepInfer and the usefulness/popularity of DeepSeek 4.1 Flash, I expect it is inevitable that more companies are going to consider the self-hosting option seriously, and perhaps play with the idea of offering inference as a vendor.
What happens when your competitors invest in better and more expensive tools or materials in general? They (potentially) make better products, but those products also cost more. It's turtles all the way down.
I’ve been in tech for a long time… I have no idea what IBM does. Now, I get that I’m not the customer, that their customer is “business” but not my business… no… bigger business. The biggest of businesses I assume. I think it must either be a club I’m not in, or IBM is three half racks in a trenchcoat submitting invoices for millions to companies where people just pay it because “it’s IBM, it must be important”.
Ironically one of the smaller places I've worked at tried to use an IBM product once - I'm still not clear on what it was meant to do or how it was better than any cheaper alternative, so I assume someone in leadership was fooled by a very impressive pitch.
Lots of work went into integration and it never went live.
Transaction can be a nebulous term; think of credit card transactions, ach and wire transfers, mutual fund buy/sells, bank deposits and withdrawals, etc.
Review the specifications for the latest Telum II chip if you need a hint. 5.5ghz base clock.
If you absolutely need to scale something vertically, IBM Z is the final stop on the line.
X86 is great at hosting an entire farm of applications. You would never want to install salesforce or gitlab on a mainframe. That would completely defeat its purpose. If you've never encountered an appropriate workload, IBM might never make intuitive sense.
You can try open models via OpenRouter and omp without having to buy or rent any GPU. I routinely try open models to see where they are. GLM 5.3 with DSFV4.1 as advisor via omp feels Opus 4.6 level, i.e. totally able to ship stuff.
I do use Anthropic and OpenAI mostly because it costs me 1/40 and 1/50 of the api prices, which is way cheaper than Chinese subscriptions, but I’m ready to jump ship when they pull the plug. Fortunately they have nothing that forces me to keep paying them, and I don’t pay the crazy enterprise subscription costs, just the personal account prices.
I also die to be able to run the OSS setup locally to stop being spied on all the time.
For context, I averaged 10 hours a day prompting LLMs from when ChatGPT came out. At the start it was just function generation, then single file modules, then modules, and now I reason, design, implement and test with them.
An interesting thing is that with omp you can plan with a smart model and then implement with a dumber one. So for example, plan with Kimi 3.1 via OpenRouter and then implement with local DSFV4 with zero data retention active, sorted by cheapest, and minimal tps accepted.
I get the people who are pissed about their cushy jobs evaporating, but if you know how to use these technologies it’s very freeing because it’s like having a company of people working for you, except they are not human so they’re not wasting their life working for you instead of using it for themselves.
You can make them do pretty much everything that is fuzzy-verifiable, you don’t have to worry about their feelings or that they won’t do what you asked them to do or stop, and you can activate them when you want at scale in parallel or not use them at all for days.
For me it’s a dream honestly. I never felt so much empowered in my life while being enabled to unlock my creativity with almost no boundaries.
In OMP just setting the "task" role (which controls subagent model choice) to some cheap Flash model like Deepseek or Gemini goes a long way towards making better use of your quota.
Sol and Opus both are quite eager to use subagents and since the subagent tasks are often well specified and then verified by the main model there's hardly any advantage to using anything better for the subagent anyway, so IMO it's a no-brainer.
For me switching the main model to Kimi K3.1 Pro would put me towards 100% open, but I don't know if it uses subagents effectively. Any other open models I noticed just don't utilize them very much.
Maybe it’s 1% or 10% today, but a decade ago, having/assembling the technical chops to know how to build something was a huge part of the meat / barrier to entry.
It can still be true though, because entry is nothing without sales and a continued search for product market fit. I don’t know about 1% but if you say ‘for every 100 apps launched in 2015-2020 only 1 turned into a sustainable business’ I’d say that’s in the ballpark.
If code becomes cheaper that may change the business fundamentals but it also means capturing customers becomes way harder.
Even a decade ago marketing/growth was the massive lion's share. IMO just building something has been near meaningless since at least post-Ruby on Rails. The app stores have been inundated since the late 2010s.
In a former life I was a freelancer who built first version mobile apps for seed and pre-seed startups. I’m a decent dev, so what I built was always good enough. Virtually none of these startups survived, and there were dozens of them.
Think bigger. Don't think in features. Think in business workflows.
You are a consumer, so you naturally think like the consumer you are. But consumers want cheap stuff and only big big businesses are able to provide cheap stuff at massive scale. So stop thinking in b2c. For us little businesses the money is in b2b. Businesses pay lots of money to have problems solved. They want to focus on their core business and outsource the rest.
Consumers hate paying subscriptions. Businesses love paying subscriptions. Because then they can make their problems someone else's problem and focus on what their actual business is.
As the OP said really. It is very very freeing to just be able to give a task that in ye olden days you knew was going to be a pain in the arse to do, and it will do it without you. Amazing.
It's like having extra copies of yourself that you can just fork and get stuff done. E.g. the other day I was in a meeting and there were some heavy discussions about a few gnarly issues. I fired off a few prompts in parallel while the AI researches options, approaches, made a prototype etc all while I was present and paying attention still in the face to face meeting. I am so much so much more efficient as an engineer now - getting better feedback, more bonuses etc. It's never been better - I worry about the computer science and software engineering big picture, and the AI handles the syntax trivia.
The models are really really good now.
The only downside is that as a result of more stuff getting done, we need to have more meetings to discuss what actually needs to be worked on and how.
People at work are cutting and pasting Claude output for use in routine messages on Teams, troubleshooting suggestions, and proposed architecture designs, and honestly it's some of the dumbest shit I've ever seen. I have to tell people to reword it in their own words, and I believe they often can't.
I've seen developers use Claude to make suggestions for infrastructure design that made absolutely no sense and did not consider our environment. I've seen people write English that grammatically made sense but logically did not.
I don't see enough complaints about this and it can't be just my company.
Is anyone else seeing really severe Dunning-Kruger?
I always check what AI says to me, and I use my own words. I think the average engineer is not.
That it's a degradation in some environments? There's not a lot being said about this. The laments are usually developers sad that they're prompting instead of coding. My lament isn't about the art of it, it's the human reasoning that seems to have gone away in my workplace when it comes to engineers.
You didn't really answer the question though, how are you not nervous about your job security and how do you still see any value in software that you make with almost 0 effort?
I've hit a weird point myself. I'm not overly interested in building software for other people now. What I'm able to do is build all my stupid hobby project ideas for my very specific use cases and not have to worry about making them profitable in order to justify the time spent on them. I just built my own point of sale system because nothing on the market meets my needs, and my needs are extremely niche.
I sell vintage cameras in an antiques centre, but only 3 days a week. I’d like to sell online too but I have no ability to put aside stock for that, and similarly I only have access to the stock a few days a week, which makes shipping awkward. Pretty much every item is unique and needs listing separately, which is a pain. This is probably doable on some existing POS but it’s almost certainly much more expensive than just building my own (I think Shopify starts around £40 per month for what I’d need, then still require a lot of customisation). A major aspect of my business is buying stuff in the stall and tracking its eventual sale price, which tends to be a pain on other systems when it’s all one-off unique items.
I also do repairs and so I recently added a job management system into the POS, which has been really useful so far. Next step is to sort out the online store/eBay integration aspect!
Take your time to set up the tools and just build something small. Create a personal website, or a calendar that helps to track friends' birthdays, or something else of that sort. It's a good way to calm yourself down, because it will give you a good picture of both the capabilities and the limitations of AI.
I've found it very liberating that I can handle some of the tedious parts of a project to AI. In the end, I spend perhaps the same amount of time on a project, but more of it goes to important work and less gets wasted on plumbing.
People are underestimating the lock-in of Claude Code and Codex. Most companies are not buying a model, they are buying the ecosystem.
But the lock-in is not _that_ big that a big co could not pull off a migration, but it would be like migrating from Teams to Slack, not like migrating away from Adobe.
They do when they see the cost savings and ROI. It’s the one reliable thing that makes a company switch from any tech to another one.
Looking at our AI spend today vs 1 year ago we have massive changes. Anything that can bring this under control and potentially save millions a year is being explored.
We will probably end up with a combo of open weight models for most work and frontier models for key projects. For now. Who knows what the landscape looks like in 1 year.
This is exactly how I got people at work onboarded to our local models. Everyone was comfortable with Claude or codex CLI so i gave them prebaked config files to use them with our local models. Ezpz
Of course IBM - the guys that sell you mainframes - would be saying this.
This is a win though. But not for the reasons people might think it is.
It is a win because it's a massive player now countering the new (temporary) behemoths. With all the leverage, budget and connections.
And contrary to the AI Labs, IBM isn't just that one SV rockstar dude, but a whole org with tons of people. Meaning that they're going to be far closer to the baseline of sane humans just from averaging alone.
The guy that sells mainframes also gives away the Granite models for free. I've found them to be quite capable in content retrieval and long-running workflows, even (or especially?) the small ones.
> And contrary to the AI Labs, IBM isn't just that one SV rockstar dude, but a whole org with tons of people
IBM of today is a hallow corpse of what it was. The hardware business remains but they’ve mastered extracting rent from imprisoned customers long ago. No sane person should willing go into business with them, same as Oracle.
I think, despite the desperate attempt at IBM to remain relevant (and in business), their CEO has a point here that do you really need SOTA models for everything?
I had a really nice time with GLM5.2 for a month or two - it was, in my experience, approx Opus 4.5ish level of capability when I think Opus 4.8 was the new hotness
Even today, I use Gemini 3.8 Flash as my daily driver for most coding. Its fast and good, even if it is not the absolute SOTA at coding benchmarks it really is very good. Yes I have tried Opus 5.5 and 4 Argon etc and yes the are better, but do I need that?
I think that we are going to be at the stage soon-enough where models are "good enough" for 99% of people's needs. The SOTA will keep on marching ever-onwards I am sure, but I would not be surprised if people stick with previous-gen/previous-previous-gen/etc models because why upgrade?
If this does play out like this, I do wonder how that works out for the frontier labs (specifically OpenAI and Anthropic, less so Meta & Google) - if open models or even just cheaper Sonnet-level models from a few generations ago are good enough for basically everyone's daily work, where is the revenue? how can you justify pumping billions and billions into new generation models and datacenters etc when there is no driving need to use them?
I love 3.8 Flash, if only it weren’t unusably flaky. Sometimes simple responses take minutes or I get rate limited after sending a single message (I guess it’s a shared pool rate limit). Have you not experienced this (would love to know if there’s a way to actually use it reliably!)
I've personally experienced three big jumps in LLM capability: o3, opus 4.6, and opus 5.5. I could go back to opus 4.6, but it did require way more involvement from my end.
For work in which you are in the loop, absolutely. Do people not value their time or something? Why in the world would you use less than the best LLM available to you? You're not going to run out of the $20 plan even if you use Opus 5.5 at xhigh and code 8 hours/day (assuming you're doing it piecewise using a Claude product).
For large volumes of programmatically submitted LLM tasks, sure.
I have single sentence prompts in one of my legacy codebases that it takes Opus 5.5 over $1000 in API spend to return an answer correctly on. I am not getting by with a $20 plan.
That's my point - do you need the best available to you? What if tomorrow your employer said no more 5.5, or Anthropic IPOs and suddenly investors want to see a profit and that 20USD a month for Opus 5.5 becomes 20USD per-day or even per-hour? We're already seeing corps restrict their employees token usage.
Could your survive on say Opus 4.8? Or GLM 5.3?
I bet you could if you had to.
I think it's the same argument about cars. Why wouldn't you want a Ferrari? Do you not value your time? Why not go as fast as physically possible?! You need the absolute fastest, most-capable, best-handling car for every single journey you take! Or... Back in reality not every journey you take is trying to lap the track as quickly as possible and just any reasonable car will get you where you need to go.
This is my point. Will Anthropic and OpenAI continue to exist selling Ferraris when actually people realise they can get by with a Toyota Corolla 99% of the time? Ferrari exists in the car world so I guess there is an argument for "yes". I am not sure myself - Ferraris are after all rare and exotic cars ... I suspect that even if we are not there now, we are already seeing current SOTA models that are mostly equivalent/better than the average engineer - another generation or two they'll be waaaaay better than the average engineer. How much more car do you need, and how much are you prepared to pay for it?
Currently the difference in my work between the best models and okay models is pretty night and day to the point its not worth using the older models. Now I haven't used the newest top open source models. But if I give the same code base to say Qwen 3.6 and GPT 5.6, Qwen will analyze the source of the problems wrong, provide actually wrong solutions, write code that takes 100x as long to run. Its not even worth it outside of querying for snippets of code. GPT 5.6 will help actually root cause the problems.
I think the point is that you are not everyone. Not everyone has your requirements, sounds like paying the premium is worth it for you, but others simply don't need it.
Ah but if the price is 10x - as the IBM CEO suggests - your employer might just ask you to use that, and then expect you to make up the gap between the cheap and expensive models using your own brain instead of just pissing money up the wall on high token costs AND your salary. You know, like the old days when we actually had to do the work ourselves.
How good will be the business of dealing with 2nd hand GPUs for open weights model in the future? What will the AI labs do with their GPUs once they are beyond their EOL? Can somebody please advise?
He is right. I’ve been a big proponent of local AI for a long time. But I had accepted the common wisdom that local inference is not economical compared to APIs. I bought a DGX Spark anyway, because I value independence and Qwen 3.8 Flash Next looked like a big deal.
A few days ago, I pulled my usage metrics to see how much I’m really getting out of this. Couldn’t believe it, but even accounting for cache discounts and typical cache hit rates, I would be spending ~$400 a month for a similar Chinese model through a ZDR-enabled API (eg, GLM 5.3-Flash on a US host) or ~$6,000 a month with a similar closed model (Opus 4.6 through Bedrock). I was pretty shocked because I only use it more or less synchronously, with no batch jobs or anything like that.
So yes, subscriptions are cheap if you’re willing to sell your data, but APIs add up really fast.
Qwen 3.8 Flash with Strata here, the problem with Strata is that you cannot easily share the server as the performance will collapse outside of a single user.
It's going to boil down to cost of the hardware vs. pay per use cloud models. I'm on the 100$/month chatgpt plan (~88euro). With all the resets, credits, and all the rest I typically don't run out of tokens. This is fine for me. I might go to the 200$/month plan at some point.
In comparison, my highend mac book pro costs 105 euro per month (leasing cost). That's not quite enough to run the bigger models and besides running the GPU on a laptop constantly makes it a bit hostile for my lap. And the model I would need a lot more memory for the bigger models. It's nice for experimenting though.
If you get your own GPUs in some kind of server form, you could run those locally or in a data center. Or you just get hardware resources from your favorite cloud provider. Hetzner (one of the more affordable ones) GPU servers start at around 200 euro per month. I know some people that use that with open weight models. Works fine, especially if you can share the resources between multiple users.
At which point of course you are doing the same as cloud based providers are doing. But probably with a bit worse economics. It all boils down to whether you are actually saving money or paying for a lot of underutilized hardware that's running 24x7 when your people actually only show up about 8x5.
I don't think most companies should be doing that. It sounds like they'd end up with a lot of idling over capacity at a premium. The only valid reasons to do this would be securing access to GPUs or some very urgent need to keep things on premise (which a lot of companies insist is true for them until they realize how much they are going to pay).
In this context yes, but for example at my job we are using LLMs to evaluate some outputs of cheaper/simpler models and to add context to them. This gets expensive very quickly with per-token pricing and would pin our entire business on the whims of one or two providers. With self-hosted open-weight LLMs we can just operate like we always have, it's a different kind of processing but otherwise we are free to do as we please wrt compute.
So it matters a lot that these open-weight models are good enough; we would likely not be rolling out this new processing if it was fully dependent on Anthropic and OpenAI.
Are there inference products focused on say, optimizing frequently used internal contexts, so that they are compact in the kv cache? Perhaps there rep could be optimized over hundreds+ of distinct real use cases?
Maybe your law firm doesn't want your data going outside your business. But you obviously want high quality analysis.
There was another comment in the IBM earnings call on enterprise AI spend [1]. Specifically, concerns around ROI for AI experiments in the enterprise and difficulty incorporating AI into enterprise workflows. This was surprising given the hype around AI, and it came across as a negative indicator.
184 comments
[ 5.9 ms ] story [ 186 ms ] threadIf you have ever worked at IBM, it makes total sense. Completely dysfunctional middle management and on up.
This is more of a statement for their consulting customers, right?
I can picture that they’d have the software side sorted out with RedHat. But I don’t know what hardware they’d push to support local on-premises models. It looks like when you buy Fusion HCI, you can choose from Dell or Lenovo as the underlying hardware supplier.
The optimal inference configuration varies according to the model, the specific traffic (which changes in time), and the specific hardware it's running to. The "knowledge" for it doesn't exist yet.
Usually in similar scenarios an agent will try to experiment with various configurations and get to a good result. However, you can't have an agent test out different inference serving configurations/architectures, since every experiment might involves 10-60 minutes of spinning up a model (on very expensive GPUs)...
I can't wait to use models that are 'slightly less good' when I am at work instead of something like Fable that lets me focus on the problem at hand as much as possible rather than also spending additional effort correcting for the tool I am using...
Over time NeoGeo realized they could sell you the hardware to play at home (“on-prem”) at arcade quality albeit at a price out of reach to most people. Alongside them, Nintendo and Atari gave you 80% of the fun at 20% of the cost. History will view games arcades as a flash in the pan compared to gaming at home.
Namco, Capcom, and Konami went into selling software instead of hardware but it feels hard to imagine buying a home copy of Fable** for $X,000 any time soon.
* That’s for video games. In fairground arcades you’d literally pay for tokens then use tokens for play. The analogy almost writes itself.
** The Anthropic model, not the video game.
And at Q2, I think you're better of with Qwen3-coder-next on one DGX spark (or much better: on any apple silicon max, we are currently in the ridiculous situation that a second hand m1 max 64G will beat a DGX spark that's at least triple the price)
From a fair bit of coding with the agent, my experience with Qwen 3.8 (Qwen3.8-Flash-Next-GSQ-RCO-IQ3_S-0 on Strata, Qwen 3.8 27B on ninfer). In my particular case, I was trying to clone a recent HN post which allowed you to visualize how out of tune your singing was relative to a song, but as a web app or portable native app.
My experience is that both those models will generate an enormous amount of noise (literally: it was running audible unit tests on my machine to test the note-detection algorithms) while coming to a solution, if they do find a solution- while Claude (whatever the latest model is) or ChatGPT (ditto) will do 1/10th the amount of work to get the right solution. I suspect if I had the resources to run GLM 5.3, it would probably be significantly faster to get to a working solution than Qwen-3.8 27B, but likely still somewhat inferior to Claude or ChatGPT solutions, and also would generate a much larger energy bill on my part.
But also look at live service games and the shit modern game studios are doing to try to go back to the old world of rent seeking.
Game industry has first manipulated the economic conditions by restricting access to game content (skins, modding capabilities, always online drm, etc.) that previously was included in the price (buy game, you own that game outright).
Manipulation of public policy was a more recent thing with the laws surrounding underage gambling via 'loot-boxes'.
So I think the onus is on you to show us why these behaviors, which increase the wealth the companies are able to extract per game, are not rent seeking behavior.
As for the laws on underage gambling, countries all over the world are outlawing that, suing gaming companies, etc. The public is not in favor of underage gambling, and it's unclear to me how banning it increases the wealth of these companies? So I think the onus is on you to explain how this is an example of rent seeking.
Consider the use of microtransactions, battle passes, paid progression, and downloadable content. These systems often exploit the time, money, and emotional investment players have already made in a game. By deliberately slowing progression, withholding content, or introducing artificial scarcity, companies can encourage consumers to spend more money not because they are receiving a better product, but because they want to avoid frustration or access something they already value. The company is monetizing the player's existing commitment rather than necessarily creating new value.
This does not mean every gaming company or microtransaction is inherently rent seeking. Selling meaningful new content is a legitimate business activity. The criticism is that parts of the modern gaming industry increasingly prioritize extracting revenue from an existing player base over developing better games. When companies use their control over popular franchises and digital ecosystems to extract additional money from consumers who have few alternatives, they exhibit the very characteristics of rent seeking that economists criticize in other industries.
No, it’s not, you just made that up. Why doesn’t this match your previous definition?
The rest of your long AI-written comment has nothing to do with the principle of economic rent seeking.
Not sure why I’ve been wasting my time, but I’m out. Peace!
(I'll run for cover now as I'm sure neogeo fans will disagree!)
(I also adore MS)
And the AI debt bubble price spike will probably stay there for a while.
Still (as bit over) 1/2 the ram bus width of a single HBM4 stack though! (1280b vs 2048b).
Uber took a decade but eventually, the real cost of a taxi service went up for many use cases.
Your argument is only solid if we are headed to the singularity.
Perfect encapsulation of the problem. I may steal the wording you used for future conversations.
It reminds me that coin-op SF2 champion edition was like $6000 new, and you rented time with tokens. Eventually you could play a very good but more limited SNES version of SF2 at home for no tokens. And then PC hardware matured to the point where you can play the exact bits via emulator at better than original fidelity.
It will be very nice when AI gets to the emulator stage.
Do you think we'll be longing for the nostalgia hit of LLM models from 2025 in 2050?
Maybe we will collectively miss being completely right and inappropriate em—dashes.
Those early GAN image generators were so weird that they produced art that felt genuinely creative. I don't know that whatever people are using to create modern Tung-Tung-Tung-Sahur instagram reel slop is even capable of producing similar things.
in the early days, 2 plays for a quarter was not unusual (but you couldn't buy one)
I mean IBM already has a track record of building gigantic amounts of garbage and over charging large corps and govts for it coasting on their brand name and networks to get deals. So using some cheap ass model fits their MO. Having used their shit mountain called websphere, what an IBM CEO says is not something I take too seriously.
Then again, I run a business and for the business it is absolutely manageable, it is roughly the price of a single job. Privately this would be a different matter but I assume that if you would do this privately you'd simply get a single (but powerful) GPU second hand and offload to RAM. Not as fast but with llama.cpp, 'ninfer', DwarfStar and various other clones of those two you can get some pretty decent performance out of single GPUs.
I've been buying powerful hardware since the 90's, starting with SGI and then ever larger Linux boxes. I make my living with them so it's never been a huge decision to invest some of the proceeds back into the tools. But on an absolute scale, yes, it is a lot of money.
I’m running local models more and more and starting to move to just tailscaling into everything
What's your use case where the open models are working well?
I’ve always wondered about those models. Are we just sending all of our data over to China, instead of ChatGPT or Claude? Can we trust them any more than we do ChatGPT or Claude?
we technologists tend to extend what we see and think to other realms where it doesn't make sense
Does Ferrari have any moat if it can't convince the government to ban Ford?
Lots of work went into integration and it never went live.
For one thing, amongst many, IBM mainframes process 70% of global transactions on a value basis
If you absolutely need to scale something vertically, IBM Z is the final stop on the line.
X86 is great at hosting an entire farm of applications. You would never want to install salesforce or gitlab on a mainframe. That would completely defeat its purpose. If you've never encountered an appropriate workload, IBM might never make intuitive sense.
Also, the roadmap:
https://newsroom.ibm.com/2026-08-24-ibm-unveils-next-generat...
Honest question: What would such a workload look like? Why would Salesforce or GitLab not make sense on a mainframe?
I do use Anthropic and OpenAI mostly because it costs me 1/40 and 1/50 of the api prices, which is way cheaper than Chinese subscriptions, but I’m ready to jump ship when they pull the plug. Fortunately they have nothing that forces me to keep paying them, and I don’t pay the crazy enterprise subscription costs, just the personal account prices.
I also die to be able to run the OSS setup locally to stop being spied on all the time.
For context, I averaged 10 hours a day prompting LLMs from when ChatGPT came out. At the start it was just function generation, then single file modules, then modules, and now I reason, design, implement and test with them.
An interesting thing is that with omp you can plan with a smart model and then implement with a dumber one. So for example, plan with Kimi 3.1 via OpenRouter and then implement with local DSFV4 with zero data retention active, sorted by cheapest, and minimal tps accepted.
I get the people who are pissed about their cushy jobs evaporating, but if you know how to use these technologies it’s very freeing because it’s like having a company of people working for you, except they are not human so they’re not wasting their life working for you instead of using it for themselves.
You can make them do pretty much everything that is fuzzy-verifiable, you don’t have to worry about their feelings or that they won’t do what you asked them to do or stop, and you can activate them when you want at scale in parallel or not use them at all for days.
For me it’s a dream honestly. I never felt so much empowered in my life while being enabled to unlock my creativity with almost no boundaries.
Sol and Opus both are quite eager to use subagents and since the subagent tasks are often well specified and then verified by the main model there's hardly any advantage to using anything better for the subagent anyway, so IMO it's a no-brainer.
For me switching the main model to Kimi K3.1 Pro would put me towards 100% open, but I don't know if it uses subagents effectively. Any other open models I noticed just don't utilize them very much.
I geniuinely don't see what I could make with this new tech, my brain immediately goes to "this app idea is worthless, anybody can make it now".
Maybe it’s 1% or 10% today, but a decade ago, having/assembling the technical chops to know how to build something was a huge part of the meat / barrier to entry.
If code becomes cheaper that may change the business fundamentals but it also means capturing customers becomes way harder.
Now that opportunity cost is much lower so experimentation is more practical than before.
You are a consumer, so you naturally think like the consumer you are. But consumers want cheap stuff and only big big businesses are able to provide cheap stuff at massive scale. So stop thinking in b2c. For us little businesses the money is in b2b. Businesses pay lots of money to have problems solved. They want to focus on their core business and outsource the rest.
Consumers hate paying subscriptions. Businesses love paying subscriptions. Because then they can make their problems someone else's problem and focus on what their actual business is.
It's like having extra copies of yourself that you can just fork and get stuff done. E.g. the other day I was in a meeting and there were some heavy discussions about a few gnarly issues. I fired off a few prompts in parallel while the AI researches options, approaches, made a prototype etc all while I was present and paying attention still in the face to face meeting. I am so much so much more efficient as an engineer now - getting better feedback, more bonuses etc. It's never been better - I worry about the computer science and software engineering big picture, and the AI handles the syntax trivia.
The models are really really good now.
The only downside is that as a result of more stuff getting done, we need to have more meetings to discuss what actually needs to be worked on and how.
I've seen developers use Claude to make suggestions for infrastructure design that made absolutely no sense and did not consider our environment. I've seen people write English that grammatically made sense but logically did not.
I don't see enough complaints about this and it can't be just my company.
Is anyone else seeing really severe Dunning-Kruger?
I always check what AI says to me, and I use my own words. I think the average engineer is not.
They call this "slop" - I think it is well known at this point?
That it's a degradation in some environments? There's not a lot being said about this. The laments are usually developers sad that they're prompting instead of coding. My lament isn't about the art of it, it's the human reasoning that seems to have gone away in my workplace when it comes to engineers.
I also do repairs and so I recently added a job management system into the POS, which has been really useful so far. Next step is to sort out the online store/eBay integration aspect!
I've found it very liberating that I can handle some of the tedious parts of a project to AI. In the end, I spend perhaps the same amount of time on a project, but more of it goes to important work and less gets wasted on plumbing.
I am however worried that for the kind of work that the majority of the worlds tech workers do, this is a direct replacement for.
I guess we'll have to live to see.
But the lock-in is not _that_ big that a big co could not pull off a migration, but it would be like migrating from Teams to Slack, not like migrating away from Adobe.
Looking at our AI spend today vs 1 year ago we have massive changes. Anything that can bring this under control and potentially save millions a year is being explored.
We will probably end up with a combo of open weight models for most work and frontier models for key projects. For now. Who knows what the landscape looks like in 1 year.
This is a win though. But not for the reasons people might think it is.
It is a win because it's a massive player now countering the new (temporary) behemoths. With all the leverage, budget and connections.
And contrary to the AI Labs, IBM isn't just that one SV rockstar dude, but a whole org with tons of people. Meaning that they're going to be far closer to the baseline of sane humans just from averaging alone.
IBM of today is a hallow corpse of what it was. The hardware business remains but they’ve mastered extracting rent from imprisoned customers long ago. No sane person should willing go into business with them, same as Oracle.
They are many people and none of them are in some sort of AI death cult like the EA polycule.
They're not suddenly good guys. They're just sane
I had a really nice time with GLM5.2 for a month or two - it was, in my experience, approx Opus 4.5ish level of capability when I think Opus 4.8 was the new hotness
Even today, I use Gemini 3.8 Flash as my daily driver for most coding. Its fast and good, even if it is not the absolute SOTA at coding benchmarks it really is very good. Yes I have tried Opus 5.5 and 4 Argon etc and yes the are better, but do I need that?
I think that we are going to be at the stage soon-enough where models are "good enough" for 99% of people's needs. The SOTA will keep on marching ever-onwards I am sure, but I would not be surprised if people stick with previous-gen/previous-previous-gen/etc models because why upgrade?
If this does play out like this, I do wonder how that works out for the frontier labs (specifically OpenAI and Anthropic, less so Meta & Google) - if open models or even just cheaper Sonnet-level models from a few generations ago are good enough for basically everyone's daily work, where is the revenue? how can you justify pumping billions and billions into new generation models and datacenters etc when there is no driving need to use them?
For work in which you are in the loop, absolutely. Do people not value their time or something? Why in the world would you use less than the best LLM available to you? You're not going to run out of the $20 plan even if you use Opus 5.5 at xhigh and code 8 hours/day (assuming you're doing it piecewise using a Claude product).
For large volumes of programmatically submitted LLM tasks, sure.
Could your survive on say Opus 4.8? Or GLM 5.3?
I bet you could if you had to.
I think it's the same argument about cars. Why wouldn't you want a Ferrari? Do you not value your time? Why not go as fast as physically possible?! You need the absolute fastest, most-capable, best-handling car for every single journey you take! Or... Back in reality not every journey you take is trying to lap the track as quickly as possible and just any reasonable car will get you where you need to go.
This is my point. Will Anthropic and OpenAI continue to exist selling Ferraris when actually people realise they can get by with a Toyota Corolla 99% of the time? Ferrari exists in the car world so I guess there is an argument for "yes". I am not sure myself - Ferraris are after all rare and exotic cars ... I suspect that even if we are not there now, we are already seeing current SOTA models that are mostly equivalent/better than the average engineer - another generation or two they'll be waaaaay better than the average engineer. How much more car do you need, and how much are you prepared to pay for it?
I think this is a very high likelihood scenario.
A few days ago, I pulled my usage metrics to see how much I’m really getting out of this. Couldn’t believe it, but even accounting for cache discounts and typical cache hit rates, I would be spending ~$400 a month for a similar Chinese model through a ZDR-enabled API (eg, GLM 5.3-Flash on a US host) or ~$6,000 a month with a similar closed model (Opus 4.6 through Bedrock). I was pretty shocked because I only use it more or less synchronously, with no batch jobs or anything like that.
So yes, subscriptions are cheap if you’re willing to sell your data, but APIs add up really fast.
DeepSeek v4.1 Flash or GLM 3 are already very very capable models.
Only bottleneck at the moment is the hardware prices.
In comparison, my highend mac book pro costs 105 euro per month (leasing cost). That's not quite enough to run the bigger models and besides running the GPU on a laptop constantly makes it a bit hostile for my lap. And the model I would need a lot more memory for the bigger models. It's nice for experimenting though.
If you get your own GPUs in some kind of server form, you could run those locally or in a data center. Or you just get hardware resources from your favorite cloud provider. Hetzner (one of the more affordable ones) GPU servers start at around 200 euro per month. I know some people that use that with open weight models. Works fine, especially if you can share the resources between multiple users.
At which point of course you are doing the same as cloud based providers are doing. But probably with a bit worse economics. It all boils down to whether you are actually saving money or paying for a lot of underutilized hardware that's running 24x7 when your people actually only show up about 8x5.
I don't think most companies should be doing that. It sounds like they'd end up with a lot of idling over capacity at a premium. The only valid reasons to do this would be securing access to GPUs or some very urgent need to keep things on premise (which a lot of companies insist is true for them until they realize how much they are going to pay).
So it matters a lot that these open-weight models are good enough; we would likely not be rolling out this new processing if it was fully dependent on Anthropic and OpenAI.
Maybe your law firm doesn't want your data going outside your business. But you obviously want high quality analysis.
Is it though? You’re likely paying more on prem per unit of electricity and the racks of NVIDIA cards also aren’t exactly cheap.
I don’t see how that leads to 10%. I can see it being a good pitch if you’re in the biz of selling on prem servers though
[1] - https://www.ibm.com/downloads/documents/us-en/17f55bd310d5fd...