Is $0.30 per million tokens that wild? Haiku 5.5 is $0.10 per million input and $0.50 per million output. And that's before you get onto the cheaper cache pricing. Surely a self hosted GLM isn't better than Haiku 5.5 at coding?
This is in/out cost, caching being mentioned as virtually free. Haiku is $0.50 up to 100k context window, $2.50 afterwards which for coding is not a high bar to cross.
Regarding model quality GLM 5.3 is past the good enough bar for most coding tasks. I wouldn't be surprised if the limiting adoption by companies is price not frontier model quality for most of their tasks.
Besides that, since this is a bank, you can see how significant the requirements are to run sensitive workflows within your own infrastructure premises.
> Per David Abitbol, who leads AI ops there: 56 Nvidia B200s, a blended 30 cents per million tokens, roughly a tenth of what they paid Anthropic and OpenAI.
So they've invested around $2.5M-3M into hardware + monthly operating costs (electricity, updates, cooling). I wonder how much did they spend on tokens to decide to make such a switch. I mean if they used to spend $500k-1M per month just on tokens then it would pay off in a year or so, assuming their infra will be stable and efficient.
We'll see if that's a trend for mid-big companies or just an experiment.
All that math is assuming that it's all self-hosted in their data center or so and not paying to a cloud provider, which will be probably more expensive.
Personally, I think Anthropic and OpenAI will introduce enterprise plans such as $2,000/month/engineer plans that will give 100x more tokens than the current max plans and then charge API pricing.
Yes, margins will go down a bit but the frontier labs still have some leverage.
The B200 was launched in 2024 if I'm not mistaken. The big gamble with this might be how fast these will deprecate in value/use as soon as Nvidia launches a new and better product.
At the moment at least Nvidia doesn't have a incentive to release a new product because it's sold out as it is and probably doesn't want to plunge their customers in depth with outdated products?
In related news, Core 4 Solutions listed a decommissioned $60 million AI supercomputer cluster featuring 9,600 Nvidia A100 GPUs on eBay. It's six years old. Nvidia isn't Intel with tick and tock, but they're still going to come out with newer hardware every year.
The gamble in this case is pocket change. They could comfortably spend this amount every year and just use precious years for overnight / batch processing jobs when they become less useful for dev work but remain operational.
> So they've invested around $2.5M-3M into hardware + monthly operating costs (electricity, updates, cooling).
Are you assuming that they bought the hardware? because they could also be renting the metal off a provider, in which case they either pay by the hour or have a yearly commitment to save on costs.
yeah, seems like they're renting (according to the info from the video), but doesn't it make the bill worse? I mean the provider makes money somehow, so it should be more expensive for revolut. I guess it's a lot anyway
Maintenance, capacity management and spares all add up, not to mention labour. If they have a requirement to maintain geographical diversity then that moves the needle significantly if they’re primarily based on the other side of the country.
I've quickly checked and guestimate of the rental would be ~250-300k per month for this kind of usage they claim to have. It's all based on open prices from together.ai. So probably it worth the switch for them afterall, I can imagine that 56 engineers can burn tokens for more than 300k per month
Unless you have very strict requirements, it does not make any sense to invest in hardware.
Most companies just rent it. Also, it's pretty hard to get the hardware.
It is in the article: "the gains showed up when tokens stopped being rationed" - so it is not just a "cost per token before/after" calculation. Once you told your employees "token are free, flatrate for all - go nuts!" they started to use it for all sorts of things and increased productivity. That is not easy to calculate.
And if the market teaches us one thing, it is that GPU prices will drop over time. The current absurd prices is attracting competitors, and the more competitors, the lower the price.
I would not be surprised if, in 5 years, there's a neat, low-power GPU in high end laptops, alternatively, a cheap, say 500 USD or so, GPU in a box, that you can use for your AI-center at home or in the office.
MoE models can be largely offloaded to system RAM - recent versions of llama.cpp automatically distribute parts of the model, with all the dense parts going into VRAM and just the routed experts residing in system RAM.
I'm currently running Qwen3 Coder Next, which is a 80B A3B model, at 15tps on a 8GB 7700S and 64GB RAM.
It's an idiot (no Thinking), so it has to go over its work several times to get it right, but it's the largest idiot I can feasibly run at the moment.
Also by accident I'm undercutting the only provider of this model on OpenRouter by 40%, doing $0.48/mTok.
Of course at my token rate it would take an entire day to even reach a million tokens, but it's the principle that matters.
This. I don't know why OpenAI, Anthropic and others have these high valuations. We have a temporary demand shock on hardware (GPUs, RAM, Storage) but the market dynamics will create more supply in the next years. Once everybody and their mother can get their hands on hardware, we will see tons of token-per-dollar inference services competing against each others. None of these providers will do any training, just hosting open-weights model. Why would anyone give OpenAI and Anthropic their subscription money then.
And we haven't even scratched the surface on highliy-specialized distillations. All those generall-purpose open-weight model will see distilled variations for their respective industry niches.
"Frontier labs" makes it sound like there's some special sauce to these companies. There isn't, other than very smart people working on problems.
Guess who else has that? The Chinese companies developing open weight models. In fact, the vast majority of AI science comes from China. They might still be playing catch up in some areas, but they're also leading in others (famously DeepSeek's inference architecture as an example).
There is IMO a non-trivial chance those labs will eventually surpass the US ones and become themselves "frontier", and that may happen sooner than a lot of people expect.
Of course, it still remains to be seen if they will keep publishing the weights at that point.
> Of course, it still remains to be seen if they will keep publishing the weights at that point.
I mean if they don't then the US companies could just release their weights and chop the Chinese off at the knee's at that point (reversing what is currently happening).
Opening the weights isn't just a technical thing, it's a geopolitical decision which likely in part comes from the top of the CCCP (by either request or just implicitly allowing it).
This is nonsense, at a policy level the Chinese government have actually been moving towards restricting larger models. I'd eat my hat if Europeans in particular aren't paying licenses for weights in a few years if it means getting to host the models on home soil, there is draft legislation (CADA) making its way through at the moment that would even require this in many circumstances.
We did the same thing, on much less powerful hardware but it has served us very well and we don't need to worry at all about the question whether or not the suppliers of inference services can be trusted long term with our data. The answer to that question in my opinion is 'absolutely not'.
Your inference server is just an inference server that spits out tokens. You can use the same harness, etc. you’re basically just changing the API base URL.
By pulling the plug. This box simply has no pathway to the internet. I do pretend I have one and it's interesting to see what it tries to do. For instance, Qwen tends to try to hit ali related URLs for websearch and is very disappointed when the mothership is not responding. I've yet to catch a model on an exfiltration attempt, either they are so good at this that it escapes my scrutiny (possible but unlikely) or they are simply not doing it. I would bet on the last one. Just don't say you're working for the Tibet Liberation Frontier or the Keep Taiwan Independent movement.
I have downloaded a number of abliterated models and done very brief tests on them (just to check for basic functionality) but on local models I have yet to run into refusals or output where I thought it might make sense to retry with an abliterated model. It was more out of a precaution that NV might drop those models from HF.
Why would you sandbox the runtime? The harness is where things go awry when it does, when the agents have access to more than they should and start deleting stuff. People aren't expecting the weights themselves to be "infected by malware" or whatever.
This is the right call - there's little proof that the quality gap between open models and OAI/Anthropic stuff translates to business results, but you surrender all control of your data to a third party.
The last cause might be a straight up blocker - certain financial regulations straight up prevent you from giving third parties to your data (esp outside national/EU jurisdiction), no matter what documents you sign.
This is probably only relevant if you go over the 150 engineers putting you in the enterprise bucket and forcing you to pay by the token rather than the subsidized Claude code plans. If you need 150 engineers all with frontier model accounts maybe you are doing something wrong ;)
For the past several months about 90% of the work we do goes to open source models. If you ignore the noise around 'cheap' etc, you might find that with a little extra effort to more precisely define what you need not only do these models generally work as well, and they will generally more closely follow instructions - on longer timeframes this is actually pretty helpful.
They are less good on ambiguous prompts like 'fix all the problems' (which to my chagrin is a very common prompt it turns out..)
Yes, that's a good point: you really need to work on the prompts, even if the harness is the same. Ambiguity is a real problem as is what we call 'the negative space', what isn't in the prompt but what is implied by the prompt to be undesirable. You need to spell that out in more detail it is not enough just to say 'do x' but you also need to make sure to say 'and not y or z', and knowing what y or z are requires a bit more insight into what hobby horses are encoded in a model.
To give one example: I'm doing a lot of stuff that involves hardware. When settings things up for fixing a bug I have to make sure to include that the hardware has been tested and works well. If I don't do that the model will almost certainly spend a ridiculous amount of time speculating about all the different ways in which the hardware could be broken (including theorizing about chip mask errors and all kinds of other obscure and rarely seen in practice defects). It's going to end up telling itself that for sure the 500 lines it just produced are absolutely perfect and the device that is on the i2c bus is broken in a way that it just really can't progress with solving the issue because there is nothing wrong with the code.
These kinds of blind-spots are funny when they happen once in a month but when they happen regularly it begins to be irritating. Adding a line or two to the prompt will take care of the issue. Frontier models tend to be better at this sort of thing.
Us too! Hardware is very interesting - I find in fact that 'smart' (aka expensive) models can overthink complex hardware state machines and end up doing atrocious code because it will try to handle every possible edge case (unbidden, mind you).
What has worked very well is harvesting and aggregating lessons and feeding this automatically into our pi sessions, this allows the models which listen to instructions to avoid a lot of time burning tokens on internal dialogue to ascertain the nature of reality.. but the expensive ones:
a) they dont listen
b) they ironically know too much!
That is very interesting. So far I am extremely careful to avoid putting negative statements into my instructions, to avoid poisoning the AI's thinking with exactly the concepts I do not want it to have. This might be from my use of much older models where if you say "do not X", then "X" gets put into the context and the "do not" kind of gets lost. It was especially visible with stable diffusion where you would ask for a picture without a feature and get exactly that feature. Maybe current models are better at that.
"the gains showed up when tokens stopped being rationed" - I can totally see that. I am leading (as an external consultant) a larger dev team for a huge insurance company. The devs are quite junior or in general more the "I became a sw-eng here because that company is the only big employer in a 1h radius" type of people (meaning: No rockstar developers). Most of them do not play with AI in their spare time, and they have been told to spend claude tokens wisely. Everybody is too afraid now to play with it and explore during work hours for fear of being token shamed.
At the same time, most of their junior devwork can be replaced completely by AI - so they really should start to learn how to work with it. I recommended everybody to get a private 20€/month claude subscription and use it for toy projects. I have younger devs (early 20s) in the team that told me they won't spend private money, or simply have no usecase in their private life for AI. We talk about software engineers here, and IMHO most of them are in denial of the fact that their skills have mostly become worthless if they don't start to pivot to something valueable.
Are you not worried about this being a sort of one-way street? Do you have any strategies to avoid ending up with a codebase you cannot work on without "AI"? when I think insurance company, I would assume the value seems to be more in the trust you build up over decades of with customers, not producing lots of code faster.
Well I am still in the loop as a senior dev and I get to review everyones code before it gets merged. Official guidelines is that everybody is responsible for the code they produce (with or without AI) and must understand it when they give it to the reviewer. I've been doiing this 3+ years at this company, so before the whole AI thing landed. I am explicitly not allowed to implement features myself (because the idea is the juniors learn and get trained).
Now, within the last 6-12 months of AI advancement, I think you could lay off half or even 3/4 of the team if I was explicitly allowed to implement features myself using AI. But that is not the current policy.
As for worries "ending up on a codebase you can't manage without AI" - my personal take is: No, I am not worried. Because going forward AI will always be available in the development workflow. That is my own opinion, not the corporation standpoint, however.
Being unable to work on a codebase without AI is not necessary. Working with models you can fit on consumer hardware is already significantly more productive than rawdogging code. It's similar to "you won't always have a calculator with you" from math teachers, see how that turned out.
A way bigger problem is a codebase that you can't effectively work on with AI, which is a lot of legacy humanslop. And things that make a codebase more suited for AI, like documentation that provides relevant domain knowledge and context for requirements, automated quality control, and test coverage, also make a codebase easier to work on for a human too.
It's not spite towards humans. It's spite towards typical state of pre-AI codebases.
It's not even a matter of incompetence, most of the humanslop came from the fact that you could just never implement anything properly in the time you were expected to do it.
> ... and IMHO most of them are in denial of the fact that their skills have mostly become worthless if they don't start to pivot to something valueable.
But your comment says it several time: the valuable skill is being already a seasoned dev.
Then AI is superpower.
It's for the junior that things sucks big times.
I see it all around me: senior devs are empowered while juniors are not even hired anymore.
>I recommended everybody to get a private 20€/month claude subscription and use it for toy projects.
That's about 20€/month more than I care to give Anthropic. They already stole my data and now they want my money too? No thanks. I'll stick to running Gemma or Qwen on my Mac.
GDPR mostly. In general, the easiest way to get ZDR for the models was through the hyperscalers. Additionally they're a German company so it's probably about all the data and tools being hosted in the EU, which Anthropic direct don't provide.
Compliance is an industry term. I'll save you the trouble of looking it up:
> Company compliance is the practice of ensuring an organization adheres to external laws, government regulations, industry standards, and internal company policies
Well that is a great idea, and if I were management, I would give everyone a 240€ claude-giftcard-subscription that they can use privately on their private computers. There is however the danger that they would start using their private accounts to fill with company data (which potentially is against GDPR regulations - hence the amazon bedrock deployment).
Money is money. Why is it a GDPR risk for one bank account paying for your proposed use case of toy projects and not another? I don't see how that changes the trust relationship for the team to do the right thing.
> most of their junior devwork can be replaced completely by AI - so they really should start to learn how to work with it. I recommended everybody to get a private 20€/month claude subscription
I'm struggling to wrap my head around this. "This technology will completely replace you and lead to your unemployment, so you should fund it out of your own pocket"? Is that what you're saying? If so, can you please explain your reasoning?
AI will not replace everyone. There will be people steering the AI, and using it. If you are not one of those people, you WILL be laid off. It is like office workers in the 90s thinking they don't need to learn computers and hold on to their typewriters.
Why would you want a junior steering the AI? Hell, why would any developer need to steer the AI at all? Just lay off the entire dev team and let the non-technical managers/execs tell the AI what they want directly.
"AI can do all the development work on its own, and you will be replaced by it, but I won't because my position as a meat proxy relaying instructions makes me special" is just you trying to cope with the inevitability of your own prophecy. You WILL be laid off too.
Why would you want to keep the non technical managers and execs too? Their work is way more trivial to replace once there is no dev team under them. Shareholders can just ask the AI to run a profitable business. /s?
it baffles me how some devs do not seeing that being a meat proxy reliant on the capability of the latest model makes themselves the prime candidate to be replaced .
Would enjoy chatting offline if up for it. I am an external consultant acting as more of an operations/product lead to insurance, but come the insurance side.
Curious about consulting model? Are you external (full-time) lead or fractional? Embedded in their tech and security stack or just helping organize/lead through communication channels?
A year or eighteen months ago, if you were to ask me how much I would personally pay to have access to a then-frontier model for work, I would have said: not much. They were frustrating to instruct and use. The cost of play was too high.
Six months ago, I think I would have paid $100 per month.
Today, for a current frontier model and harness with all-I-can-eat, I think I would pay a big chunk of my salary. Easily more than $2,000 per month.
Everything improved: models, harnesses, and instructions from me.
This makes me realize that I have no advantage over LLMs in general software development.
Think of using an LLM like driving a car. Easy enough, yeah? Do you say the car drove you to the airport or your friend the driver did? Let's watch four time World Champion Max Verstappen absolutely tear up 100 amateur go-kart drivers, courtesy of Red Bull: https://youtu.be/7GIULwETan4
LLMs are better than they first were when ChatGPT hit the scene in 2022, and you're not the Max Verstappen of LLM prompting or software development (it's okay, neither am I). But it's still food for thought, at least while their true capabilities are being held back by their progenitors and we're forced to ration their use.
This is a sane version of what “sovereign” AI can look like for many. In Canada we’ve just had this national plan to overpay for an inferior AI product in the name of sovereignty. Even if GLM is slightly behind, this is likely close to the Pareto frontier.
They also could consider putting this on openrouter when they have spare capacity to make the most effective use of the hardware.
At dayjob we're also using self-hosted GLM-5.3 via our flavour of pi.dev but teams still have discretionary power over using Claude Code or not (there's a team's budget to spend on whatever the team decides). Some teams like mine still like to use Claude Code for a lot of the work but most stuff doesn't require the latest and greatest LLM and can be done with "lesser" LLMS by simply throwing a lot of tokens and compute to it.
What does "blended $0.30 per 1M tokens" mean - is that input or output or average across the two? (asking as am trying to benchmark that cost against public GLM providers on Openrouter)
One would still have to guess what the ratio of input tokens to output tokens is. (I would say 10:1 off the cuff, assuming this is editing an existing codebase)
I would sacrifice a few benchmark points for higher tok/s actually, especially if more bandwidth as well so I can scale up swarms, then I can likely compensate for any drop and get more mileage out of it.
112 comments
[ 6.5 ms ] story [ 145 ms ] threadThe relevant question is how easy is to setup 56x B200s in a cluster, are we talking about a adhoc infra team job or clicking a button on AWS?
Regarding model quality GLM 5.3 is past the good enough bar for most coding tasks. I wouldn't be surprised if the limiting adoption by companies is price not frontier model quality for most of their tasks.
Besides that, since this is a bank, you can see how significant the requirements are to run sensitive workflows within your own infrastructure premises.
So they've invested around $2.5M-3M into hardware + monthly operating costs (electricity, updates, cooling). I wonder how much did they spend on tokens to decide to make such a switch. I mean if they used to spend $500k-1M per month just on tokens then it would pay off in a year or so, assuming their infra will be stable and efficient.
We'll see if that's a trend for mid-big companies or just an experiment.
All that math is assuming that it's all self-hosted in their data center or so and not paying to a cloud provider, which will be probably more expensive.
Yes, margins will go down a bit but the frontier labs still have some leverage.
At the moment at least Nvidia doesn't have a incentive to release a new product because it's sold out as it is and probably doesn't want to plunge their customers in depth with outdated products?
Are you assuming that they bought the hardware? because they could also be renting the metal off a provider, in which case they either pay by the hour or have a yearly commitment to save on costs.
I would not be surprised if, in 5 years, there's a neat, low-power GPU in high end laptops, alternatively, a cheap, say 500 USD or so, GPU in a box, that you can use for your AI-center at home or in the office.
I'm currently running Qwen3 Coder Next, which is a 80B A3B model, at 15tps on a 8GB 7700S and 64GB RAM.
It's an idiot (no Thinking), so it has to go over its work several times to get it right, but it's the largest idiot I can feasibly run at the moment.
Also by accident I'm undercutting the only provider of this model on OpenRouter by 40%, doing $0.48/mTok.
Of course at my token rate it would take an entire day to even reach a million tokens, but it's the principle that matters.
And we haven't even scratched the surface on highliy-specialized distillations. All those generall-purpose open-weight model will see distilled variations for their respective industry niches.
But the goal posts will move and I think frontier labs will find new areas that have a need for absolute frontier.
Guess who else has that? The Chinese companies developing open weight models. In fact, the vast majority of AI science comes from China. They might still be playing catch up in some areas, but they're also leading in others (famously DeepSeek's inference architecture as an example).
There is IMO a non-trivial chance those labs will eventually surpass the US ones and become themselves "frontier", and that may happen sooner than a lot of people expect.
Of course, it still remains to be seen if they will keep publishing the weights at that point.
At some point, they'll stop releasing models for free and will need a license.
Is this a guess, or do you have any information to back that inference (no pun intended)?
There are already rumblings of CCP wanting to control export of top tier frontier models.
I mean if they don't then the US companies could just release their weights and chop the Chinese off at the knee's at that point (reversing what is currently happening).
Opening the weights isn't just a technical thing, it's a geopolitical decision which likely in part comes from the top of the CCCP (by either request or just implicitly allowing it).
On both frontiers and open. And lately the gap has been closing so if anything yeeting Anthropic etc is getting more viable over time
And they will keep it to themselves to extract the value for as long as possible.
Has abliteration/decensoring been at all useful or relevant in your setup?
The last cause might be a straight up blocker - certain financial regulations straight up prevent you from giving third parties to your data (esp outside national/EU jurisdiction), no matter what documents you sign.
They are less good on ambiguous prompts like 'fix all the problems' (which to my chagrin is a very common prompt it turns out..)
To give one example: I'm doing a lot of stuff that involves hardware. When settings things up for fixing a bug I have to make sure to include that the hardware has been tested and works well. If I don't do that the model will almost certainly spend a ridiculous amount of time speculating about all the different ways in which the hardware could be broken (including theorizing about chip mask errors and all kinds of other obscure and rarely seen in practice defects). It's going to end up telling itself that for sure the 500 lines it just produced are absolutely perfect and the device that is on the i2c bus is broken in a way that it just really can't progress with solving the issue because there is nothing wrong with the code.
These kinds of blind-spots are funny when they happen once in a month but when they happen regularly it begins to be irritating. Adding a line or two to the prompt will take care of the issue. Frontier models tend to be better at this sort of thing.
At the same time, most of their junior devwork can be replaced completely by AI - so they really should start to learn how to work with it. I recommended everybody to get a private 20€/month claude subscription and use it for toy projects. I have younger devs (early 20s) in the team that told me they won't spend private money, or simply have no usecase in their private life for AI. We talk about software engineers here, and IMHO most of them are in denial of the fact that their skills have mostly become worthless if they don't start to pivot to something valueable.
Now, within the last 6-12 months of AI advancement, I think you could lay off half or even 3/4 of the team if I was explicitly allowed to implement features myself using AI. But that is not the current policy.
As for worries "ending up on a codebase you can't manage without AI" - my personal take is: No, I am not worried. Because going forward AI will always be available in the development workflow. That is my own opinion, not the corporation standpoint, however.
So your AI subscription is basically a subscription to your own code. Are you sure the tokens will always flow for you?
A way bigger problem is a codebase that you can't effectively work on with AI, which is a lot of legacy humanslop. And things that make a codebase more suited for AI, like documentation that provides relevant domain knowledge and context for requirements, automated quality control, and test coverage, also make a codebase easier to work on for a human too.
And that whole "legacy humanslop" paragraph is just stating the obvious with an air of spite towards humans.
People who can't bother to reason get angry when you say "hey, why put all eggs into one basket", episode N. Boring.
It's not even a matter of incompetence, most of the humanslop came from the fact that you could just never implement anything properly in the time you were expected to do it.
But your comment says it several time: the valuable skill is being already a seasoned dev.
Then AI is superpower.
It's for the junior that things sucks big times.
I see it all around me: senior devs are empowered while juniors are not even hired anymore.
That's about 20€/month more than I care to give Anthropic. They already stole my data and now they want my money too? No thanks. I'll stick to running Gemma or Qwen on my Mac.
> Company compliance is the practice of ensuring an organization adheres to external laws, government regulations, industry standards, and internal company policies
I'm struggling to wrap my head around this. "This technology will completely replace you and lead to your unemployment, so you should fund it out of your own pocket"? Is that what you're saying? If so, can you please explain your reasoning?
"AI can do all the development work on its own, and you will be replaced by it, but I won't because my position as a meat proxy relaying instructions makes me special" is just you trying to cope with the inevitability of your own prophecy. You WILL be laid off too.
Here is €1B in initial capital. Go!
Curious about consulting model? Are you external (full-time) lead or fractional? Embedded in their tech and security stack or just helping organize/lead through communication channels?
A year or eighteen months ago, if you were to ask me how much I would personally pay to have access to a then-frontier model for work, I would have said: not much. They were frustrating to instruct and use. The cost of play was too high.
Six months ago, I think I would have paid $100 per month.
Today, for a current frontier model and harness with all-I-can-eat, I think I would pay a big chunk of my salary. Easily more than $2,000 per month.
Everything improved: models, harnesses, and instructions from me.
This makes me realize that I have no advantage over LLMs in general software development.
LLMs are better than they first were when ChatGPT hit the scene in 2022, and you're not the Max Verstappen of LLM prompting or software development (it's okay, neither am I). But it's still food for thought, at least while their true capabilities are being held back by their progenitors and we're forced to ration their use.
They also could consider putting this on openrouter when they have spare capacity to make the most effective use of the hardware.
I have no clue how do the lunatics think there's any moat in models.
One would still have to guess what the ratio of input tokens to output tokens is. (I would say 10:1 off the cuff, assuming this is editing an existing codebase)
My whole company (Amazon) has unlimited tokens, and the only thing it has led to is insane amounts of slop. Illusion of productivity.