The comments are so ignorant it's unreal. I think the OpenAI team is doing their best. I've heard it's actually hard to scale an app to 100+ million users
I think it’s because people associate OpenAi with Microsoft so they think that one of the largest cloud computing providers in the world ought to be able to scale to these numbers.
Imagine people saying 'Google is doing their best' whilst GCP goes down every week with startups like Anthropic and others getting investment from Google.
It is also like accepting the hilarious availability times of GitHub and then saying 'GitHub is doing their best' whilst they have an outage every week.
Lets just say that a service that claims to scale to more than 50 million or 100 million users going down each week is not acceptable or being an apologist for large companies unable to scale with the so-called best engineers working for them.
GPT3 has 175 billion parameters and Altman said 4 would use way more compute than 3.
Let's just look at GPT 3. Each forward pass requires 2N=350B flops per token. The computational overhead of the attention mechanism is negligible but the memory overhead of the attention for all users is not, but we can ignore that for now. Let's assume each query involves ~200 tokens of combined input and output. That's 70T flops per query. Let's say the cost of compute is 1 cent per 100B flops (probably a lowball). That's $7 per query for compute alone.
I mean, that might be the cost for a onesie-twosie in Azure, but if you're microsoft and you own the hardware, the cost may be one or more orders of magnitude less than that. (Of course, there's just a limit on the number of GPUs that exist and that Nvidia can pump out.)
Microsoft's profit margin for "intelligent cloud" in their most recent earnings was less than 50%. Impressive, but not nearly enough to make GPT subscriptions make financial sense.
And a lot of that is marketing, keeping unused capacity in reserve, etc. Microsoft can probably afford a very low price without losing money if they price the compute at-cost. Certainly less than $7/request. 70 teraflop per request, and an $8000 GPU, with $2k of server overhead, $10k in power and cooling costs, $20k total for 312 teraflops of compute, leased for 2 years gives you an at-cost hardware price of $0.00007/request: https://www.google.com/search?q=%2420000*70/(312*3600*24*365...
So even assuming just a 1% utilization rate, you’re still talking less than 1 cent per request.
Remember, inference runs at reduced precision. FP16 (or even just 8 bit) is WAY cheaper than fp64 bit. More than an order of magnitude cheaper. 9.7Tflops vs 312Tflops, factor of 32 improvement.
With sparse matrix and 8-bit precision, you can effectively get over 100x the performance on an A100 than with FP64, plus the memory requirements are more than an order of magnitude less. Factor of 128 improvement in processing speed plus at least a factor of 8-16 or more improvement in tokens per memory, depending on sparsity. (Sparsity and reduced precision do reduce performance for the same number of operations and weights, but the overall effect on performance from reducing precision is beneficial probably at least to 4-bit quantization… You get diminishing or negative returns if you go less than that with current architectures, but in principle we can keep going.)
I think your compute cost figure is both very old and probably assumes double precision whereas GPT-3 uses at most FP16 at inference and GPT-4 may go beyond that.
Most obviously this part must be off by 3-4 orders of magnitude:
> Let's say the cost of compute is 1 cent per 100B flops (probably a lowball)
That's like 0.5ms worth of compute on A100. VMs with an A100 cost a few dollars per hour at cloud providers, i.e. about a cent per second or 0.001 cents per ms. OpenAI would obviously get a far better deal than somebody renting a single GPU for an hour.
Look, we know what OpenAI is charging for their commercial service. We don't know how it maps to their actual serving costs; maybe they're breaking even, making a small profit, or making a small loss. But just basic common sense should tell you that they are not operating at -10000% gross margins.
I suspect that just working backwards from the prices to a reasonable range of gross margins is going to produce much better estimates than your methodology would even after the inputs are tuned to be more realistic.
Basic common sense would tell you that starting with an assumption and working backwards to justify that assumption is called succumbing to confirmation bias.
I'll acknowledge that a fleet entirely made up of A100 GPUs running at full capacity 100% of the time with Azure's 3-year commitment pricing (likely very low margin) would make this 5 orders of magnitude cheaper. But the sequential nature of transformers makes it difficult to achieve perfect utilization, and there aren't enough A100 GPUs in the Azure fleet to serve ChatGPT's billion users, so they're certainly running on mostly lesser hardware. Also, my estimate was for GPT-3 not 4, and neglected to address any memory or bandwidth costs, not to mention the costs to train and iterate on the models in the first place, including legions of data curators and many researchers commanding million dollar salaries.
If ChatGPT plus is $20/mo and each user uses it ~10 times per day, then MS/OpenAI would break even if my estimate were 2 orders of magnitude too high. Within the year, we'll have a much better idea of the real cost of these models.
You can be sure Open AI isn't even paying Azure's punlishedy 3-year commitment pricing rate.
The other thing to note is that OpenAI limits paying users to X queries per Y hours in GPT-4, something like 3 per 5. Which means the GPT-4 fleet is quite limited.
Uhh... I'm pretty sure that page is about the price of buying an entire GPU that does N GFLOP/S in perpetuity. It has no relevance to the calculation you did, unless you assume the GPU is used for one second and then thrown in the bin.
> I'll acknowledge that a fleet entirely made up of A100 GPUs running at full capacity 100% of the time with Azure's 3-year commitment pricing (likely very low margin) would make this 5 orders of magnitude cheaper. But the sequential nature of transformers makes it difficult to achieve perfect utilization,
I don't think you've thought this through. There's no realistic level of utilization that's going to make up for 5 orders of magnitude. An unimaginably crappy 1% utilization would still leave you off by 3 orders of magnitude.
This very submission about OpenAI having to suspend selling subscriptions makes it pretty clear they're not having trouble with low utilization...
> If ChatGPT plus is $20/mo and each user uses it ~10 times per day, then MS/OpenAI would break even if my estimate were 2 orders of magnitude too high.
You do understand that this doesn't actually support your initial estimate, right? It just makes it more obvious how ridiculous it was.
They are just giving away GPT-3 grade queries. In terms of revenue, that's NaN% gross margin! Thus, -10,000% gross margin is quite the improvement. While I personally don't think that they are spending quite that much on compute, burning investor capital to build a customer base is a time honored tradition. Which means we can't really deduce that because -10,000% is such a big number, that can't possibly be their compute costs.
If you ask ChatGPT what 1/0 is, you get a paragraph as to why it's undefined. Nowhere does it use the phrase "NaN". Perhaps you need to adjust your humor filter, LLMs aren't quite there enough to be able to use NaN% in context as a joke like that.
Also:
> Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something.
It really shouldn't use the word NaN in any case, since 1/0 would be +Inf. Meanwhile, here's what GPT-4 had to say (after refusing to talk about percentages at all at first because it is "impossible to calculate"):
>> ... suppose you're describing this to a technical audience - software engineers, specifically. There might be some way to talk about percentages concisely in that particular context.
> In that case, you could describe the situation to a technical audience by saying, "The company's profit margin is effectively -∞% due to offering the product for free while incurring expenses."
“But just basic common sense should tell you that they are not operating at -10000% gross margins.”
I’ll be honest, I laughed out loud at this.
Pretty sure I read somewhere that Bing had around 1 billion searches per day BEFORE the hype of bingchat. At $7/query that makes MSFT’s $10 billion investment basically peanuts. Less than 2 days of Bings GPT usage payments. Lmao.
If they charged 1 cent per 100B flops that pricing would be $35.000 / 1K tokens. They may be taking a loss, but I doubt it would be at a 1 to 17,500 ratio.
Even GPT-4 with 32k token window is only $0.12 / 1K tokens
This is why I laugh when people say AI can be factually incorrect. Just look at this comment made by a human!
I've heard from insiders that the cost is roughly a couple cents per ChatGPT message but that was back with the legacy GPT-3.5 model, not the Turbo model or GPT-4
>Let's say the cost of compute is 1 cent per 100B flops (probably a lowball).
I might be totally misunderstanding this, but from the source you posted elsewhere[1], GPU Sale Price per Flops is as a little bit higher than 1*10^-11 dollars per flop, or 10^11 flops per dollar, or 10^9 flops per cent. So 1GFlops per cent. But Flops are Floating point operations per second. So you aren't paying 1cent for 100B flops, you're paying 1 cent for a card capable of doing 100B floating point operations per second. So let's say the card lasts 3 years and runs at 100% utilization (I know, it doesn't we can multiply by some efficiency metric later), there's about 9.5*10^7 seconds in 3 years. So that dollar cost isn't for 100B flops, it's for 100B * 9.5 *10^7 ( *some efficiency number). So it's many orders of magnitude cheaper than you're saying.
In reality, the cost of running these things isn't the cost of buying the GPUs, it's the cost of the power to run the GPUs, and that's a totally different calculation we can do.
There's no way it's profitable. I'd wager this will materially affect Microsoft's next earnings report. Hard to tell how open ai itself is affected financially by the explosive growth but a rude awakening is coming for many users and shareholders alike
They're footing the bill for this whole operation. They provide the compute and storage and they're also so heavily invested in OpenAI after their recent $10B round that they can't offload the costs. See my other comment on this post for a conservative cost estimate for each query.
It's not necessarily a bad move to get them back in the game after many years of being useless. Deepmind is one of only a few thin threads that may hold Google together in the coming years, but it really doesn't look good for them
It won't show up on their financials I reckon, they made equity investment into OpenAI - their P&L won't show on Microsoft's. If they have a sweetheart deal with MSFT, Azure revenues aren't going to change. Even if they built a new DC or bought a boatload of GPUs for OpenAI, it will get mixed into their massive Capex and they probably are using existing infrastructure
Creating credits out of thin air is impossible as all financial transactions must be accounted for. In the case of Azure credits, they will appear as increased revenue on Microsoft's balance sheet. However, if Microsoft decides to price Azure services below cost as part of their agreement with OpenAI, their profit margins will be affected. I highly doubt that is the case because the alleviation of cost is in the form of credits.
I would wager against comment 0 that said that this will materially affect earnings.
It will, hence the MSFT layoffs happening at the same time.
People here wanted Google scale of operations and replacing Google with ChatGPT as their search engine.
But it seems that some here have forgotten about the high availability requirements and incalculable operational costs of running a search engine. Google.com's availability is a very high barrier of serving in the multi-billions of searches a day.
Azure has lots of experience in unreliability and running services at a massive loss whilst minting credits to compete on cost.
You realise that openAI is just one customer of Azure, the second largest cloud provider in the world. How much revenue do you think they are forgoing? 50m is a rounding error there, their guidance for next quarters revenue has a range of a billion
The value to microsoft is in the marketing around a premier training company using their hardware... their actual hardware in this case is a bespoke supercomputer, but performance watchers like me are paying attention. From what I can tell, of the top 3 cloud providers, azure is doing the best work with high speed interconnects right now.
But what are the workloads costing MSFT? If the credits are part of the acquisition deal, maybe they're considered capex not opex, and MSFT gets to amortize the credits' operating cost over decades.
What is the true marginal cost to MSFT for gpt workloads anyway? As long as it's not bumping paying customers then it's what the cost of electricity?
It probably is. I'd bet for 99.99% of their users, simply using the API directly would be much cheaper. The ChatGPT premium subscription is overpriced if you compare it to the cost of each API call.. at least according to OpenAI's pricing.
Yeah, I'm waiting for one more month for the API-bots to mature. Then I'll cancel my ChatGPT premium subscription and put up a private Discord server just for myself and a GPT4 bot.
It’s pretty easy to spend a couple of dollars with a few interactions with the API. I tried one of those langchain type agents and it spent $2 in 10 minutes. My guess is they make more money per user from the API than they do from ChatGPT.
I have found Bing and Bard a lot more efficient for tasks where you need to look stuff up on the web, probably because they both have local access to a crawler index. They do tend to need more guidance though.
I'm sure serious competitors are just around the corner, at least Bard will catch up, No doubt Apple has something in the works, there's little to zero doubt about that.
So do they make a huge capital investment hoping that growth will continue at this trajectory, or do they back off the gas pedal ?
I bet there's also some contentious ideas going on internally too. If they go on a huge hiring spree, does that kind of invalidate the product which is supposed to replace everyone? How do the optics look there? Do they hire and fire once GPT-5 becomes a sentient AGI?
Google already has a huge engineering team, operates are unprecedented scale, and IMO has a much more diverse set of products to integrate with, they also solve a lot of novel solutions which they require specialized engineers that can't be replaced easily by "AI".
Be interesting to see how it plays out. Maybe a victim of their own success ?
The cynic in me sees they are making a vendor lock-in play, the API is dirt cheap so much so that every other bespoke chatbot or language model is now obsolete - every chat startup is now switching to OpenAI.
Once everyone has their prompts and embeddings tuned and is serving customers, OpenAI can dial up the price and customers chose to keep paying or redo their prompts/embeddings to switch to Bard or the next competitor. It does seem like business as usual if you relied on GPT-3 now ChatGPT, then whatever iteration comes next, probably won't work the same, so its not a heavy heavy lift. Decent platform risk, I believe OpenAI shutdown Codex so now everyone who had a product based on Codex has to switch to ChatGPT
From my experience with their Plus plan GPT 4 usually just times out no matter what the prompt, forcing me to revert to 3.5. I'm not sure why I haven't asked for a refund yet.
As for Bing, despite the claims that it is GPT-4, it's clearly inferior to what OpenAI is offering. It's probably an older iteration of the model, and I wonder if it might also be a scaled-down version to run it cheaper at scale.
The Playground certainly does have GPT-4. I've been using it there for a couple of weeks. Maybe you need to have applied to the beta and been given access?
As for Bing? Who knows what's going on behind the scenes. I've read that the "Creative" mode uses GPT-4 while the others might use a faster model, but that may well be nonsense.
Ah, indeed, that access is still rather limited; most can only have a peek a GPT-4 through ChatGPT Plus.
As for Bing, I don't know what's behind the scenes, but it does noticeably worse on tasks in "Creative" mode than ChatGPT backend in GPT-4 mode, in my limited experiments. For example, try this:
> A is 1m left of B, B is 1m above C, D is 1m right of C, E is 1m below D, and E is 1m right of F. Where is F located relative to C?
GPT-4 can usually solve this correctly. Bing is usually wrong even when it tries to solve it step by step (and it often won't unless you prompt it).
Maybe it depends on time of day? I've been using GPT-4 mostly in the evenings CET and it's been responsive and fast the whole time, I've never seen a timeout.
62 comments
[ 2.7 ms ] story [ 135 ms ] threadcommit author: GPT-4
commit message: ":rocket_emoji: Set all Azure clusters to autoscale in order to resolve flooded message queue"
It is also like accepting the hilarious availability times of GitHub and then saying 'GitHub is doing their best' whilst they have an outage every week.
Lets just say that a service that claims to scale to more than 50 million or 100 million users going down each week is not acceptable or being an apologist for large companies unable to scale with the so-called best engineers working for them.
> I bought it and it was removed off of my account. I paid money for a feature I never got. Anyone else get this?
It look like a very bad experience. if i were him, i would issue a credit card charge back
Let's just look at GPT 3. Each forward pass requires 2N=350B flops per token. The computational overhead of the attention mechanism is negligible but the memory overhead of the attention for all users is not, but we can ignore that for now. Let's assume each query involves ~200 tokens of combined input and output. That's 70T flops per query. Let's say the cost of compute is 1 cent per 100B flops (probably a lowball). That's $7 per query for compute alone.
And that's just GPT 3.
With sparse matrix and 8-bit precision, you can effectively get over 100x the performance on an A100 than with FP64, plus the memory requirements are more than an order of magnitude less. Factor of 128 improvement in processing speed plus at least a factor of 8-16 or more improvement in tokens per memory, depending on sparsity. (Sparsity and reduced precision do reduce performance for the same number of operations and weights, but the overall effect on performance from reducing precision is beneficial probably at least to 4-bit quantization… You get diminishing or negative returns if you go less than that with current architectures, but in principle we can keep going.)
https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Cent...
I think your compute cost figure is both very old and probably assumes double precision whereas GPT-3 uses at most FP16 at inference and GPT-4 may go beyond that.
> Let's say the cost of compute is 1 cent per 100B flops (probably a lowball)
That's like 0.5ms worth of compute on A100. VMs with an A100 cost a few dollars per hour at cloud providers, i.e. about a cent per second or 0.001 cents per ms. OpenAI would obviously get a far better deal than somebody renting a single GPU for an hour.
Look, we know what OpenAI is charging for their commercial service. We don't know how it maps to their actual serving costs; maybe they're breaking even, making a small profit, or making a small loss. But just basic common sense should tell you that they are not operating at -10000% gross margins.
I suspect that just working backwards from the prices to a reasonable range of gross margins is going to produce much better estimates than your methodology would even after the inputs are tuned to be more realistic.
I sourced my cost/flop estimate from here: https://aiimpacts.org/2019-recent-trends-in-gpu-price-per-fl...
I'll acknowledge that a fleet entirely made up of A100 GPUs running at full capacity 100% of the time with Azure's 3-year commitment pricing (likely very low margin) would make this 5 orders of magnitude cheaper. But the sequential nature of transformers makes it difficult to achieve perfect utilization, and there aren't enough A100 GPUs in the Azure fleet to serve ChatGPT's billion users, so they're certainly running on mostly lesser hardware. Also, my estimate was for GPT-3 not 4, and neglected to address any memory or bandwidth costs, not to mention the costs to train and iterate on the models in the first place, including legions of data curators and many researchers commanding million dollar salaries.
If ChatGPT plus is $20/mo and each user uses it ~10 times per day, then MS/OpenAI would break even if my estimate were 2 orders of magnitude too high. Within the year, we'll have a much better idea of the real cost of these models.
The other thing to note is that OpenAI limits paying users to X queries per Y hours in GPT-4, something like 3 per 5. Which means the GPT-4 fleet is quite limited.
It also keeps saying "expect significantly lower caps, as we adjust for demand", but it's been three weeks now and they haven't lowered it yet.
> I'll acknowledge that a fleet entirely made up of A100 GPUs running at full capacity 100% of the time with Azure's 3-year commitment pricing (likely very low margin) would make this 5 orders of magnitude cheaper. But the sequential nature of transformers makes it difficult to achieve perfect utilization,
I don't think you've thought this through. There's no realistic level of utilization that's going to make up for 5 orders of magnitude. An unimaginably crappy 1% utilization would still leave you off by 3 orders of magnitude.
This very submission about OpenAI having to suspend selling subscriptions makes it pretty clear they're not having trouble with low utilization...
> If ChatGPT plus is $20/mo and each user uses it ~10 times per day, then MS/OpenAI would break even if my estimate were 2 orders of magnitude too high.
You do understand that this doesn't actually support your initial estimate, right? It just makes it more obvious how ridiculous it was.
Also:
> Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something.
https://news.ycombinator.com/newsguidelines.html
If you're going to accuse me of using ChatGPT, at least bother to say why.
>> ... suppose you're describing this to a technical audience - software engineers, specifically. There might be some way to talk about percentages concisely in that particular context.
> In that case, you could describe the situation to a technical audience by saying, "The company's profit margin is effectively -∞% due to offering the product for free while incurring expenses."
I’ll be honest, I laughed out loud at this.
Pretty sure I read somewhere that Bing had around 1 billion searches per day BEFORE the hype of bingchat. At $7/query that makes MSFT’s $10 billion investment basically peanuts. Less than 2 days of Bings GPT usage payments. Lmao.
If they charged 1 cent per 100B flops that pricing would be $35.000 / 1K tokens. They may be taking a loss, but I doubt it would be at a 1 to 17,500 ratio.
Even GPT-4 with 32k token window is only $0.12 / 1K tokens
I've heard from insiders that the cost is roughly a couple cents per ChatGPT message but that was back with the legacy GPT-3.5 model, not the Turbo model or GPT-4
I might be totally misunderstanding this, but from the source you posted elsewhere[1], GPU Sale Price per Flops is as a little bit higher than 1*10^-11 dollars per flop, or 10^11 flops per dollar, or 10^9 flops per cent. So 1GFlops per cent. But Flops are Floating point operations per second. So you aren't paying 1cent for 100B flops, you're paying 1 cent for a card capable of doing 100B floating point operations per second. So let's say the card lasts 3 years and runs at 100% utilization (I know, it doesn't we can multiply by some efficiency metric later), there's about 9.5*10^7 seconds in 3 years. So that dollar cost isn't for 100B flops, it's for 100B * 9.5 *10^7 ( *some efficiency number). So it's many orders of magnitude cheaper than you're saying.
In reality, the cost of running these things isn't the cost of buying the GPUs, it's the cost of the power to run the GPUs, and that's a totally different calculation we can do.
[1]: https://aiimpacts.org/2019-recent-trends-in-gpu-price-per-fl...*
How?
I would wager against comment 0 that said that this will materially affect earnings.
People here wanted Google scale of operations and replacing Google with ChatGPT as their search engine.
But it seems that some here have forgotten about the high availability requirements and incalculable operational costs of running a search engine. Google.com's availability is a very high barrier of serving in the multi-billions of searches a day.
Azure has lots of experience in unreliability and running services at a massive loss whilst minting credits to compete on cost.
What is the true marginal cost to MSFT for gpt workloads anyway? As long as it's not bumping paying customers then it's what the cost of electricity?
And save a ton of money at the same time.
I have found Bing and Bard a lot more efficient for tasks where you need to look stuff up on the web, probably because they both have local access to a crawler index. They do tend to need more guidance though.
I'm sure serious competitors are just around the corner, at least Bard will catch up, No doubt Apple has something in the works, there's little to zero doubt about that.
So do they make a huge capital investment hoping that growth will continue at this trajectory, or do they back off the gas pedal ?
I bet there's also some contentious ideas going on internally too. If they go on a huge hiring spree, does that kind of invalidate the product which is supposed to replace everyone? How do the optics look there? Do they hire and fire once GPT-5 becomes a sentient AGI?
Google already has a huge engineering team, operates are unprecedented scale, and IMO has a much more diverse set of products to integrate with, they also solve a lot of novel solutions which they require specialized engineers that can't be replaced easily by "AI".
Be interesting to see how it plays out. Maybe a victim of their own success ?
Once everyone has their prompts and embeddings tuned and is serving customers, OpenAI can dial up the price and customers chose to keep paying or redo their prompts/embeddings to switch to Bard or the next competitor. It does seem like business as usual if you relied on GPT-3 now ChatGPT, then whatever iteration comes next, probably won't work the same, so its not a heavy heavy lift. Decent platform risk, I believe OpenAI shutdown Codex so now everyone who had a product based on Codex has to switch to ChatGPT
Plus, you can always use Bing Chat. It’s more locked down, but depending on what you’re doing, it’s pretty good and backed by GPT-4.
As for Bing, despite the claims that it is GPT-4, it's clearly inferior to what OpenAI is offering. It's probably an older iteration of the model, and I wonder if it might also be a scaled-down version to run it cheaper at scale.
https://ibb.co/f2KJm4z
As for Bing? Who knows what's going on behind the scenes. I've read that the "Creative" mode uses GPT-4 while the others might use a faster model, but that may well be nonsense.
As for Bing, I don't know what's behind the scenes, but it does noticeably worse on tasks in "Creative" mode than ChatGPT backend in GPT-4 mode, in my limited experiments. For example, try this:
> A is 1m left of B, B is 1m above C, D is 1m right of C, E is 1m below D, and E is 1m right of F. Where is F located relative to C?
GPT-4 can usually solve this correctly. Bing is usually wrong even when it tries to solve it step by step (and it often won't unless you prompt it).