That's because they are. Ever since the mining crash Nvidia has been selling all its GPUs for exorbitant prices, much to the detriment of people who need the VRAM for machine learning. Oh well, guess you're buying the (checks notes) $13,000 A100, just because you need 3.3 times the VRAM of the $1600 4090 (which could be $5280 instead, if price corresponds exactly to VRAM).
At the risk of sounding like a broken record: Once others get their software shit together, they can have some of those profits. Nvidia has had a competent software game for a decade.
They still haven't gotten their hardware shit together. As a consumer I can't buy a PCIe card that slots into my system with more than 24GB of VRAM, whether it's from Nvidia, AMD, or Intel. Nvidia's prices will continue to be high while there is literally no competition.
I guess before datacenter class cards can trickle down to the consumer, they do have to get their software shit together first.
If I go to the page for the W7900, I can download the datasheet. I can look at the marketing pages. I can look at the specifications and download drivers. But I can't buy the card.
If I look for one on eBay, they start at $4600, which is still far too expensive for 48GB. At that point, just buy an Apple Silicon machine and you can get far more memory than that for even cheaper. (M3 Max MacBook Pro with 96GB memory is just $4,000!)
Alright, let's try the RTX A6000! Oop, it's $4,000. Cheaper than AMD, but wow! Oh, and I can only find these prices on eBay, you can't buy them direct from Nvidia as a consumer.
I'm comparing the cost of a GPU with 48GB VRAM to the cost of an Apple Silicon machine with at least 48GB of Unified Memory, yes. (The ASi machine having 96GB of memory instead is just because it's still cheaper than the card, so why not.)
A normal PC with 96GB of regular RAM wouldn't allow you to use that RAM for GPU workloads, at least not with a dedicated GPU.
My comment was made 10 hours before that reply, so I didn't have that context. Sorry.
> Would an ASi machine and Dedicated GPU with the same amount of respective RAM have the same performance compute-wise?
Not sure, but performance comes after making sure you have enough VRAM to run your workload in the first place. I doubt an ASi chip would beat most datacenter-class training cards, but if a consumer just wants to train a model locally without much regard for getting the best possible performance, there is more value-for-money.
> My comment was made 10 hours before that reply, so I didn't have that context. Sorry.
Sorry, i phrased that poorly, i was referencing my own comment, not because i thought you had ignored it but because i had just replied to the other comment.
It probably shouldn't have reference the other reply.
> Not sure, but performance comes after making sure you have enough VRAM to run your workload in the first place. I doubt an ASi chip would beat most datacenter-class training cards, but if a consumer just wants to train a model locally without much regard for getting the best possible performance, there is more value-for-money.
That mostly makes sense.
Though personally,if i'm dropping car money on hardware to run a model then i'd really want to know the type of difference in power, are we talking about the ASi taking twice as long or are we talking orders of magnitude?
Not a question for you specifically, just a thought.
I'll see if i can find any benchmarks for the differences in compute.
NVidia primarily doesn't sale to consumers. You'll have to get from 3rd-party like PNY. It is costly because they have customers lining up to pay that price. If demand goes down price will follow.
Well, my (totally made-up) calculation conservatively assumes that multiplying the VRAM multiplies the price by the same amount. In practice I don't think it's even that expensive to add more VRAM, since most of the rest of the card should stay the same?
Market power just means having a negotiating position that allows you to determine prices, regardless of how that negotiating position is obtained. It can either come from illicit means (say cartels for example), or from true innovation and investing into the right things.
I think that both the customers of Nvidia as well as the competition are keen on replacing it (given the scale that many of the big customers work at, I'm sure they can afford the engineers to migrate the LLM models to ROCm if it were a viable alternative), but it seems that they cannot manage to make replacements fast enough.
> The most interesting detail is that Nvidia managed to double sales without incurring additional costs
market power also comes from a material log jam because we can't make GPUs fast enough to satisfy demand, so the wealthiest customers choose the price because the GPUs built essentially go to auction.
"In economics, market power refers to the ability of a firm to influence the price at which it sells a product or service by manipulating either the supply or demand of the product or service to increase economic profit.[1] In other words, market power occurs if a firm does not face a perfectly elastic demand curve and can set its price (P) above marginal cost (MC) without losing revenue."
> Nvidia is now the tech equivalent of an oil producer during an energy shortage, seemingly able to charge what it wants and pocket outrageous profits
The other "oil producer", AMD, does not make a good competition. All cheap graphics cards are Nvidia.
This is the same situation like with Windows: when everyone use it at home (pirated), or school (free), they will use it also at work ($$$).
If you watched any serious review last year you'll notice the AMD GPUs were consistently better value than their Nvidia counterparts. Though the 7000 generation was still very disappointing regardless.
That was a similar business model of Sun Microsystems and SunOS/Solaris operating system, except it failed to be a business desktop OS for the workplace... and then Oracle came along and murdered any chance of that idea taking effect at a later date.
The gap between Nvidia and AMD isn't that big. At the moment if you want to be involved with cutting edge AI you have to go with Nvidia. There is no evidence at all what the situation will be like in 5 years though.
Not to mention that the clock is ticking until other market players start really responding. The situation is very fluid and Nvidia was ready for it, but they haven't discovered a magic formula that lets them overcome commodity economics.
Yes, but unlike behemoths like Broadcom who have successfully locked down certain types of software through patent-fu, Nvidia does not (currently) have a way to stop AMD from creating their own software.
This means that with enough incentives - and there are plenty right now - the competition can reasonably catch up in the next 2-3 years.
Sigh. I mean may be HN want to take a look at [1] NVIDIA Profit Margin 2010-2023 | NVDA, which include Net Profit Margin, Gross Margin as well as Operating Margin and make up their own mind. People tends to have very short memory, so many may also want to Google what happened to Nvidia in 2022 and what it did to their stock price.
38 comments
[ 3.8 ms ] story [ 90.8 ms ] threadMarket power implies it’s because nvidia is using methods other than better products to keep its prices high.
I guess before datacenter class cards can trickle down to the consumer, they do have to get their software shit together first.
They just want more money.
If I go to the page for the W7900, I can download the datasheet. I can look at the marketing pages. I can look at the specifications and download drivers. But I can't buy the card.
If I look for one on eBay, they start at $4600, which is still far too expensive for 48GB. At that point, just buy an Apple Silicon machine and you can get far more memory than that for even cheaper. (M3 Max MacBook Pro with 96GB memory is just $4,000!)
Alright, let's try the RTX A6000! Oop, it's $4,000. Cheaper than AMD, but wow! Oh, and I can only find these prices on eBay, you can't buy them direct from Nvidia as a consumer.
Because if so you can get a good pc with 96GB of machine ram for significantly less than $4000.
Unless the claim is that an M3 Max MacBook has equivalent GPU type compute power as a dedicated GPU?
A normal PC with 96GB of regular RAM wouldn't allow you to use that RAM for GPU workloads, at least not with a dedicated GPU.
Would an ASi machine and Dedicated GPU with the same amount of respective RAM have the same performance compute-wise?
My comment was made 10 hours before that reply, so I didn't have that context. Sorry.
> Would an ASi machine and Dedicated GPU with the same amount of respective RAM have the same performance compute-wise?
Not sure, but performance comes after making sure you have enough VRAM to run your workload in the first place. I doubt an ASi chip would beat most datacenter-class training cards, but if a consumer just wants to train a model locally without much regard for getting the best possible performance, there is more value-for-money.
Sorry, i phrased that poorly, i was referencing my own comment, not because i thought you had ignored it but because i had just replied to the other comment.
It probably shouldn't have reference the other reply.
> Not sure, but performance comes after making sure you have enough VRAM to run your workload in the first place. I doubt an ASi chip would beat most datacenter-class training cards, but if a consumer just wants to train a model locally without much regard for getting the best possible performance, there is more value-for-money.
That mostly makes sense.
Though personally,if i'm dropping car money on hardware to run a model then i'd really want to know the type of difference in power, are we talking about the ASi taking twice as long or are we talking orders of magnitude?
Not a question for you specifically, just a thought.
I'll see if i can find any benchmarks for the differences in compute.
Thanks
I'm not aware of any benchmarks comparing Apple Silicon to datacenter-class chips, unfortunately.
I think that both the customers of Nvidia as well as the competition are keen on replacing it (given the scale that many of the big customers work at, I'm sure they can afford the engineers to migrate the LLM models to ROCm if it were a viable alternative), but it seems that they cannot manage to make replacements fast enough.
> The most interesting detail is that Nvidia managed to double sales without incurring additional costs
https://en.wikipedia.org/wiki/Market_power
The other "oil producer", AMD, does not make a good competition. All cheap graphics cards are Nvidia. This is the same situation like with Windows: when everyone use it at home (pirated), or school (free), they will use it also at work ($$$).
If you watched any serious review last year you'll notice the AMD GPUs were consistently better value than their Nvidia counterparts. Though the 7000 generation was still very disappointing regardless.
Not to mention that the clock is ticking until other market players start really responding. The situation is very fluid and Nvidia was ready for it, but they haven't discovered a magic formula that lets them overcome commodity economics.
This means that with enough incentives - and there are plenty right now - the competition can reasonably catch up in the next 2-3 years.
Then how long after that for similar levels of adoption among devs?
The fact their margins are so high is a measure of how much their competitors thought that Nvidia was wrong.
The funny thing to me is that crypto crashed immediately before LLMs exploded, WTF!?!
A timeline where they both simultaneously happen or LLMs come first would have changed the future quite significantly.
[1] https://www.macrotrends.net/stocks/charts/NVDA/nvidia/profit...