Here’s the thing, what if memory manufacturers take this opportunity to collude and basically never reduce the price of memory below the current levels since it’s too hard for a new competitor to just rise up and undercut them? Everything I hear about is how hard and risky it is to spin up a new fab.
And by doing this, they ensure local LLMs never become feasible for the vast majority of people and AI companies solidify subscriptions forever.
Bought a second hand Dell server a week ago. The entire rig with a 12-core CPU and 32GB DDR4 ecc RAM cost as much as I'd pay to buy 64 GB of DDR RAM alone. I hope there's an end to this absurdity soon enough otherwise the pain will affect other markets too. I read the other day that PC case sales have collapsed by more than 40%.
I'm not moving past my DDR4 build (and the 32 GB of DDR4 2133 MHz backup chips I still have around from way back, before I got the current 3200 MHz ones) until the prices go back to being at least partially sane. This also means that CPU manufacturers are not getting my money (since the 5800X is fine for now) and I have no reason to get a new GPU either (though admittedly the B580 isn't perfect).
It's still unclear to me: the shortage is semiconductor boules / wafers? or the shortage is semiconductor fab process step availability?
As long as the discussion seems focused on memory, I'd suspect the latter, but if its really the semiconductor boules/wafers, then I'd expect the boule growers to profit, not the memory makers, who just pass on the cost.
I wonder why the hyperscalers aren't vertically integrating more and building their own fabs. Sure, a fab costs a billion dollars, but they're currently spending hundreds of billions of dollars purchasing chips from NVidia and others.
Fab margins are on average super thin compared to the margins of big tech companies, and come with a lot of risk because of that. It's not something they are likely to be keen to integrate.
Another guy answered it ITT. Intel did that, it’s not great because fabs are expensive and risky and it’s less risky to amortize the cost across multiple customers instead of just yourself
Everything I read seems to suggest that RAM capacity is going to grow at 20-25% a year, which just doesn't seem good enough. Even in consumer use cases, phones and laptops would benefit greatly by double the amount of RAM. And then obviously, the AI need is gigantic.
I don't see it going away. I mean, it may not grow as fast as now, but I don't see it growing away either. I get why the memory makers do not want to bankrupt themselves, but it feels like there's got to be some way to push that risk off onto model providers and other people in the ecosystem to allow us to grow ram capacity more like 50% per year.
I really don’t want to give anyone ideas, but doesn’t this make the Nvidia 5090 an unbelievably good deal right now?
The VRAM in the 5090 is only made by one country in the world.
The 50xx series is special, because its ram is so dependent on a single commodity. It’s not like a 4090 or a 3090; their VRAM chips have been around for years.
If there’s a shortage or interruption in DDR7 VRAM, it seems like every GPU that requires it would explode in value.
I hope I don’t regret posting this because I’d really like to buy one myself…
And the max storage in pre-built computers has stagnated at 2010 levels (~1TB). This was first due to the switch to the much more expensive and much faster charge trap flash. In the 2020s it finally started to approach 2010 sizes in pre-builts but then the corporate finance wars re: fab capacity happened.
for the most part, unless soldered down, it has been hard to find higher than dual channel (maybe quad for a massive odm gaming laptop). each stick and platform having set maximum memory capacity has put a glass ceiling for those machines.
doesn't matter anyway when things are not reasonably priced. i am stuck at the same memory capacity in my personal system for the better part of two decades, partially due to the above and the current pricing today.
An interesting implication of this is that AI inference and training has a path to a ~3x hardware cost reduction (and maybe ~2x total cost reduction) without any technical innovation whatsoever, we just need to wait for dram supply to meet demand (either by manufacturing scaling or just waiting for the current rate of manufacturing to fill the demand spike).
If you factor in Nvidia’s profit margin due to the scarcity of the current bleeding-edge chips there is a path to a much larger cost reduction still.
There’s a lot to criticize Sam Altman for saying or popularizing culturally but I’ve come to think his “this is the worst it will ever be” is, in the long run, actually a very intriguing and underrated point.
In a decade training LLMs to the current level of sophistication, which is in my opinion rather advanced and probably has lots of additional upside just from constructing better RL training regime independently of hardware advancement, will become just as table stakes as running a database is now. I highly recommend everyone look into the Allen Institute’s projects in GitHub and HF because they have open source training materials (including an LLM from scratch off common crawl, and some quite interesting tunes of qwen) to get a taste for what will be in the near future afternoon projects or educational material. The future is going to be wild
What demand? Can't shake the notion that it's fictive considering the amount od data centers being built and GPUs sitting in containers, where they will spend quite some time before being even integrated, even more until used...
This line of thinking makes sense if we're talking about opex like power usage. This is capex though and we'll be financing this overpaying for a long time after the hardware has "aged out". Not really sure there is an upside to it.
Also, inference cost predictions were made before this price jump, so we really haven't started paying for it yet. Inference will not be getting cheaper.
I think the companies that drive up the prices here, need
to pay an extra-tax to all of us. I fail to see why I now
have to pay more due to the AI monster companies ruining
the economy.
65 comments
[ 2.9 ms ] story [ 57.2 ms ] threadi compensate by never paying for AI
And by doing this, they ensure local LLMs never become feasible for the vast majority of people and AI companies solidify subscriptions forever.
NVIDIA in their recent quarterly report stopped categorizing "Geforce" as a single category, and merged it into "Edge-Computing".
If you are a PC Gamer or PC Enthusiast as I am, then we have some dark times ahead.
As long as the discussion seems focused on memory, I'd suspect the latter, but if its really the semiconductor boules/wafers, then I'd expect the boule growers to profit, not the memory makers, who just pass on the cost.
So which is it?
I don't see it going away. I mean, it may not grow as fast as now, but I don't see it growing away either. I get why the memory makers do not want to bankrupt themselves, but it feels like there's got to be some way to push that risk off onto model providers and other people in the ecosystem to allow us to grow ram capacity more like 50% per year.
Why were tech savy investors unable to figure this out when the datacenter craze had already started?
How to explain this lag between quickly rising demand for all datacenter components besides memory?
The VRAM in the 5090 is only made by one country in the world.
The 50xx series is special, because its ram is so dependent on a single commodity. It’s not like a 4090 or a 3090; their VRAM chips have been around for years.
If there’s a shortage or interruption in DDR7 VRAM, it seems like every GPU that requires it would explode in value.
I hope I don’t regret posting this because I’d really like to buy one myself…
doesn't matter anyway when things are not reasonably priced. i am stuck at the same memory capacity in my personal system for the better part of two decades, partially due to the above and the current pricing today.
There’s a lot to criticize Sam Altman for saying or popularizing culturally but I’ve come to think his “this is the worst it will ever be” is, in the long run, actually a very intriguing and underrated point.
In a decade training LLMs to the current level of sophistication, which is in my opinion rather advanced and probably has lots of additional upside just from constructing better RL training regime independently of hardware advancement, will become just as table stakes as running a database is now. I highly recommend everyone look into the Allen Institute’s projects in GitHub and HF because they have open source training materials (including an LLM from scratch off common crawl, and some quite interesting tunes of qwen) to get a taste for what will be in the near future afternoon projects or educational material. The future is going to be wild
Or the more likely scenario that the AI bubble bursts and the hyperscalars realize they have built too many data centers.
Also, inference cost predictions were made before this price jump, so we really haven't started paying for it yet. Inference will not be getting cheaper.
we are going to have amazing cheap used hardware for a decade
If you made it 10x cheaper right now you would see a truly unimaginable wave of bot slop.