Not sure I buy that? There's a hardware/process argument that DRAM scaling is hitting a wall (for largely the same reason that logic scaling is, even though the fabs are different).
But DRAM has always been almost completely dominated by consumer device needs, and consumer requirements leveled off hard maybe a decade ago. Once you had a 4G system, things were mostly OK. Pervasive VM use and general bloat pushed that up by maybe a factor of two or three since, but that's it.
AI is different. It wants that capacity for different reasons and puts it in different places. And almost none of us really saw that coming more than ~18 months ago or so. Certainly not to the extend that you'd conclude "we've been ignoring it".
How so? Memory bandwidth and capacity has been steadily increasing. On desktop for example, 10 years ago an i7-5960X was the absolute top of the line and it had a limit of 64GB, with most motherboards in practice being limited to 32-48GB, and this was an unthinkable amount of RAM at the time for desktop. Now the top end desktop chips support 192-256GB, and a year ago (before the RAM price spike) you could realistically buy that much if you wanted for a reasonable price.
Has it? The i7-5960X was 8C/16T, Ryzen 9 9950X (and variants) are 16C/32T. Even the Threadripper PRO 9000 WX is 96C/192T and supports a staggering 2TB of RAM.
My health insurance company recently rolled out a major revamp of their website, and it now uses GiB as if they were KiB.
At my knitting group last night one of the conversation topics was how everyone's computer feels really unstable these days. One person stopped volunteering at a local charity because, after a recent website rewrite, she has too much trouble with the page locking up when she's trying to sign up for hours.
Last weekend I got talking to another parent at my kid's gymnastics class about how her computer's really struggling since they rolled out some "modernization" rewrites of internal business applications, and it's making it hard to get her work done.
What I'm getting at is, if we do have to get used to memory being more expensive, that might be very disruptive. Programmers will have to do a major about-face on how they've been writing software in the post-AI era.
... for what were then 'advanced' weapons capability in high-cost units produced and expended in relatively small numbers with high yield payloads.
The modern drone warfare version of expendable, high volume, high technology small units with two way communication via fibre-optic and RF jamming resistant multiple channels, and the ability to do on-device video object identification and terminal flight control when active electronic counter measures degrade operator control...
That seems like exactly the kind of gear using commodity DDR4/5 sodimm and dimm modules and literally burning them up at then end of nearly every sortie. Even if they are using much smaller units (smartphone grade RAM) it's still cooking the limited output of the global memory fab capacity, and from the fields covered in fibre-optic blankets, I think it's safe to say there is a significant volume of electronics equipment being consumed.
So yeah, AI might be ordering memory and putting it in datacenter, but eventually that material will be available for e-waste recovery (repurpose first, then recycle later). Both are competing for the same limited fab capacity.
Yes it is, they're talking about induced demand. There are lots of resources where, when usage of the resource becomes more efficient, people end up using more rather than less.
If training and inference is hardware constrained, and you can train and server better and bigger models on the same hardware with memory optimizations, that's exactly what I would expect companies to do.
What was the benefit of all this AI stuff again? Because all I see is middle and lower class being fucked over, repeatedly. Increasing prices of consumer electronics, job losses, suffering quality of goods and services, consolidation of power in favor of elites.
Think about the app sitting on some developer's laptop, necer getting deployed anywhere else! Or the thousands getting uploaded to app stores to slurp and sell data!
or you can wait until one of those frontier labs IPO goes bust in a really bad way where it pumps 10x on day 1 and then everyone dumps every share out there and half a million people are holding bags. that ll surely wipe out all the AI hype
Oracle's recent force majeure invocation shines a spotlight on what seems to be an endemic problem: datacenter builders have been signing commitments to bring a certain amount of capacity online by a certain date, without first conducting sufficient due diligence about whether they would be able to secure all the resources needed to meet those commitments.
This means they are now caught in a mad scramble to source all sorts of stuff, including RAM, potentially as a matter of life and death.
What's less clear to me, though, is whether datacenter builders will be able to make good on these purchase agreements. It seems these projects have every opportunity to fail, and the path to success keeps getting narrower. What are the odds that chip manufacturers ultimately get caught holding a bag of chips that nobody can buy in one hand, and a bag of IOUs from bankrupt customers in the other?
There are signs that we are, and are not, in an AI bubble.
Compared to the dotcom boom, firms are well positioned in terms of capital - they have revenue and profits.
However, there's still the absurd speculative valuations, risk, and high expenditure (on said data centres). Also, Andrew Bailey from the Bank of England, personal opinion - https://www.bbc.co.uk/news/articles/cv8e30enrkxyo
Remember when software was eating the world? The gravy train ran out on commodity hardware and now we are paying the price for almost 3 decades of underinvestment.
The real problem is not that they are buying all the hardware, but that they are buying it and keeping it in bloody warehouses without bringing it online. Because they haven't got DC capacity for it.
Has anyone else here been trying to rent some cloud gpus for reasonable money? I have. I have signed up to a dozen "providers" that advertise everything from tesla cards to b200 only to find stuff is either out of capacity or double/triple the price.
43 comments
[ 1.5 ms ] story [ 34.9 ms ] threadBut DRAM has always been almost completely dominated by consumer device needs, and consumer requirements leveled off hard maybe a decade ago. Once you had a 4G system, things were mostly OK. Pervasive VM use and general bloat pushed that up by maybe a factor of two or three since, but that's it.
AI is different. It wants that capacity for different reasons and puts it in different places. And almost none of us really saw that coming more than ~18 months ago or so. Certainly not to the extend that you'd conclude "we've been ignoring it".
At my knitting group last night one of the conversation topics was how everyone's computer feels really unstable these days. One person stopped volunteering at a local charity because, after a recent website rewrite, she has too much trouble with the page locking up when she's trying to sign up for hours.
Last weekend I got talking to another parent at my kid's gymnastics class about how her computer's really struggling since they rolled out some "modernization" rewrites of internal business applications, and it's making it hard to get her work done.
What I'm getting at is, if we do have to get used to memory being more expensive, that might be very disruptive. Programmers will have to do a major about-face on how they've been writing software in the post-AI era.
Why? Weapons are low-volume compared to commercial products, and they probably use trailing edge or weird bespoke components.
... for what were then 'advanced' weapons capability in high-cost units produced and expended in relatively small numbers with high yield payloads.
The modern drone warfare version of expendable, high volume, high technology small units with two way communication via fibre-optic and RF jamming resistant multiple channels, and the ability to do on-device video object identification and terminal flight control when active electronic counter measures degrade operator control...
That seems like exactly the kind of gear using commodity DDR4/5 sodimm and dimm modules and literally burning them up at then end of nearly every sortie. Even if they are using much smaller units (smartphone grade RAM) it's still cooking the limited output of the global memory fab capacity, and from the fields covered in fibre-optic blankets, I think it's safe to say there is a significant volume of electronics equipment being consumed.
So yeah, AI might be ordering memory and putting it in datacenter, but eventually that material will be available for e-waste recovery (repurpose first, then recycle later). Both are competing for the same limited fab capacity.
This means they are now caught in a mad scramble to source all sorts of stuff, including RAM, potentially as a matter of life and death.
What's less clear to me, though, is whether datacenter builders will be able to make good on these purchase agreements. It seems these projects have every opportunity to fail, and the path to success keeps getting narrower. What are the odds that chip manufacturers ultimately get caught holding a bag of chips that nobody can buy in one hand, and a bag of IOUs from bankrupt customers in the other?
Compared to the dotcom boom, firms are well positioned in terms of capital - they have revenue and profits.
However, there's still the absurd speculative valuations, risk, and high expenditure (on said data centres). Also, Andrew Bailey from the Bank of England, personal opinion - https://www.bbc.co.uk/news/articles/cv8e30enrkxyo
Has anyone else here been trying to rent some cloud gpus for reasonable money? I have. I have signed up to a dozen "providers" that advertise everything from tesla cards to b200 only to find stuff is either out of capacity or double/triple the price.