What an excellent simple post. Amazing how our of sync, how ridiculous the situation is. The board real-estate for DIMMs is out of control. I'd say smaller form factors would be so excellent, but short of serious change it feels like an incommensurate ask, like it's not going to get us to really better.
Yesterday there was an article on Intel's newest integrated silicon photonics, this time having some of the photonics on stacked chiplets on the chip[1]. It's a different sort of integration and micro-miniaturization, but feels very much the same. Not needing a big ole NIC, not needing to have extra, power sucking PCIe transcievers at each end, by moving capabilities onto the chip complex would be amazing. I'd love to see ultra-dense servers of the future where there are dozens of cpus on a sled, with basically just power & optical io feeding each chip. We've bounced off the "disaggregated rack" a couple times now, but it also feels like a major phase transition has been building for a while to.
This gives me a vision of a "slotcket" thing with CPU and a bunch of RAM on it, like 128GB for today's CPUs. gang these up on your expensive bus for bigger systems, package them in ceramic blocks with fluid ports, and... oh that's what they're talking about for "HPC" now innit.
human manipulable components are too big for the interconnect and density we want now... and those are already too small for easy cooling.
If bandwidth isn't the bottleneck in real world workloads, then I am not sure what point Marcan is trying to make by highlighting this particular characteristic.
Except ONE of those sockets has the same memory ceiling as an M1 Studio Ultra. So while you need a lot of space to get the same bandwidth you still need a lot of space to get a high memory ceiling. DRAM chips are only so small.
Right now the Mac Pro has 2TB memory limit. Apple's either going to have to commit considerable board space to DRAM, use sockets or just completely obliterate the max amount of RAM a M* series Mac Pro can have.
That's ok. I'll choose the ability to self-repair and upgrade, even with the "limited" bandwidth of user-replaceable RAM, over soldered or "integrated" RAM and SSD for planned obsolescence anytime.
The base model aluminium Intel Mac Minis started out with 2GB RAM, then 4 GB and finally 8 GB - all over a span of 8 years. Many of these models would be useless today if you couldn't upgrade the RAM at all. In fact, the 2014 Mac Mini models with soldered RAM (LPDDR) received a very poor reception and Apple was forced to backtrack and go back to user-replaceable RAM with the next Mac Minis ( https://everymac.com/systems/apple/mac_mini/mac-mini-aluminu... ).
Many of the early adopters of Apple Silicon M1 devices noticed heavy usage of their SSD, which many attributed to heavy swapping due to the limited (non-upgradable) RAM in the M1 devices. Apple finally said it was a bug ( https://appleinsider.com/articles/21/06/04/apple-resolves-m1... ) and recommended upgrading macOS. But many of these early adopters are less than satisfied and still spend a lot of anxious time worrying whether their soldered SSD's life has reduced by a year or two because of the increased usage. Their concerns are real because with soldered SSDs, you obviously have to do a very, very costly repair through Apple if the soldered SSD chips conk out.
Imagine if you have to replace your whole CPU / SoC to upgrade just your RAM because the RAM is now integrated with the CPU / SoC!? (This is ignoring that even CPUs are now unnecessarily being soldered in many Apple and non-Apple desktops and not easily upgradeable ...).
At some point, the cost-benefit analysis of such soldered / integrated parts doesn't work out in favour of us consumers. And that's what we need to consider before blindly giving up ease of repair for some brief performance increase. (While I have specifically highlighted Apple because they are the only one currently using integrated RAM with their CPU / SoCs, my criticism is directed at the others too, including Dell, HP, Lenovo etc. who are following the same path of planned obsolescence by soldering components to make them less repairable and reduce their life. Unless we start supporting repairability - like with https://frame.work/ devices - we consumers are in a very slippery slope of losing our right to own and repair our devices).
That's a bit disingenuous, isn't it? Of course at some point you'll need a newer processor and chipset to make use of the latest features and innovations.
But you can just move your unsoldered M.2 SSD to the new board. And when properly supported by OEMs, motherboards can last you for generations of CPUs.
There are B350 and X370 boards that were released with the Ryzen 7 1800X in 2017, that support the recently released 5800X3D, a processor two generations of microarchitecture newer on an entirely different node. And while that processor still has the same amount of L2, you get six times the L3 cache of it's predecessor.
No. The reason the average consumer accepts limited cache memory in CPU / GPU / HDD / SSD etc, despite the high bandwidth they offer is simply because of the cost - they are very expensive. Thus, consumers accept the reasonable compromise to sacrifice some performance to match their budget. Otherwise, why wouldn't we all purchase server grade hardware? Or we could all be using high-speed SRAM for our RAM without even the need for cache memory in our CPUs. Hell, if SSDs get as fast as even dynamic RAM then we could even do away with the RAM completely - it would be a significant departure / upgrade from the Princeton and Harvard computer architecture that we currently use.
Yes, we expect cache to be built-in because the manufacturers of high-speed memory realistically don't expect the average consumers to be their target market, and thus chose the B2B model to sell their product. Nobody (that I know of) has also explored offering upgradeable cache memory as an option to consumers (hard to know if it even possible with current CPU / SoC hardware).
As for consumers changing their expectation to accepting soldered RAM and SSDs, that remains to be seen - anything is possible with powerful lobbying and PR. Personally, I think Intel and AMD would be fools to not treat any "integrated" RAM in their future CPUs as "L4 / L5 cache memory" while still fully supporting DIMM DDR RAM as that would make their CPUs more powerful and more desirable than Apple Silicon SoCs. Note that unlike Apple, they don't sell a closed-box solution and have to support multiple OS platform. Thus, it would be more difficult for them to offer a "one size fits all" built-in RAM with their CPUs.
> No. The reason the average consumer accepts limited cache memory in CPU / GPU / HDD / SSD etc, despite the high bandwidth they offer is simply because of the cost - they are very expensive
The 486.
> Yes, we expect cache to be built-in because the manufacturers of high-speed memory realistically don't expect the average consumers to be their target market
The 486.
> Nobody (that I know of) has also explored offering upgradeable cache memory as an option to consumers
I don't really understand the point being made here. Is it that high-end bespoke Apple hardware is faster than the equivalent commodity hardware? Or is the person trying to explain why the memory chips are integrated directly into the chipset (making them extremely fast at the expense of being not so easy to upgrade/replace)?
If you need a ton of memory bandwidth and have the money then you build your product around HBM2 or similar. There are routers, switches, IPS, etc built like this today. Most consumer computer systems only need big memory bandwidth for graphics, which is why video cards have fast on-board memory. Apple of course doesn't want to buy another vendor's graphics hardware and has experience designing their own bespoke hardware, so they designed an integrated system which of course then needs big memory bandwidth. It's an interesting bet that is certainly paying out in terms of power efficiency, but it's not some impossible feat of engineering.
Intel is married to the commodity server market and their laptop/mobile products have always been trickle-down, because they are a giant megacorp whose roadmaps are dictated by their biggest profit centers. Even if they designed this kind of system, they wouldn't be able to capture most of the profit from it like Apple can, since they don't sell memory or systems, they sell CPUs. After the mobile debacle, Microsoft pivoted to SaaS and is way too chickenshit to start designing hardware. Apple is the only company incentivized to produce a product like this for consumers. This is not going to suddenly become profitable for the commodity server market; and it doesn't have to; it's not a market Apple is particularly interested in.
Indeed, any spec of HBM is not going to be comparable to an ondie memory controller having to reach out to individual DIMMS on a traditional motherboard. I appreciate the work this person is putting in to Asahi linux, but on the whole this is an unfair comparison.
> If you need a ton of memory bandwidth and have the money then you build your product around HBM2 or similar
What does similar mean? I'm not sure I agree.
The M1 is basically a classic big-iron architecture, miniaturized with chiplets. The top M1 Max config is effectively 8x channel/512-bit-wide, just like a current epyc. It's faster though because it's 5.2GT/s LPDDR5 memory (400GBps) vs 3.2GT/s DDR5, and it's much smaller because it appears as 4x stacks on package. (Next gen Genoa is up to 12x channels of 5.2GT/s DDR5 fwiw.) But it's still a conventional play, basically, with new packaging.
> Intel is married to the commodity server market and their laptop/mobile products have always been trickle-down, because they are a giant megacorp whose roadmaps are dictated by their biggest profit centers.
Generally, yes! Notably sometimes innovation is first delivered in mobile and "trickles up," but basically it's because mobile is a safer easier space to innovate/explore/learn in (as it's much less investment cost to make the much smaller/lighter chips there). Generally you're right about what leads the business.
> Even if they designed this kind of system, they wouldn't be able to capture most of the profit from it like Apple can, since they don't sell memory or systems, they sell CPUs.
It's definitely a little intimidating figuring out how to sell bundled on ram. The technical benefits are hard to ignore though. Total cost of ownership should improve for customers notably, via less intense/simpler motherboards, via low power consumption stemming from much more direct channels & stronger signal integrity to main memory. But you're right, I think there's a lot of unappealing & unwanted aspects to becoming a company that ships ram. It'll be hard to resist/ignore for long though.
AMD hasn't shown DRAM related interest yet, but their 5800X3D has 96MB stacked L3 v-cache on a separate chiplet. Their upcoming Genoa cpus will have ~1GB stacked L3 V-cache, again at absurd terabits-per-second speed, also via chiplets. AMD is pushing the memory heirarchy wider, is doing something like this M1-ening, but at a at higher-throughput lower-latency but smaller-capacity points, while leaving the conventional RAM channels also available. But they could, pretty easily I'd wager, snap back & deliver integrated DDR if they wanted, saw the need.
Now that there are groups like Universal Chiplet Interconnect Express, this tech is kind of semi-commodified, available for purchase, to a degree. Yes, the mainstream market may lag. But with the capabilities arriving, it's hard to imagine this tech staying out of the data-centers for too long. Once we see some hyperscalers or upstarts delivering chips with integrated DDR, I expect the appeal will be irresistible/unsuppressable. It'll become obvious/known that this is good tech that is table stakes for selling chips, like it or not.
I was thinking that about HMC and thinking that HBM was still Samsung / SK's thing. I see now everyone is making HBM. You'd build your system around HBM.
> Notably sometimes innovation is first delivered in mobile and "trickles up,"
Not at Intel :P but I agree in general.
Apple would never use HBM because consumer margins and they have been building SoC based on LPDDR for years. Hyperscalers already do their compute on "bundled ram" systems when they buy NV Tesla or AMD ROCm cards, both based on HBM. AMD is targeting the slow&wide "million VMs" market, and DDR is doing great in their niche.
I do agree we will start seeing weirder server boards sold with soldered DDR for cost and power reasons, probably by Intel or AMD. I think it will be an even more specific niche, sitting between off the shelf commodity servers and expensive appliances like NV DGX. Again Intel/AMD are in a bad spot to capture the fat margin here.
UCIE is classic giving away old stuff to extinguish competition. It's good for bespoke builders but meant to ward off actual competition from selling something similar and then eventually becoming competitive with IF or NVLink etc. Note Apple made their own fancy interconnect thing too for M1.
I've seen HBM as growingly niche & less interesting, but upcoming HBM3 seems to be quite a turn towards a more DDR like behavior: 5.2+ GT/s from 2.0GT/s, a huge jump, and channel width dropping from 8x gargantuan 128-bit channels to 16x reasonable-size-ish 64-bit channels. I'm not sure what if anything else would be required to escape HBM2's reputation for being high latency, but hopefully the situation is just better.
AMD hasn't really sold HBM as far as I've seen since ~2018. Counter-point, upcoming Intel data-center GPUs seem loaded with HBM. I tend to believe this use case will be more permissive of the the higher latency, wider width HBM, that deep number crunching is less affected. Nvidia has HBM on their contemporary Axx Tesla accelerators- again, a case where aggregate number crunching with some internal latency is ok, unlike general GPU and CPU work.
A combination of capacitance that has to be charged, crosstalk and characteristic impedance discontinuities.
The signals move between the cpu and the ram at about one nanosecond per foot. A nanosecond is one full cycle of a 1GHz clock.
The characteristic impedance of the connection is set by the physical size of the pcb track and the dielectric constant of the pcb material.
As the signals move between the cpu and ram at nearly the speed of light if they hit an impedance discontinuity some of the energy is reflected back. That messes up the signal.
Making a connector with hundreds of pins that works reliably and does not cause a significant impedance discontinuity is hard.
The lower the impedance of the connection, the more power is required to drive data through it.
A bit error rate of one in a billion would corrupt data or cause a software crash daily so signal integrity is a hard problem.
I still don’t get it after reading this tweet. What would be more informative to me - a table for M1 and a system with the same RAM size but DIMMs which would contain: 1. random access latency 2. linear access bandwidth 3. price
I'm so tired of this person's constant fawning over the shiny and fast but proprietary ARM Macs like the raw performance is the only criterion that should matter to everyone else too.
I mean it's his job to literally work with this hardware every day. It's not surprising that he sometimes explains stuff like this. Especially considering the misinformation that's out there where people are saying: "Apple is evil for the mac studio not having DIMMs.".
it is so 2020 wannabe era… this motherboad dimm config purpose is to put TB of ram, they dont care to much of pure bandwidth… they permit 2TB ram servers when people where happy with 4GB
I can buy the idea of POP ram for SOC with decent GPU despite the fact consoles deliver much beefier GPU performance and wider/faster memory busses off chip.
What I cant forgive is Apples insistence on forced planned obsolescence by not including memory slot for additional expandable second level ram. They even got a stupid and probably invalid patent for that exact thing https://www.techpowerup.com/277760/apple-patents-multi-level... . You could have 8GB of normal/fast ram + 32GB sodimm buffer instead of burning SSD down with paging.
34 comments
[ 3.0 ms ] story [ 89.2 ms ] threadYesterday there was an article on Intel's newest integrated silicon photonics, this time having some of the photonics on stacked chiplets on the chip[1]. It's a different sort of integration and micro-miniaturization, but feels very much the same. Not needing a big ole NIC, not needing to have extra, power sucking PCIe transcievers at each end, by moving capabilities onto the chip complex would be amazing. I'd love to see ultra-dense servers of the future where there are dozens of cpus on a sled, with basically just power & optical io feeding each chip. We've bounced off the "disaggregated rack" a couple times now, but it also feels like a major phase transition has been building for a while to.
[1] https://news.ycombinator.com/item?id=31924382
human manipulable components are too big for the interconnect and density we want now... and those are already too small for easy cooling.
Right now the Mac Pro has 2TB memory limit. Apple's either going to have to commit considerable board space to DRAM, use sockets or just completely obliterate the max amount of RAM a M* series Mac Pro can have.
The base model aluminium Intel Mac Minis started out with 2GB RAM, then 4 GB and finally 8 GB - all over a span of 8 years. Many of these models would be useless today if you couldn't upgrade the RAM at all. In fact, the 2014 Mac Mini models with soldered RAM (LPDDR) received a very poor reception and Apple was forced to backtrack and go back to user-replaceable RAM with the next Mac Minis ( https://everymac.com/systems/apple/mac_mini/mac-mini-aluminu... ).
Many of the early adopters of Apple Silicon M1 devices noticed heavy usage of their SSD, which many attributed to heavy swapping due to the limited (non-upgradable) RAM in the M1 devices. Apple finally said it was a bug ( https://appleinsider.com/articles/21/06/04/apple-resolves-m1... ) and recommended upgrading macOS. But many of these early adopters are less than satisfied and still spend a lot of anxious time worrying whether their soldered SSD's life has reduced by a year or two because of the increased usage. Their concerns are real because with soldered SSDs, you obviously have to do a very, very costly repair through Apple if the soldered SSD chips conk out.
Imagine if you have to replace your whole CPU / SoC to upgrade just your RAM because the RAM is now integrated with the CPU / SoC!? (This is ignoring that even CPUs are now unnecessarily being soldered in many Apple and non-Apple desktops and not easily upgradeable ...).
At some point, the cost-benefit analysis of such soldered / integrated parts doesn't work out in favour of us consumers. And that's what we need to consider before blindly giving up ease of repair for some brief performance increase. (While I have specifically highlighted Apple because they are the only one currently using integrated RAM with their CPU / SoCs, my criticism is directed at the others too, including Dell, HP, Lenovo etc. who are following the same path of planned obsolescence by soldering components to make them less repairable and reduce their life. Unless we start supporting repairability - like with https://frame.work/ devices - we consumers are in a very slippery slope of losing our right to own and repair our devices).
How do you upgrade your L2 cache? You're not still using a 486, are you?
Did you protest the Pentium as well?
But you can just move your unsoldered M.2 SSD to the new board. And when properly supported by OEMs, motherboards can last you for generations of CPUs.
There are B350 and X370 boards that were released with the Ryzen 7 1800X in 2017, that support the recently released 5800X3D, a processor two generations of microarchitecture newer on an entirely different node. And while that processor still has the same amount of L2, you get six times the L3 cache of it's predecessor.
No, it's not. Well, at least it wasn't intended to be.
The whole point of my post is: What were you supposed to do with all of the expensive cache chips you bought for your 486 when the Pentium came out?
--> They weren't soldered on, but you couldn't use them anymore if you wanted to upgrade your Pentium. <--
How is the situation with RAM today any different?
The only difference I see is that we now expect cache to be soldered on. I imagine in the future we'll expect the same of RAM too.
Yes, we expect cache to be built-in because the manufacturers of high-speed memory realistically don't expect the average consumers to be their target market, and thus chose the B2B model to sell their product. Nobody (that I know of) has also explored offering upgradeable cache memory as an option to consumers (hard to know if it even possible with current CPU / SoC hardware).
As for consumers changing their expectation to accepting soldered RAM and SSDs, that remains to be seen - anything is possible with powerful lobbying and PR. Personally, I think Intel and AMD would be fools to not treat any "integrated" RAM in their future CPUs as "L4 / L5 cache memory" while still fully supporting DIMM DDR RAM as that would make their CPUs more powerful and more desirable than Apple Silicon SoCs. Note that unlike Apple, they don't sell a closed-box solution and have to support multiple OS platform. Thus, it would be more difficult for them to offer a "one size fits all" built-in RAM with their CPUs.
The 486.
> Yes, we expect cache to be built-in because the manufacturers of high-speed memory realistically don't expect the average consumers to be their target market
The 486.
> Nobody (that I know of) has also explored offering upgradeable cache memory as an option to consumers
The 486.
Intel is married to the commodity server market and their laptop/mobile products have always been trickle-down, because they are a giant megacorp whose roadmaps are dictated by their biggest profit centers. Even if they designed this kind of system, they wouldn't be able to capture most of the profit from it like Apple can, since they don't sell memory or systems, they sell CPUs. After the mobile debacle, Microsoft pivoted to SaaS and is way too chickenshit to start designing hardware. Apple is the only company incentivized to produce a product like this for consumers. This is not going to suddenly become profitable for the commodity server market; and it doesn't have to; it's not a market Apple is particularly interested in.
What does similar mean? I'm not sure I agree.
The M1 is basically a classic big-iron architecture, miniaturized with chiplets. The top M1 Max config is effectively 8x channel/512-bit-wide, just like a current epyc. It's faster though because it's 5.2GT/s LPDDR5 memory (400GBps) vs 3.2GT/s DDR5, and it's much smaller because it appears as 4x stacks on package. (Next gen Genoa is up to 12x channels of 5.2GT/s DDR5 fwiw.) But it's still a conventional play, basically, with new packaging.
> Intel is married to the commodity server market and their laptop/mobile products have always been trickle-down, because they are a giant megacorp whose roadmaps are dictated by their biggest profit centers.
Generally, yes! Notably sometimes innovation is first delivered in mobile and "trickles up," but basically it's because mobile is a safer easier space to innovate/explore/learn in (as it's much less investment cost to make the much smaller/lighter chips there). Generally you're right about what leads the business.
> Even if they designed this kind of system, they wouldn't be able to capture most of the profit from it like Apple can, since they don't sell memory or systems, they sell CPUs.
It's definitely a little intimidating figuring out how to sell bundled on ram. The technical benefits are hard to ignore though. Total cost of ownership should improve for customers notably, via less intense/simpler motherboards, via low power consumption stemming from much more direct channels & stronger signal integrity to main memory. But you're right, I think there's a lot of unappealing & unwanted aspects to becoming a company that ships ram. It'll be hard to resist/ignore for long though.
AMD hasn't shown DRAM related interest yet, but their 5800X3D has 96MB stacked L3 v-cache on a separate chiplet. Their upcoming Genoa cpus will have ~1GB stacked L3 V-cache, again at absurd terabits-per-second speed, also via chiplets. AMD is pushing the memory heirarchy wider, is doing something like this M1-ening, but at a at higher-throughput lower-latency but smaller-capacity points, while leaving the conventional RAM channels also available. But they could, pretty easily I'd wager, snap back & deliver integrated DDR if they wanted, saw the need.
Now that there are groups like Universal Chiplet Interconnect Express, this tech is kind of semi-commodified, available for purchase, to a degree. Yes, the mainstream market may lag. But with the capabilities arriving, it's hard to imagine this tech staying out of the data-centers for too long. Once we see some hyperscalers or upstarts delivering chips with integrated DDR, I expect the appeal will be irresistible/unsuppressable. It'll become obvious/known that this is good tech that is table stakes for selling chips, like it or not.
I was thinking that about HMC and thinking that HBM was still Samsung / SK's thing. I see now everyone is making HBM. You'd build your system around HBM.
> Notably sometimes innovation is first delivered in mobile and "trickles up,"
Not at Intel :P but I agree in general.
Apple would never use HBM because consumer margins and they have been building SoC based on LPDDR for years. Hyperscalers already do their compute on "bundled ram" systems when they buy NV Tesla or AMD ROCm cards, both based on HBM. AMD is targeting the slow&wide "million VMs" market, and DDR is doing great in their niche.
I do agree we will start seeing weirder server boards sold with soldered DDR for cost and power reasons, probably by Intel or AMD. I think it will be an even more specific niche, sitting between off the shelf commodity servers and expensive appliances like NV DGX. Again Intel/AMD are in a bad spot to capture the fat margin here.
UCIE is classic giving away old stuff to extinguish competition. It's good for bespoke builders but meant to ward off actual competition from selling something similar and then eventually becoming competitive with IF or NVLink etc. Note Apple made their own fancy interconnect thing too for M1.
AMD hasn't really sold HBM as far as I've seen since ~2018. Counter-point, upcoming Intel data-center GPUs seem loaded with HBM. I tend to believe this use case will be more permissive of the the higher latency, wider width HBM, that deep number crunching is less affected. Nvidia has HBM on their contemporary Axx Tesla accelerators- again, a case where aggregate number crunching with some internal latency is ok, unlike general GPU and CPU work.
The signals move between the cpu and the ram at about one nanosecond per foot. A nanosecond is one full cycle of a 1GHz clock.
The characteristic impedance of the connection is set by the physical size of the pcb track and the dielectric constant of the pcb material.
As the signals move between the cpu and ram at nearly the speed of light if they hit an impedance discontinuity some of the energy is reflected back. That messes up the signal.
Making a connector with hundreds of pins that works reliably and does not cause a significant impedance discontinuity is hard.
The lower the impedance of the connection, the more power is required to drive data through it.
A bit error rate of one in a billion would corrupt data or cause a software crash daily so signal integrity is a hard problem.
2. basically the same
3. basically the same
The advantage to Apple's soldered RAM is that the motherboard is 10x smaller.
What I cant forgive is Apples insistence on forced planned obsolescence by not including memory slot for additional expandable second level ram. They even got a stupid and probably invalid patent for that exact thing https://www.techpowerup.com/277760/apple-patents-multi-level... . You could have 8GB of normal/fast ram + 32GB sodimm buffer instead of burning SSD down with paging.