In case it saves anyone some time (from the article):
"The AMD Ryzen AI Max+ 395(Strix Halo) processor has been available since Spring 2025 and the Halo doesn’t offer anything new on that front."
It has the same 256 GB/s memory bandwidth limit as every board previously, not sure why this is even being released right now as if it's some new fangled thing - you can go get a Framework Desktop for roughly the same price or a GMKtec EVO-X2 for a bit cheaper.
Framework, weirdly, overcharges considerably for their SSDs. You can currently get a Samsung 990 Pro 2TB on Amazon for $390; Framework charges $625 for the Sandisk 850x 2TB, which has similar performance (and is being sold on Amazon for $530).
If you DIY your own SSD, you can spec a Framework Desktop for below $4k; but not much below. Roughly the same price.
I got the EVO-X2 for $1,599! In 44+ years of buying computers, I've seen some appreciation, but nothing like this. Going from $1,599 to ~$3,500 in a year is just insane.
128GB of the most in demand consumer chip when all the manufacturers are switching to HBM manufacturing because the margins are far better there will do that to you. It sucks.
I was looking at buying hardware for me and as testing ground to learn. With those prices I just nipped out and will just rent in demand at runpod or so.
And GMKtec just released an EVO-X3 [0] based on the same board but with Oculink. When AMD launched this I thought it was the new chipset that was going to have 192GB of unified memory, but they look to be capitalizing on the inflated premium these "dev" focused desktop solutions.
Regretting not buying the Framework back when it was around $2k.
I had a pre-order in for the 128GB DIY motherboard only model last Oct 2025, but cancelled it and just upgraded my current 5900x rig to 64GB instead and replaced the x370 board with a used x470. Really with I had pulled the trigger, but I honestly am just fine with the performance as is on my current system for now. The Framework should have lasted me just as long, but who knows. Hoping my current rig lasts at last another 4 years..
It's being released right now because it's massively profitable and in high demand and has actually gone up in price over the past year so obviously AMD wants to cash in on that instead of selling these units to PC manufacturers at a lower price.
I also think this is forward looking and helps reanchor expectations for RAM and memory bandwidth, which historically have been areas where companies do value engineering.
The memory shortages won't last forever; when companies start adding capacity I wouldn't be surprised to see massive sticks of RAM being sold for consumers.
Before AI I kinda thought more ram and faster ram was kind of useless because I had written off increased speeds as sort of a luxury whose only application is bloated unoptimized software. Boy was I wrong about that. But then why stop there?
I mean, there's a difference between "my IDE opens 0.1s faster" and "my desktop can now write my code for me" and only one of those justifies increasing RAM production.
It’s tragic that memory is so expensive, and yet, we don’t have chipsets to exploit the bandwidth possibilities of the memory being burnt on these devices.
Absolutely no reason that these need to be capped at 256 GB/s other than shortsighted design from three years ago.
The extremely charitable answer is that this is AMD's chance to control the built-in software, so they can ship a bunch of built-in AI stuff and a controlled environment similar to what NVIDIA does with their DGX OS, rather than e.g. Framework which is going to give you a stock Linux distro.
(I agree that you almost certainly don't need this)
$4k is pretty darn spendy. I recently purchased a refurbished Corsair AI Workstation with almost the same hardware (same chip, same 128GB RAM, but only 1TB storage) for $2160. Pretty good deal! Codex and I wrote a Linux driver to report the power mode of the device:
How much are we going to pay for "AI kits" once the DRAM shortage is over? Will we be able to run a local model equivalent to the current AI frontier in sub $1000 hardware, even if dedicated, in 5 years?
When this hardware was announced, it was expected to be in the $1200-1400 range new... so, maybe. The real question is will the powers that be let this bubble burst, and how painful will the fallout be... I have a feeling it will be worse than 2001-2002.
Was “only” $2k in its previous form but even in this updated box the mem bandwidth is woefully inadequate.
There’s a few models with space for a dedicated GPU for hybrid inference but imo not worth it.
Save your money for a Xeon or EPYC build
I have another strix halo that I got for half the price (before this price increase world wide). AMD making lemonade is one of the best reasons to get a strix halo. Lemonade + qwen3.6 35B MTP @ Q8_0 + anythingLLM (in docker) replaced 90%+ of my AI usage. And it’s fully local! Setting everything up took less than 3 hours total, including installing the OS
Hardware is the exact same as what used to be available for $2K last year (and is still $1K cheaper from Chinese OEMs).
LTT Lab's LLM testing is getting more sophisticated, which is great - I think it's worth noting that ROCm/Vulkan versions and llama.cpp build versions are going to have some big differences for numbers.
For those wanting to get the most out of their Strix Halos, there's both kernel tweaks and utilities like ryzenadj that can help you get the most out of it. ( http://strixhalo.wiki/ has most of that documented). Also, if you're running for coding or agentic work, if you model supports MTP, that's mature and should give you a decent (30%?) decode boost.
it's worth noting that AMD's software is universally weak and not worth any degree of reliance. and it's not just their AI software, every few months they merge a serious regression into amdgpu and sometimes even backport it into stable. they are amateurs.
just a few weeks ago they backported a null deference kernel oops into stable, it's still not fixed.
The "ROCm" situation with Strix Halo was pretty bad for a while. I think it finally stabilized late last year. You needed the right combo of ROCm, Linux kernel, and kernel firmware for it to work reliably.
Whenever I rebuild llama.cpp, I wind up using the Vulkan build anyway.
And thats a pretty big annoyance. You need to perfectly line up all the holes in the swiss cheese for the AMD stack to work, then their dev team kicks you in the nuts anyway.
They have made a few attempts at investing in the hardware, but the software side is letting them down hard, and that part is almost entirely their own fault. They have underinvestment in their own stack, and into popular standard and Community libraries that would make it easy to use their gear.
Yep, it definitely wasn't an "out-of-box" experience. This was on Ubuntu 25.10. Maybe other distros are better?
Also, when you do Python work, you have to remember to install dependencies like pytorch from an alternate repo, otherwise you wind up with the CUDA versions that only use the CPU and not the GPU through ROCm.
I started a project w/ the assumption that fully tuned for HIP kernels could get closer to roofline and beat llama.cpp's performance (and vs their ROCm engine it does!) but after thousands of iterations/experiments my suspicion is that RADV/ACO is actually just much better than LLVM/HIP when it comes to a lower-level scheduling etc for RDNA3.
I haven't finished comprehensive tests but I found:
- Vulkan is up to 2.25X faster for most coalesced, strided and interleave variants for memory-side scheduling/access shapes
- 3.3X faster on specific dot-path sweeps, including for scalar-dequant
- For matched LDS, Vulkan can be 8-14X+ faster (!!!) than matched HIP LDS
HIP doesn't always win against RADV/ACO, but on dispatch/runtime, it does appear to be quite a bit faster than HIP/LLVM on gfx1151 (Strix Halo). I'll be publishing sharing full data once I also run vs gfx1100...
biggest mistake was buying an amd laptop. this thing had gpu crashes from day one, and it got worse as time went on. now just playing a video and closing the lid will crash the kernel 100% of the time. amd - never again
Without further info that’s a lot to pin on the bottom layer of the stack. I have an Asus ProArt Ryzen AI HX 390 with 64GB RAM. Running CachyOS and Pop! the thing kept falling over at random times. Running Fedora workstation, it’s rock solid. YMMV.
I recently bought a few sparks from Micro Center for the exact same price and it comes with ConnectX-7 200Gbps inter-connectivity. Not sure how AMD feels it can charge exactly the same for less.
The CPU of Ryzen is better than that of DGX Spark, especially for modern programs that have been updated to use AVX-512 (i.e. it has a significantly higher multithreaded performance).
Only for GPU applications the NVIDIA system is likely to be better.
For inference, the prefill on the DGX Spark is like 5x the speed. Decode about the same (both bandwidth limited to about the same memory speed, unfortunately).
Difference is on the Spark you can use and learn CUDA, vLLM, SGlang etc which is the industry standard.
I bought mine (ASUS version) back in December with the intent to learn that stack of stuff. I've been on and off with it but it seems to have paid off, looks like I might be getting work in the inference serving space.
Wow the prices on these have really come up.. Got my Framework desktop mainboard (Just the motherboard + CPU + soldered 128gb RAM) in Dec 2025 for ~1900 EUR
I had hoped this was about Medusa Halo, but unfortunately, it's about 2025 technology. It's the same as Framework Desktop was at the end of last summer, which would have been a slightly silly but fun buy at $2k... I'd hope Mark Cerny / Sony launch PS6 sooner rather than later, as together with the upcoming LPDDR6 standard, it should trickle down to us in the local LLM mud eventually?
It would be really nice if they included clustering support like a blueprint on how to buy several of these and cluster them to run the really large models in the best way possible.
Any performance gains caused by the internal bandwidth of the card will evaporate once you spill into system RAM, because now your bottleneck is probably a slow PCI lane.
And if your jobs do fit onto a 24GB card, then you are not the target user for the "AI mini PC" niche that these guys are trying to carve out
I imagine there may be users who can’t use macOS, or maybe they want the ability to upgrade storage.
The framework desktop even has a usable PCIe 4x slot available if you put the board in a different case. They sell the 128GB board on its own for $3150.
All the gpu makers make all their profit selling datacenter products. They don’t want consumer/home lab stuff with lower margins to replace their data center products so they handicap the vram in those products to make them less enticing for datacenter use.
Perhaps if less spending went towards their private aviation interests LTT labs could review a piece of hardware that was released _this_ year, or maybe extend their narrow testing process to cover real-world use metrics like TTFT. Not to mention the lack of real value-perf comparison to CUDA
I really want a 128gb+ machine but it's brutal to be at only 256gb/s for $4k (especially with the drawbacks of both ARM and AMD).
I fear that by the time the RTX Spark comes out it'd have to be $6k, and by the time a 128gb or more machine with 700+gb/s comes out it'd be at $10k, way out of most consumers' hands.
A Mac Studio is a much better buy in terms of memory bandwidth, but impossible to buy in a 128 GB configuration. Honestly there aren’t great options right now and it’s probably better to wait for the market to be less insane.
Late last year I was debating a Framework desktop vs waiting for a M5 studio. I went with the former in December 2025, glad I did now as everything has gone up in price and if I had put off the decision I’d probably still be putting it off.
Apple rumor mill is suggesting that we may see M5 Mac Studio announced in September at that event. Apple just increased their pricing across the board, so I'm not holding my breath that they will be reasonably priced. They also cancelled development of their M6 in favor of the future M7 which I suspect will be a major, AI-focused upgrade compared to the M4->M5 upgrade
The Mac Studio M3 Ultra with 96GB of RAM with 1Tb SSD costs $6799, but has triple the memory bandwidth [1]. It's almost twice the price of Strix Halo and when the M5 Ultras come out it will be interesting to see how much they go for. I expect a lot of demand for them!
There was a post recently, that showed the DGX spark is twice as fast as the M3 Ultra at prompt processing, but half as fast at output tokens [2]. They used gps-oss-120 for that test with a small context.
I’ve got a 128gb m5 max mbp and two sparks. For my real-world use cases, a single spark running DS4 Flash will have fully responded by the time my Mac has even started generating tokens. I figured I’d have more generation heavy work when I also bought the Mac, but it has done very little work running LLMs since I got the first Spark.
I mostly run them clustered for DS4 and am quite happy with the performance, and the cost isn’t that much more for two than the MBP while giving me double the unified memory.
I’ll probably pick up a third to run multiple smaller models. I don’t understand why people would buy a halo over a spark at comparable prices, particularly because if you want to cluster, the cx7 be beats the shit out of them when it comes to latency and throughput
Yeah, folks should be aware that if you're filling up the memory on a Strix Halo for an inference workload, you're going to be getting uncomfortably slow token rates. Like, DS4 (a 1-bit quantization of DeepSeek V4 Flash) runs at something like 9-13 tokens/second, with a loooong time to first token. It is not a realistic interactive coding model for agentic use.
I like my Strix Halo and keep it chewing on stuff, mostly non-interactive workloads (security audits of software mostly, training experiments, etc.), I get a lot of use out of it. If you want to experiment with AI, it is a good platform for that, though at $4k you can get an Nvidia-based Asus Ascend GX10, which is probably better. But, if you want a local model for interactive agentic use, you're going to be running either Qwen 3.6 or Gemma 4, which will fit comfortably on 2x64GB GPUs (even old GPUs will run them faster than the Strix Halo...I have dual Radeon Pro V620s which are faster, and they're six years old), or snugly on 32GB. A 48GB or 64GB Mac would run them well. Two Radeon AI Pro R9700 GPUs is probably the sweet spot, right now for GPUs. Not the cost of a good used car, like a 5090 or 4090, but plenty of memory and performance for local inference. Also, not finicky and weird and needing custom 3D printed fan shrouds like the old server GPUs on eBay.
At the moment, there just isn't a model that works better on a 128GB inference machine like this that don't also work fine on 64GB machines, which may be faster (very few 32GB GPUs will be slower, though I wouldn't recommend buying any GPU that isn't currently actively supported by the vendor drivers and CUDA or ROCm...so probably don't buy an MI50 or V100 or whatever).
I said, "Like, DS4 (a 1-bit quantization of DeepSeek V4 Flash) runs at something like 9-13 tokens/second, with a loooong time to first token."
Which almost exactly matches the benchmark you just linked. Looks like it's possible to goose it to 15 tokens per second with a tiny context, but why would I want a giant model with a 2k context? DeepSeek is too big to be fast enough on a Strix Halo.
To be clear, if you think that's comfortable for interactive use, more power to you. But, I'm not waiting for that. I'll pay DeepSeek to host it for me. Their token prices are quite cheap and their cached tokens are even cheaper...and they have the most effective caching in the business, as far as I can tell. Even naively using the API you get 80-90% cached token rate. If you use Reasonix, you get ~98% cached token rate. I just built a feature for an app I'm working on for $0.10 for 20 minutes of work. Not bad at all.
i wish there was a system like strix halo, but with enough lanes for a dedicated PCIe 5.0 x16 slot so you can have the best of both worlds: large sparse models on CPU with unified memory, dense models on GPU with real tensors and higher bandwidth memory.
32 Gb DDR4 RAM module has a bandwidth of 25 Gb/s and costs $160. If you buy 8 of these, you get 256 Gb RAM with 200 Gb/s bandwidth at $1280. And if you buy 16 x 16 Gb modules (each at $60) then you can get 400 Gb/s of bandwidth for $960.
The only problem, you need 8 or 16 memory controllers. So isn't it better to make a cheap CPU with 16 DRAM controllers instead of this $4K gear having only 128 Gb? Or maybe 2 CPUs each having 8 RAM channels?
DDR5 costs 2 times more ($360 for 32 Gb) while not even having 2 times the bandwidth so it is not worth buying. It is more reasonable to make more RAM channels and stuff them with DDR4.
So what I am trying to say, industry took a wrong turn. Instead of moving to over-priced DDR5, they should just make even cheapest CPUs support 8/16 DDR4 channels. Because a 32Gb DDR5-4800 module costs $360, and two 32Gb DDR4-3200 modules cost $320, so you get twice more size, more bandwidth and it costs you less. DDR5 is just a rip off.
Each memory controller interface is a not-insignificant number of PCB traces. Increasing the number of memory controllers may dramatically increase the number of PCB layers (or may not, it really depends on the CPU pinout) but it definitely will increase the number of pins on the CPU socket.
This is one of the main reasons (the other is the number of PCIe lanes) why high end desktop and server CPUs have like double the number of pins and so much bigger sockets as compared to consumer desktop CPUs.
To get memory mapping across systems you'd want to look into infiniband (https://en.wikipedia.org/wiki/InfiniBand) but the adapters generally aren't cheap. Common to find in super computer clusters.
If you have a desktop CPU with 2 memory channels and 2 DIMMs per channel, then the 2 DIMMs on each single channel share all of the address and data lines, there's just a chip select difference to pick which DIMM you're addressing. There is some extra loading on the lines, since you have 2 DIMMs, hence the supported speeds are usually slightly lower, but you only add 1-2 more actual PCB traces to have 2 DIMMs per channel vs 1 DIMM per channel. In contrast, adding another memory controller would add upwards of almost 100 additional PCB traces from the CPU.
Well to be honest, there are a lot of NOOP pins on CPUs, but using them basically means fabbing a new die altogether, which is basically making a whole new CPU altogether.
This is an odd comment. Pins are seriously expensive. Companies don't just throw them in for fun. Usually they're necessary for signal integrity or packaging constraints.
In the alternate reality where this happened, wouldn't the price of DDR4 still be sky high? We'll ignore any costs for CPU, chip set, and motherboard redesign. You're just pushing the demand somewhere else.
The problem is if DDR4 was viable here it would also be absurdly expensive. There’s no world in which we survive this demand without a corresponding increase in supply. It’s not like there are zettabytes of spare cheap DDR4 sitting around.
In addition to cost and possibly market segmentation, presumably the additional power consumption wouldn't be worth the tradeoff for laptop parts. If you want high channel count you've always been able to purchase power hungry datacenter gear. You can also pick up surplus 10 GbE or even 100 GbE fiber NICs to link up your beowulf cluster. You'll probably have to run a few new electrical circuits and a small bit of HVAC if you're installing this in a residential home but it's entirely doable to cram 10 kW in a closet if you feel like it.
Alternatively you could just rent cloud instances by the hour to avoid the hassle.
166 comments
[ 0.22 ms ] story [ 80.3 ms ] threadIt has the same 256 GB/s memory bandwidth limit as every board previously, not sure why this is even being released right now as if it's some new fangled thing - you can go get a Framework Desktop for roughly the same price or a GMKtec EVO-X2 for a bit cheaper.
If you DIY your own SSD, you can spec a Framework Desktop for below $4k; but not much below. Roughly the same price.
Absolute insanity
Regretting not buying the Framework back when it was around $2k.
[0] https://www.gmktec.com/products/gmktec-evo-x3-ai-mini-pc-amd...
The memory shortages won't last forever; when companies start adding capacity I wouldn't be surprised to see massive sticks of RAM being sold for consumers.
Well, almost. Maybe. Not all units. But probably.
Absolutely no reason that these need to be capped at 256 GB/s other than shortsighted design from three years ago.
(I agree that you almost certainly don't need this)
https://github.com/pettijohn/corsair-ai-workstation-performa...
https://lemonade-server.ai/
Hardware is the exact same as what used to be available for $2K last year (and is still $1K cheaper from Chinese OEMs).
LTT Lab's LLM testing is getting more sophisticated, which is great - I think it's worth noting that ROCm/Vulkan versions and llama.cpp build versions are going to have some big differences for numbers.
For those wanting to get the most out of their Strix Halos, there's both kernel tweaks and utilities like ryzenadj that can help you get the most out of it. ( http://strixhalo.wiki/ has most of that documented). Also, if you're running for coding or agentic work, if you model supports MTP, that's mature and should give you a decent (30%?) decode boost.
just a few weeks ago they backported a null deference kernel oops into stable, it's still not fixed.
Whenever I rebuild llama.cpp, I wind up using the Vulkan build anyway.
They have made a few attempts at investing in the hardware, but the software side is letting them down hard, and that part is almost entirely their own fault. They have underinvestment in their own stack, and into popular standard and Community libraries that would make it easy to use their gear.
Also, when you do Python work, you have to remember to install dependencies like pytorch from an alternate repo, otherwise you wind up with the CUDA versions that only use the CPU and not the GPU through ROCm.
I haven't finished comprehensive tests but I found:
- Vulkan is up to 2.25X faster for most coalesced, strided and interleave variants for memory-side scheduling/access shapes
- 3.3X faster on specific dot-path sweeps, including for scalar-dequant
- For matched LDS, Vulkan can be 8-14X+ faster (!!!) than matched HIP LDS
HIP doesn't always win against RADV/ACO, but on dispatch/runtime, it does appear to be quite a bit faster than HIP/LLVM on gfx1151 (Strix Halo). I'll be publishing sharing full data once I also run vs gfx1100...
Personally I’ll never buy AMD again.
As traditionally AMD was a supplier of parts.
But when they cost the same price (unless the Spark has shot up too), there's no reason to buy this over a Spark.
The Spark is literally a faster version of this, with better software support.
Edit: And I say that as an owner of a Ryzen AI Max 395 device.
The CPU of Ryzen is better than that of DGX Spark, especially for modern programs that have been updated to use AVX-512 (i.e. it has a significantly higher multithreaded performance).
Only for GPU applications the NVIDIA system is likely to be better.
Difference is on the Spark you can use and learn CUDA, vLLM, SGlang etc which is the industry standard.
I bought mine (ASUS version) back in December with the intent to learn that stack of stuff. I've been on and off with it but it seems to have paid off, looks like I might be getting work in the inference serving space.
it allows you to run smaller models much better
imo 3090s make the most sense if you can buy at least 2x ideally 4x but of course we're talking about a completely different budget at that point
And if your jobs do fit onto a 24GB card, then you are not the target user for the "AI mini PC" niche that these guys are trying to carve out
"The Apple Silicon Mac Studios outperform the AMD Ryzen AI Max+ 395 machines"
The framework desktop even has a usable PCIe 4x slot available if you put the board in a different case. They sell the 128GB board on its own for $3150.
128 bit: 96 GB?
256 bit: 192 GB
512 bit: 384 GB?
1024 bit: 768 GB?
I fear that by the time the RTX Spark comes out it'd have to be $6k, and by the time a 128gb or more machine with 700+gb/s comes out it'd be at $10k, way out of most consumers' hands.
There will be a basic M6.
There was a post recently, that showed the DGX spark is twice as fast as the M3 Ultra at prompt processing, but half as fast at output tokens [2]. They used gps-oss-120 for that test with a small context.
[1] https://gpuquicklist.com/apus?models=GB10%20Grace%20Blackwel...
[2] https://aimultiple.com/dgx-spark-alternatives , https://news.ycombinator.com/item?id=48732679
I mostly run them clustered for DS4 and am quite happy with the performance, and the cost isn’t that much more for two than the MBP while giving me double the unified memory.
I’ll probably pick up a third to run multiple smaller models. I don’t understand why people would buy a halo over a spark at comparable prices, particularly because if you want to cluster, the cx7 be beats the shit out of them when it comes to latency and throughput
I like my Strix Halo and keep it chewing on stuff, mostly non-interactive workloads (security audits of software mostly, training experiments, etc.), I get a lot of use out of it. If you want to experiment with AI, it is a good platform for that, though at $4k you can get an Nvidia-based Asus Ascend GX10, which is probably better. But, if you want a local model for interactive agentic use, you're going to be running either Qwen 3.6 or Gemma 4, which will fit comfortably on 2x64GB GPUs (even old GPUs will run them faster than the Strix Halo...I have dual Radeon Pro V620s which are faster, and they're six years old), or snugly on 32GB. A 48GB or 64GB Mac would run them well. Two Radeon AI Pro R9700 GPUs is probably the sweet spot, right now for GPUs. Not the cost of a good used car, like a 5090 or 4090, but plenty of memory and performance for local inference. Also, not finicky and weird and needing custom 3D printed fan shrouds like the old server GPUs on eBay.
At the moment, there just isn't a model that works better on a 128GB inference machine like this that don't also work fine on 64GB machines, which may be faster (very few 32GB GPUs will be slower, though I wouldn't recommend buying any GPU that isn't currently actively supported by the vendor drivers and CUDA or ROCm...so probably don't buy an MI50 or V100 or whatever).
ctx_tokens,prefill_tokens,prefill_tps,gen_tokens,gen_tps,kvcache_bytes 2048,2048,202.02,128,15.31,52184460 4096,2048,211.03,128,14.64,80373132 6144,2048,208.04,128,14.59,108561804 8192,2048,200.78,128,14.43,136750476 10240,2048,203.04,128,14.37,164939148 12288,2048,200.82,128,14.27,193127820 14336,2048,198.62,128,14.22,221316492 16384,2048,196.14,128,14.20,249505164 18432,2048,189.48,128,14.13,277693836 20480,2048,186.59,128,14.06,305882508 22528,2048,183.88,128,13.99,334071180 24576,2048,183.38,128,13.92,362259852 26624,2048,181.57,128,13.87,390448524 28672,2048,183.46,128,13.80,418637196 30720,2048,181.80,128,13.73,446825868 32768,2048,175.93,128,13.55,475014540 34816,2048,175.42,128,13.46,503203212
https://kyuz0.github.io/strix-halo-ds4-toolbox/
Which almost exactly matches the benchmark you just linked. Looks like it's possible to goose it to 15 tokens per second with a tiny context, but why would I want a giant model with a 2k context? DeepSeek is too big to be fast enough on a Strix Halo.
To be clear, if you think that's comfortable for interactive use, more power to you. But, I'm not waiting for that. I'll pay DeepSeek to host it for me. Their token prices are quite cheap and their cached tokens are even cheaper...and they have the most effective caching in the business, as far as I can tell. Even naively using the API you get 80-90% cached token rate. If you use Reasonix, you get ~98% cached token rate. I just built a feature for an app I'm working on for $0.10 for 20 minutes of work. Not bad at all.
I think once someone comes up with a machine which has both it will easily sell for $10000 and people will be queueing to buy it.
https://github.com/jamesob/local-llm
Plus a reasonably inexpensive super low-latency interconnect.
The only problem, you need 8 or 16 memory controllers. So isn't it better to make a cheap CPU with 16 DRAM controllers instead of this $4K gear having only 128 Gb? Or maybe 2 CPUs each having 8 RAM channels?
DDR5 costs 2 times more ($360 for 32 Gb) while not even having 2 times the bandwidth so it is not worth buying. It is more reasonable to make more RAM channels and stuff them with DDR4.
This is one of the main reasons (the other is the number of PCIe lanes) why high end desktop and server CPUs have like double the number of pins and so much bigger sockets as compared to consumer desktop CPUs.
And as for DRAM channels, typical cheap motherboard has 2 channels and 4 slots, it should not be super difficult to add 2 more channels.
If you have a desktop CPU with 2 memory channels and 2 DIMMs per channel, then the 2 DIMMs on each single channel share all of the address and data lines, there's just a chip select difference to pick which DIMM you're addressing. There is some extra loading on the lines, since you have 2 DIMMs, hence the supported speeds are usually slightly lower, but you only add 1-2 more actual PCB traces to have 2 DIMMs per channel vs 1 DIMM per channel. In contrast, adding another memory controller would add upwards of almost 100 additional PCB traces from the CPU.
Isn't adding pins kind of expensive?
Yes they spent those costs to switch from DDR4 to over-priced DDR5 and I suggested the cost could be spent on adding more DDR4 channels instead.
In addition to cost and possibly market segmentation, presumably the additional power consumption wouldn't be worth the tradeoff for laptop parts. If you want high channel count you've always been able to purchase power hungry datacenter gear. You can also pick up surplus 10 GbE or even 100 GbE fiber NICs to link up your beowulf cluster. You'll probably have to run a few new electrical circuits and a small bit of HVAC if you're installing this in a residential home but it's entirely doable to cram 10 kW in a closet if you feel like it.
Alternatively you could just rent cloud instances by the hour to avoid the hassle.