Aren't these specs already outdated? Seems meh. I would think players would like to bring something new to the table instead of bringing what was okay two years ago or so.
Give me 1TB RAM and 5TB/s or something that exciting.
What exactly do you propose they do? Do you think they are holding back supply? Should they sell at less than market price? That's just handing the profits to scalpers/middle men.
I'm sure NVIDIA, AMD, Intel, Qualcomm, etc. would sell more hardware if they could, but they can't.
For most purposes, this seems reasonable? This looks like a single-height card, so fitting four of these in a workstation or server would give 512 GB RAM and 2 TB/s total bandwidth. That's pretty attractive for a lot of current LLMs, even if one wishes for 1 TB capacity at 5 TB/s bandwidth.
But this gives you significantly more memory capacity for a compute/vram bandwidth tradeoff (i.e. arithmetic intensity) that's ultimately comparable to Blackwell, perhaps even more skewed towards compute. This seems good if you care about being able to store larger sparse models and more KV caches because that ultimately bottlenecks your ability to do inference.
Maybe. I don't know power for the MI350, but the RTX PRO 6000 is normally a 200-300W card, so you pay 2x the power for 3/4th the RAM and 3x the bandwidth. The performance frontier looks very different in terms of power to hold a mid-size model and throughout per user or across users. Price and inference stack would be big factors in what I would choose.
MI350P, H200 NVL and Nvidia RTX PRO 6000 are all 600W but compute wise all of them should be 3 to 4 times more performant than A100 Ultra according to their specifications.
600w is way off the vf curve though... Tiny bit of tuning and you'll get ~80-90% of the performance at ~150w nominal. At least that's what I got out of some pro 6000s when I spent 15 mins to underclock it.
> I would think players would like to bring something new to the table instead of bringing what was okay two years ago or so.
Speaking of... from [0]:
Introducing Qualcomm Cloud AI 100 Ultra
High performance and cost-optimized cloud AI inferencing for generative AI
Written by
Rashid Attar
Nov 13, 2023
Today we announce the Qualcomm Cloud AI 100 Ultra, a new member of our portfolio of cloud artificial intelligence (AI) Inference cards, purpose-built for generative AI and large language models (LLMs). ...
Ok so it’s significantly worse than Apple M6 and we don’t know if it’s cheaper or not
Edit: Ah sorry you’re right, the bandwidth is lower but the TOPS is a lot higher. I’m not sure how much effect you get from higher TOPS with such low bandwidth though.
> I’m not sure how much effect you get from higher TOPS with such low bandwidth though.
Raised arithmetic intensity. It would be quite good for LLM prefill, and decode would want to adapt by pushing parallelism higher (saving on memory bandwidth by reusing the same layer weights for multiple streams) or, if unable to do so (e.g. due to KV cache constraints) by resorting to speculative decoding (where the expensive 'validate' step handles tokens in parallel within a single stream). It does not make the platform inherently unviable. It would also do well at other tasks such as image/video generation, which is already quite compute bound.
I'm starting to believe in Qualcomm! The Snapdragon X2 Elite Extreme is alas only available in a single system, with it's tri-channel on-package (!) memory good for 228 GB/s. But it's a beast. The core is just good, even in the mainstream form in other systems: some of the best single core performance you can get, shy of Apple.
And they are upstreaming their stuff much better! They hired a bunch of really good known FOSS developers and it seems to be working!
Now they are talking up High Bandwidth Compute as their next big thing. And I want to believe, even if it makes me nervous that they too intend to leave consumer computing completely in the dust. https://www.qualcomm.com/news/onq/2026/11/qualcomm-hbc-valid...
This is, as commented, at M4 Max levels of GB/s. It is not impressive. It has some uses! But if this was their modern opening salvo I would not be impressed.
> tri-channel on-package (!) memory good for 228 GB/s
Why is on-package memory impressive/better in this case? It's one channel more than your typical consumer x86 system (192-bit bus vs 128-bit) which is something but you could technically have quad channel with two LPCAMM2 sockets?
As a computer engineer, I can barely state how excited I am for systems with high bandwidth where I don't have to route gobs and gobs of very high speed ram.
It's radically simpler and smaller a computer to build. That has huge upsides. For laptops maybe this isn't a huge win, but for tablets and smaller computers, this is such an incredible win. And for making it simpler to build computers, to expand the range of who can make systems? It's everything.
Small devices are the best use case for on package memory, yes. But at the same time I don't know of any that go beyond a 128-bit bus. Most are 64-bit.
58 comments
[ 2.7 ms ] story [ 45.4 ms ] threadGive me 1TB RAM and 5TB/s or something that exciting.
Plenty of businesses want in local inference and are priced out.
Nobody helps China like American corps helps China. Domestic anticompetitive behavior spurs demand for international competition
I'm sure NVIDIA, AMD, Intel, Qualcomm, etc. would sell more hardware if they could, but they can't.
But this gives you significantly more memory capacity for a compute/vram bandwidth tradeoff (i.e. arithmetic intensity) that's ultimately comparable to Blackwell, perhaps even more skewed towards compute. This seems good if you care about being able to store larger sparse models and more KV caches because that ultimately bottlenecks your ability to do inference.
https://www.servethehome.com/the-amd-instinct-mi350p-is-a-hb...
The thing you are describing is much closer to AMD and Nvidia latest generation server offerings.
See MI455X for the absolute cutting edge and MI350P for something you can use outside of a datacenter.
Speaking of... from [0]:
[0] <https://www.qualcomm.com/news/onq/2023/11/introducing-qualco...>Edit: Ah sorry you’re right, the bandwidth is lower but the TOPS is a lot higher. I’m not sure how much effect you get from higher TOPS with such low bandwidth though.
Raised arithmetic intensity. It would be quite good for LLM prefill, and decode would want to adapt by pushing parallelism higher (saving on memory bandwidth by reusing the same layer weights for multiple streams) or, if unable to do so (e.g. due to KV cache constraints) by resorting to speculative decoding (where the expensive 'validate' step handles tokens in parallel within a single stream). It does not make the platform inherently unviable. It would also do well at other tasks such as image/video generation, which is already quite compute bound.
I'm starting to believe in Qualcomm! The Snapdragon X2 Elite Extreme is alas only available in a single system, with it's tri-channel on-package (!) memory good for 228 GB/s. But it's a beast. The core is just good, even in the mainstream form in other systems: some of the best single core performance you can get, shy of Apple.
And they are upstreaming their stuff much better! They hired a bunch of really good known FOSS developers and it seems to be working!
Now they are talking up High Bandwidth Compute as their next big thing. And I want to believe, even if it makes me nervous that they too intend to leave consumer computing completely in the dust. https://www.qualcomm.com/news/onq/2026/11/qualcomm-hbc-valid...
This is, as commented, at M4 Max levels of GB/s. It is not impressive. It has some uses! But if this was their modern opening salvo I would not be impressed.
Someone drop the YouTube link of their robot presentation
https://youtube.com/shorts/aZmt5E151Bk
Why is on-package memory impressive/better in this case? It's one channel more than your typical consumer x86 system (192-bit bus vs 128-bit) which is something but you could technically have quad channel with two LPCAMM2 sockets?
It's radically simpler and smaller a computer to build. That has huge upsides. For laptops maybe this isn't a huge win, but for tablets and smaller computers, this is such an incredible win. And for making it simpler to build computers, to expand the range of who can make systems? It's everything.
It's a trend I like.