It really is not. The 70b model just does not fit into RTX 4090. As seen in the same video for models that fit 4090 is about 2 times faster.
Basically the task is memory bandwidth bound and 4090 has 2x bandwidth of M3 Max. Incidentally you can get ~ the same memory bandwidth as M3 Max with newly released 7945WX Threadripper for $1400 + the cost of DDR5. And you will be able to scale the amount of RAM as much as you want for bigger models (currently up to 2TB) by just buying new RAM.
It really is not. The 70b model just does not fit into RTX 4090. As seen in the same video for models that fit 4090 is about 2 times faster.
It really is. The LLM is benefiting from the high amount of high bandwidth RAM, which is a benefit of UMA in Macs. To get 128GB or more RAM from Nvidia, you'd have to combine two H100 GPUs, which retail for $30k+ each.
Incidentally you can get ~ the same memory bandwidth as M3 Max with newly released 7945WX Threadripper for $1400 + the cost of DDR5. And you will be able to scale the amount of RAM as much as you want for bigger models (currently up to 2TB) by just buying new RAM.
The CPU would have less compute power. I'd be curious to see the test results of an Epyc 128-core CPU with 12 memory channels though.
MI250 (which is two GPUs) has 128 GB and H100 NVL (also two GPUs) has 188 GB at 7.8 TB/s. Both are probably an order of magnitude more expensive than a 4090 so they're not really in the same class.
This comparison is not fair, since the VRAM in the RTX4090 is not enough to hold the whole model in VRAM.
I have tested llama.cpp both on an M2 and in a 4090:
- The prompt ingestion time in M2 is pretty slow.
- The extra memory of the M2 allows one to try more interesting models (Mixtral) and run multiple models at the same time.
6 comments
[ 3.1 ms ] story [ 17.9 ms ] threadBasically the task is memory bandwidth bound and 4090 has 2x bandwidth of M3 Max. Incidentally you can get ~ the same memory bandwidth as M3 Max with newly released 7945WX Threadripper for $1400 + the cost of DDR5. And you will be able to scale the amount of RAM as much as you want for bigger models (currently up to 2TB) by just buying new RAM.
I have tested llama.cpp both on an M2 and in a 4090:
- The prompt ingestion time in M2 is pretty slow. - The extra memory of the M2 allows one to try more interesting models (Mixtral) and run multiple models at the same time.