[dead]
With 128 GB of unified memory, large models can fit into memory, but memory bandwidth quickly becomes the limiting factor. With long contexts in particular, the KV cache can consume a lot of memory and bandwidth.…
[dead]
With 128 GB of unified memory, large models can fit into memory, but memory bandwidth quickly becomes the limiting factor. With long contexts in particular, the KV cache can consume a lot of memory and bandwidth.…