Do you have more info on that? Their support site[1] claims their data collection is much more constrained than that (although still seems like a lot).
Ah, you’re right! I’m basically repeating stuff I got from a somewhat recent HN thread [1] which I found shocking when I first read it, except it’s not true and I did not bother checking it because it was not unthinkable and… confirmation bias, I guess. Now, if only I could go back and still edit the original post…
Thanks for catching my bullshit, I stand corrected (and should have been wiser).
TechPowerUp's nvcleanstall is a pretty lean NVidia driver installer. It fetches from TPUs servers, and installs only user-selected driver components. The screenshots linked do a good job of communicating what it does.
Has a scheduled update-check if you want that too.
I'm looking forward to CodeLlama being usable on a Dev system using some CPU offloading and only 32GB _while developing in an IDE_. Code Llama is great but using it alongside an IDE just kicks swapping into overdrive, and at least this will mitigate some of that loss on the LLM generation side (at least for the part in GPU ram).
Where is AMD in all this? Like M1 Macs, their Ryzen APUs have access to the whole system memory and AI accelerators, so I would like to get a laptop with 32-64GB of RAM just to fit giant LLMs in it, instead of splurging a lot more on RTX cards with large VRAM even if I don't game, but AMD isn't even trying.
It appears that it's already in there. The trick is that you have to install an extension in some apps to make it work. For instance, there's a TensorRT accelerator extension for Stable Diffusion WebUI that takes advantage of it. I'm installing it right now, though the docs leave a bit to be desired.
I got it installed and tested with an SD 1.5 model in Stable Diffusion Webui using a 4090. (SDXL models did not work.) I generated the same test prompt with and without their TensorRT acceleration. The generated images are nearly identical, with very minor, almost imperceptible differences, which is pretty standard for techniques that optimize the attention layers.
It's faster, but clearly it's not 4x faster. I suppose they cherrypicked benchmarks against generation techniques not using xformers or SDP Attention. Also, it appears to be limited to a max batch size of 4.
These types of optimizations are what I figured Carmack would be doing at Keen. He's so good at optimization and low-level hardware understanding that I thought he might be able to wring some small extra percentage out of existing hardware for AI tasks. With compute currently as expensive as it is, that would be extremely valuable.
34 comments
[ 4.7 ms ] story [ 93.6 ms ] thread1: https://nvidia.custhelp.com/app/answers/detail/a_id/3188/~/w...
Thanks for catching my bullshit, I stand corrected (and should have been wiser).
[1] https://news.ycombinator.com/item?id=37033859
Maybe someone wrote a script that automatically scrapes the website and downloads the newest driver automatically?
Has a scheduled update-check if you want that too.
https://www.techpowerup.com/nvcleanstall/
Haven’t found any good setup instructions for Linux or my Google skills are failing me.
https://github.com/ggerganov/llama.cpp/issues/34
If you meant eGPU support, IIRC that is beta for everyone right now.
I mean I guess they go a lot wider on a GPU, but there's no reason an APU couldn't do that too.
https://github.com/NVIDIA/Stable-Diffusion-WebUI-TensorRT
Prompt: A cat wearing pajamas Model: deliberate_v2 Size: 768x512 Batch Size: 4 Clip Skip: 2 Samler: DPM++ 2S a Karras Steps: 70 Seed: 2024828515
Generation times avg over 5 runs
================================
without TensorRT: 19.2 seconds
with TensorRT: 12.3 seconds
Benchmarks
Batch Size 1 / 2 / 4
=======================================
without TensorRT: 31.11 / 36.23 / 42.34
with TensorRT: 55.32 / 58.06 / 62.27
It's faster, but clearly it's not 4x faster. I suppose they cherrypicked benchmarks against generation techniques not using xformers or SDP Attention. Also, it appears to be limited to a max batch size of 4.