text-generation-webui supports state of the art 4bit GPTQ quantization for LLaMA[0], reducing VRAM overhead by 75% with no output performance loss compared to baseline fp16.[1]
LLaMA-13B, rivaling GPT-3 175B, requires only 10GB* of VRAM with 4bit GPTQ quantization.
LLaMA-30B fits on a 24GB* consumer video card with no output performance loss, beating GPT-3 175B.
Multi-GPU support[2] means LLaMA-65B, rivaling PaLM-540B, runs on 2x3090.
*Further improvements in active development will reduce VRAM requirements another 30-40% with no performance loss (ex. flash attention).
Yes, text-generation-webui has multiGPU support regardless of your GPU(s). They do not even need to be the same GPU. You can have an AMD 6900xt combined with an RTX 4090, combined with an RTX 3050, if you want. It makes no difference.
6 comments
[ 15.3 ms ] story [ 554 ms ] threadLLaMA-13B, rivaling GPT-3 175B, requires only 10GB* of VRAM with 4bit GPTQ quantization.
LLaMA-30B fits on a 24GB* consumer video card with no output performance loss, beating GPT-3 175B.
Multi-GPU support[2] means LLaMA-65B, rivaling PaLM-540B, runs on 2x3090.
*Further improvements in active development will reduce VRAM requirements another 30-40% with no performance loss (ex. flash attention).
[0] https://github.com/qwopqwop200/GPTQ-for-LLaMa
[1] https://arxiv.org/abs/2210.17323
[2] https://github.com/oobabooga/text-generation-webui/issues/14...
For LLaMA set-up instructions, including GPTQ 4bit, refer to this wiki article: https://github.com/oobabooga/text-generation-webui/wiki/LLaM...
https://github.com/zphang/minimal-llama#peft-fine-tuning-wit...