Activation-Aware Weight Quantization for LLM Compression Outperforms GPTQ (arxiv.org)

Better quantization will have a direct and meaningful impact for everyone running local LLMs. The technique has already been applied to both Vicuna and the multimodal LLaMA variant LLaVA.

https://github.com/mit-han-lab/llm-awq

They claim average 1.45x speedup and maximum 1.7x speedup over GPTQ.

High-level idea: About 1% of the weights contribute greatly to quantization error. So skip the quantization of these weights.