Is LMDeploy the Ultimate Solution? Why It Outshines VLLM, TRT-LLM, TGI, and MLC (bentoml.com) 16 points by helloericsf 2y ago ↗ HN
[–] timliu9 2y ago ↗ Why was onnx not part of the tested runtimes? Seems like an oversight [–] chaoyu 2y ago ↗ onnx is not a good option for LLM type of autoregressive generation [–] helloericsf 2y ago ↗ Personally, I never seen onnx used for LLM.
[–] ssheng 2y ago ↗ How does Exllama rank among these? Heard good things about it. [–] helloericsf 2y ago ↗ Seems interesting! https://github.com/turboderp/exllama "A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights." [–] helloericsf 2y ago ↗ 4-bit quantization tends to come at the cost of output quality losses. https://github.com/ggerganov/llama.cpp/issues/9 [–] ssheng 2y ago ↗ Quality loss with quantization is expected. It seems like with GPTQ the loss is within acceptable range based on the perplexity score shown.
[–] helloericsf 2y ago ↗ Seems interesting! https://github.com/turboderp/exllama "A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights."
[–] helloericsf 2y ago ↗ 4-bit quantization tends to come at the cost of output quality losses. https://github.com/ggerganov/llama.cpp/issues/9 [–] ssheng 2y ago ↗ Quality loss with quantization is expected. It seems like with GPTQ the loss is within acceptable range based on the perplexity score shown.
[–] ssheng 2y ago ↗ Quality loss with quantization is expected. It seems like with GPTQ the loss is within acceptable range based on the perplexity score shown.
[–] ShawnBasquiat 2y ago ↗ Why aren't there more of these benchmark studies? How did TGI make the cut?
9 comments
[ 3.4 ms ] story [ 29.0 ms ] thread