Faster Mixtral inference with TensorRT-LLM and quantization (baseten.co)

TensorRT-LLM plus int8 quantization = cheaper, minimal quality loss. Some nice tables in the post