GCP system to run llama2 Chat model 7B 13B and 70B

8 points by Hammad_Ahmad 2y ago ↗ HN

which system is best to run llama2 Chat 7B 13B and 70B model on GCP for production? So it can handle multiple requests and mention multiple user chat sessions at the same time. And do load balancing effectively

4 comments

[ 3.1 ms ] story [ 18.7 ms ] thread

tikkun 2y ago ↗

Look into vllm and text-generation-inference

Why not use an API though?

Some to check out are:

Replicate, Together AI, Fireworks AI, Deepinfra, Anyscale Endpoints

extasia 2y ago ↗

I can personally recommend deepinfra. Been using it the past few days and it's served me very well.

yeesian 2y ago ↗

https://cloud.google.com/vertex-ai/docs/start/explore-models...

gongy 2y ago ↗

You can have load balanced inference and also fine tune models on Modal (disclaimer: I wrote this guide)

You can get a load balanced endpoint serving hundreds of tok/s per GPU using vLLM or Text Generation Inference in a few lines of code:

https://modal.com/docs/guide/ex/text_generation_inference