1. rent 4x H100 instance on http://app.hyperbolic.xyz/compute
(Llama 4 Scout has 109B parameters in bf16, so the weights are already 218GB)
2. open a terminal tool and SSH into the machine
3. run the following commands:
>> sudo apt-get update && sudo apt-get install -y python3-pip
>> pip install -U vllm
>> pip install -U "huggingface_hub[cli]"
4. get an access token on u/huggingface website and run
>> huggingface-cli login
5. use @vllm_project to serve Llama 4
>> vllm serve meta-llama/Llama-4-Scout-17B-16E-Instruct --tensor-parallel-size 4 --max-model-len 10000
6. open a new terminal and call the API to know "What can I do in SF?":
>> curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Llama-4-Scout-17B-16E-Instruct",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What can I do in SF?"}
]
}'
It's just that simple ;) A big thank you to @AIatMeta and @vllm_project for making it easy to access the best open intelligence!
1 comment
[ 3.1 ms ] story [ 11.1 ms ] thread2. open a terminal tool and SSH into the machine
3. run the following commands: >> sudo apt-get update && sudo apt-get install -y python3-pip >> pip install -U vllm >> pip install -U "huggingface_hub[cli]"
4. get an access token on u/huggingface website and run >> huggingface-cli login
5. use @vllm_project to serve Llama 4 >> vllm serve meta-llama/Llama-4-Scout-17B-16E-Instruct --tensor-parallel-size 4 --max-model-len 10000
6. open a new terminal and call the API to know "What can I do in SF?": >> curl http://localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "meta-llama/Llama-4-Scout-17B-16E-Instruct", "messages": [ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What can I do in SF?"} ] }'
It's just that simple ;) A big thank you to @AIatMeta and @vllm_project for making it easy to access the best open intelligence!