Lindon4290
No user record in our sample, but Lindon4290 has activity below (stories or comments). Likely we have partial data — the full bulk-load will fill profiles in.
No user record in our sample, but Lindon4290 has activity below (stories or comments). Likely we have partial data — the full bulk-load will fill profiles in.
Yes, a single sequence with 256 prompt tokens and 256 output tokens. This is a batch size = 1. No one is saying anything about 256 batches. The first step in understanding this is to notice that the model (llama2)…
Yeah, I've asked questions along those lines as well. Something sketchy is going on. See here - https://news.ycombinator.com/item?id=40833109
Now, I don't have any MI300X, so I can't make any definite claims here. I am hoping someone else can replicate the results shown here or at the least educate me on how this is possible. Good part is the docker container…
Yeah, the rumours(?) are a groq system required to produce 300+ t/s on a Llama-2 70B (bs=1) requires 576 chips (9 racks) [1] So, that's like $10M+ for serving bs=1 Llama-2 70B vs whatever a single MI300X costs? [1]…
Going by the results from the article/video, a single MI300X is even outperforming a Groq system [1] The video shows that the optimized run with Llama-2 70B gives 314 tokens/s for a bs=1 with 256 prompt + 256…
Right. On that track, I want to confirm something. Maybe I am doing my math wrong or don't understand how transformers work. There is a video about the bs=1 case, i.e. a single prompt with input 256 tokens and output…
Looks their performance is better than llama.cpp - https://news.ycombinator.com/item?id=37018989 - and scales to batches of prompts.