Lindon4290
No user record in our sample, but Lindon4290 has activity below (stories or comments). Likely we have partial data — the full bulk-load will fill profiles in.
No user record in our sample, but Lindon4290 has activity below (stories or comments). Likely we have partial data — the full bulk-load will fill profiles in.
Now, I don't have any MI300X, so I can't make any definite claims here. I am hoping someone else can replicate the results shown here or at the least educate me on how this is possible. Good part is the docker container…
You should make a whole post about this! Like how a single MI300X outperforms groq at bs=1. 300 tokens/s with bs=1 for a llama-2 70B on a single card is no joke.
Yeah, the rumours(?) are a groq system required to produce 300+ t/s on a Llama-2 70B (bs=1) requires 576 chips (9 racks) [1] So, that's like $10M+ for serving bs=1 Llama-2 70B vs whatever a single MI300X costs? [1]…
Going by the results from the article/video, a single MI300X is even outperforming a Groq system [1] The video shows that the optimized run with Llama-2 70B gives 314 tokens/s for a bs=1 with 256 prompt + 256…
Right. On that track, I want to confirm something. Maybe I am doing my math wrong or don't understand how transformers work. There is a video about the bs=1 case, i.e. a single prompt with input 256 tokens and output…