I checked some logs from my past experiments, the decoding went for about 400 tps over a ~3k token query, so about 7 seconds to process it, and then the generation speed was about 28 tokens.
llama.cpp + llama-3-8b in Q8 run great on a single T4 machine. Cannot remember the TPS I got there, but it was much above 6 mentioned in the article.
Munich airport T2 has water fountains for at least the last 2 years
In theory it’s easier/possible with some types of models, harder/impossible with others, but only if the model and the data processing around it is disclosed. The bigger issue here is that some seemingly unrelated…
I checked some logs from my past experiments, the decoding went for about 400 tps over a ~3k token query, so about 7 seconds to process it, and then the generation speed was about 28 tokens.
llama.cpp + llama-3-8b in Q8 run great on a single T4 machine. Cannot remember the TPS I got there, but it was much above 6 mentioned in the article.
Munich airport T2 has water fountains for at least the last 2 years
In theory it’s easier/possible with some types of models, harder/impossible with others, but only if the model and the data processing around it is disclosed. The bigger issue here is that some seemingly unrelated…