Yeah that makes sense. I'm more familiar with seeing tok/s/user + TTFT rather than the total node throughput.
Oh, that's much better then. A good metric to share is the tokens per second per user for the node rather than the total throughput of the node. It disambiguates what's being optimized for much better than your blog…
This is very interesting and yet not at the same time. This looks to be optimized for single-stream LLM traffic which is not viable to serve in a production setting. It's only interesting to hobbyists that want to run…
700 customers on a direct to consumer API is not a lot.
Yeah that makes sense. I'm more familiar with seeing tok/s/user + TTFT rather than the total node throughput.
Oh, that's much better then. A good metric to share is the tokens per second per user for the node rather than the total throughput of the node. It disambiguates what's being optimized for much better than your blog…
This is very interesting and yet not at the same time. This looks to be optimized for single-stream LLM traffic which is not viable to serve in a production setting. It's only interesting to hobbyists that want to run…
700 customers on a direct to consumer API is not a lot.