My issue is figuring out how to identify how many concurrent users you can support on average on a given GPU.
Understanding the vram to simply load the weights is easy enough. When you are allowing for something like content generation with varying lengths of input/output tokens, how do you even begin to identify the GPUs you need?
2 comments
[ 6.7 ms ] story [ 20.1 ms ] threadWrote a blog post to demystify the process of GPU memory usage estimating.
Understanding the vram to simply load the weights is easy enough. When you are allowing for something like content generation with varying lengths of input/output tokens, how do you even begin to identify the GPUs you need?