5 comments

[ 0.23 ms ] story [ 18.3 ms ] thread
the useful split for me is interactive vs batch. keep a small model warm for private jobs and measure queue time, not just tokens/sec.
agreed queue time should be measured and added to the total wait time for the end user
No. There is no evidence for AI profitability which can support this investment.