HN Explorer

11 comments

[ 3.2 ms ] story [ 38.8 ms ] thread
metadat
How does 12 tokens/sec equate to satisfactorily severing thousands of end-customere?

I did enjoy the headline, and the register has formerly been an incredibly good news outlet. Prior to the founder passing away last week.

atherton33
From the article, it's not 12 total, but 12 per user for 100 concurrent generations.
metadat
Thank you atherton friend.
washadjeffmad
Capacity doesn't equate to quality, but I could easily see an 8B finetune with exl2 at low context working for short, simple customer interactions, akin to oversubscribing a 1Gbit uplink for 100 customers at 50Mbps.
Cordiali
> "even an old Nvidia RTX 3090"

Saying that like it's mediocre... Maybe I'll have to benchmark my old 1050, see what it can do!

fooblaster
Guess what Nvidia won't let you deploy in a data center!
stevenhuang
This is wildly misleading as the benchmarks make use of batching. It will entirely fall apart in real workloads where each prompt is different. If you're doing batch processing with a fixed prompt, the results will be more applicable.
PeterStuer
It depends. For batching to be viable each prompt has to share some similarities of context/intention, which quite often is the case in specific applications (as opposed to say general chats)
Havoc
Bought a 3090 because they are good value for this, but this logic is frankly a little ridiculous:

> Since only a small fraction of users are likely to be making requests at any given moment

So what if 5 out of the thousands happen to coincide?

kristoo
The benchmark result is for 100 concurrent users