Time to First Intelligence
Slowly but surely we are now in the era or AI 2.0 where applications are being built on top of these models. "Time to first token" or ttft becomes less important because your output could be structured and you might need to do additional parsing on top of it to get meaningful UX out of it.
In such a world I believe something akin to "Time to First Intelligence" is much more important. To put it simply: How long does it take after the user has sent a request for the system to display a sign of intelligence?
Moving away from the chat paradigm, its more akin to, how quickly you can grab useful information from the AI generated stream of output something intelligent to the user. Often times when building LLM/LRM powered applications we do not think too much about it, but as LLM/LRM become staple, the right UX which can show signs of intelligence more quickly (and is correct) would win over a system which is slower.
Let's take a look at Perplexity or the new o1-preview model which OpenAI released. Perplexity before giving you the final answer, shows the steps its taking to generate the answer. As a user you still feel in the loop even tho these intermediate queries might not be the answer you want, but being able to glean through them allows you to very quickly understand the direction of the answer.
o1-preview on OpenAI's chat website also exposes the reasoning threads. Reading through them has been fascinating since you can kind of understand the direction or intent of the model before getting the final answer.
On the flip side of this is the API endpoints, o1-preview does not stream which means Application developers have to wait for a minute+ and get a final wall of text, Anthropic on the other hand streams their output tokens. I strongly believe part of the reason people still prefer sonnet3.5 over o1-preview has nothing to do with the level of intelligence (both are on similar levels with o1-preview ahead of sonnet3.5) but the fact that the user feels in control and engaged throughout the response.
To summarize, optimize your applications for "time to first intelligence", this can often times be harder to do and requires more rigour but your end users will love the extra speed and feel more engaged.
What do you all think? Does this resonate with how everyone is thinking about AI first applications now a days
2 comments
[ 4.0 ms ] story [ 36.4 ms ] threadI think in the same vein, the first token generated is also not as important as the final answer, without the full context it generally gives no signal.
To measure the intelligence of a string of text, the context in which it has been generated is far more important.
I do think coming up with better ways to measure intelligence in any part of the answer O(10 tokens), O(20 tokens)... would be useful