As this article is from some weeks ago and Huggingface has now implemented Paged Attention in text-generation-inference[1], I would assume the benchmark results would be quite different if done today. Would be very interesting to see more recent benchmarks if anyone has done any!
As always the LLM hamster wheel is moving too fast. I’d love to matrix download half this stuff (I still enjoy learning so classic-learn the other half)
> For example, suppose you prompt with a sentence "What is the capital of California: ", it would take ten forward pass iterations to get back the full response of ["S", "a", "c", "r", “a”, "m", "e", "n", "t", "o"]
From the content I have been reading trying to understand LLMs, I thought that the output was a token and not a string of chars. What am I missing here?
The tokens are chosen by a method called BPE, and there are single letter tokens. IOW you can encode the same text many ways. That said, this is probably just shown like this for illustrative purposes.
It's mentioned in the article: "This example simplifies things a little bit because in actuality tokens do not map 1:1 to ASCII characters (a popular token encoding technique is Byte-Pair Encoding which is beyond the scope of this blog post), but the iterative nature of generation is the same regardless of how you tokenize your sequences."
It's an extremization that still is true for character-based models.
Is that because four letters happen to pack into a word on most architectures or is there some other underlying reason? Also Sac and rament together already form a word so it's kind of logical have the final 'o' as a separate token.
The underlying reason is closer to your second observation. Statistically chosen tokens are most likely to be reusable and composable. Much less to do with hardware architecture. Token vocabulary happens before any quantization where processor word size may play a role.
It's just a function of finding the most efficient solution for encoding a corpus of text into a given vocabulary size. Using something like SentencePiece you can define how big you want your vocabulary to be (the number of discrete tokens), and it will find the best solution of subwords/characters in a sample set.
It's very hard to filter blatant self-promotion and marketing jargon from actual innovation (which goes at a much slower pace) these days.
The speed people are just creating wrappers or minor changes and using words like "disruptive", "game-changing", "democratising <something>" just feel so inflated and boring at this point.
"The speed people are just creating wrappers or minor changes"
To be fair, the overhead of running LLMs at scale -- the cost per interaction -- is the limiting factor for commercial deployments. Efficiency is an enormous advantage.
Your account reliably and almost exclusively posts in AI related topics, or brings AI topics into other threads with predictably negative commentary. I would have assumed this would have been right up your alley.
27 comments
[ 3.0 ms ] story [ 61.2 ms ] thread[1] https://github.com/huggingface/text-generation-inference/iss...
> For example, suppose you prompt with a sentence "What is the capital of California: ", it would take ten forward pass iterations to get back the full response of ["S", "a", "c", "r", “a”, "m", "e", "n", "t", "o"]
From the content I have been reading trying to understand LLMs, I thought that the output was a token and not a string of chars. What am I missing here?
It's an extremization that still is true for character-based models.
LLMs use a vocabulary of statistically chosen tokens. GPT 3 vocab, for instance, splits Sacramento into three tokens-
Sac - 38318
rament - 15141
o - 78
There is a rule of thumb that about every four letters in English text becomes a token but that's just the average.
California is a single token (25284). As is Canada (17940). And so on.
In the case of GPT-3 the vocabulary has a size of 50,257 tokens. GPT-4 increases that past 100k (see https://gist.github.com/s-macke/ae83f6afb89794350f8d9a1ad8a0...).
It's very similar to compression algos, really. Find recurring sets of characters.
The speed people are just creating wrappers or minor changes and using words like "disruptive", "game-changing", "democratising <something>" just feel so inflated and boring at this point.
I hope this gets very soon into the next phase of the hype cycle[1], so the we can talk about something else. [1] https://en.wikipedia.org/wiki/Gartner_hype_cycle
To be fair, the overhead of running LLMs at scale -- the cost per interaction -- is the limiting factor for commercial deployments. Efficiency is an enormous advantage.
/runs
Kinda hard to believe, but I have no intuition about this.