Can you LLM a custom language?
If token limit and accuracy are important, it seems English (or other spoken languages) are no optimal.
They're a butchered product of history and easy verbal noises.
A new custom language seems inevitable, that is concise, unambiguous, rooted in relation with custom words. Replacing common sentences with simple strings such as "Once upon a time..." to "a1"
Most likely alpha-numeric, to minimise tokens, and generate an order of magnitude increase in context window.
Followed by translation back to {language}
Is this possible? Anyone working on it?
(here to be educated)
2 comments
[ 4.7 ms ] story [ 11.4 ms ] threadThis is what byte-pair encoding does. It doesn't go quite so far as to allocate only a single token to "Once upon a time", because that string isn't actually that common, but in principle it could.
Trying to get humans to produce content directly in such a concise representation is a waste of time, since LLMs heavily rely on the ability to take whatever content is already available on the internet, which drastically reduces the labor cost of acquiring training data.