There has been some reporting that Mandarin and languages similar to it provide several advantages over using English. First, the theory is that a language like Mandarin can encode more complex ideas using much less memory than it would take in a language like English.
Some reports, which I can find and post here if necessary, claim this can lead to a 40% or so overall performance difference.
There is also a view that due to the way complex meanings are encoded in a pictograph type language that it improves the inference stage and ultimately greatly reduces hallucinations.
There has been some work from Microsoft and others to compress tokens from the user side. Other papers have suggested the advantages are so great that a new kind of symbol based language should be created to use for all of the training data.
Does anyone have any experience with this sort of LLM optimization? Is Mandarin and similar languages more efficient for LLMs?
Take a language that is equidistant from english and chinese, like Malay, and translate that dataset into both languages. Then train a model and benchmark them. I suspect the differences would be irrelevant.
2 comments
[ 4.5 ms ] story [ 16.3 ms ] threadSome reports, which I can find and post here if necessary, claim this can lead to a 40% or so overall performance difference.
There is also a view that due to the way complex meanings are encoded in a pictograph type language that it improves the inference stage and ultimately greatly reduces hallucinations.
There has been some work from Microsoft and others to compress tokens from the user side. Other papers have suggested the advantages are so great that a new kind of symbol based language should be created to use for all of the training data.
Does anyone have any experience with this sort of LLM optimization? Is Mandarin and similar languages more efficient for LLMs?