What's the difference between a standard and distributionally robust language model? The paper didn't say anywhere.
[edit] After some quick reading:
A language model is a model which (roughly speaking) tries to predict the next character in a text string, given a text string as input. For example: "Let there be l" -> "Let the be li".
A distributionally robust language model is one whose training objective is supposed to help it generalise to topics which weren't in its training set. For instance: A model trained on the news won't necessarily work correctly on Amazon reviews. It seems that there ought to be many ways of achieving this objective; and the details of how it does it ought to affect how well it achieves this goal.
I'm still not sure what distribution changes the article authors wanted their models to be robust to. Can anyone explain?
Hey there! I'm the first author and I'm very shocked to see this on HN. To answer your question -- the distribution change that the models should be robust to is the multilingual lexicon, as discussed with the Singlish example. We know that a speaker's linguistic background can influence how they communicate in Creole. So with the Singlish example, a sentence with the same meaning can be really different if its a person who also speaks Mandarin, versus someone who speaks Malay. Maybe in a training set, we'll see that a dataset is actually biased towards examples containing Mandarin, and it then becomes important to be robust towards the lesser represented Malay then, for example.
Singlish and Nigerian Pidgin are a really good examples of Creoles where we would expect robustness to multilingual vocabulary to be important, as these Creoles are are "linguae francae" within their respective countries (although, these two languages have VERY different levels of acceptance within their respective societies). Looking back on this work, Haitian Creole makes less sense, as Haiti is largely a monolingual country, with only a few also speaking French. But with the history of the language, the vocabulary is still mixed between French origin and like Akan/Igbo/Yoruba etc.
2 comments
[ 3.0 ms ] story [ 16.4 ms ] thread[edit] After some quick reading:
A language model is a model which (roughly speaking) tries to predict the next character in a text string, given a text string as input. For example: "Let there be l" -> "Let the be li".
A distributionally robust language model is one whose training objective is supposed to help it generalise to topics which weren't in its training set. For instance: A model trained on the news won't necessarily work correctly on Amazon reviews. It seems that there ought to be many ways of achieving this objective; and the details of how it does it ought to affect how well it achieves this goal.
I'm still not sure what distribution changes the article authors wanted their models to be robust to. Can anyone explain?
Singlish and Nigerian Pidgin are a really good examples of Creoles where we would expect robustness to multilingual vocabulary to be important, as these Creoles are are "linguae francae" within their respective countries (although, these two languages have VERY different levels of acceptance within their respective societies). Looking back on this work, Haitian Creole makes less sense, as Haiti is largely a monolingual country, with only a few also speaking French. But with the history of the language, the vocabulary is still mixed between French origin and like Akan/Igbo/Yoruba etc.
Hope this helps, thanks for taking a look :-)