I generally agree with point made in the article that too much NLP research is focused only on English and a small number of other high-resourced languages. To me, this is part of a larger problem with natural language processing's obsession with "state-of-the-art" metrics and general abandonment of broader research in linguistics.
The situation for those who, like myself, work in applied linguistics is not actually so dire... At least when working with the languages that have a reasonable amount of training data. Decent enough treebanks exist for lemmatisation, POS-tagging, and dependency parsing for dozens of languages [0]. Fast tools such as spaCy (15 languages) [1] and udpipe (40+ languages) [2] are freely available and work well for most applied tasks. There are even decent word embeddings available trained on various versions of Wikipedia [3].
Of course, some of the issue is that these very tasks are biased towards the specific structures of Indo-European languages. However, for getting work done (building sentiment classifiers, document clustering, NER, ect.), currently available tools make it possible to work with a large proportion of the currently available data.
There is still a lot of work to be done own regards to computational research on non-English languages, but a lot of the problem right now is recognition of applied work by the top NLP conferences rather than a complete lack of quality work currently being done.
One thing i think about is what where would NLP and linguistics research gone if another language (other than english) was the predominant language. Granted there is quite of bit interesting fundamental research in other countries and languages. But it seems many of problems and approaches developed in language understanding and NLP target specific idiosyncracies of the English language. If NLP researched with say German (which has a deep and overly explicit vocabulary) or Hindi or Mandarian, would the fundamental NLP approaches and problem areas be different and potentially better?
I recommend you to check out NLP research regarding the Czech language. It has high quality researchers keen on making use of the language's features (it's a highly regular and expressive grammar). Sadly not much research is done, but what is done is interesting.
Agreed. The way I see it, current research directions have focused on approaches that work best with large datasets because such datasets exist for English, and because English just happens to be a language that is possible to represent well by training on large datasets.
It helps of course that English is the de facto lingua franca of most of the technologically developed world (if not the rest of the world also). If another language was the lingua franca, current approaches would not work so well. The article makes a point about how we assume that pre-trained embeddings encode all relevant information about a language when this may not be the case for arbitrary languages. The article also makes a point about the morphological simplicity of English which is of course a great asset for a lingua franca, and also simplifies the language modelling task. Imagine if the lingua franca was, e.g. ancient Greek (as it was, in ancient times, all around the Mediterrannean) or something with similar morphological intricacies (and Greek is still an Indo-European language; I just don't know much about, e.g. Finno-Ugric to give an example about them).
Is it practical to do NLP research using a language you're not fluent in? Like, does most research require the experimenter to be able to judge fluency and ambiguity themselves?
For sure. Many NLP scientific benchmarks are multilingual today and you need to evaluate on X languages. You are of course not expected to be fluent in all of them. Most of the deep learning based approaches are quite language-universal, i.e. you have a pre-trained language model or word embeddings and you just slap a sequence modeling model on top of that and you let the neural network do the magic.
However, you can still incorporate language-specific knowledge if you want to get the best performance for that one particular language. This is mainly true for academic research. If you are developing a rule-based or keyword-based application (which is still often the case outside of academia), it can actually help to be fluent to get the rules right.
>> Recent models have repeatedly matched human-level performance on
increasingly difficult benchmarks—that is, in English using labelled datasets
with thousands and unlabelled data with millions of examples. In the process,
as a community we have overfit to the characteristics and conditions of
English-language data. In particular, by focusing on high-resource languages,
we have prioritised methods that work well only when large amounts of labelled
and unlabelled data are available.
This is an interesting observation. Training on large datasets tends to be
framed as a strength, e.g. we have recently seen articles praising OpenAI's
GPT-3 for being, well, big. The truth is that model size (and the assorted
dataset size) is a bug, not a feature. It is the result of a dearth of
research on approaches with low sample complexity. Or in other words, it's the
result of consistently picking the low-hanging fruit and calling sour grapes
on any domain for which there isn't sufficiently large data ("who cares about
Swahilli?").
>> In contrast, most current methods break down when applied to the
data-scarce conditions that are common for most of the world's languages. Even
recent advances in pre-training language models that dramatically reduce the
sample complexity for downstream tasks (Peters et al., 2018; Howard and Ruder,
2018; Devlin et al., 2019; Clark et al., 2020) require massive amounts of
clean, unlabelled data, which is not available for most of the world's
languages (Artetxe et al., 2020). Doing well with few data is thus an ideal
setting to test the limitations of current models—and evaluation on
low-resource languages constitutes arguably its most impactful real-world
application.
And this is interesting to read in the context of the recent manic hype about
GPT-3, itself described in a paper titled "Language models are few-shot
learners", hilariously attempting to sweep under the carpet the amount of data
and compute required to get to the point where a language model can do "few"
shot learning.
In general, the excitement about the successes that have come from training
large, deep neural nework models with very large datasets has only served to
avoid posing the obvious question about such data-hungry approaches: what do
we do when there isn't a lot of data? Human language with all its minutely
fragmented diversity, turns out to be just that kind of domain.
I see this argument a lot. I have essentially zero ml experience, but isn't it fair to say that DNA indeed carries a similarly large amount of data? Also, it's not like any human is just born fully capable of inventing an entire language. It seemingly takes entire societies to pull this off.
Granted, our ml isn't even close to human intelligence yet. But why is the assumption made that humans are "trained" with very little data? We've collected millions of years of data to accomplish what we have.
Humans seem to come into the world with very good quality biases that allow us to learn any human language but I don't think anyone has ever found DNA encoding information about, say, the English language. An I misunderstanding your comment? If so, I apologise- I'm not sure where DNA comes in.
>> Also, it's not like any human is just born fully capable of inventing an entire language. It seemingly takes entire societies to pull this off.
This is a complicated matter and as I'm not a linguist I don't know what theories there even exist. However, I think a human child is perfectly capable of inventing an entire language. Small groups of children are certainly capable of doing that in the early stages of their lives. For example, see Creole languages [1] which are the languages invented by the children of people who only have a pidgin as a common language. A pidgin is not a fully developed language and pidgins are used e.g. for communication between migrant communities with different origins and who do not have a common language, while a Creole is a fully developed natural language that is invented by the migrants' children, who have their parents' share pidgin as a maternal language. The children start with their parents' pidgin and develop it into a fully formed natural language with all the bells and whistles. It's quite a thing to read about really, have a look at the wikipedia article I link above it's very interesting.
Also, as far as I understand it there's a lot that we don't know about how the first human languages were created. From some short readings, the current understanding is that at some point, humans must have become capable of producing natural language whereas before they weren't. The way things usually work the turning point for this was most likely some random mutation that affected a very few individuals, possibly no more than two. So the first natural language was probably invented not by an entire society but by a couple of people, probably a pair of siblings. The idea is that the ability to use natural language gave those very few individuals an important advantage that ensured its propagation through generations until our time.
In any case that all goes to invention of a new language. Learning an existing language is quite another matter. Children who grow up to learn their maternal language don't have access to "millions of years of data". They certainly don't have access to a few billion examples of utterances in their maternal language, like the datasets that language models are trained on. That is to say, kids learn whatever language is their maternal language very, very quickly as they grow up and from very few examples. And they're capable of learning any of the few thousand natural languages, beyond English. For me at least this lends credence to an idea of an innate bias to learn a certain type of language, a type broad enough to cover every human language. To put a name on this idea, that's through and through Noam Chomsky's idea of an innate "Universal Grammar". See wikipedia about criticisms of UG and linguistic nativism in general [2].
>> English and the small set of other high-resource languages are in many ways
not representative of the world's other languages. Many resource-rich
languages belong to the Indo-European language family, are spoken mostly in
the Western world, and are morphologically poor, i.e. information is mostly
expressed syntactically, e.g. via a fixed word order and using multiple
separate words rather than through variation at the word level.
I thought I'd give an example of this for readers who don't speak many
languages other than English. My native language is Greek, which is an
Indo-European language, but it has some morphological intricacies that make it
more complicated to learn and use (as a second language and from what I'm told
of course) than English.
For example, in Greek, nouns have a gender [1]: masculine, feminine or neuter.
For instance, "the dog" is "ο σκύλος" ("the male dog") while "the cat" is "η
γάτα" (the female cat). However, grammmatical gender is not necessarily fixed
so one can also say "η σκύλα" ("the female dog") or "ο γάτος" ("the male
cat"). So there is a word-root ("σκυλ-" for dog, "γατ-" for cat) that is then
modified by a termination, typically -ος, -η -ο for the nominative of each
gender. The terminations change depending on the declention, for example the
genetive for "σκύλος" is "σκύλου" and the generative for "σκύλα" is "σκύλας",
etc. The termination also changes to denote number, singular or plural
(ancient Greek also used to have a dual number, used to refer to pairs of
nouns). Terminations vary depending on number according to gender and
declention. So for example the plural of "σκύλος" is "σκύλοι", genetive
"σκύλων" and the plural of "σκύλα" is "σκύλες", genetive "σκυλών".
Compare this with English where a dog is a dog is a dog, there is only one
plural form, "dogs" and there is only one genetive form "dog's" or "dogs'" for
each number. And a female dog is either a circumlocution, "female dog" or an
entirely new word, "bitch", with its own simple set of transformations for
number and genitive "bitches" and "bitch's" or "bitches'". Basically, changing
meaning in English can be represented by adding a few words to a vocabulary-
but changing meaning in Greek requires manipulating words at the structural
level.
There are probably a few counter-examples to the above but my understanding is
that this is how it works for the most part. And this, in particular the
abilty of English to represent more meanings with more words, might go some
way to explain why approaches that work best with large datasets tend to be
favoured in English NLP.
Edit: not an NLP expert or a linguist so corrections welcome, of course!
______________
[1] Other Indo-European languages also have genders but the interesting thing
is that those genders don't always match, between languages. This is an
endless source of confusion for foreign language learners. For example I speak
French from an early age and I only realised much later in life that Greek and
French do not always use the same gender for the same nouns, even though I'd
been using the right genders in my speech. For example, both "dog" and "cat"
are masculine in French: "le chien" and "le chat". But while I always used "le
chat" correctly, I always thought of it in my mind as "η γάτα", the feminine
Greek noun. Human language is weird!
The cheap money, as in Oligarchy backed petrodollar slush funds, is residing mainly in the US. The 'defacto' second language of the world is English. Combine those two and it is not hard to see why English NLP is so dominant.
I'm not yet sure whether that is a good or a bad thing for non-English-as-a-first-languege regions.
I don't have a horse in this race, but surely it could only be bad for those regions, right? There's a whole world of applications involving NLP which non-English speakers don't have access to.
On the other hand, a huge amount of those applications target consumers to get them to misspend more money, so not having them might be a benefit?
Surveillance capitalism relies on getting ever more info about you, and exploiting it or selling it on to all that want to exploit it. NLP is one vector to extract that info from raw data.
It's not just NLP, English is pretty bad as an intermediate language for translations from language A to language B.
If I try translating a Russian word "пружина" ("a mechanical spring") to German using Google translate, I end up with "Frühling", which in fact means "springtime".
This is an obvious artefact of transitive translatiion: Russian -> English -> German.
Providing context may help, but still translation to English strips important pieces of information.
I don't think this is inherent to English, or any other language (perhaps in more specific cases when there is no word with the similar meaning).
I think in general we pack a concept in a word and lose some information this way, so when you want to be precise with what you are saying you have to bring your definitions with you. Essentially with translation you take a concept and "pack" it in a word, then look for an equivalent packing in different language, then unpack. Naturally, this process is prone to losing information.
I think it is inherent to English at least in degree.
Reading translated books from Polish to Spanish or Russian to Spanish conveys a lot more information, than reading the same book in their English translation.
There could be multiple factors I think, like: your language skill level, translation quality, "closeness" of languages, psychological predisposition towards your native language.
Depending on the form of the thing you are translating it can be simply impossible to translate properly (like poetry).
I'm currently reading English translation of "Black Obelisk" which I believe is written in German and it isn't any worse than Russian translation (my native language) to me.
In any case, what was originally asserted is that English is somehow worse than other languages as a "transitional" translation language for words or simple phrases, so I argued with that specific idea. Translating literary works is a subject of it's own and where the quality is much harder to measure.
Looking at translations as if they represent languages seems like a common beginner trope. -- I certainly made that mistake.
Funny that the upstream commenter essentially praises Spanish as superior to English, Spanish being the language I dismissed as less expressive than English when I was a noob.
A bad Netflix or literary Spanish translation, for example, is full of frustrating ambiguity ('who is the antecedent of "su" here? "Bajó"? Who bajó?? At least in English we have to use "he" or "she"!'). And, with experience, you realize that native Spanish writers will keep things disambiguated, it's just crappy translation shortcuts that don't. And trying to compare translations is only an exercise in comparing translators.
I cannot agree. Not only English seems to have many homonyms (the word "spring" alone has more than 2 meanings), its grammar is also somewhat primitive.
Let me bring one more example, this time another way around: Google Translate from German to Russian.
The verb "tragen" ("to wear") is translated as "износ" ("a wear") which is a noun. Using English we lose important knowledge: we have no clue what part of speech the word "tragen" is.
This isn't an issue for any considerably long fragment of text, it will be properly translated due to context analysis. Still if the text would be analyzed using German in the first place, this would become less of a problem.
You're confusing two separate things: linguistic complexity and ambiguity.
Linguistic complexity is hard to measure, but it's not hard to show that at least morphologically, English is undercomplex compared to many other languages.
This doesn't necessarily mean that English is more ambiguous, though. Unlike German, English typically has very rigid word order, so in the context of a sentence, you'll know if a specific word is a noun or a verb.
The problem here is that many NLP models inadequately capture syntactic structure.
> So you say you are able to guess from the word order what part of speech a particular word is. But with German you hardly need all this guessing.
Not really. It's not about guessing: in English, the part of speech really is mostly determined by its syntactic structure.
> If you compare two marginal examples: - English "time flies like an arrow" - German "Wenn Fliegen fliegen hinter Fliegen..."
Not sure what you're trying to say here. The English example is ambiguous, yes (and only strictly grammaticaly; semantically the meaning is clear, unless you're using it in the phrase "time flies like an arrow, fruit flies like a banana", which is meant as a linguistic joke). It's also very easy to come up with examples of phrases or sentences that are ambiguous in German, or in any language for that matter. Here are some fun examples:
"Er liest das Buch seiner Schwester vor" (could either mean "he's reading the book to his sister" or "he's reading his sister's book to someone")
"der weiße Schimmel" ("white mould", or "white horse")
"wilde Tiere jagen" ("to hunt wild animals", or "wild animals are hunting")
and don't even get me started on the ambiguity of compound words or phrases with a genitive, where there are often tons of potential interpretations depending on the intended relationship between head and dependent noun.
And also the German example you gave (fully: "Wenn Fliegen hinter Fliegen fliegen, fliegen Fliegen Fliegen nach", or "if flies fly behind flies, flies fly after flies") is a) another joke sentence nobody uses in practice, and b) is exactly a case where you can only distinguish the part of speech (and the grammatical case) of a word from the syntactic structure and not from its morphology, something you claimed doesn't happen in German, but here it clearly does.
Look, you may make a case that it's easier for English sentences to be ambiguous than for some other languages, but I would need to see some good data before I believed that claim, because it's just not something that is immediately obvious.
I still think you're missing my point, although I am impressed by your German skills ("der Schimmel" is BTW just a homonym, it's hardly related to the topic of syntactic ambiguity).
> is a) another joke sentence nobody uses in practice, and b) is exactly a case where you can only distinguish the part of speech (and the grammatical case) of a word from the syntactic structure and not from its morphology, something you claimed doesn't happen in German, but here it clearly does.
I didn't make such a strong claim. All I wanted to say in German syntactic ambiguities are much less of a problem than in English. I've brought two anecdotal evidences to let you compare possible ambiguities in both of them, these two are indeed nothing but jokes.
But let's take a closer look at them once again.
a) "Time flies like an arrow": the word "time" can be 1) a noun 2) an adjective 3) a verb in declarative form 4) a verb in imperative form. This gives us a factor of 4 on the very first word of the sentence.
b) "Wenn Fliegen hinter Fliegen fliegen" - ambiguitity exists just between "fliegen" as a verb and "Fliegen" as a plural noun, thus the "ambiguity factor" of the word "f/Fliegen" is just 2.
> but I would need to see some good data before I believed that claim, because it's just not something that is immediately obvious.
>> I think in general we pack a concept in a word and lose some information this way, so when you want to be precise with what you are saying you have to bring your definitions with you. Essentially with translation you take a concept and "pack" it in a word, then look for an equivalent packing in different language, then unpack. Naturally, this process is prone to losing information.
This is a plausible description of how humans perform translation, but it does not apply to machine translation, because we have no good way to represent the meaning of a word other than with the word itself. Consequently machine translation systems can't distinguish between different meanings of the same word and instead try to produce a correct translation by relying on frequency-based heuristics: faced with two likely translations of a word, a system will try to determine the context of the word (in terms of its collocation with other words) and then assign to the word the meaning it has in the context that happens to be the most common according to its training dataset. Clearly, that is like "flying blind"; sometimes it will work, sometimes it will fail and there's no way to predict beforehand which.
The comment above gave the "spring" example, my routine example is asking Google Translate to translate Greek "χελιδόνι" (the bird, swallow) to French and getting "avaler" (the verb, to swallow) instead of the correct "hirondelle", again because translation goes from Greek to French via English, introducing ambiguity about the intended meaning of "swallow" that does not exist in either Greek or French. Note that this doesn't happen when the word "χελιδόνι" is used in a sentence (e.g. "ένα το χελιδόνι" translates to "un l'hirondelle", which is ungrammatical and nonsensical but at least gets the right noun), but it's a good test to show that Google Translate is really incapable of recognising the meaning of words and so cannot use such information to make translations. Note that the same goes for machine translation in general, i.e. Google Translate is a state of the art system.
> In contrast, most current methods break down when applied to the data-scarce conditions that are common for most of the world's languages. Even recent advances in pre-training language models that dramatically reduce the sample complexity for downstream tasks (Peters et al., 2018; Howard and Ruder, 2018; Devlin et al., 2019; Clark et al., 2020) require massive amounts of clean, unlabelled data, which is not available for most of the world's languages (Artetxe et al., 2020).
Languages are generally not unique and this is how humans are able to deal with sparse data as well. So you can use a language model that is trained on text across all languages and then the model will infer common structures to languages with sparse data. XLM did this and seems to perform fairly well although I don't know how sparse you can go before it breaks down.
> Doing well with few data is thus an ideal setting to test the limitations of current models—and evaluation on low-resource languages constitutes arguably its most impactful real-world application.
This is what ML and Stats has done for the last X decades and it has limitations due to the need to constrain the problem with human generated rules/knowledge/axioms. The new large scale language models skip the human rules step and are thus able to learn the long tail of language structure. I don't see how you can have it both ways as the unconstrained problem is infeasible to learn on small data as there isn't enough to generalize from.
> I don't see how you can have it both ways as the unconstrained problem is infeasible to learn on small data as there isn't enough to generalize from.
I mostly agree with you, but it does seem like current models require more data to learn language than baby humans.
This would suggest there is a middle ground between old fashioned data starved learning, and current overfed models.
But baby humans tend to associate phonetic symbols (spoken words) to what they are seeing, and the learning method is very different. Their context is much broader, helped by multiple senses. NLP is extremely contextless compared to what humans do. You can't talk about "more data" so easily.
One random note I heard on a podcast about child development is that children do not improve their language skills by watching tv or educational material and that they need feedback to learn.
Contrast that with language models which do nothing besides watching what people are writing, it potentially points to a reason who the amount of data is not comparable.
Even more crucial is that children learn language in situations of actual language use. The things and situations that are spoken about are often at hand in some way, and there is a large social element.
No matter how close you might get to "the language learning algorithm", if all you're feeding it is text, it's not going to learn the same thing that kids learn. The data is simply not there.
My native language is Dutch. Google handles it poorly. As a result people have adapted to either search in English, or to morph the original search (that google does not understand) into sometime that has a higher changes of yielding good results. I'm pretty sure Dutch isn't the only language that fares this way.
Now the main reason (at least, I think it is) it's handled poorly is that Dutch has an infinite number of words ( * ) and order in compositions matters a lot. This conflicts with the typical precomputations that try to reduce the number of words etc.
( * ) take any 2 nouns A & B. you can concatenate them into a new noun AB or a new noun BA (with completely different meaning). German has this concept too.
Google does a pretty decent job for me in Dutch; Duckduckgo and Bing are awful to the point of being useless. Don't agree with your hypothesis either, results are bad even for short simple sentences with common non-combined nouns.
They aren't that common in Dutch anyway unless they have evolved into a more or less standalone noun. Dutch nor English people really think of a flamethrower as a thrower of flames, it's just another noun.
> take any 2 nouns A & B. you can concatenate them into a new noun AB or a new noun BA (with completely different meaning). German has this concept too.
This is also how it works in English, except the concatenation may be written with a space as "A B". If anything, writing it as "AB" should make search more accurate, since you're less likely to get results with "BA" than getting results with "B A" when searching for "A B".
Only if your NLP models actually decompose words. Simpler ones don't.
But while it's technically true that English also has compounds, it just writed them apart, it is also the case that German (don't know about Dutch) uses compounds, and especially long compounds with more than two nouns, a lot more than English, where you might more often use a whole sentence.
I am not a native English-speaker, but my dream is that the world gradually converges on a single standard for international communication, and that this standard is English, so that an ever-growing proportion of the population will become fluent in it. I understand various reasons for why this is unlikely to happen; and I realise that there've been lots of centrifugal forces lately that make this vision ever less likely; but I can't help but smile inwardly when I hear that NLP is so much focused on English, and can't help rooting for it to remain so.
I think this is a horrible idea; unifying the mindsets of all people via the same language would mean that we might get forever stuck in some local optimum and there won't be any other culture that could push humanity forward. People tend to become complacent if they aren't feeling competition, leading to a downfall, and that is true for all aspects of life.
It's like we are all in some sort of Reinforcement Learning algorithm and unifying the language portion would significantly restrict the exploration phase, the one that moves world forward. There are many concepts unique to each language, different emphasis on different things, different emotional response to different sentences, leading to unique ways of exploring the universe, and that all would be lost.
> unifying the mindsets of all people via the same language
There is a Sapir-Whorfian assumption here that it is the language that enslaves the mind, rather than the collective minds that are constantly shaping, morphing and bending the language. To me, the prospect of cultures having to communicate with each other via such a broken mechanism as translation is far more repugnant than a linguistically unified superculture within which ideas flow unimpeded.
The more language users there are, the more capable of expression the language will become; so no, I am not particularly worried about getting stuck in a local maximum.
The Sapir-Whorf hypothesis has been widely debunked. It feels right, so it keeps getting spread around as a fact.
The world is ultra connected now. It's fine for people to be bilingual and continue to communicate and respect their local history and culture. There's also a premium on being able to communicate wherever you go and with anyone you might meet. That's where English is already filling a dominant niche.
The concern that some countries have with children learning English (when it's not their native language), is that their children might also grow up being heavily influenced by American/British culture (as they'll grow up watching English TV shows, reading English books, etc). That's a legitimate concern, where it's difficult to decouple the language from the associated culture. That said, I think there's a value in being able to communicate with everyone else and those countries could do other things to either limit screen time or increase the amount of historical and cultural content they give to their kids (this seems mostly like a problem for people who just want to sit their kids in front of a laptop and hope for the best - active parenting and teaching of local culture seems effective).
> The Sapir-Whorf hypothesis has been widely debunked. It feels right, so it keeps getting spread around as a fact.
You say this as if there were consensus around that, but that's not true. The very strong, original formulation of Sapir-Whorf probably doesn't hold much water, but you'll still find enough linguists claiming that language influences thought nontrivially. But of course, you'll also find linguists arguing the opposite.
I think it is true in an obvious but meaningful sense, though - word choices available impact meaning. If it's idiomatic to use a colourful dramatic phrase, a normally benign saying has subtlety pushed towards different connotations in people's minds etc. Now, I think this is true in pockets. In the larger sense, our available linguistic tools continues to grow, and speakers have no taboos even playing with basic grammar for effect
>There is a Sapir-Whorfian assumption here that it is the language that enslaves the mind, rather than the collective minds that are constantly shaping, morphing and bending the language.
You don't need Sapir-Whorf to see the point. Even if language is purely a tool, only relying on one language narrows cultural expression. Korean for example contains honorifics baked into the grammar of the language to express social status, this is not trivially doable in English, at least in any version that could be considered unified with what is called English today.
> only relying on one language narrows cultural expression
I am not sure I understand what you mean.
Sure, there are languages whose grammars require speakers to choose a more polite or a less polite form of a word; but I doubt this presents an insurmountable obstacle to the other culture for expressing the attitude of politeness, or the concept of hierarchy. Being speaker of a language that has two forms for the word "you" — a polite, plural one, and a familiar, singular one (I suspect that this grammatical behavior was imported from French in around 18th century; the distinction is similar to what Middle English had with you vs thou), I am sure that our concept of politeness is not markedly different from cultures whose languages do not have this distinction in second-person pronoun. The distinction cannot be trivially translated into modern English, true, but I don't think it matters a whole lot.
Besides, if a culture really starts to feel a need for expressing a certain concept, they will find a way for doing that with the language they have.
Korean has seven different levels of speech depending on the level of casualness one engages in and the Japanese language has features to raise or lower the status of the addresse as well encoding different feelings in the language.
There is no analog for this in English, and also not really in Western culture. There's a very distinct notion of politeness in several Asian cultures that has no corolarry in Western societies because we don't tie it formally into social status.
There's also a lot of other issues. A lot of major religious traditions rely on revealed scripture whose meaning is only precisely accessible through a particular language. In Islamic theology this would be Arabic and for Jews it is Hebrew of course. It would be an extreme cultural loss for billions of people to lose this access to their traditions.
> There's also a lot of other issues. A lot of major religious traditions rely on revealed scripture whose meaning is only precisely accessible through a particular language. In Islamic theology this would be Arabic and for Jews it is Hebrew of course. It would be an extreme cultural loss for billions of people to lose this access to their traditions.
I don't see it as a problem in the slightest, and not just because I personally think religious traditions are better left behind. No-one these days speaks classical Latin or ancient Greek; and yet we have some knowledge about the ancient tradition through commentaries and translations (regardless of how imprecise those translations are). We do not experience this as a tragedy or as an extreme cultural loss, despite us not being able to read those texts. Nor is anyone bemoaning the inaccessibility of Beowulf and the inadequacy of existing translations. And if someone is especially interested in the topic — well, there are always scholars who are free to learn those languages and to study the originals.
You clearly haven't talked to Latin or Greek teachers if you think nobody bemoans our lack of fluency in those languages nowadays. It isn't that long ago that knowing Latin was almost a prerequisite for calling yourself educated.
I do believe that we miss out on a lot of nuances by not being able to read Latin or Greek. But I also accept that, pragmatically, we can't all be learning these languages anymore, so yeah, I'm not one of those grumpy Latin teachers. That doesn't mean it's not still a loss.
I used to placed more emphasis unification of standards (as in the case, unification of a natural language) but the truth is: humans will always diverge for various (conscious and less conscious) reasons.
Imagine a world of 7billion English speakers: there would obviously be a huge range of accents and standards. In fact, compared to other languages, English is much less standardized and I'd argue that the premise that there's a single "English" for NLP to focus on.
So, I'd rather NLP scientists find fulfilling work in exploring existing languages diversity.
For some reason the thought of English become a standard made me feel sad.
While standardization certainly lends convenience and efficiency, diversity leads to more robustness and effective exploration — instead of everything being caught up in the same bubble. The web as a platform might become unmanageably “hot” (in a McLuhan-esque sense) if that were to happen; we’d just have to find other arbitrary ways of differentiating ourselves to form more convivial communities.
I don’t mind if English becomes the common interchange format (lingua franca, ha) but I hope people continue to be multilingual and conserving and growing the cultures of other tongues.
My amateur observations, I wonder what others think:
English is already something of a standard for international communication I think, at least in some domains.
It looks like a successor to Latin in many regards.
Nonetheless I don't think English can function as an universal language because it is specialized in some regards. It is better for thinking about and capturing some things and ideas, but not for everything. Other languages, it seems to me, are optimized better for certain use cases.
Everything of course cross-pollinates and, thanks to that, flexibility of different languages increases. But some fundamental rules must remain in place to maintain coherence of a language. If you change the basic grammar rules, you get a different one. In this way achieving universality within a single lanugage is either impossible or very difficult. Hard to tell if lanugages are evolving towards convergence into a single universal superlanguage or if the optimum is specialization.
Programming languages seem to follow similar patterns.
Nonetheless I don't think English can function as an universal language because it is specialized in some regards. It is better for thinking about and capturing some things and ideas, but not for everything. Other languages, it seems to me, are optimized better for certain use cases.
Curious as to what you had in mind (assuming it was broader than Eskimos and snow!)?
> Nonetheless I don't think English can function as an universal language because it is specialized in some regards. It is better for thinking about and capturing some things and ideas, but not for everything. Other languages, it seems to me, are optimized better for certain use cases.
Care to back up this claim with examples? To me, it sounds like you're advocating for linguistic relativism [1], which doesn't really hold up to our modern understanding of language and thought.
If you read that Wikipedia article you linked carefully, you'll find reference to the more recent (last 30 years or so) research made by cognitive linguists in how for example, spatial or colour perception is influenced by the language you speak. One of the most striking examples is that of the Guugu Yimitthir people who don't use terms like "left" and "right", but only "north", "south", etc. [1]
In general, it's not fair to say that linguistic relativism doesn't hold up to modern understanding. The debate is still on-going.
> Care to back up this claim with examples? To me, it sounds like you're advocating for linguistic relativism [1], which doesn't really hold up to our modern understanding of language and thought.
As I said in another comment, I think there is a feedback loop between language and mind, so IMO strong relativism or determinism creates a false dichotomy. As Tainnor mentioned, the article you linked provides some examples that suggest the weak form of relativism or however you want to call it exists. At the same time it agrees that determinism is false.
To my amateur eye this just shows that a feedback loop exists. Anecdata to back it up would include writers, scholars, philosophers, many of them polyglots, like Conrad, Watts, Nabokov, Huxley, and many others who seem to express a similar sentiment. In a polyglot context this hypothesis seems to arise often, see for example this interview:
> I can draw polar opposite philosophical lessons from my experience. On the one hand, languages are only fascinating (or frustrating, depending on your point of view) as long as they are foreign – once they become familiar, that is, once they are simply used as a vehicle for communication, then they all do the same thing. So, in this sense, no matter how different they might seem, they are all the same. Obviously, this is true of us as human beings as well.
> On the other hand, each language has certain unique characteristics, words, and concepts that do not exist in other languages. Furthermore, there is an enormous variety of ways to express ideas and thoughts. Many of these ways of expressing what is in the brain seem absolutely essential to those who use them, but in point of fact they are not, as other languages do without them entirely.
> Finally, as far as culture is concerned, all languages are created equal and all languages do carry the culture of their speakers if by culture we mean simply the ways and traditions of a group of people.
> Each language resonates with me in a distinct way, bringing out a different part of my character. Russian makes me more melancholic because of its minor tone. In French, I am super pensive. Brazilian Portuguese is a very flirtatious and sweet language.
> But there is more to it than that though – each language has its own way of expressing thoughts and ideas, so you get a real insight into diverse thinking. Language carries the culture of the country that uses it and when you internalise it, it becomes a part of you too.
etc.
My personal experience with Polish and English also aligns with this. Switching between these languages influences how I build thoughts. Sometimes I seem to make mistakes in Polish because my mind is more used to thinking about a certain area in English and the other way around as well.
For expressing some thoughts I find Polish more suitable with its ubiquitous prefixes and suffixes and flexible word order. For other thoughts English seems better with its ubiquitous homonyms and simplicity. Sometimes it's easier to make leaps of understanding and connection between different concepts in one, sometimes in the other.
These are the two languages that I think and speak daily with something close to 50/50 ratio, perhaps more English, even though it is my second language.
Other languages that I've been exposed to also seem to show me different ways of thinking. For example German feels more rigid then either English or Polish but also more lego-like. Chinese or Japanese seem to be quite different still, with ...
I am a native English speaker. The problem I have with English as the Universal International Language is that it gives enormous economic and political power to the United States. I am not convinced that the United States can be relied upon to use this power responsibly.
> is that it gives enormous economic and political power to the United States
All the best to them :-) I think having models in nations that actually use a language as their native tongue is far better than choosing a language for which there is no living model (such as Latin or Esperanto).
But then, why don't you also add Britain, Australia, Canada or South Africa to your list?
I am aware that there are other countries that have English as their primary language. According to Wikipedia https://en.wikipedia.org/wiki/List_of_countries_by_GDP_(nomi... the US GDP is approximately 3.45 times the combined GDP of the other countries you named. I stand by my statement that the US has "enormous economic and political power".
edit: I personally think Esperanto would be a good choice as an International Language. I believe it's much easier to learn than English.
Moreover, it is well known to sociolinguists that political and economic power is something that makes a language spoken by that nation more prestigious and more desirable to attain, and, therefore, contributes to the spread of that language. So, for the purposes of my dream, I can only wish that the US remains as influential as it is (and doesn't switch to Spanish in the process) :-)
I guess my point is that I am not sure that, say, having other countries where English is the primary language (as listed in the previous comment) contributes to the power of the US. Likewise, I am not sure that the further spread of English as an international language will by itself contribute significantly to the power of the US.
> I personally think Esperanto would be a good choice as an International Language
Ugh, not a fan. English is a vibrant, living language that already is being used by huge swathes of people for various purposes; Esperanto is not.
You've got this backwards: English is the Universal International Language because the US has enormous economic and political power (and before that, the British Empire).
This has nothing to do with communication. The article makes the case that overemphasising English for NLP has produced approaches that are limited in what they can do and can only solve some of the problems of English language modelling. In short, they're "one trick ponies" that only work well in one kind of problem. Further progress requires the ability to tackle more diverse problems and that's what the article advocates for. So I'm sorry to say but your "vision" of a united world under English is a vision of a world united under technological stagnation, a world where our NLP (and possibly AI, in general) approaches can only handle a limited set of "low hanging fruit" problems associated with English.
The opinion expressed in your comment, that we should adjust ourselves to our technology, rather than improving our technology to suit our needs, is not entirely new. For example, Andrew Ng has said that rather than trying to make self-driving cars capable of dealing with an unexpected situation such as a pedestrian on a pogo stick, we should make sure that people don't engage in such unexpected behaviour that self-driving cars can't handle [1]. Obviously, this is a recipe not just for suppressing much of the diversity of human behaviour with unpredictable results for our own cognitive capacity, it's also backwards and antithetical to the motive of improving technology to make it capable of new applications.
___________________
[1] Rodney Brooks, Bothersome Bystanders and Self Driving Cars:
Though i agree with the main point, the blog post went off on a tangent about how some other languages are in danger because of English. IMO if the speakers of the language do not:
1. Provide materials in their own language
2. Consume materials in their own language
3. Actively and exclusively contribute to the world in English
it is already a statement about the necessity of their language: it isn't. Why should non speakers of a certain language invest money and/or time on a language its own people don't even invest in? If its own speakers kill the language (actively or passively) who are we to say they shouldn't?
> Why should non speakers of a certain language invest money and/or time on a language its own people don't even invest in?
Because rational choices at an individual level do not necessarily align with rational choices at a social collective level. "Meditations on Moloch" is a good read on this topic. You can easily see how collectively the language diversity is positive, but individually it kinda makes sense to bet for the biggest language instead. And to be fair, there's a lot of people investing in their own languages, just not enough, or not with enough power.
A continuing lack of technological inclusion will not only exacerbate the language divide but it may also drive speakers of unsupported languages and dialects to high-resource languages with better technological support, further endangering such language varieties. To ensure that non-English language speakers are not left behind and at the same time to offset the existing imbalance, to lower language and literacy barriers, we need to apply our models to non-English languages.
Have you considered what this means, concretely, for a speaker of an endangered language? Suppose I want to "invest" into my language and
1. Provide materials:
- Written? I need some kind of keyboard support. In the worst case, this might entail getting new characters added to Unicode. Or I could just use paper, but then I lose out on all the benefits of digital text editing.
- Spoken/Signed? In some ways, that's easier, since I can just record myself. But if I want closed captions or some kind of transcript, I can't make use of automatic tools, I have to transcribe it completely manually.
2. Consume materials:
- But how do I find them? What if searching for words in my language mostly turns up results in other languages because the search engine assumes I misspelled my query?
- It might happen that the text I'm trying to read uses characters that were considered "too rare" to be included in the default font on my device. On most versions of Android, it's not possible to install a new font to fix that. I'm not sure about other mobile operating systems.
Those are problems that don't require speakers of the language to do something differently (such as just using English instead...) but instead need technical solutions that can very well be implemented by someone who doesn't speak the language. I don't speak an endangered language myself, but I've written code to support dozens of languages I don't speak, because someone who did speak the language asked for a particular change that would make life easier for them.
I'm reading lots of comments reacting against the premise of the article in name of some kind of efficiency or inherent value of a universal standar.
I wonder if commenters have considered that Research is useful to understand the present and more importantly prepare for the future. While English being useful língua Franca today seems to back decision to just focus on English research the point is: will this English uptake continue at same rate in future? Isn't there some kind of bubble effect where here in hn many think that (as non English native speakers) we can all engage so we think English is the only useful thing for us to understand?
What about all possible unknown Mandarin NLP research (which I think many agree) may be useful for non-mandarin speakers?
Also, isn't one of the points of Information Technology to enable/create value in a long tail of experiences? Focusing on a single attribute of a substantial part of the world seems counter productive and shortsighted...
With new direction of research, like pre-taining and subword vocab, the models now are mostly language agnostic. English or not, it is just sequence of tokens.
Difference mainly lies in data abundance, where the distribution is severly skewed.
Can anyone chime in to how Chinese NLP compares? I speak Mandarin as a second language at an intermediate level, and from my perspective it has some interesting properties compared to English (my native language) that seem like they might make NLP easier in ways that English NLP might be hard:
- The grammar for standard Mandarin is much simpler than English. No verb conjugations or strange tense rules ("go" -> "went")!
- Tense and even voice are explicit and additive - i.e. adding "呢" or "吧" to a sentence.
- Written, it seems less ambiguous. i.e. "duck" or "grave" have different meanings in English based on context, but that seems much much rarer in Chinese - two words spoken the same way but with different meanings usually have different written forms.
Mandarin has even less morphology than English[1], so to some extent, working on Mandarin would not "fix" some of the issues of NLP models often performing poorly on morphology-rich languages.
But it's possible that focusing more on Mandarin would still lead to improvements in other areas, because the two languages are otherwise quite different (I wonder how well speech recognition works for tonal languages, for example?).
[1] Possibly not without reason: there is some indication that languages spoken by a large number of speakers tend to become structurally simpler, while very small isolated languages can often develop surprising complexity.
68 comments
[ 2.2 ms ] story [ 132 ms ] threadThe situation for those who, like myself, work in applied linguistics is not actually so dire... At least when working with the languages that have a reasonable amount of training data. Decent enough treebanks exist for lemmatisation, POS-tagging, and dependency parsing for dozens of languages [0]. Fast tools such as spaCy (15 languages) [1] and udpipe (40+ languages) [2] are freely available and work well for most applied tasks. There are even decent word embeddings available trained on various versions of Wikipedia [3].
Of course, some of the issue is that these very tasks are biased towards the specific structures of Indo-European languages. However, for getting work done (building sentiment classifiers, document clustering, NER, ect.), currently available tools make it possible to work with a large proportion of the currently available data.
There is still a lot of work to be done own regards to computational research on non-English languages, but a lot of the problem right now is recognition of applied work by the top NLP conferences rather than a complete lack of quality work currently being done.
[0] https://universaldependencies.org
[1] https://spacy.io/models
[2] https://ufal.mff.cuni.cz/udpipe/models
[3] https://github.com/facebookresearch/fastText/blob/master/doc...
It helps of course that English is the de facto lingua franca of most of the technologically developed world (if not the rest of the world also). If another language was the lingua franca, current approaches would not work so well. The article makes a point about how we assume that pre-trained embeddings encode all relevant information about a language when this may not be the case for arbitrary languages. The article also makes a point about the morphological simplicity of English which is of course a great asset for a lingua franca, and also simplifies the language modelling task. Imagine if the lingua franca was, e.g. ancient Greek (as it was, in ancient times, all around the Mediterrannean) or something with similar morphological intricacies (and Greek is still an Indo-European language; I just don't know much about, e.g. Finno-Ugric to give an example about them).
However, you can still incorporate language-specific knowledge if you want to get the best performance for that one particular language. This is mainly true for academic research. If you are developing a rule-based or keyword-based application (which is still often the case outside of academia), it can actually help to be fluent to get the rules right.
This is an interesting observation. Training on large datasets tends to be framed as a strength, e.g. we have recently seen articles praising OpenAI's GPT-3 for being, well, big. The truth is that model size (and the assorted dataset size) is a bug, not a feature. It is the result of a dearth of research on approaches with low sample complexity. Or in other words, it's the result of consistently picking the low-hanging fruit and calling sour grapes on any domain for which there isn't sufficiently large data ("who cares about Swahilli?").
>> In contrast, most current methods break down when applied to the data-scarce conditions that are common for most of the world's languages. Even recent advances in pre-training language models that dramatically reduce the sample complexity for downstream tasks (Peters et al., 2018; Howard and Ruder, 2018; Devlin et al., 2019; Clark et al., 2020) require massive amounts of clean, unlabelled data, which is not available for most of the world's languages (Artetxe et al., 2020). Doing well with few data is thus an ideal setting to test the limitations of current models—and evaluation on low-resource languages constitutes arguably its most impactful real-world application.
And this is interesting to read in the context of the recent manic hype about GPT-3, itself described in a paper titled "Language models are few-shot learners", hilariously attempting to sweep under the carpet the amount of data and compute required to get to the point where a language model can do "few" shot learning.
In general, the excitement about the successes that have come from training large, deep neural nework models with very large datasets has only served to avoid posing the obvious question about such data-hungry approaches: what do we do when there isn't a lot of data? Human language with all its minutely fragmented diversity, turns out to be just that kind of domain.
Granted, our ml isn't even close to human intelligence yet. But why is the assumption made that humans are "trained" with very little data? We've collected millions of years of data to accomplish what we have.
>> Also, it's not like any human is just born fully capable of inventing an entire language. It seemingly takes entire societies to pull this off.
This is a complicated matter and as I'm not a linguist I don't know what theories there even exist. However, I think a human child is perfectly capable of inventing an entire language. Small groups of children are certainly capable of doing that in the early stages of their lives. For example, see Creole languages [1] which are the languages invented by the children of people who only have a pidgin as a common language. A pidgin is not a fully developed language and pidgins are used e.g. for communication between migrant communities with different origins and who do not have a common language, while a Creole is a fully developed natural language that is invented by the migrants' children, who have their parents' share pidgin as a maternal language. The children start with their parents' pidgin and develop it into a fully formed natural language with all the bells and whistles. It's quite a thing to read about really, have a look at the wikipedia article I link above it's very interesting.
Also, as far as I understand it there's a lot that we don't know about how the first human languages were created. From some short readings, the current understanding is that at some point, humans must have become capable of producing natural language whereas before they weren't. The way things usually work the turning point for this was most likely some random mutation that affected a very few individuals, possibly no more than two. So the first natural language was probably invented not by an entire society but by a couple of people, probably a pair of siblings. The idea is that the ability to use natural language gave those very few individuals an important advantage that ensured its propagation through generations until our time.
In any case that all goes to invention of a new language. Learning an existing language is quite another matter. Children who grow up to learn their maternal language don't have access to "millions of years of data". They certainly don't have access to a few billion examples of utterances in their maternal language, like the datasets that language models are trained on. That is to say, kids learn whatever language is their maternal language very, very quickly as they grow up and from very few examples. And they're capable of learning any of the few thousand natural languages, beyond English. For me at least this lends credence to an idea of an innate bias to learn a certain type of language, a type broad enough to cover every human language. To put a name on this idea, that's through and through Noam Chomsky's idea of an innate "Universal Grammar". See wikipedia about criticisms of UG and linguistic nativism in general [2].
____________
[1] https://en.wikipedia.org/wiki/Creole_language
[2] https://en.wikipedia.org/wiki/Universal_grammar#Criticisms
I thought I'd give an example of this for readers who don't speak many languages other than English. My native language is Greek, which is an Indo-European language, but it has some morphological intricacies that make it more complicated to learn and use (as a second language and from what I'm told of course) than English.
For example, in Greek, nouns have a gender [1]: masculine, feminine or neuter. For instance, "the dog" is "ο σκύλος" ("the male dog") while "the cat" is "η γάτα" (the female cat). However, grammmatical gender is not necessarily fixed so one can also say "η σκύλα" ("the female dog") or "ο γάτος" ("the male cat"). So there is a word-root ("σκυλ-" for dog, "γατ-" for cat) that is then modified by a termination, typically -ος, -η -ο for the nominative of each gender. The terminations change depending on the declention, for example the genetive for "σκύλος" is "σκύλου" and the generative for "σκύλα" is "σκύλας", etc. The termination also changes to denote number, singular or plural (ancient Greek also used to have a dual number, used to refer to pairs of nouns). Terminations vary depending on number according to gender and declention. So for example the plural of "σκύλος" is "σκύλοι", genetive "σκύλων" and the plural of "σκύλα" is "σκύλες", genetive "σκυλών".
Compare this with English where a dog is a dog is a dog, there is only one plural form, "dogs" and there is only one genetive form "dog's" or "dogs'" for each number. And a female dog is either a circumlocution, "female dog" or an entirely new word, "bitch", with its own simple set of transformations for number and genitive "bitches" and "bitch's" or "bitches'". Basically, changing meaning in English can be represented by adding a few words to a vocabulary- but changing meaning in Greek requires manipulating words at the structural level.
There are probably a few counter-examples to the above but my understanding is that this is how it works for the most part. And this, in particular the abilty of English to represent more meanings with more words, might go some way to explain why approaches that work best with large datasets tend to be favoured in English NLP.
Edit: not an NLP expert or a linguist so corrections welcome, of course!
______________
[1] Other Indo-European languages also have genders but the interesting thing is that those genders don't always match, between languages. This is an endless source of confusion for foreign language learners. For example I speak French from an early age and I only realised much later in life that Greek and French do not always use the same gender for the same nouns, even though I'd been using the right genders in my speech. For example, both "dog" and "cat" are masculine in French: "le chien" and "le chat". But while I always used "le chat" correctly, I always thought of it in my mind as "η γάτα", the feminine Greek noun. Human language is weird!
I'm not yet sure whether that is a good or a bad thing for non-English-as-a-first-languege regions.
Surveillance capitalism relies on getting ever more info about you, and exploiting it or selling it on to all that want to exploit it. NLP is one vector to extract that info from raw data.
Providing context may help, but still translation to English strips important pieces of information.
I think in general we pack a concept in a word and lose some information this way, so when you want to be precise with what you are saying you have to bring your definitions with you. Essentially with translation you take a concept and "pack" it in a word, then look for an equivalent packing in different language, then unpack. Naturally, this process is prone to losing information.
Reading translated books from Polish to Spanish or Russian to Spanish conveys a lot more information, than reading the same book in their English translation.
It's like every subtle nuance is lost in English.
Depending on the form of the thing you are translating it can be simply impossible to translate properly (like poetry).
I'm currently reading English translation of "Black Obelisk" which I believe is written in German and it isn't any worse than Russian translation (my native language) to me.
In any case, what was originally asserted is that English is somehow worse than other languages as a "transitional" translation language for words or simple phrases, so I argued with that specific idea. Translating literary works is a subject of it's own and where the quality is much harder to measure.
Funny that the upstream commenter essentially praises Spanish as superior to English, Spanish being the language I dismissed as less expressive than English when I was a noob.
A bad Netflix or literary Spanish translation, for example, is full of frustrating ambiguity ('who is the antecedent of "su" here? "Bajó"? Who bajó?? At least in English we have to use "he" or "she"!'). And, with experience, you realize that native Spanish writers will keep things disambiguated, it's just crappy translation shortcuts that don't. And trying to compare translations is only an exercise in comparing translators.
Though you said it better than I did.
This isn't an issue for any considerably long fragment of text, it will be properly translated due to context analysis. Still if the text would be analyzed using German in the first place, this would become less of a problem.
Linguistic complexity is hard to measure, but it's not hard to show that at least morphologically, English is undercomplex compared to many other languages.
This doesn't necessarily mean that English is more ambiguous, though. Unlike German, English typically has very rigid word order, so in the context of a sentence, you'll know if a specific word is a noun or a verb.
The problem here is that many NLP models inadequately capture syntactic structure.
> Unlike German, English typically has very rigid word order, so in the context of a sentence, you'll know if a specific word is a noun or a verb.
So you say you are able to guess from the word order what part of speech a particular word is. But with German you hardly need all this guessing.
If you compare two marginal examples: - English "time flies like an arrow" - German "Wenn Fliegen fliegen hinter Fliegen..."
you'll find out the English one has way more possible interpretations.
Not really. It's not about guessing: in English, the part of speech really is mostly determined by its syntactic structure.
> If you compare two marginal examples: - English "time flies like an arrow" - German "Wenn Fliegen fliegen hinter Fliegen..."
Not sure what you're trying to say here. The English example is ambiguous, yes (and only strictly grammaticaly; semantically the meaning is clear, unless you're using it in the phrase "time flies like an arrow, fruit flies like a banana", which is meant as a linguistic joke). It's also very easy to come up with examples of phrases or sentences that are ambiguous in German, or in any language for that matter. Here are some fun examples:
"Er liest das Buch seiner Schwester vor" (could either mean "he's reading the book to his sister" or "he's reading his sister's book to someone")
"der weiße Schimmel" ("white mould", or "white horse")
"wilde Tiere jagen" ("to hunt wild animals", or "wild animals are hunting")
and don't even get me started on the ambiguity of compound words or phrases with a genitive, where there are often tons of potential interpretations depending on the intended relationship between head and dependent noun.
And also the German example you gave (fully: "Wenn Fliegen hinter Fliegen fliegen, fliegen Fliegen Fliegen nach", or "if flies fly behind flies, flies fly after flies") is a) another joke sentence nobody uses in practice, and b) is exactly a case where you can only distinguish the part of speech (and the grammatical case) of a word from the syntactic structure and not from its morphology, something you claimed doesn't happen in German, but here it clearly does.
Look, you may make a case that it's easier for English sentences to be ambiguous than for some other languages, but I would need to see some good data before I believed that claim, because it's just not something that is immediately obvious.
> is a) another joke sentence nobody uses in practice, and b) is exactly a case where you can only distinguish the part of speech (and the grammatical case) of a word from the syntactic structure and not from its morphology, something you claimed doesn't happen in German, but here it clearly does.
I didn't make such a strong claim. All I wanted to say in German syntactic ambiguities are much less of a problem than in English. I've brought two anecdotal evidences to let you compare possible ambiguities in both of them, these two are indeed nothing but jokes.
But let's take a closer look at them once again.
a) "Time flies like an arrow": the word "time" can be 1) a noun 2) an adjective 3) a verb in declarative form 4) a verb in imperative form. This gives us a factor of 4 on the very first word of the sentence.
b) "Wenn Fliegen hinter Fliegen fliegen" - ambiguitity exists just between "fliegen" as a verb and "Fliegen" as a plural noun, thus the "ambiguity factor" of the word "f/Fliegen" is just 2.
> but I would need to see some good data before I believed that claim, because it's just not something that is immediately obvious.
Fair enough.
This is a plausible description of how humans perform translation, but it does not apply to machine translation, because we have no good way to represent the meaning of a word other than with the word itself. Consequently machine translation systems can't distinguish between different meanings of the same word and instead try to produce a correct translation by relying on frequency-based heuristics: faced with two likely translations of a word, a system will try to determine the context of the word (in terms of its collocation with other words) and then assign to the word the meaning it has in the context that happens to be the most common according to its training dataset. Clearly, that is like "flying blind"; sometimes it will work, sometimes it will fail and there's no way to predict beforehand which.
The comment above gave the "spring" example, my routine example is asking Google Translate to translate Greek "χελιδόνι" (the bird, swallow) to French and getting "avaler" (the verb, to swallow) instead of the correct "hirondelle", again because translation goes from Greek to French via English, introducing ambiguity about the intended meaning of "swallow" that does not exist in either Greek or French. Note that this doesn't happen when the word "χελιδόνι" is used in a sentence (e.g. "ένα το χελιδόνι" translates to "un l'hirondelle", which is ungrammatical and nonsensical but at least gets the right noun), but it's a good test to show that Google Translate is really incapable of recognising the meaning of words and so cannot use such information to make translations. Note that the same goes for machine translation in general, i.e. Google Translate is a state of the art system.
Languages are generally not unique and this is how humans are able to deal with sparse data as well. So you can use a language model that is trained on text across all languages and then the model will infer common structures to languages with sparse data. XLM did this and seems to perform fairly well although I don't know how sparse you can go before it breaks down.
> Doing well with few data is thus an ideal setting to test the limitations of current models—and evaluation on low-resource languages constitutes arguably its most impactful real-world application.
This is what ML and Stats has done for the last X decades and it has limitations due to the need to constrain the problem with human generated rules/knowledge/axioms. The new large scale language models skip the human rules step and are thus able to learn the long tail of language structure. I don't see how you can have it both ways as the unconstrained problem is infeasible to learn on small data as there isn't enough to generalize from.
I mostly agree with you, but it does seem like current models require more data to learn language than baby humans.
This would suggest there is a middle ground between old fashioned data starved learning, and current overfed models.
Contrast that with language models which do nothing besides watching what people are writing, it potentially points to a reason who the amount of data is not comparable.
No matter how close you might get to "the language learning algorithm", if all you're feeding it is text, it's not going to learn the same thing that kids learn. The data is simply not there.
Now the main reason (at least, I think it is) it's handled poorly is that Dutch has an infinite number of words ( * ) and order in compositions matters a lot. This conflicts with the typical precomputations that try to reduce the number of words etc.
( * ) take any 2 nouns A & B. you can concatenate them into a new noun AB or a new noun BA (with completely different meaning). German has this concept too.
They aren't that common in Dutch anyway unless they have evolved into a more or less standalone noun. Dutch nor English people really think of a flamethrower as a thrower of flames, it's just another noun.
This is also how it works in English, except the concatenation may be written with a space as "A B". If anything, writing it as "AB" should make search more accurate, since you're less likely to get results with "BA" than getting results with "B A" when searching for "A B".
But while it's technically true that English also has compounds, it just writed them apart, it is also the case that German (don't know about Dutch) uses compounds, and especially long compounds with more than two nouns, a lot more than English, where you might more often use a whole sentence.
It's like we are all in some sort of Reinforcement Learning algorithm and unifying the language portion would significantly restrict the exploration phase, the one that moves world forward. There are many concepts unique to each language, different emphasis on different things, different emotional response to different sentences, leading to unique ways of exploring the universe, and that all would be lost.
There is a Sapir-Whorfian assumption here that it is the language that enslaves the mind, rather than the collective minds that are constantly shaping, morphing and bending the language. To me, the prospect of cultures having to communicate with each other via such a broken mechanism as translation is far more repugnant than a linguistically unified superculture within which ideas flow unimpeded.
The more language users there are, the more capable of expression the language will become; so no, I am not particularly worried about getting stuck in a local maximum.
The world is ultra connected now. It's fine for people to be bilingual and continue to communicate and respect their local history and culture. There's also a premium on being able to communicate wherever you go and with anyone you might meet. That's where English is already filling a dominant niche.
The concern that some countries have with children learning English (when it's not their native language), is that their children might also grow up being heavily influenced by American/British culture (as they'll grow up watching English TV shows, reading English books, etc). That's a legitimate concern, where it's difficult to decouple the language from the associated culture. That said, I think there's a value in being able to communicate with everyone else and those countries could do other things to either limit screen time or increase the amount of historical and cultural content they give to their kids (this seems mostly like a problem for people who just want to sit their kids in front of a laptop and hope for the best - active parenting and teaching of local culture seems effective).
You say this as if there were consensus around that, but that's not true. The very strong, original formulation of Sapir-Whorf probably doesn't hold much water, but you'll still find enough linguists claiming that language influences thought nontrivially. But of course, you'll also find linguists arguing the opposite.
You don't need Sapir-Whorf to see the point. Even if language is purely a tool, only relying on one language narrows cultural expression. Korean for example contains honorifics baked into the grammar of the language to express social status, this is not trivially doable in English, at least in any version that could be considered unified with what is called English today.
I am not sure I understand what you mean.
Sure, there are languages whose grammars require speakers to choose a more polite or a less polite form of a word; but I doubt this presents an insurmountable obstacle to the other culture for expressing the attitude of politeness, or the concept of hierarchy. Being speaker of a language that has two forms for the word "you" — a polite, plural one, and a familiar, singular one (I suspect that this grammatical behavior was imported from French in around 18th century; the distinction is similar to what Middle English had with you vs thou), I am sure that our concept of politeness is not markedly different from cultures whose languages do not have this distinction in second-person pronoun. The distinction cannot be trivially translated into modern English, true, but I don't think it matters a whole lot.
Besides, if a culture really starts to feel a need for expressing a certain concept, they will find a way for doing that with the language they have.
https://en.wikipedia.org/wiki/Honorifics_(linguistics)
Korean has seven different levels of speech depending on the level of casualness one engages in and the Japanese language has features to raise or lower the status of the addresse as well encoding different feelings in the language.
There is no analog for this in English, and also not really in Western culture. There's a very distinct notion of politeness in several Asian cultures that has no corolarry in Western societies because we don't tie it formally into social status.
There's also a lot of other issues. A lot of major religious traditions rely on revealed scripture whose meaning is only precisely accessible through a particular language. In Islamic theology this would be Arabic and for Jews it is Hebrew of course. It would be an extreme cultural loss for billions of people to lose this access to their traditions.
I don't see it as a problem in the slightest, and not just because I personally think religious traditions are better left behind. No-one these days speaks classical Latin or ancient Greek; and yet we have some knowledge about the ancient tradition through commentaries and translations (regardless of how imprecise those translations are). We do not experience this as a tragedy or as an extreme cultural loss, despite us not being able to read those texts. Nor is anyone bemoaning the inaccessibility of Beowulf and the inadequacy of existing translations. And if someone is especially interested in the topic — well, there are always scholars who are free to learn those languages and to study the originals.
I do believe that we miss out on a lot of nuances by not being able to read Latin or Greek. But I also accept that, pragmatically, we can't all be learning these languages anymore, so yeah, I'm not one of those grumpy Latin teachers. That doesn't mean it's not still a loss.
Imagine a world of 7billion English speakers: there would obviously be a huge range of accents and standards. In fact, compared to other languages, English is much less standardized and I'd argue that the premise that there's a single "English" for NLP to focus on.
So, I'd rather NLP scientists find fulfilling work in exploring existing languages diversity.
While standardization certainly lends convenience and efficiency, diversity leads to more robustness and effective exploration — instead of everything being caught up in the same bubble. The web as a platform might become unmanageably “hot” (in a McLuhan-esque sense) if that were to happen; we’d just have to find other arbitrary ways of differentiating ourselves to form more convivial communities.
I don’t mind if English becomes the common interchange format (lingua franca, ha) but I hope people continue to be multilingual and conserving and growing the cultures of other tongues.
English is already something of a standard for international communication I think, at least in some domains. It looks like a successor to Latin in many regards.
Nonetheless I don't think English can function as an universal language because it is specialized in some regards. It is better for thinking about and capturing some things and ideas, but not for everything. Other languages, it seems to me, are optimized better for certain use cases.
Everything of course cross-pollinates and, thanks to that, flexibility of different languages increases. But some fundamental rules must remain in place to maintain coherence of a language. If you change the basic grammar rules, you get a different one. In this way achieving universality within a single lanugage is either impossible or very difficult. Hard to tell if lanugages are evolving towards convergence into a single universal superlanguage or if the optimum is specialization.
Programming languages seem to follow similar patterns.
Curious as to what you had in mind (assuming it was broader than Eskimos and snow!)?
Care to back up this claim with examples? To me, it sounds like you're advocating for linguistic relativism [1], which doesn't really hold up to our modern understanding of language and thought.
[1]: https://en.wikipedia.org/wiki/Linguistic_relativity
In general, it's not fair to say that linguistic relativism doesn't hold up to modern understanding. The debate is still on-going.
[1] https://anthrosource.onlinelibrary.wiley.com/doi/abs/10.1525...
As I said in another comment, I think there is a feedback loop between language and mind, so IMO strong relativism or determinism creates a false dichotomy. As Tainnor mentioned, the article you linked provides some examples that suggest the weak form of relativism or however you want to call it exists. At the same time it agrees that determinism is false.
To my amateur eye this just shows that a feedback loop exists. Anecdata to back it up would include writers, scholars, philosophers, many of them polyglots, like Conrad, Watts, Nabokov, Huxley, and many others who seem to express a similar sentiment. In a polyglot context this hypothesis seems to arise often, see for example this interview:
http://writeparagraphs.blogspot.com/2014/02/a-stroll-with-al...
> I can draw polar opposite philosophical lessons from my experience. On the one hand, languages are only fascinating (or frustrating, depending on your point of view) as long as they are foreign – once they become familiar, that is, once they are simply used as a vehicle for communication, then they all do the same thing. So, in this sense, no matter how different they might seem, they are all the same. Obviously, this is true of us as human beings as well.
> On the other hand, each language has certain unique characteristics, words, and concepts that do not exist in other languages. Furthermore, there is an enormous variety of ways to express ideas and thoughts. Many of these ways of expressing what is in the brain seem absolutely essential to those who use them, but in point of fact they are not, as other languages do without them entirely.
> Finally, as far as culture is concerned, all languages are created equal and all languages do carry the culture of their speakers if by culture we mean simply the ways and traditions of a group of people.
or this article:
https://www.theguardian.com/education/2013/sep/05/multilingu...
> Each language resonates with me in a distinct way, bringing out a different part of my character. Russian makes me more melancholic because of its minor tone. In French, I am super pensive. Brazilian Portuguese is a very flirtatious and sweet language.
> But there is more to it than that though – each language has its own way of expressing thoughts and ideas, so you get a real insight into diverse thinking. Language carries the culture of the country that uses it and when you internalise it, it becomes a part of you too.
etc.
My personal experience with Polish and English also aligns with this. Switching between these languages influences how I build thoughts. Sometimes I seem to make mistakes in Polish because my mind is more used to thinking about a certain area in English and the other way around as well.
For expressing some thoughts I find Polish more suitable with its ubiquitous prefixes and suffixes and flexible word order. For other thoughts English seems better with its ubiquitous homonyms and simplicity. Sometimes it's easier to make leaps of understanding and connection between different concepts in one, sometimes in the other.
These are the two languages that I think and speak daily with something close to 50/50 ratio, perhaps more English, even though it is my second language.
Other languages that I've been exposed to also seem to show me different ways of thinking. For example German feels more rigid then either English or Polish but also more lego-like. Chinese or Japanese seem to be quite different still, with ...
All the best to them :-) I think having models in nations that actually use a language as their native tongue is far better than choosing a language for which there is no living model (such as Latin or Esperanto).
But then, why don't you also add Britain, Australia, Canada or South Africa to your list?
edit: I personally think Esperanto would be a good choice as an International Language. I believe it's much easier to learn than English.
Moreover, it is well known to sociolinguists that political and economic power is something that makes a language spoken by that nation more prestigious and more desirable to attain, and, therefore, contributes to the spread of that language. So, for the purposes of my dream, I can only wish that the US remains as influential as it is (and doesn't switch to Spanish in the process) :-)
I guess my point is that I am not sure that, say, having other countries where English is the primary language (as listed in the previous comment) contributes to the power of the US. Likewise, I am not sure that the further spread of English as an international language will by itself contribute significantly to the power of the US.
> I personally think Esperanto would be a good choice as an International Language
Ugh, not a fan. English is a vibrant, living language that already is being used by huge swathes of people for various purposes; Esperanto is not.
The opinion expressed in your comment, that we should adjust ourselves to our technology, rather than improving our technology to suit our needs, is not entirely new. For example, Andrew Ng has said that rather than trying to make self-driving cars capable of dealing with an unexpected situation such as a pedestrian on a pogo stick, we should make sure that people don't engage in such unexpected behaviour that self-driving cars can't handle [1]. Obviously, this is a recipe not just for suppressing much of the diversity of human behaviour with unpredictable results for our own cognitive capacity, it's also backwards and antithetical to the motive of improving technology to make it capable of new applications.
___________________
[1] Rodney Brooks, Bothersome Bystanders and Self Driving Cars:
https://rodneybrooks.com/bothersome-bystanders-and-self-driv...
1. Provide materials in their own language
2. Consume materials in their own language
3. Actively and exclusively contribute to the world in English
it is already a statement about the necessity of their language: it isn't. Why should non speakers of a certain language invest money and/or time on a language its own people don't even invest in? If its own speakers kill the language (actively or passively) who are we to say they shouldn't?
Because rational choices at an individual level do not necessarily align with rational choices at a social collective level. "Meditations on Moloch" is a good read on this topic. You can easily see how collectively the language diversity is positive, but individually it kinda makes sense to bet for the biggest language instead. And to be fair, there's a lot of people investing in their own languages, just not enough, or not with enough power.
A continuing lack of technological inclusion will not only exacerbate the language divide but it may also drive speakers of unsupported languages and dialects to high-resource languages with better technological support, further endangering such language varieties. To ensure that non-English language speakers are not left behind and at the same time to offset the existing imbalance, to lower language and literacy barriers, we need to apply our models to non-English languages.
Have you considered what this means, concretely, for a speaker of an endangered language? Suppose I want to "invest" into my language and
1. Provide materials:
- Written? I need some kind of keyboard support. In the worst case, this might entail getting new characters added to Unicode. Or I could just use paper, but then I lose out on all the benefits of digital text editing.
- Spoken/Signed? In some ways, that's easier, since I can just record myself. But if I want closed captions or some kind of transcript, I can't make use of automatic tools, I have to transcribe it completely manually.
2. Consume materials:
- But how do I find them? What if searching for words in my language mostly turns up results in other languages because the search engine assumes I misspelled my query?
- It might happen that the text I'm trying to read uses characters that were considered "too rare" to be included in the default font on my device. On most versions of Android, it's not possible to install a new font to fix that. I'm not sure about other mobile operating systems.
Those are problems that don't require speakers of the language to do something differently (such as just using English instead...) but instead need technical solutions that can very well be implemented by someone who doesn't speak the language. I don't speak an endangered language myself, but I've written code to support dozens of languages I don't speak, because someone who did speak the language asked for a particular change that would make life easier for them.
I wonder if commenters have considered that Research is useful to understand the present and more importantly prepare for the future. While English being useful língua Franca today seems to back decision to just focus on English research the point is: will this English uptake continue at same rate in future? Isn't there some kind of bubble effect where here in hn many think that (as non English native speakers) we can all engage so we think English is the only useful thing for us to understand?
What about all possible unknown Mandarin NLP research (which I think many agree) may be useful for non-mandarin speakers?
Also, isn't one of the points of Information Technology to enable/create value in a long tail of experiences? Focusing on a single attribute of a substantial part of the world seems counter productive and shortsighted...
Difference mainly lies in data abundance, where the distribution is severly skewed.
- The grammar for standard Mandarin is much simpler than English. No verb conjugations or strange tense rules ("go" -> "went")!
- Tense and even voice are explicit and additive - i.e. adding "呢" or "吧" to a sentence.
- Written, it seems less ambiguous. i.e. "duck" or "grave" have different meanings in English based on context, but that seems much much rarer in Chinese - two words spoken the same way but with different meanings usually have different written forms.
But it's possible that focusing more on Mandarin would still lead to improvements in other areas, because the two languages are otherwise quite different (I wonder how well speech recognition works for tonal languages, for example?).
[1] Possibly not without reason: there is some indication that languages spoken by a large number of speakers tend to become structurally simpler, while very small isolated languages can often develop surprising complexity.
NLP in other languages can help sell better ads in those languages.
You all can follow that chain of thought down to it's corporate conclusion =)