Machine translation feels like one of the quiet victories of the current crop of machine learning products.
It's getting to be within spitting distance of being completely adequate (though not excellent) for translating long-form articles, which is amazing considering where things stood ten years ago.
I find that machine translation is getting better at producing what reads as grammatically correct text, but that it still misses a lot of what is said, even when the idioms and expressions used are common. Often what is produced is grammatical, but opposite/orthogonal of the initial intent.
In a sense, I feel that this is an even worse situation, because the word-salad signal that it isn't an accurate-meaning translation isn't as strong.
I think the problem is when direct translation is done. Netflix paper using simplify then translating to me seems to be the better approach when you want to convey the meaning rather than just translating the exact words
https://arxiv.org/abs/2005.11197
That's what MT based on neural networks will do: their output can look great, but the correspondence with the input can be highly inaccurate. On the other hand, traditional methods based on syntactic analysis tend to yield less grammatically correct and more literal translations, but the literalness can also result in better accuracy.
Depends what you optimise for. Neural translation fairly consistently produces incorrect results that read like they could be correct, for me. If your goal is "text that makes sense" they seem to be doing great. If your goal is "text that conveys the same meaning as the source" they're universally rubbish, at least for the language pairs I need (mostly english <-> east Asian languages).
I'm curious what the domain is and what languages you're talking about, it's hard for me to evaluate your claims otherwise. I would buy it for English -> Tibetan for instance.
Mostly Chinese, Japanese, and Vietnamese, and mostly business and lightly-technical text.
My observation of the failure rate (if you define failure as "meaning has been lost") is well above 80% with Google Translate for these languages and this type of text.
It's literally unusable because you have no confidence that the translation has not completely reversed or garbled the meaning of the text, and the failure mode is particularly unpleasant because you may be unaware that it has failed to translate correctly, since it produces text that appears to have meaning - just the wrong meaning.
Kinda funny when people say "East Asia" it mostly means Japan and its isolationism, with China mixed in just to change the tone a bit. I don't find it offensive, just kind of funny/interesting.
Do those languages really have that relationship? When I scratched surface of Chinese language in university it felt almost English written in Kanji. No resemblance to Japanese, and it even made the classical Chinese interpretation methods taught in Japanese education rather dubious to me.
Although they belong to historically disparate language families, modern Vietnamese and Japanese vocabulary both have significant influence from Chinese languages. I know little about Korean but I imagine it's the case there too.
Your comment about Chinese feeling like English written in Kanji is really interesting to me, perhaps it's a case of different perspectives - I tend to focus on grammar when learning a language, and so Chinese feels very different to English from my perspective.
> The Mainland Southeast Asia linguistic area is a sprachbund including languages of the Sino-Tibetan, Hmong–Mien (or Miao–Yao), Kra–Dai, Austronesian and Austroasiatic families spoken in an area stretching from Thailand to China. Neighbouring languages across these families, though presumed unrelated, often have similar typological features, which are believed to have spread by diffusion.
In a European context, this is similar to the unusual closeness between the "unrelated" English and French, known to be due to intensive contact between the languages post-William-the-Conqueror. (In a world context, English and French are closely related regardless.)
It's not "literally unusable." Even for language pairs with poor translation, it considerably saves on manual translation time if you can have a translator look over sets of already translated (but lower-quality) sentences and correct those with errors.
Plus, there are actual studies out there comparing human translations to machine translations in a double-blinded evaluation by translators and the gap is not as high as your comment suggests.
For chinese-english (the other direction), bilingual evaluators have had difficulty differentiating between the machine translation and the human translation.
> For chinese-english (the other direction), bilingual evaluators have had difficulty differentiating between the machine translation and the human translation.
Having attempted to use CN->EN machine translation many times, I find this very surprising. Have you got a reference I can have a look at?
-> Human evaluation does not agree that PBSMT is more accurate than NMT. Sure, the fluency/"sounds good" gains from NMT are much larger than the accuracy gains, but evaluations I've seen put NMT ahead on both metrics.
Fair enough, this is just my sentiment, where accurate was not a term of art, but meant "honest where it is weak and where it is confident."
Before, I could skim outputs, and get a fair idea of which areas were poorly translated by where it was garbage. Now, I get the impression that (like with GPT or style transfer) something always comes out, whether it's related (or even necessarily has the right negation parity) or not.
You're absolutely right with your impression of GPT/these models - they will generally generate plausible text.
And it is also worth noting (as you've sort of mentioned) that appearance of fluency can bias translators who are instructed to just evaluate accuracy, or to evaluate accuracy and fluency separately. There is ongoing research into that problem.
For some language pairs, maybe. The problem is that as you translate between more distant languages, you find that something which is contextual in one language is textual in the other, and vice versa.
The neural networks are great at faking it, and they'll make up text that looks completely reasonable all day long. If that means inventing subjects and choosing random verb tenses, that's what they'll do. If you pick closely-related languages, you might be able to get a near perfect translation just by putting each word through a dictionary, and the neural networks can disambiguate things and smooth over the problems in the output by looking at local context. If you pick distantly-related languages, you might routinely need context from paragraphs away, or context which isn't present in the text at all but relies on some kind of human experience.
low resource translation is definitely still bad, you're right.
> near perfect translation just by putting each word through a dictionary, and the neural networks can disambiguate things and smooth over the problems in the output
Funnily enough, this is not a terrible explanation of how the current sort of approaches to low-resource MT work.
> low resource translation is definitely still bad, you're right.
No, you've misunderstood. Languages that are closely related to each other are fairly easy to translate between. Languages that aren't related are not, regardless of whether you have a large corpus to work with. English - Mandarin is not an example of "low resource translation".
> Languages that aren't related are not, regardless of whether you have a large corpus to work with
The size of the corpus makes a huge difference. I don't see how can both be true that a. there is this massive failure of translation engines on these language pairs,
b. human translators were unable to differentiate between human translated chinese-english sentences and machine translated sentences.
It's not incredible - but it's also not awful or unusable.
> It's not incredible - but it's also not awful or unusable.
If your use case is (1) being vocally excited about machine translation, or (2) having fun in your spare time, then machine translation is good enough for you.
If you have a genuine need to know what something in a foreign language says, machine translation is unusable.
I really have to disagree. Machine translation is utterly terrible, and appears to be optimising for the wrong thing. All the progress seems to be in making it produce output that makes sense, rather than producing output that conveys the same meaning as the source document. Almost every time I've tried to use machine translation for something serious, the meaning of the text has been garbled or reversed. This is not "adequate" unless your only goal is to produce something that looks like it could be correct, without regard for whether it is.
What now? How? It’s getting better by the day but translation is one of the places where it has done more damage than good. It teaches language students incorrect grammar and hinder students from changing their thought pattern.
“- Milk tea or regular tea? -Yes!” Yes is the proper response in some languages. You can either change the language or your mindset.
And it’s not like these things are one offs, all languages are basically a bunch of quirks.
No doubt it’s the way of the future but I’d wait for the researchers to improve translation a lot.
I thought the "large digital platform" would be AliExpress (which from experience I know does use MT extensively in seller-buyer communications), but it's actually referring to eBay. My experience with the former is that the MT is far from perfect, and even frequently causes much amusement, but definitely useful.
Another one is when one language uses a single word for two meanings which the other language has split. An example I cam across is when I translated the word "amber" meaning the orange color and the translation came out the other end as a word that purely means "fossilized sap"
I generally learn proper nouns first (following news is my biggest use of MT) and I've found it often helps to replace them by placeholders such as X and Y when translating.
As to sibling comment, IIRC Yandex translate has a nice feature where they'll provide a number of meanings for words when clicked upon in the source text.
> And for the purpose of deterrence and rehabilitation of re-delinquency of juvenile delinquents, as the process of until a decision, " the fact delinquency to the family court -up Reel-notice - family court investigators (referred to hereinafter as" investigators "), or the like by the survey - survey It is customary to go through the flow of " judgment based on the result- decide protective measures or protective measures as necessary ".
That's barely recognizable as English. The translation is hot garbage. Perhaps something you might use if you had an emergency.
Just to point out here... the text on the front page of Wikipedia tends to be very neutral, matter-of-fact, and well written. There are no characters (like in novels), no use of slang, and nothing else that should be tricky for translation, and yet we get this garbage.
It's a miracle that machine translation works at all, and when it does work, it is usually because you are translating between languages that are fairly similar to begin with. For example, English and Spanish.
One issue with Japanese specifically is that the subject is often omitted from the sentence and inferred from context. It's less common to see the equivalent pronouns of "I" or "they" in Japanese, but it's required a lot more in English. A machine translation would have to determine what pronoun to use when there isn't one in the original sentence to go off of, which is hard.
Then there are other things besides pronouns that require inference. For example, adding な to the end of a sentence could mean either a modifier expressing admiration or confirmation, or it could be indicate a demand or order not to do something. In machine translation it almost always seems to be translated to "don't" no matter the context. That only makes the translation less accurate.
The most common language people would want to translate from online platforms is most likely English, given it's ubiquitous use on the internet and the fact that there are more people who don't speak English than people who do.
Of course the details of how English is turned into incomprehensible garbage by machine translation always depend on the target language in question.
Yeah, Google Translate is hot garbage when it comes to Japanese. DeepL does quite a good job though!
> The purpose of this process is to deter and rehabilitate the juvenile delinquent, and the process of making a decision on the delinquency is usually conducted in the following manner: referral and notification of the fact of the delinquency to the family court; investigation by an investigator of the family court (hereinafter referred to as "investigator"); trial based on the results of the investigation; and, if necessary, a decision on protective measures or protective measures. There is.
Translated with www.DeepL.com/Translator (free version)
Right from the start, DeepL manages to drop information, making it seem as if it's describing a program that deters juvenile delinquency.
DeepL is doing a better job of generating text that seems to make sense, but this just helps to more effectively communicate the wrong meaning, and in a manner less detectable to people who would use it. This is a much more insidious failure mode.
> DeepL is doing a better job of generating text that seems to make sense, but this just helps to more effectively communicate the wrong meaning, and in a manner less detectable to people who would use it. This is a much more insidious failure mode.
This is something I don't understand about the use of autocorrect. Autocorrect takes misspelled words, which are essentially always easy to understand, and replaces them with random other words. It's frequently impossible to guess what original word got mangled by autocorrect into appearing in a suddenly unintelligible sentence. This is a problem that misspellings just don't have.
Autocorrect is actively making communication worse. What's it supposed to be doing? Why is it there at all?
As I type on my phone it auto-guesses the word. I can choose to pick the guessed word, and almost always do so. I call that a lot more than spell-check and yet no words that I did not actively choose sneak in.
For anyone interested in a correct translation, the Japanese reads:
"Its purpose is recidivism prevention and rehabilitation for youth offenders; the usual process leading up to a decision is: Notice or report of the fact of the youth crime to the family court > Investigation by the family court investigating officer (the "Investigator") and/or other parties > Judgment based on the results of the investigation > Protective measures or a disposition for rehabilitation, as necessary."
>Just to point out here... the text on the front page of Wikipedia tends to be very neutral, matter-of-fact, and well written
This is true in general, though perhaps not always of Japanese wikipedia, and certainly not on legal topics. I have no love lost with machine translation (hard agree on naniwaduni's comment about "insidious failure modes"), but there are very few humans who would be able to translate this particular excerpt effectively. Legal translation is my job and I still found this sentence to be reasonably challenging.
Admittedly I am usually translating English and east Asian languages, like this example, but I've basically given up on Google Translate (and other machine translation) at this point. It's worse than useless, because it often produces a translation that reads like it could be correct, but has the meaning reversed or otherwise garbled.
Uhhh.. I just translated the exact same sentence from wikipedia in Google translate and got:
"The juvenile protection procedure aims to deter and rehabilitate juvenile delinquents, and as a process up to the decision, "send and notify the facts of delinquency to the family court-family court investigator (hereinafter abbreviated as" investigator "). It is customary to go through the flow of "investigation by, etc.-a referee based on the investigation results-determine protective measures or protective measures as necessary"."
Not incredible (especially when it starts listing the process), but not as god awful as whatever you got. I'm curious if you omitted some part of the sentence when pasting in.
It's also worth noting that this is a pretty out-of-domain legal Japanese sentence.
> do you have any idea of what things machine translation still consistently gets wrong?
My biggest pet peeve with MT is the incredibly brain-dead choice of using English as an intermediate language.
This effectively prevents MT from ever becoming useful for any A→B translation where neither A nor B are English.
Example: put "пружи́на" into Google translate (or Bing translate, doesn't matter)and translate to German. You'll get "Frühling" as the suggested translation.
This is due to the fact that internally "пружи́на" is translated to English first, resulting in "spring". This is then translated to German with the statistically highest probability being "Frühling", even though "Feder" would be the only correct match ("Frühling" would be "весна" in Russian).
If you now say "But wait! This should clear itself up once you add some context, right?" you'd be wrong.
Try "Die Sprungfeder kam gerade rechtzeitig." ("The coil spring arrived just in time,") and you'll get "Весна пришла как раз вовремя." ("Springtime came just in time").
It's ridiculous!
EDIT: happens with DeepL as well; only Yandex gets it right (presumably because they actually use an actual German/Russian-model.
It's not brain-dead, it's a result of limited training data.
For N languages you need to train N models if you go through English as an intermediate language, but N^2 if you don't use an intermediate language model. OK, computer time is cheap these days, but more importantly, for each language pair you need D high-quality documents translated in each of your two languages.
There are far fewer pieces of writing that exist in both German and Russian than in both German and English or both Russian and English.
There is cool-looking research on zero-shot language translation where the model learns its own internal "intermediate language". But you can be sure that if it produced reliably better results with the data available, the likes of Google would already be using it.
> It's not brain-dead, it's a result of limited training data.
That's quite literally a cheap excuse. It'snot lack of training data (remember, you still need translations in the target language anyway!), it's a result of Anglo-Saxon cultural hegemony.
> There are far fewer pieces of writing that exist in both German and Russian than in both German and English or both Russian and English.
Right. And that's why Russia-based Yandex gets excellent results for German/Russian translation, I see...
> But you can be sure that if it produced reliably better results with the data available, the likes of Google would already be using it.
Nah, you're being way too optimistic there. Google is a business and as such, they don't chase SOTA outside of research (which doubles as advertisement). Good enough is the name of the game and it's simply infinitely cheaper to just train and maintain N-models (i.e. English/X and X/English) than to do the same for all combinations of X/Y-pairs (which would be N² models).
So as long as there is no incentive to do better, they simply won't, because they rely on the fact that English is the de-facto Lingua Franca and people translate to and from English more often than directly between other languages.
This would be an actual chance for companies from other countries (especially European ones) to step in and do exactly that, but again - "good enough" and reliance on English stops that from happening anytime soon. It's a missed opportunity especially with low-resource languages as demonstrated by the recent "Scottish" Wikipedia fiasco...
> This is due to the fact that internally "пружи́на" is translated to English first, resulting in "spring"
Do you have a source that that is actually what is happening behind the scenes? I work in this field and would be surprised if that is what Google is doing. My guess is this phenomenon is more due to a model that was trained on an Russian->English corpora first and then trained on Russian->German due to low resource, or something like that.
Someone like Yandex is going to have better translations because (shocker) they have larger russian corpora than google.
> What this basically means is: If during training you provide it examples of English->Japanese & English->Korean translations, GNMT automatically does Japanese->Korean reasonably well! In fact, this is the biggest achievement of GNMT as a project.
It's exactly what Google have been doing for years now: train on X-to-English and English-to-Y only to get X-to-Y for free. Since the intermediate language only ever sees English as a source or target language, ambiguities like "spring" literally get lost in translation.
1. It is pretty out of date (ML is a fast moving field) - I doubt Google is using LSTMs for translation in 2020.
> Since the intermediate language only ever sees English as a source or target language, ambiguities like "spring" literally get lost in translation.
2. This article is trying to dumb down what Google is doing - but you're right that this is why ambiguities get lost in translation, due to pre-training on a different language pair.
That said, there isn't literally a process of "translate into english" and then "translate english into german". These models are trained on Russian-German corpora, but because there is little resources for that, they are supplementing with Russian-English and English-German.
> It is pretty out of date (ML is a fast moving field) - I doubt Google is using LSTMs for translation in 2020.
Sure, but that doesn't change the fact the training data is focused on English-to-X and X-to-English corpora. The underlying architecture of the model is just an implementation detail that doesn't really affect this as demonstrated by my example.
> These models are trained on Russian-German corpora, but because there is little resources for that, they are supplementing with Russian-English and English-German.
This is exactly what I'd argue isn't the case at all. Otherwise words that have a direct 1:1 translation wouldn't be mistranslated and companies like Yandex wouldn't be able to deliver so much better results.
German-Russian isn't low-resource at all, given 95M and 150M native speakers respectively and a close history for the past 150 years. [edit]The rich cultural history of both countries resulting in a vast library of literature, theatre plays, news publications, films and the general cultural relevance of both languages is even more important.[/edit] It's simply (quite comprehensible) bias towards English for research taking place in the USA and the fact that it's much easier to compile English-to-X and X-to-English corpora in a predominantly English-speaking country.
There are tons of translated books, films, news paper articles, scientific papers, etc. available for Russian-German and Yandex, being a Russian company, naturally has no problem compiling a Russian-German corpus (since they're not biased towards English).
It’s useful in some situations, but they often tend to be the situations where it’s least needed. There’s plenty of languages that MT is very bad at translating (to the point of being counterproductive), and a lot of those languages are natively spoken by people who don’t also speak English. MT is quite bad at translating Hindi, but that’s not so bad because English is (relatively) quite common in India. MT is absolutely terrible at translating Indonesian, which isn’t great for Indonesian trade, because English is rather uncommon in Indonesia.
> MT is far from perfect ... but definitely useful
This is, I think, the best way to look at MT.
As many people here have commented, machine translations are often incorrect, misleading, or just incomprehensible garbage. However, in situations with enough context, people can use MT to accomplish tasks that they would not be able to do without it. Some examples, and a discussion of implications for foreign-language education, appear in a paper I wrote last year [0].
Well I, for one, have long believing that this mouth-watering chicken recipe isn't complete/authentic without the grandma spitting the last ingredient into the glorious final mix.
Another one: the Chinese word for product/good, which is a bit more generic than its English counterpart and might be translated as "it/thing". One of the definitions under that broad umbrella is "baby" (analogous to "little thing" I suppose) -- and for the longest time, despite year after year of Google publishing advanced neural translation models, google translate reliably chose "baby" as the translation in the context of online commerce.
I bought babies, got good deals on babies, and even tried to return a defective baby once, but it didn't work out.
It looks like this has now been fixed, but it was good fun for a long time.
Shameless plug for my open source neural machine translation project. A Python library and GUI with pre-trained models that packages the files needed for machine translation and makes them usable through a simple interface.
Yes Apertium is great! If you go back in the commit logs my project actually started as a GUI for doing Apertium translations. Apertium does "rules-based" translations which works best for translating between very similar languages like Spanish-Catalan (it was initially funded by the Spanish government), and supports a number of languages. However, currently statistical and neural net approaches have better performance for most language pairs. My initial goal was to write the GUI for Apertium, release it on the Snap store, and then try to figure out OpenNMT. I had it working inside of a snap package but ran into an issue with uploading to the Snap store so I moved on: https://forum.snapcraft.io/t/unable-to-upload-to-snap-store-...
I think we need a machine translation library with multiple backends (Apertium, local ML based MT tools, the online ML based MT APIs) and then have toolkits, desktops, email clients, browsers and other things that encounter untranslated text use that library to convert that text into the user's language.
I agree I think especially for supporting a large number of languages your going to want to use online APIs so you don't need to store a lot of models locally. For common language pairs, or ones you use frequently it would be nice for privacy and offline use to be able to seamlessly do it locally.
I think it's time to give up on machine translation. Let people do it. Maybe try and teach the computers how to do something we can't already do ourselves.
For low-resource languages, or anything of great linguistic distance, neural MT is still not there. That is to say, you may find some language pairs where it works OK, but there's a huge sea of pairs where all you get is gibberish – and still they publish these systems and let people use them to produce more gibberish which they put on their blogs and spam sites, making the non-English web into a corpus of dissociated press ramblings.
Baidu recently added Saami to their system. They manage to make a Wikipedia article about wolverines into one about spreadsheets, depressed buttons and famous chickens: https://imgur.com/a/m91SbRw
A rough translation of what Baidu gave:
> No one is greater than zero. There are new things for spreadsheet[here Baidu used the English word for some reason] and all the things in it.
> English is an active wolf, but the background as wolf indicates that the active button is depressed.
> Too many places to install. Sweden was released from 1968 and July 1973. Probably number of times the alarm is to be reapeated within 39 minutes. 2010 is in total 66 chickens, but later a famous chicken and chickens 12th of march 2010, with a total of 54 chickens
Apertium's translation is often stilted, but it gets the meaning across: Wolverines are the largest animals in the marten family, etc.
Google Translate isn't half bad at Norwegian→English, so you can even combine them to get a sort of Saami→English:
https://translate.google.com/#view=home&op=translate&sl=no&t....
I tried Baidu again now with the same input, just to see what they would give into English, and now it's about old ports, finished volcanos and civic computers:
That's "neural network hallucinations" --- and often a sign of insufficient training material. You can see pieces of its training material showing through in the output --- and no doubt the translation would've been far better with similar input, which is why they thought it was great in the first place.
78 comments
[ 2.6 ms ] story [ 52.6 ms ] threadIt's getting to be within spitting distance of being completely adequate (though not excellent) for translating long-form articles, which is amazing considering where things stood ten years ago.
In a sense, I feel that this is an even worse situation, because the word-salad signal that it isn't an accurate-meaning translation isn't as strong.
Maybe this was true 4-6 years ago, but no longer.
I'm curious what the domain is and what languages you're talking about, it's hard for me to evaluate your claims otherwise. I would buy it for English -> Tibetan for instance.
My observation of the failure rate (if you define failure as "meaning has been lost") is well above 80% with Google Translate for these languages and this type of text.
It's literally unusable because you have no confidence that the translation has not completely reversed or garbled the meaning of the text, and the failure mode is particularly unpleasant because you may be unaware that it has failed to translate correctly, since it produces text that appears to have meaning - just the wrong meaning.
Your comment about Chinese feeling like English written in Kanji is really interesting to me, perhaps it's a case of different perspectives - I tend to focus on grammar when learning a language, and so Chinese feels very different to English from my perspective.
But Asian languages are nevertheless related in more mysterious ways. Compare the intro to https://en.wikipedia.org/wiki/Mainland_Southeast_Asia_lingui... :
> The Mainland Southeast Asia linguistic area is a sprachbund including languages of the Sino-Tibetan, Hmong–Mien (or Miao–Yao), Kra–Dai, Austronesian and Austroasiatic families spoken in an area stretching from Thailand to China. Neighbouring languages across these families, though presumed unrelated, often have similar typological features, which are believed to have spread by diffusion.
In a European context, this is similar to the unusual closeness between the "unrelated" English and French, known to be due to intensive contact between the languages post-William-the-Conqueror. (In a world context, English and French are closely related regardless.)
It's not "literally unusable." Even for language pairs with poor translation, it considerably saves on manual translation time if you can have a translator look over sets of already translated (but lower-quality) sentences and correct those with errors.
Plus, there are actual studies out there comparing human translations to machine translations in a double-blinded evaluation by translators and the gap is not as high as your comment suggests.
For chinese-english (the other direction), bilingual evaluators have had difficulty differentiating between the machine translation and the human translation.
Having attempted to use CN->EN machine translation many times, I find this very surprising. Have you got a reference I can have a look at?
https://news.ycombinator.com/item?id=24302564
Bonus clip: https://www.youtube.com/watch?v=0E4aYgKmIko
-> Human evaluation does not agree that PBSMT is more accurate than NMT. Sure, the fluency/"sounds good" gains from NMT are much larger than the accuracy gains, but evaluations I've seen put NMT ahead on both metrics.
Before, I could skim outputs, and get a fair idea of which areas were poorly translated by where it was garbage. Now, I get the impression that (like with GPT or style transfer) something always comes out, whether it's related (or even necessarily has the right negation parity) or not.
And it is also worth noting (as you've sort of mentioned) that appearance of fluency can bias translators who are instructed to just evaluate accuracy, or to evaluate accuracy and fluency separately. There is ongoing research into that problem.
The neural networks are great at faking it, and they'll make up text that looks completely reasonable all day long. If that means inventing subjects and choosing random verb tenses, that's what they'll do. If you pick closely-related languages, you might be able to get a near perfect translation just by putting each word through a dictionary, and the neural networks can disambiguate things and smooth over the problems in the output by looking at local context. If you pick distantly-related languages, you might routinely need context from paragraphs away, or context which isn't present in the text at all but relies on some kind of human experience.
low resource translation is definitely still bad, you're right.
> near perfect translation just by putting each word through a dictionary, and the neural networks can disambiguate things and smooth over the problems in the output
Funnily enough, this is not a terrible explanation of how the current sort of approaches to low-resource MT work.
> low resource translation is definitely still bad, you're right.
No, you've misunderstood. Languages that are closely related to each other are fairly easy to translate between. Languages that aren't related are not, regardless of whether you have a large corpus to work with. English - Mandarin is not an example of "low resource translation".
The size of the corpus makes a huge difference. I don't see how can both be true that a. there is this massive failure of translation engines on these language pairs,
b. human translators were unable to differentiate between human translated chinese-english sentences and machine translated sentences.
It's not incredible - but it's also not awful or unusable.
If your use case is (1) being vocally excited about machine translation, or (2) having fun in your spare time, then machine translation is good enough for you.
If you have a genuine need to know what something in a foreign language says, machine translation is unusable.
This isn't really a good usability profile.
“- Milk tea or regular tea? -Yes!” Yes is the proper response in some languages. You can either change the language or your mindset.
And it’s not like these things are one offs, all languages are basically a bunch of quirks.
No doubt it’s the way of the future but I’d wait for the researchers to improve translation a lot.
Oh fascinating, do you have any idea of what things machine translation still consistently gets wrong?
As to sibling comment, IIRC Yandex translate has a nice feature where they'll provide a number of meanings for words when clicked upon in the source text.
"Invisible, Insane"
Here's part of the Google translation of the front page of ja.wikipedia.org into English:
> 非行少年の再非行の抑止や更生を目的としており、決定までの過程として、「非行事実を家庭裁判所に送致・通告 - 家庭裁判所調査官（以下「調査官」と略称する）等による調査 - 調査結果をふまえた審判 - 必要に応じて保護的措置あるいは保護処分を決定」という流れを経るのが通例である。
> And for the purpose of deterrence and rehabilitation of re-delinquency of juvenile delinquents, as the process of until a decision, " the fact delinquency to the family court -up Reel-notice - family court investigators (referred to hereinafter as" investigators "), or the like by the survey - survey It is customary to go through the flow of " judgment based on the result- decide protective measures or protective measures as necessary ".
That's barely recognizable as English. The translation is hot garbage. Perhaps something you might use if you had an emergency.
Just to point out here... the text on the front page of Wikipedia tends to be very neutral, matter-of-fact, and well written. There are no characters (like in novels), no use of slang, and nothing else that should be tricky for translation, and yet we get this garbage.
It's a miracle that machine translation works at all, and when it does work, it is usually because you are translating between languages that are fairly similar to begin with. For example, English and Spanish.
Then there are other things besides pronouns that require inference. For example, adding な to the end of a sentence could mean either a modifier expressing admiration or confirmation, or it could be indicate a demand or order not to do something. In machine translation it almost always seems to be translated to "don't" no matter the context. That only makes the translation less accurate.
And I’d assume that could be the most common language people would want to translate from online platforms in this day and age.
Of course the details of how English is turned into incomprehensible garbage by machine translation always depend on the target language in question.
> The purpose of this process is to deter and rehabilitate the juvenile delinquent, and the process of making a decision on the delinquency is usually conducted in the following manner: referral and notification of the fact of the delinquency to the family court; investigation by an investigator of the family court (hereinafter referred to as "investigator"); trial based on the results of the investigation; and, if necessary, a decision on protective measures or protective measures. There is.
Translated with www.DeepL.com/Translator (free version)
DeepL is doing a better job of generating text that seems to make sense, but this just helps to more effectively communicate the wrong meaning, and in a manner less detectable to people who would use it. This is a much more insidious failure mode.
This is something I don't understand about the use of autocorrect. Autocorrect takes misspelled words, which are essentially always easy to understand, and replaces them with random other words. It's frequently impossible to guess what original word got mangled by autocorrect into appearing in a suddenly unintelligible sentence. This is a problem that misspellings just don't have.
Autocorrect is actively making communication worse. What's it supposed to be doing? Why is it there at all?
The first has your stated effect. The second does not.
Also, it has "bugs" that sometimes it will just drop an entire sentence or half-sentence for no reason.
"Its purpose is recidivism prevention and rehabilitation for youth offenders; the usual process leading up to a decision is: Notice or report of the fact of the youth crime to the family court > Investigation by the family court investigating officer (the "Investigator") and/or other parties > Judgment based on the results of the investigation > Protective measures or a disposition for rehabilitation, as necessary."
>Just to point out here... the text on the front page of Wikipedia tends to be very neutral, matter-of-fact, and well written
This is true in general, though perhaps not always of Japanese wikipedia, and certainly not on legal topics. I have no love lost with machine translation (hard agree on naniwaduni's comment about "insidious failure modes"), but there are very few humans who would be able to translate this particular excerpt effectively. Legal translation is my job and I still found this sentence to be reasonably challenging.
"The juvenile protection procedure aims to deter and rehabilitate juvenile delinquents, and as a process up to the decision, "send and notify the facts of delinquency to the family court-family court investigator (hereinafter abbreviated as" investigator "). It is customary to go through the flow of "investigation by, etc.-a referee based on the investigation results-determine protective measures or protective measures as necessary"."
Not incredible (especially when it starts listing the process), but not as god awful as whatever you got. I'm curious if you omitted some part of the sentence when pasting in.
It's also worth noting that this is a pretty out-of-domain legal Japanese sentence.
My biggest pet peeve with MT is the incredibly brain-dead choice of using English as an intermediate language.
This effectively prevents MT from ever becoming useful for any A→B translation where neither A nor B are English.
Example: put "пружи́на" into Google translate (or Bing translate, doesn't matter)and translate to German. You'll get "Frühling" as the suggested translation.
This is due to the fact that internally "пружи́на" is translated to English first, resulting in "spring". This is then translated to German with the statistically highest probability being "Frühling", even though "Feder" would be the only correct match ("Frühling" would be "весна" in Russian).
If you now say "But wait! This should clear itself up once you add some context, right?" you'd be wrong.
Try "Die Sprungfeder kam gerade rechtzeitig." ("The coil spring arrived just in time,") and you'll get "Весна пришла как раз вовремя." ("Springtime came just in time").
It's ridiculous!
EDIT: happens with DeepL as well; only Yandex gets it right (presumably because they actually use an actual German/Russian-model.
For N languages you need to train N models if you go through English as an intermediate language, but N^2 if you don't use an intermediate language model. OK, computer time is cheap these days, but more importantly, for each language pair you need D high-quality documents translated in each of your two languages.
There are far fewer pieces of writing that exist in both German and Russian than in both German and English or both Russian and English.
There is cool-looking research on zero-shot language translation where the model learns its own internal "intermediate language". But you can be sure that if it produced reliably better results with the data available, the likes of Google would already be using it.
That's quite literally a cheap excuse. It'snot lack of training data (remember, you still need translations in the target language anyway!), it's a result of Anglo-Saxon cultural hegemony.
> There are far fewer pieces of writing that exist in both German and Russian than in both German and English or both Russian and English.
Right. And that's why Russia-based Yandex gets excellent results for German/Russian translation, I see...
> But you can be sure that if it produced reliably better results with the data available, the likes of Google would already be using it.
Nah, you're being way too optimistic there. Google is a business and as such, they don't chase SOTA outside of research (which doubles as advertisement). Good enough is the name of the game and it's simply infinitely cheaper to just train and maintain N-models (i.e. English/X and X/English) than to do the same for all combinations of X/Y-pairs (which would be N² models).
So as long as there is no incentive to do better, they simply won't, because they rely on the fact that English is the de-facto Lingua Franca and people translate to and from English more often than directly between other languages.
This would be an actual chance for companies from other countries (especially European ones) to step in and do exactly that, but again - "good enough" and reliance on English stops that from happening anytime soon. It's a missed opportunity especially with low-resource languages as demonstrated by the recent "Scottish" Wikipedia fiasco...
Do you have a source that that is actually what is happening behind the scenes? I work in this field and would be surprised if that is what Google is doing. My guess is this phenomenon is more due to a model that was trained on an Russian->English corpora first and then trained on Russian->German due to low resource, or something like that.
Someone like Yandex is going to have better translations because (shocker) they have larger russian corpora than google.
Indirectly yes, I do: https://codesachin.wordpress.com/2017/01/18/understanding-th...
> What this basically means is: If during training you provide it examples of English->Japanese & English->Korean translations, GNMT automatically does Japanese->Korean reasonably well! In fact, this is the biggest achievement of GNMT as a project.
It's exactly what Google have been doing for years now: train on X-to-English and English-to-Y only to get X-to-Y for free. Since the intermediate language only ever sees English as a source or target language, ambiguities like "spring" literally get lost in translation.
1. It is pretty out of date (ML is a fast moving field) - I doubt Google is using LSTMs for translation in 2020.
> Since the intermediate language only ever sees English as a source or target language, ambiguities like "spring" literally get lost in translation.
2. This article is trying to dumb down what Google is doing - but you're right that this is why ambiguities get lost in translation, due to pre-training on a different language pair.
That said, there isn't literally a process of "translate into english" and then "translate english into german". These models are trained on Russian-German corpora, but because there is little resources for that, they are supplementing with Russian-English and English-German.
Sure, but that doesn't change the fact the training data is focused on English-to-X and X-to-English corpora. The underlying architecture of the model is just an implementation detail that doesn't really affect this as demonstrated by my example.
> These models are trained on Russian-German corpora, but because there is little resources for that, they are supplementing with Russian-English and English-German.
This is exactly what I'd argue isn't the case at all. Otherwise words that have a direct 1:1 translation wouldn't be mistranslated and companies like Yandex wouldn't be able to deliver so much better results.
German-Russian isn't low-resource at all, given 95M and 150M native speakers respectively and a close history for the past 150 years. [edit]The rich cultural history of both countries resulting in a vast library of literature, theatre plays, news publications, films and the general cultural relevance of both languages is even more important.[/edit] It's simply (quite comprehensible) bias towards English for research taking place in the USA and the fact that it's much easier to compile English-to-X and X-to-English corpora in a predominantly English-speaking country.
There are tons of translated books, films, news paper articles, scientific papers, etc. available for Russian-German and Yandex, being a Russian company, naturally has no problem compiling a Russian-German corpus (since they're not biased towards English).
This also happens between e.g. Spanish and Bulgarian.
Definitely anything where a word has two or more different meanings.
Paper jam being my favourite example.
https://translate.google.com/#view=home&op=translate&sl=en&t...
It is translated as "(fruit) jam made out of paper"
This is, I think, the best way to look at MT.
As many people here have commented, machine translations are often incorrect, misleading, or just incomprehensible garbage. However, in situations with enough context, people can use MT to accomplish tasks that they would not be able to do without it. Some examples, and a discussion of implications for foreign-language education, appear in a paper I wrote last year [0].
[0] https://researchmap.jp/multidatabases/multidatabase_contents...
And I understand Chinese pretty well.
I bought babies, got good deals on babies, and even tried to return a defective baby once, but it didn't work out.
It looks like this has now been fixed, but it was good fun for a long time.
https://github.com/argosopentech/argos-translate
https://www.apertium.org/
Baidu recently added Saami to their system. They manage to make a Wikipedia article about wolverines into one about spreadsheets, depressed buttons and famous chickens: https://imgur.com/a/m91SbRw
A rough translation of what Baidu gave:
> No one is greater than zero. There are new things for spreadsheet[here Baidu used the English word for some reason] and all the things in it.
> English is an active wolf, but the background as wolf indicates that the active button is depressed.
> Too many places to install. Sweden was released from 1968 and July 1973. Probably number of times the alarm is to be reapeated within 39 minutes. 2010 is in total 66 chickens, but later a famous chicken and chickens 12th of march 2010, with a total of 54 chickens
Apertium's translation is often stilted, but it gets the meaning across: Wolverines are the largest animals in the marten family, etc. Google Translate isn't half bad at Norwegian→English, so you can even combine them to get a sort of Saami→English: https://translate.google.com/#view=home&op=translate&sl=no&t....
I tried Baidu again now with the same input, just to see what they would give into English, and now it's about old ports, finished volcanos and civic computers: