> During training, we use monolingual speech-text datasets
So there's still a way till machines learn language as humans do, i.e. with sounds as primary modality. But nowadays I won't bet as to how long any ml task for language will take to be solved
If I understood correctly, to me there seem to be two keys to the proposed method:
1) they use a single, shared embedding space for the two languages, forcing the model to learn "semantics" independently (or rather, interdependently) of language
2) using back-translation for training. I'm not sure that I got this right, but this seems to be round-trip translation? So the model can self-assess its performance by checking the spanish->english->spanish difference.
Sounds very promising and interesting! However, it seems they only tested on spanish and english. I wonder if the similarity of the languages at the lexical level made these results possible.
I've wondered for years how far you could get just checking perplexity. English -> internal rep, and x-> internal rep. Then mapping between the internal reps such that English -> another language has low perplexity. That is, a sensible sentence in English should result in a sensible sentence in another language.
The wiki link is good. In the context here it's easy to picture it as how weird a sentence would sound to a native speaker. Low perplexity means what was generated would be unsurprising if you saw it in the dataset.
Some form of internal representation is crucial. Translation is a n^2 problem where some nodes like Chinese, English and Spanish have much thicker arrows, which makes traditional approaches awful for less common languages-pairs.
Aside from the lack of training data in many languages, I get the impression that tech companies like Google have been anglocentric in their approach, resulting in ok results only if at least one of the languages are “big”. That’s one thing that’s amazing about ChatGPT, it doesn’t discriminate between languages much, or, at least it seems like it’s able to transfer knowledge really well between languages. It seems it finds the higher level patterns of human knowledge to the point where language or even style is basically just a frontend.
Ironically, it seems the less you bother to teach computers about linguistics, the better they perform at language.
I really hope LLMs become mainstream in non-tech fields soon. I would love to see its impacts on the social sciences, linguistics, diplomacy (and yes, animal communications!), etc.
Translation is just a special case of general conversation. Just as LLMs can do translation just by asking, we ought to train a general speech-to-speech model for conversation, then we can simply ask it to translate for us.
AudioLM was developed more than a year ago and was an "audio continuation" model which is basically what you describe. However it was not used for conversational purposes afaik. Maybe it just needed some RLHF tuning?
Looking forward to the debate about real-time translators censoring or altering people's speech.
Also the debate about whether all human speech need be piped through such a preemptive filter. (actually not looking forward to this one). Suddenly everything that anyone says will be couched with "it is important to consult a professional to ensure safety and compliance with local regulations".
Machine translation is generally regarded as pretty bad when you speak both languages. I've heard really accurate (in terms of voice and intonation) speech to speech translation which tries preserve the speaker's voice, and I fear this will create a false sense of security about accuracy of translation. Similar to when chatgpt gives us confidently false information that is then trusted.
LLMs are _very_ good at translating the languages that I speak. They handle slang and colloquialisms very well, and have been surprisingly good at extracting very context-dependant meaning from the examples that I’ve tested.
They’re obviously not perfect, but this is a task that’s impossible to do perfectly. To translate some speech accurately you first have to decipher what the speaker was trying to say, and with most speech coming from human beings, it’s not always clear exactly what it is they’re trying to express, and that’s before you even address trying to translate it.
> you first have to decipher what the speaker was trying to say
This is also where the quality is varying a lot between languages.
I'm a native French speaker with an heavy non-standard accent and no speech recognition currently works for my normal voice.
I also speak Vietnamese to an okay level and while the speech recognition is usually good, the fact that the language is not based on words but syllables usually trips machine attempts at translation and the result is generally pretty bad.
These kind of live translations while impressive require the full pipeline to work, speech recognition and then proper translation. If any of them fail, the whole thing fails. On my case, it only works with my English for now then.
> These kind of live translations while impressive require the full pipeline to work, speech recognition and then proper translation.
That's a good point. Machine translation of text input by LLMs is pretty good for the languages I speak--English and Japanese--but it can get much worse with spoken input if the speech recognition isn't very good.
That said, OpenAI's Whisper is able to transcribe my spoken English nearly perfectly, and GPT-4's translation of that transcription into Japanese is usually very good. The main failures are with proper names.
Quality seems to depend on the language. For languages with less online presence, you get absurd errors very frequently, especially if there are any typos or regional variation in spelling (common in phonetic languages, in an informal context).
There was a thread today about LLMs' robustness to scrambled data; this seems heavily dependent on the quantity of training data.
That's for text to text though, not sure if voice to voice would be better or worse in this regard.
Amazing. And frustratingly light on details about this magical "MUSE loss" which seems to do a lot of the heavy lifting here, will have to read the paper to see how that works. Anyone have a tldr?
I cannot imagine that this can work well: my personal speaking (and also writing) in my native language is often full of puns that obscure side-meanings or homophones of words, or subly alter common phrases.
I cannot even imagine how this translation service might be capable of translating this. A human translator (translating it from speech or text to text) would likely add lots of footnotes with explanations so that the reader is capable of understanding the intended side-meanings that are hard to translate.
31 comments
[ 3.2 ms ] story [ 86.6 ms ] threadIn any case, text seems to stil form a part:
> During training, we use monolingual speech-text datasets
So there's still a way till machines learn language as humans do, i.e. with sounds as primary modality. But nowadays I won't bet as to how long any ml task for language will take to be solved
1) they use a single, shared embedding space for the two languages, forcing the model to learn "semantics" independently (or rather, interdependently) of language 2) using back-translation for training. I'm not sure that I got this right, but this seems to be round-trip translation? So the model can self-assess its performance by checking the spanish->english->spanish difference.
Sounds very promising and interesting! However, it seems they only tested on spanish and english. I wonder if the similarity of the languages at the lexical level made these results possible.
Aside from the lack of training data in many languages, I get the impression that tech companies like Google have been anglocentric in their approach, resulting in ok results only if at least one of the languages are “big”. That’s one thing that’s amazing about ChatGPT, it doesn’t discriminate between languages much, or, at least it seems like it’s able to transfer knowledge really well between languages. It seems it finds the higher level patterns of human knowledge to the point where language or even style is basically just a frontend.
Ironically, it seems the less you bother to teach computers about linguistics, the better they perform at language.
I really hope LLMs become mainstream in non-tech fields soon. I would love to see its impacts on the social sciences, linguistics, diplomacy (and yes, animal communications!), etc.
https://google-research.github.io/seanet/audiolm/examples/
Also the debate about whether all human speech need be piped through such a preemptive filter. (actually not looking forward to this one). Suddenly everything that anyone says will be couched with "it is important to consult a professional to ensure safety and compliance with local regulations".
[Obama's] "Anger translator" (2012) https://www.youtube.com/results?sp=mAEA&search_query=anger+t...
A citizen ostensibly forfeits their right to sue for defamation when they become a public figure; but counter-non-fraud isn't fraud either then eh.
Say "that's not enhanced" just like the old one please.
Have you worked at a company with a PR, Comms, and HR department?
They’re obviously not perfect, but this is a task that’s impossible to do perfectly. To translate some speech accurately you first have to decipher what the speaker was trying to say, and with most speech coming from human beings, it’s not always clear exactly what it is they’re trying to express, and that’s before you even address trying to translate it.
This is also where the quality is varying a lot between languages.
I'm a native French speaker with an heavy non-standard accent and no speech recognition currently works for my normal voice.
I also speak Vietnamese to an okay level and while the speech recognition is usually good, the fact that the language is not based on words but syllables usually trips machine attempts at translation and the result is generally pretty bad.
These kind of live translations while impressive require the full pipeline to work, speech recognition and then proper translation. If any of them fail, the whole thing fails. On my case, it only works with my English for now then.
That's a good point. Machine translation of text input by LLMs is pretty good for the languages I speak--English and Japanese--but it can get much worse with spoken input if the speech recognition isn't very good.
That said, OpenAI's Whisper is able to transcribe my spoken English nearly perfectly, and GPT-4's translation of that transcription into Japanese is usually very good. The main failures are with proper names.
There was a thread today about LLMs' robustness to scrambled data; this seems heavily dependent on the quantity of training data.
That's for text to text though, not sure if voice to voice would be better or worse in this regard.
I cannot even imagine how this translation service might be capable of translating this. A human translator (translating it from speech or text to text) would likely add lots of footnotes with explanations so that the reader is capable of understanding the intended side-meanings that are hard to translate.