It’s amazing how far text to speech has come in the past few years, but what I’m wondering is when this tech will finally make it into local TTS engines baked into the OS (eg for screen readers, etc)
I'm the most excited for an open source one though, and it would be incredible if this could become it. I do 95% of my compute on desktop linux and it sucks being behind.
Differences in verb-subject-object word order will always add latency. If you want to translate from German, with the verb at the end, to Welsh, where the verb goes at the start, you'll have to wait for the complete sentence before you can begin.
Not necessarily true, for the first few sentences you won’t be able to do it. But afterwards, once the context is established you don’t really need to wait for the verb, you can predict it. For example if you are speaking about cleaning the house and you detail that you have cleaned the kitchen the stove and so on, you can predict the verb with only the start of the sentence. I don’t have any source to back this up, but it sounds plausible
A good approach might be to start with how top notch, ultra-experienced human translators handle corrections for real-time scenarios, for example, the expert translators that do the ear monitors at the United Nations. I've worked with a few such real-time translators when preparing keynote speeches and they seem to have rigorous processes that appeared quite deep. Probably a ton of domain expertise to be captured there.
That said, I suspect that real-time language translation is always going to be somewhat imperfect due to its nature. Non-real-time translation of literature is still a subjective art form even at the very high-end of human expertise.
Although true and considering what “mrob” had also replied, this will never mean full translation every time, all the time. This will work with specific environments and linguistic expectations.
I’ve been learning german since 8 years, and the amount of expressions and different ways to say things around the country is impressive. There’ll be a “interpretative” real-time translation, but it won’t guarantee fully understanding in so many cases, maybe ever.
Other thing, and we have this in common with all languages, is the context and this is difficult to address i believe.
Nevertheless, it’s impressive how far we’ve reached and i acknowledge the usability of these tools. However, human knowledge will be always crucial and primordial if we want to guarantee full understanding.
The voice cloning worked pretty well for me. From english to spanish I noticed that the first few words sounded more like me than the last few words. Also it doesn't sound like how I speak in spanish but that's expected.
Voice cloning works pretty well already but not necessarily on one 10 sec sample as the source data. If you can give it some hours of data it’ll work much better
Do you have examples of it working well? I haven't heard anything that really impressed me. Nothing close to a good human impersonator. We're a long, long way from replacing voice actors, even considering the rapid rate of progress.
Besides the obvious good news about making it easier for people to communicate with each other across languages, it's also exciting to me that we're trending towards a world where I can tap into all the knowledge that only exists on the non-English web. I'm sure there are vast troves of programming knowledge in the Japanese-only web for example. The Chinese-only and Russian-only web are obvious candidates too but presumably those are harder to access for other reasons.
It won't replace high-end talent, I don't think models can replicate the nuance for a long time, however the entire low-to-mid end of the market is going to get nuked from low earth orbit
I wonder which will happen first - AI evolves to work well at the high-end, or high-end humans retire and there’s nobody left in the low-to-mid end to fill their shoes…
It will absolutely replace high-end talent. Anything that a human can do will be able to be done 10x better by a model -- especially in such a narrow and well defined domain.
Did you hear the output examples? Yeah, I think not. I mean, definitely on the way, but there's no way if you need quality acting in your dub that you're going with this.
These are models specially tuned and sized for near real-time, instant translation. It would be naive to think that there aren't technical creatives building and training models tuned for expressiveness and nuance in a more controlled environment.
Maybe not in the current state of the model, but judging by the rate of improvement we’re all seeing it’s just a matter of time (and data+compute+research obv).
It won’t replace it, but it’s very likely to supplant it, just about destroying the segment by reducing demand by being good enough and so much cheaper, especially as people get more used to it.
Typesetting. Music engraving. Bookbinding. The quality of all these fields have been materially harmed by advancements.
Computer typesetting has, by and large, been a significant regression, though the gap has largely been made up now if you make the right choices.
Published music scores used to be set by experts. Now they’re set by novices using software that is mechanical in method and generally quite insipid. Most are atrocious compared to the old masters, and mediocre at best compared to the typical published scores from a hundred years ago; and very few popular scores are really good (… and if they are, there’s a reasonably high chance they’ve used GNU LilyPond, which has focused on this problem). But the barrier for entry is so much lower, and people have got used to the inferior results, so I don’t know if anyone engraves music the old way, and even people that know better largely just shrug and make do with the new. Like with computer typesetting, there is hope because things have slowly improved. But most will continue to be mediocre.
Books used to be bound with cold glue. It takes time to set, but the results are very good, supple and long-lasting. Then along came hot-melt glue, and it’s just so much friendlier for cheap manufacturing because books are finished within a few minutes instead of a day or two, that I don’t think anyone produces books the old way any more, even though the results are abysmal in comparison (compare the binding and reading experience of a paperback from the ’40s or ’50s with one from the turn of the century; no one after tasting the old will desire the new; for he says, the old is good). But they’re just (barely) good enough. Unlike the other two, I don’t think there’s any hope here—the regressive advancement crowded out the superior but dearer option so that no place was found for it.
You can still get relatively good published music scores from a few of the old German shops (Schirmer, Henle, etc.), but they are very expensive. They are a joy to use when playing, though, since the music is very clearly laid out and page turns are in the perfect place, etc. Finale and Sibelius are controllable enough that you can use them to do fantastic layout, but many people either do not understand how to make a score readable or don't care enough.
That, and what GP describes, is what I see as the overall trend of the market to hollow out the middle. It's not just about technology (though it plays a big role), as all optimization coming from competitive pressure - materials, processes, business models, marketing.
What seems to universally happen is, the market bifurcates - one part is in a race to the bottom, the other (much smaller) aims for super premium tier (overpriced quality), because only those two positions are sustainable, once the race-to-the-bottom side drags all the economies of scale with it. So as a consumer, you get to chose between cheap low-quality garbage that's barely fit for purpose, and rare, super-expensive, professional/elite high-end products. There is no option for "good value for reasonable price".
This has been happening to everything - software, furniture, construction, electronics, vehicles, food, you name it.
I'm using AI for training videos for my startup. Never going back to voice actors outside of primary marketing videos. tThe sheer convenience of write/listen/tweak cycle on scripts is insane. In minutes you can do a voiceover which would have have taken hours + days delay prior.
Sure the final result sounds slighty robotic. 99% of people wouldn't care, and you can get more training videos done, faster for a fraction of the cost.
[Edit] And I'll add the difference from 6 months ago is noticeable to today. I imagine every 6 months we can just re-download updated voiceovers and every 6 months will sound just slightly more polished..
> I told her then that the industry would be disrupted by AI before she retired.
Yes. I just discovered there is a text-to-speech addon [1] (now a few months old) for World of Warcraft that adds voices for every NPC in the game... It is so impressive and game changer (pun intended) that I naively asked in the chat of the Twitch stream I was watching "when did Blizzard add voices to the NPCs??". For an instant I really thought Blizzard contracted actors, but no, someone like you and me just used AI to generate realistic voices for every character in the game. I don't think it's ready yet to completely replace actors in video games (surely it will in the near future tho) but voice acting is something so expensive to do that I can see studios and developers in 2024 already use this tech for all the optional dialogues and secondary characters' voices.
I've wondered at what point this would happen. I think it could now, but from what I've read the voice actor unions are able to prevent it currently (at least for AAA games or non-indie devs). Many of them have agreements/contracts in place for the foreseeable future, and being the first big company to replace them is a heap of terrible press that nobody is going to want to touch. I think it's the same reason Hollywood reached the AI agreement recently too.
I work on seamless. You can see the results in the paper. M4Tv2 is significantly ahead (Whisper Large v3 - 16.9 BLEU vs. M4Tv2 26.6). These are averages over 81 directions X->english
This is what I'd like to build (the tutor part at least, not the VR game part yet). I'm planning to extend my current English only rough prototype[1] to support Mandarin. (I happen to be learning Mandarin myself at the moment, and there are a bunch of open source bilingual Mandarin LLMs and speech synthesizers from China to choose from.)
I think a lot of people are working on similar things right now. I know of one called http://yourteacher.ai
Is there a high quality speech synthesizer (ideally local) for Mandarin you have found? There are some subtleties with tone sandhi rules and how they interact with prosody that I feel are lacking with current TTS voices I’ve tried.
I don't have the expertise to judge the quality of Mandarin pronunciation myself, being a beginner. But it sounds OK in English and it's made by native Mandarin speakers in China so I expect that it sounds better in Mandarin than English.
I love the idea of LLMs being super-efficient language tutors. And you have a good point; coming soon: "We've been getting a lot of these tourists here lately, they're eerily fluent, but all seem to have the same minor speech impediment" (read: messed-up weights in a commonly used speech model).
I've been using ChatGPT 4 to translate and explain various texts in Mandarin and it's been very on point (checking with native speakers from time to time, or internet searches). As expected, it has trouble with slang and cross-language loanwords from time to time. However for languages with much lower information online, it hallucinates like crazy.
> coming soon: "We've been getting a lot of these tourists here lately, they're eerily fluent, but all seem to have the same minor speech impediment"
Haha, if that were to pass, that would still be a far better outcome than our current situation of completely blind machine translation (this is especially for various Asian languages that are very sensitive to phrasing) and mispronunciation by non-native speakers.
Kind of, Accents are typically derived from the intersection of natural languages, specifically which ones you learned the phonetics of first. (With the exception of the Mid-Atlantic accent...)
This would be something quite novel as the speech irregularities would not have their origin in people
I don't know what you would call it but it needs at least some adjective before accent to differentiate it IMO
the azure neural tts voices in chinese are the best i’ve heard, specifically the “xiaochen” voice. i use it in anki daily to generate sentences for my mandarin decks with an api key/plugin. it’s not something you run locally of course, but they have a decent enough free tier.
i’m hoping a voice as realistic as this becomes a local app soon, but i’ve not found anything that’s nearly as natural sounding yet. (also, honorable mention to chatgpt’s “sky.” she pronounces mandarin with a funnily american accent, but it sounds natural and not as robotic as the open-source alternatives i’ve tried)
Duo has a different problem for me. The lack of focus means some languages don't get features. Chinese still doesn't have Stories (there's an unofficial version of it, but we've been waiting years).
(Duolingo problem(, AIUI): Duolingo is designed around such premise that, by exposing your subconsciousness to such small set of words and phrases in target languages, your brain should be able to trivially construct output shims from Universal Grammar, which must exist, to desired languages; but that doesn't work in practice and you end up with small set of words and phrases your subconsciousness had recorded)
the Duolingo's problem it is not because they have a bunch of languages, it is because achieving fluency in a target language it is about been able to produce/generate phrases, and they just move you to consume and sort words and phrases. in the case of any AI Language tutor, the student must produce phrases in order to practice, and that makes them advance in the path to achieving fluency
Sure, if you're not into personal growth. Not everyone wants to become the useless bit of lard sitting in a chair while a computer does everything for them. Yet. Some of us still like to do the actual things, but just need some assistance along the way. We still have a bit of time before we're all the humanoids from Wall-E
That isn't an extreme example at all, people used to mill grain and make clothing by hand, now we don't. We somehow are not sitting around getting fat even though technology takes care of those tasks.
The parents suggestion is that if we don't have to learn languages that will lead to us all laying down drinking big gulps while robot slaves take care of us. Their take is the extreme example. People have literally made this same suggestion about every technological advance and it never comes true.
> We still have a bit of time before we're all the humanoids from Wall-E
Obligatory reminder that the movie itself explains that people are what they are not because of their lifestyle, but because of the time spent in low-gravity environment.
It depends on what your goal is; for some tasks it's possible that getting the AI to do it is best, but, e.g. the existence of auto-pilot doesn't mean that hobbyist pilots wouldn't benefit from/enjoy exercising the same skills manually.
Maybe prior to fluency, for something like an odd business or tourist trip.
But there's a point in language learning where you can come to express yourself directly in a new language without intermediary "thinking" in your first tongue. The communicative and expressive potential of that mode is much higher than trying to squeeze one's intent through any kind of translation, machine or internal.
Not necessarily. It depends on the use case. For taking a vacation, having an AI that can instantly translate to your native language would be amazing. That’d solve a lot of real world problems, no doubt.
However, translation has a great deal of subjectivity embedded in it, particularly when there aren’t 1:1 translations. Case-in-point: there are many English translations of the Christian bible, all similar enough, but there are enormous variations in some cases. And there are at least as many branches of Christianity as there are English translations of the Bible. Some of them strictly recommend the same translation, and they still disagree on the meaning of various passages.
Besides the problems inherent to translation, learning another language gives you another paradigm of thinking. The words we use, the way we construct sentences, etc., all impact our view of the world. Here’s a paper that discusses the impact of the over-reliance on English in cognitive sciences, and how this has downstream effects:
https://www.sciencedirect.com/science/article/pii/S136466132...
Learning languages as an adult also has protective benefits. It reduces the probability of Alzheimer’s (maybe dementia, overall?).
I'm learning Hindi and there are somethings that are easy (phonetic alphabet, nothing like 7 different sounds for 'ough') but the sentence structure is very different and can be hard to get right. Pronunciation isn't too bad for the most part but there a few tricky things, for example four different 't' sounds and four different 'd' sounds. The hardest part is that there really aren't that many resources. Even though Hindi is the third most spoken language in the world, you will find far more resources for many of the less spoken European languages.
To me the key functionally for any language learning app is giving you feedback on your pronunciation and general understanding. I’ve been using Duolingo to learn Mandarin and when I try to speak to anyone it’s difficult for them to understand me, because my pronunciation is all wrong. The app is just feeding info to me one way, and I can try my best to recreate what I’m hearing, but there’s no way to know if I’m messing it up. They do have a speaking feature but it doesn’t work very well, certainly not to the same level as speaking with a real person who is fluent in the language and having them correct you.
As a quick solution, you should try recording yourself speaking and then listen to it to check your pronunciation against some reference. So for example, find a YouTube video in the language you're learning that also has good subtitles (use https://filmot.com/ ) and listen to how they say the phrase and then record yourself saying the same phrase and play it back and compare.
I practiced for a long time using the below pronunciation trainer and I get a ton of compliments from native speakers on how accurate my pronunciation is.
It's the same struggle language learners have faced for a long time regardless of app or not. I did careful studies of the French grammar, read French books, listened to audio tapes in French, and I still got prononciation wrong often.
I resisted using Duolingo because I knew their speaking feature sucks. But the only reason I need an app rather than books or audio tapes is that I need something to correct my prononciation.
Absolutely, what I've noticed is that the current apps are great for beginners but after a certain point the only way to improve your ability to speak a new language is to well... speak it. I built Proseable to help people move beyond the generic how to order a coffee or ask to go to the bathroom, and have more meaningful conversations in the real world. Check it out!
Yes! Better yet, you're a spy, or a hostage negotiator, or the leader of any kind of enterprise (army, business, aid organization) ...
Programming games like that will resemble directing improv theater. You can't program every response; you'll have to instead fit each character with beliefs and motivations.
For Language Acquisition, Input Is All You Need. (Mostly)
What would be really cool is something that can autodub videos or audio into your target language. The hardest problem learning languages that aren't English is often finding content to consume in them.
Disclaimer : I am Krashenist so this take is biased
I would love a game that helped you learn a language (not necessarily VR though as I don't have that equipment). The game drops you into a world (a country of the language the game is meant to teach you) where no one speaks your language and you have to figure out what people are saying in order to fulfill quests. You get some hints, like maybe you have a simple translation guide in your inventory or sometimes you meet people who can speak a few words of your language. That would motivate me to learn faster than self-taught tutorials.
I'd love to learn French and the game would take place in locations all around modern France.
It would have to a good story. Maybe something in the style of Professor Layton series could be interesting, or something more open world.
If Professor Layton itself has a French translation then you're more than half of the way there! Existing games are already quite good for language learning. But indeed they're missing the "realistic" element that you're after.
https://chat.quazel.com teaches languages through speaking and having specific goals on where to steer the conversation. It's actually pretty nice. Has many language combinations to learn, ex. French speaker learning German. They seem to use LLMs and text to speech for now but I'm sure they'll get there with this tech progressing so fast. Duolinguo is from yesteryear in my opinion, learning by speaking is way better.
I was hoping to find out, that the actor's voice in the demo video was generated, or that he had recorded the video speaking in another language or something.
"The Babel fish is small, yellow, leech-like, and probably the oddest thing in the Universe. It feeds on brainwave energy received not from its own carrier, but from those around it. It absorbs all unconscious mental frequencies from this brainwave energy to nourish itself with. It then excretes into the mind of its carrier a telepathic matrix formed by combining the conscious thought frequencies with nerve signals picked up from the speech centres of the brain which has supplied them. The practical upshot of all this is that if you stick a Babel fish in your ear you can instantly understand anything said to you in any form of language. The speech patterns you actually hear decode the brainwave matrix which has been fed into your mind by your Babel fish.
"Now it is such a bizarrely improbable coincidence that something so mind-bogglingly useful could have evolved purely by chance that some thinkers have chosen to see it as a final and clinching proof of the non-existence of God.
"The argument goes something like this: 'I refuse to prove that I exist,' says God, 'for proof denies faith, and without faith, I am nothing.' 'But, says Man, the Babel fish is a dead giveaway, isn't it? It could not have evolved by chance. It proves you exist, and, by your own arguments, you don't. QED.' 'Oh dear,' says God, 'I hadn't thought of that,' and vanishes in a puff of logic."
Illinois has a facial recognition / cloud biometrics ban. Familiar face detection for doorbells etc. isn't allowed there. Wonder if Texas has something similar?
Likely related to biometrics laws. I know Illinois has restrictions on the collection of biometrics, not sure about Texas. Facebook in particular paid out a significant amount of money in a class action in Illinois, I know because I got a chunk of change from it.
which you mean someone took a dime and carved off a piece of it, and then sent you a piece of paper with postage that cost more than the value of that chunk? yeah, we all got hosed by that one too i'd imagine
Facebook has had to pay out hundreds of millions of dollars in settlements for related class-action lawsuits, and rather than trying to get informed consent, they’re deciding not to collect biometrics from residents of those states.
As someone working in tech and following along the progression of AI, I believe I have the right expectation. But still feels surreal seeing myself speaking a foreign language in my own speech style.
Well that was spectacularly bad. Failed to translate a single word from english->spanish. Admittedly I was using George Carlins favorites, but if you're trying to have an expressive language translator that refuses to translate "fuck" then what you've got is bullshit.
Yes, we chose not to release the watermark detector to safeguard against adversarial attacks. This decision helps prevent any attempts to erase the watermark by malicious users.
The watermark generator and detector are trained together, one can use the information in our paper to train your own generator and detector model, however in this case the watermark signature created will be distinct from the one we use to protect our seamless translation models. This approach ensures each model maintains its unique security features.
I had pretty terrible results when I tried English -> Swahili I'm using the Huggingface M4T V2 spaces, it pretty much doesn't work most of the time and I just get English back with a different voice, Expressive on the other hand only has a few languages it seems.
It would be nice if they could layout what exactly is missing in terms of data to make a language work better, while the actual AI bit is out of reach for most of us maybe we could provide more data.
There is also a 60 sec limit and wonder if this is HuggingFace limitation or Seamless?
If you want to contribute by recording yourself speaking Swahili, https://commonvoice.mozilla.org/sw is the place to go. Although Meta has access to much larger data sets, they nonetheless use Common Voice as a "known good" source. E.g. the paper on their SONAR speech encoder reports experiments on Common Voice data, coincidentally involving Swahili https://ai.meta.com/research/publications/sonar-sentence-lev...
Besides the ACCEPTABLE_USE_POLICY, there's a CC BY-NC 4.0 (NonCommercial) license, a 'SEAMLESS_LICENSE' (NonCommercial), but also an MIT license? It would seem these other licenses contradict the MIT license, could somebody help clarify how these all interact in practice?
How will Meta put these models into practice? I understand why Google and Apple have models for their mobile OS users, but I don't understand where users for Meta speech models come from. Are they planning to show Instagram videos with English narration in French or what?
Ads and Reels (their TikTok competitor) I imagine would be the primary use-case. Imagine spreading the "wonders" of TikTok-like videos to non-$native_language speaking world.
They have arguably the most diverse userbase of any company, with users from pretty much every single country + language across all their services & apps. I could easily imagine a handful of use cases having a high performing universal translation model would be incredibly useful.
I'm thrilled to see the progress made in the last 30 years.
As a student in the mid-90s I worked on a system called Verbmobil at the German Research Center for AI and it did speech-to-speech for English, German and Japanese in very limited domain.
This was done via "classical" NLP: You had to model the domain with concepts, you needed sentence parsers, semantic engines, speech-to-text hand-crafted for 3 languages etc.
Neat. How translatable are tones of voice for intent across languages? Like does a person trying to do a "nerdy" voice(nasally, whiny, etc.) in English translate to the "nerdy" stereotype for a French speaker. Seems to do very good on whispers which made me wonder what could be next.
If you don't speak the language into which these models translate your inputs, how do you know if or why the model has generated, without being commanded to do so, a campy American gay male sociolect, or an African American regional accent, or some other thing that may convey unintended meaning to native listeners?
Seems like there will always be latency, because it's not possible to easily stream over languages that have different structure. You need to wait a bit before you can start faithfully translating the meaning.
They also mention it in one of the videos about the streaming variant of their translator. But I guess 2s delay or what they mention is close enough for practical purposes.
I feel like for personal relationships where true real-time is required, having a computer intermediary would be weird anyway and you have to learn the language, at least for the time being and as long as personal relationships are still relevant (in the post-AI world they might not be).
> You need to wait a bit before you can start faithfully translating the meaning
I guess it's possible that the AI learns about a specific person over time? That way it can be confident about what's being said as soon the person starts saying it
379 comments
[ 4.0 ms ] story [ 448 ms ] threadI'm the most excited for an open source one though, and it would be incredible if this could become it. I do 95% of my compute on desktop linux and it sucks being behind.
This may be apocryphal but I’ve heard that in formal settings (e.g. UN) they won’t translate it and will instead give instruction on when to laugh.
That said, I suspect that real-time language translation is always going to be somewhat imperfect due to its nature. Non-real-time translation of literature is still a subjective art form even at the very high-end of human expertise.
It's all a trade off.
Either way it's extremely exciting that we get to even discuss this stuff as real possiblities.
I’ve been learning german since 8 years, and the amount of expressions and different ways to say things around the country is impressive. There’ll be a “interpretative” real-time translation, but it won’t guarantee fully understanding in so many cases, maybe ever.
Other thing, and we have this in common with all languages, is the context and this is difficult to address i believe.
Nevertheless, it’s impressive how far we’ve reached and i acknowledge the usability of these tools. However, human knowledge will be always crucial and primordial if we want to guarantee full understanding.
"Since", as used here, would lead me to guess you are not a native English speaker?
I told her then that the industry would be disrupted by AI before she retired.
Glad she pivoted. Really impressive results.
a few more years of improvements if they happen could be disruptive
Typesetting. Music engraving. Bookbinding. The quality of all these fields have been materially harmed by advancements.
Computer typesetting has, by and large, been a significant regression, though the gap has largely been made up now if you make the right choices.
Published music scores used to be set by experts. Now they’re set by novices using software that is mechanical in method and generally quite insipid. Most are atrocious compared to the old masters, and mediocre at best compared to the typical published scores from a hundred years ago; and very few popular scores are really good (… and if they are, there’s a reasonably high chance they’ve used GNU LilyPond, which has focused on this problem). But the barrier for entry is so much lower, and people have got used to the inferior results, so I don’t know if anyone engraves music the old way, and even people that know better largely just shrug and make do with the new. Like with computer typesetting, there is hope because things have slowly improved. But most will continue to be mediocre.
Books used to be bound with cold glue. It takes time to set, but the results are very good, supple and long-lasting. Then along came hot-melt glue, and it’s just so much friendlier for cheap manufacturing because books are finished within a few minutes instead of a day or two, that I don’t think anyone produces books the old way any more, even though the results are abysmal in comparison (compare the binding and reading experience of a paperback from the ’40s or ’50s with one from the turn of the century; no one after tasting the old will desire the new; for he says, the old is good). But they’re just (barely) good enough. Unlike the other two, I don’t think there’s any hope here—the regressive advancement crowded out the superior but dearer option so that no place was found for it.
What seems to universally happen is, the market bifurcates - one part is in a race to the bottom, the other (much smaller) aims for super premium tier (overpriced quality), because only those two positions are sustainable, once the race-to-the-bottom side drags all the economies of scale with it. So as a consumer, you get to chose between cheap low-quality garbage that's barely fit for purpose, and rare, super-expensive, professional/elite high-end products. There is no option for "good value for reasonable price".
This has been happening to everything - software, furniture, construction, electronics, vehicles, food, you name it.
Sure the final result sounds slighty robotic. 99% of people wouldn't care, and you can get more training videos done, faster for a fraction of the cost.
[Edit] And I'll add the difference from 6 months ago is noticeable to today. I imagine every 6 months we can just re-download updated voiceovers and every 6 months will sound just slightly more polished..
Yes. I just discovered there is a text-to-speech addon [1] (now a few months old) for World of Warcraft that adds voices for every NPC in the game... It is so impressive and game changer (pun intended) that I naively asked in the chat of the Twitch stream I was watching "when did Blizzard add voices to the NPCs??". For an instant I really thought Blizzard contracted actors, but no, someone like you and me just used AI to generate realistic voices for every character in the game. I don't think it's ready yet to completely replace actors in video games (surely it will in the near future tho) but voice acting is something so expensive to do that I can see studios and developers in 2024 already use this tech for all the optional dialogues and secondary characters' voices.
[1] https://www.curseforge.com/wow/addons/voiceover
https://youtu.be/kZ87wiHps9s
Everyone gets a personal tutor for hours a day.
I would absolutely love a VR game where I just need to work in China or Mexico all day and pick up the language that way.
I think a lot of people are working on similar things right now. I know of one called http://yourteacher.ai
[1] https://apps.microsoft.com/detail/9NC624PBFGB7
I don't have the expertise to judge the quality of Mandarin pronunciation myself, being a beginner. But it sounds OK in English and it's made by native Mandarin speakers in China so I expect that it sounds better in Mandarin than English.
> coming soon: "We've been getting a lot of these tourists here lately, they're eerily fluent, but all seem to have the same minor speech impediment"
Haha, if that were to pass, that would still be a far better outcome than our current situation of completely blind machine translation (this is especially for various Asian languages that are very sensitive to phrasing) and mispronunciation by non-native speakers.
Ah, that is called an accent.
This would be something quite novel as the speech irregularities would not have their origin in people
I don't know what you would call it but it needs at least some adjective before accent to differentiate it IMO
i’m hoping a voice as realistic as this becomes a local app soon, but i’ve not found anything that’s nearly as natural sounding yet. (also, honorable mention to chatgpt’s “sky.” she pronounces mandarin with a funnily american accent, but it sounds natural and not as robotic as the open-source alternatives i’ve tried)
You end up with the Duolingo problem where you know to say the names of 20 different fruits but not how to introduce yourself.
Not sure if this is a duolingo problem. There are of modules in duolingo specifically for saying your name. I think its the travel module.
The parents suggestion is that if we don't have to learn languages that will lead to us all laying down drinking big gulps while robot slaves take care of us. Their take is the extreme example. People have literally made this same suggestion about every technological advance and it never comes true.
Obligatory reminder that the movie itself explains that people are what they are not because of their lifestyle, but because of the time spent in low-gravity environment.
But there's a point in language learning where you can come to express yourself directly in a new language without intermediary "thinking" in your first tongue. The communicative and expressive potential of that mode is much higher than trying to squeeze one's intent through any kind of translation, machine or internal.
Plus, you know, it's fun.
However, translation has a great deal of subjectivity embedded in it, particularly when there aren’t 1:1 translations. Case-in-point: there are many English translations of the Christian bible, all similar enough, but there are enormous variations in some cases. And there are at least as many branches of Christianity as there are English translations of the Bible. Some of them strictly recommend the same translation, and they still disagree on the meaning of various passages.
Besides the problems inherent to translation, learning another language gives you another paradigm of thinking. The words we use, the way we construct sentences, etc., all impact our view of the world. Here’s a paper that discusses the impact of the over-reliance on English in cognitive sciences, and how this has downstream effects: https://www.sciencedirect.com/science/article/pii/S136466132...
Learning languages as an adult also has protective benefits. It reduces the probability of Alzheimer’s (maybe dementia, overall?).
https://www.parcero.ai/
I could integrate this instead of Polly pretty easily.
https://fluent-forever.com/product/fluent-forever-pronunciat...
I resisted using Duolingo because I knew their speaking feature sucks. But the only reason I need an app rather than books or audio tapes is that I need something to correct my prononciation.
https://github.com/adrianmfi/gpt-tutor
I look forward to testing if switching to Seamless can improve it further, Seamless supporting nearly 100 languages is a nice improvement.
https://www.proseable.com/
Yes! Better yet, you're a spy, or a hostage negotiator, or the leader of any kind of enterprise (army, business, aid organization) ...
Programming games like that will resemble directing improv theater. You can't program every response; you'll have to instead fit each character with beliefs and motivations.
I can hardly wait.
What would be really cool is something that can autodub videos or audio into your target language. The hardest problem learning languages that aren't English is often finding content to consume in them.
Disclaimer : I am Krashenist so this take is biased
I'd love to learn French and the game would take place in locations all around modern France.
It would have to a good story. Maybe something in the style of Professor Layton series could be interesting, or something more open world.
That would have been the knockout punch.
https://seamless.metademolab.com/expressive/?utm_source=meta...
Interesting mix.
In Texas it seems to be part of AG Paxton's culture war stuff. https://www.texastribune.org/2022/05/12/texas-face-filters-i...
> According to the Settlement Administrator, payments to class members between $200 to $400 started going in the mail May 9.
I got a $0.19 check from an iTunes settlement once, but this wasn't one of those cases.
Facebook has had to pay out hundreds of millions of dollars in settlements for related class-action lawsuits, and rather than trying to get informed consent, they’re deciding not to collect biometrics from residents of those states.
Edit: I can’t find the weights but if I’m reading the paper right anyone could train their own detector.
Yes, we chose not to release the watermark detector to safeguard against adversarial attacks. This decision helps prevent any attempts to erase the watermark by malicious users.
The watermark generator and detector are trained together, one can use the information in our paper to train your own generator and detector model, however in this case the watermark signature created will be distinct from the one we use to protect our seamless translation models. This approach ensures each model maintains its unique security features.
It would be nice if they could layout what exactly is missing in terms of data to make a language work better, while the actual AI bit is out of reach for most of us maybe we could provide more data.
There is also a 60 sec limit and wonder if this is HuggingFace limitation or Seamless?
If you want to contribute by recording yourself speaking Swahili, https://commonvoice.mozilla.org/sw is the place to go. Although Meta has access to much larger data sets, they nonetheless use Common Voice as a "known good" source. E.g. the paper on their SONAR speech encoder reports experiments on Common Voice data, coincidentally involving Swahili https://ai.meta.com/research/publications/sonar-sentence-lev...
Besides the ACCEPTABLE_USE_POLICY, there's a CC BY-NC 4.0 (NonCommercial) license, a 'SEAMLESS_LICENSE' (NonCommercial), but also an MIT license? It would seem these other licenses contradict the MIT license, could somebody help clarify how these all interact in practice?
https://github.com/facebookresearch/seamless_communication#l...
As a student in the mid-90s I worked on a system called Verbmobil at the German Research Center for AI and it did speech-to-speech for English, German and Japanese in very limited domain.
This was done via "classical" NLP: You had to model the domain with concepts, you needed sentence parsers, semantic engines, speech-to-text hand-crafted for 3 languages etc.
As it turns out, this approach is/was a dead-end.
Just text to speech has gone too far. Audio books would be mainly generated on the fly like this?
I think some RPGs in some 5 years time might have something like this:
- A text file that outlines characters and a lose plot/Story line. Human written.
- 3D Mesh Generation based on character description via Transformers based models. Auto generated.
- Dialogues for each NPC via LLM.
- This TTS engine again based on such models.
Result - almost unlimited replayability. Or even edit text file, have a new world based on a new story line with characters having different personas.
[0]. https://news.ycombinator.com/item?id=38335255
They also mention it in one of the videos about the streaming variant of their translator. But I guess 2s delay or what they mention is close enough for practical purposes.
I feel like for personal relationships where true real-time is required, having a computer intermediary would be weird anyway and you have to learn the language, at least for the time being and as long as personal relationships are still relevant (in the post-AI world they might not be).
I guess it's possible that the AI learns about a specific person over time? That way it can be confident about what's being said as soon the person starts saying it