HN Explorer

Ask HN: My wife might lose the ability to speak in 3 weeks – how to prepare?

855 points by tech4all ↗ HN
My wife will be undergoing significant oral surgery in a few weeks and there is a SMALL chance she may lose the ability to speak. I'd like to prepare, just in case, to have technology to reproduce her voice from keyboard or other input.

My ideal would be an open source "deepfake toolkit" that allows me to provide pre-recorded samples of her speech and then TTS in her voice. Unfortunately most articles and tools I'm finding are anti-deepfake. Any recommendations?

Fallback would be recording her speaking "phonetic pangrams" and then using her pre-recorded phonemes to recreate speech that sounds like her. I feel like the deepfake toolkit is the way to go. Appreciate any recommendations... There must be open source tools for this??

228 comments

[ 3.6 ms ] story [ 113 ms ] thread
fxtentacle
Record her reading the texts of a standardized text training corpus.

That way, you can retrain an existing AI to do text to speech with her own voice.

Edit: here's a link to the corpus that I believe Mozilla uses http://www.openslr.org/12/

Mo3
This right here
olivermarks
https://youtu.be/0sR1rU3gLzQ

'This AI Clones Your Voice After Listening for 5 Seconds'

joeyspn
Woah, this is definitely the solution that the OP needs. I did read about WaveNet's text-to-speech a couple of years ago but didn't know it has progressed this far. It's crazy good, mindblowing.
aerovistae
And get a high quality mic to do it with!
cabite
or better, rent a recording room for the time it takes.
trynewideas
Mozilla's is licensed CC-BY, which is pretty liberal. In case the Attribution license is a blocker, here's CMU_ARCTIC's, which is built from copyright-free sources and has no licensing restraints: http://festvox.org/cmu_arctic/
asveikau
Is she on board with this? I can imagine a lot of people being severely put off by being asked to record "a corpus of approximately 1000 hours" in advance of what sounds like a stressful surgery.
fxtentacle
It's 1000 hours because multiple speakers record the same articles.

I believe some speakers only recorded 1-2 hours, which seems doable.

jfkebwjsbx
They have 500 hours left, so that would be impossible.
joshribakoff
Seconding this, also, reproducing her voice with an AI may not be something she is on board with, it could make her feel like you don't accept her with or without a voice. It may also be unhealthy for you, similar to how spending too long on social media can become a dangerous source of dopamine.

It might make sense to consider making a recording that is more meaningful, and focus on giving her emotional support rather than building an AI that could be perceived as a replacement.

netsharc
It's not like OP is replacing her entirity with Alexa, if I were the wife I'd think "sure, let's 'backup' my voice, having it available in case I lose mine would be useful, so that people can still hear my thoughts in my voice instead of a robot's."...
badRNG
> if I were the wife I'd think "sure, let's 'backup' my voice"

That very well seems to be the OP's position as well. That's a far more generous reading of the situation. It makes sense that someone here would have the mindset of "lets keep a backup in case we want access to it later."

aspaceman
Of course, but I think it's very, very important that OP has this conversation themselves and doesn't take the word of folks on the Internet
tech4all
Good concern. We won't be doing any "hundreds of hours" solution. We've been married over 30 years so we're a pretty good team - naturally I wouldn't do it without both of us thinking it was a great idea.
josinalvo
I think this is backwards... This is a corpus to train speech to text, not text to speech, right?
joshribakoff
It's a corpus designed to capture the full breadth of combinatorial nuances of human speech in a general sense.
reubenmorais
No, it is not. For one, it's a corpus of read speech, which means it does not capture well the characteristics of conversational human speech – hesitation, disfluencies, different tones and registers, etc. LibriSpeech has a paper explaining the design of the corpus, all you need to read is the first sentence of the abstract to know what it is supposed to capture:

This paper introduces a new corpus of read English speech, suitable for training and evaluating speech recognition systems.

http://www.danielpovey.com/files/2015_icassp_librispeech.pdf

joshribakoff
That sentence alone does not establish that read speech differs from conversational speech, thanks for the information / pointing this out, though.
audiohermit
I'll push back on this. The quality of the read speech should be a higher concern than having parallel data. Unless OP's wife is a teacher or actor/voice actor, if LibriSpeech transcripts are boring, it will come out in the speech.

I think OP would ideally want the model to pick up on more natural intonation, instead of monotone dictation. Record everything from now on, as best you can with similar recording context, and hopefully that data will be enough to cover more natural nuances.

[deleted]
(comment deleted)
kerkeslager
I don't have any answers to give you, but I want to say that this is a really loving and beautiful thing you're trying to do.
Someone
Is it? My first thought was “is your ideal also her ideal?”.

We cannot rule out she wants to spend quality time with her partner instead of spending time in a recording studio, so that, if the worst outcome comes, her husband can remind her of what she lost.

kerkeslager
Presumably the guy is better at guessing what his wife wants than you are, and his wife is an adult who can tell him if he guesses wrong.
thaumasiotes
> his wife is an adult who can tell him if he guesses wrong

She can, but she might not. A lot of that depends on how he presents the idea to her -- it might seem like something that's important to him.

pugworthy
It's sad that people trying to discuss the emotional side of this are being down voted.

Honestly there is no doubt a very large emotional/personal side of this, irrespective of who's idea it is and who supports it.

Technology isn't the solution for all problems and challenges in life.

at_a_remove
No, it isn't.

But good lord, sometimes trying to get technical help on the Internet turns into this rabbithole of people who are specifically looking for ways not to be helpful. "Did you really want that?" "Did you consider alternatives?" "What you really have is an XY problem."

pugworthy
"Truly identifying a problem means looking deeper at the symptoms, the customer, the impact, the alternatives, the opportunity, and the relationships between them, while avoiding the “solution bias” (often known as “The issue is that the customer does not use my solution”)."

#1 item from https://www.molfar.io/blog/yc-questions

at_a_remove
Or not. Not everything has to be this super-deep, six whys exploration of how craving and attachment is the cause of all suffering and if you would only stop wanting a solution you would no longer be in pain.

Sometimes a cigar is just a cigar.

pugworthy
If it was a cigar he’d just ask the technical question of how to capture and simulate someone’s voice.
at_a_remove
"I'd like to prepare, just in case, to have technology to reproduce her voice from keyboard or other input."

He then goes on to say "My ideal would be an open source 'deepfake toolkit' that allows me to provide pre-recorded samples of her speech and then TTS in her voice."

That sounds like wanting to capture and simulate someone's voice.

kerkeslager
People are being downvoted because if you cannot possibly have an informed opinion on a subject, it's completely arrogant to form opinions on that subject, and even more arrogant to criticize someone publicly based on those opinions.

Literally the only person on this thread who knows anything about the OP's wife is the OP. Everyone else sharing an opinion on "the emotional side of this" is vocally ignorant.

fragmede
Presumably the wife is the best at knowing what she wants.

I'd make no presumption, good or bad, about the their relationship dynamic, however.

DaniloDias
By all means, let’s have hacker news expand this technical question into an evaluation of this guys marriage.
pezo1919
Life is complex bro.
kerkeslager
True, but I don't think it's particularly useful to get so caught up in all the possible ways that your kindly-intentioned actions could go wrong that you need permission to even try to do kind things. That's just a form of social anxiety.

And it's particularly useless when your worries are about a situation which does not concern you and which you are almost completely ignorant about.

daveFNbuck
I think the idea here is that she could use her own voice instead of a generic voice with text to speech devices. I doubt he intends to taunt her with it.
glonq
That was also my first thought, having seen far too many tech geeks inflict unwanted products and projects onto their poor partners and families.

The sentiment is admirable, but it's a lot of work considering that the probability of a negative outcome is very low.

rimliu
Depends. I'd be horrified to listen to my own voice when I am not speaking. Keep in mind, what we sound very different to ourselves.
kerkeslager
Presumably the guy is better at guessing what his wife wants than you are, and his wife is an adult who can tell him if he guesses wrong.
LeifCarrotson
From a different perspective, I'd be brokenhearted if I could not speak to my son or my wife in something they could recognize as their father or husband's voice. I know modern TTS systems are a fair sight (voice?) better than Microsoft Sam, but it would be emotionally valuable to me to have a self-trained TTS library.

I'm not sure there's a correlation to other senses, I can't see for my future self or move on his behalf. I suppose there are things I would want to taste or smell if I was going to lose those senses, but those are experiences for me, not things I'd use to communicate with loved ones.

After losing my voice in an accident, I'd be willing to spend many, many hours transcribing my own speech in the handful of scratchy family videos, voicemails, and phone logs of ordinary conversations. If I could spend a couple days prior to the event reading some books, a TTS training corpus, or anniversary/birthday/wedding/etc greetings and congratulations into a microphone and have a personal text-to-speech voice I'd be all over that.

It would be a little weird if someone else used it as their narrator, but that's not OP's goal.

Speaking of recording books and training corpuses, my grandparents (who have their voices) got a special kind of joy from reading children's books that they once read to me and that I once heard as a child to their new grandson. OP, if you and your wife have or might have kids (and she can handle it emotionally), it might be nice to record video/audio of reading children's books to future grandchildren. Even if your future grandchild knows that grandma can't read books out loud, I'd bet Grandma would be happy to silently turn the pages for a toddler on her lap until those digital recordings got worn and scratchy like an old VHS.

inanutshellus
I'd love to be able to change my mind, though.

Recording audio and then choosing not to use it later is fine.

Not recording it because I don't want it right now... maybe fine? maybe sad.

Alex3917
> Keep in mind, what we sound very different to ourselves.

This is less of a problem with modern high-quality mics than it was, say, with answering machines 30 years ago. Your voice might still sound not exactly the same, but it hopefully shouldn't be unbearably grating either.

corobo
Turn the bass up a bit

It's because reproduced audio doesn't have the bass the same as you hearing it conducted through your jawbone (though of course this will sound too bassy to everyone else!)

Alex3917
Makes sense. I think also until YouTube and podcasting became popular, most of the mics that the typical person would accidentally stumble on were probably bright rather than warm.
[deleted]
(comment deleted)
y-c-o-m-b
Was just about to post the same. Not only is this an amazing thing to do for her, but this is one of the coolest threads I've seen on HN in a long time
[dead] david_draco
Learn sign language, instead of holding on to the past.
thaumasiotes
She's not losing her knowledge of English. The odds are overwhelming that she'll prefer to communicate by English text (where she'll have native fluency) than in a new language.
gnulinux
I think most people who are not familiar with ASL don't understand that ASL is an entirely different language with completely different grammar, morphology and semantics. It's linguistically as distinct as it can be from English. It's probably closer to an English speaker learning Chinese than an English speaker learning German or French. Most people imagine it's just a relex of English where English words have "signs" and then you keep putting them the same order it's in English. Well no, ASL has it's own set of inflections, affixes, suffixes etc.
alexjm
Sign language in the US forms a sort of dialect continuum.

At one end is ASL, which its own language as you said, although with some influence from English).

On the opposite end are things like Signed Exact English that reproduce English verbatim on the hands, borrowing from ASL in some places and inventing signs in other places. For example, ASL has no sign for -ed (the past tense ending) because it marks aspect instead of tense, so SEE invented a sign for it.

And in the middle is Contact Sign or Pidgin Signed English, a sort of compromise between ASL and English grammar, which is most used when ASL speakers and non-fluent speakers communicate. You might get a mix of some ASL topic-comment word order with some English subject-verb-object word order. You'l get a lot less "simultaneous" ASL grammar like indicating the subject or object pronoun using eye gaze, or conjugating verbs by changing motion, because those are hard for non-fluent ASL speakers to understand. You'll also get a lot less of the SEE constructs like explicit -ed markings, because they're usually obvious from context.

thaumasiotes
> You might get a mix of some ASL topic-comment word order with some English subject-verb-object word order.

Not a huge shock, since you also get this between two sole English speakers communicating vocally.

klyrs
Personally speaking, as a hearing, native english speaker. I've taken classes in French, Italian, Spanish, and ASL.

The romance languages were taught in English, with a strong focus on the rules. Progress was extremely slow. I've had opportunities to practice Spanish and Italian, and that's been way more helpful than those classes ever were (and let's be honest, Spanish and Italian are too similar for my muddled brain and "communication" is more of a negotiation). I've heard from quite a few students of French that after 4 years of formal study, they can't speak a lick. I didn't hang on that long.

ASL is often taught immersively. You might have an interpreter for the first day or the first week, but after that, it's voices off (and if you're lucky, your instructor is Deaf). You start with letters, to bootstrap from English, and mimeing, because that forms a basis for a lot of informal ASL. Subsequent vocabulary is communicated through a combination of fingerspelling and mime. Grammar is considered a fairly advanced topic; and by the time your teacher gets around to that, you'll have absorbed more of it than you realize.

In my experience, learning ASL is more akin to riding a bike, than learning a romance language -- much less Chinese.

thaumasiotes
> In my experience, learning ASL is more akin to riding a bike, than learning a romance language -- much less Chinese.

The fact that you had formal language classes in romance languages doesn't mean that's the only way romance languages can be taught or learned. You can learn or teach any language however you want. Some ways are more effective; some are less effective.

One reason students can't speak any French after taking four years of school French is that effectiveness isn't really a goal of the instruction (or of the students).

gnulinux
If you learned French or Chinese the same way you learned ASL you could have made the same argument for these languages. My comment wasn't a point about how ASL is taught or the pedagogy of ASL learning -- these are things I know nothing about. I was just explaining the linguistic properties of ASL. But you're right, in retrospect I should not have used the analogy with Chinese, it really isn't a good analogy since learning a spoken language is a different experience than learning a sign language. I was just trying to point out the "linguistic distance" between English and ASL.
bluGill
For the first couple weeks, after that she will discover that sign language is easier than typing all the time. Of course this assumes both make a concentrated effort to learn the language. Once basic conversations can be had in sign language the rest comes. There is a small community of deaf people where ever you are that you can join so you won't be alone in your world which is important both for learned faster and for psychological reasons.

Fortunately the effort can be done anytime after the fact, so if she is lucky it might not be done at all.

[dead] ThePowerOfFuet
> Learn sign language, instead of holding on to the past.

What the fuck, man?

tech4all
LOL. Thank you for downvoting that for me. The Internet wouldn't be the Internet if it weren't for asshats!
sharker8
repeating the trolls is a form of feeding them.
issa
Ignoring how rude this comment is, I don't think embracing a technological solution could count as "holding on the the past".
glenpierce
Sign language is a pretty good option. There will be times when access to the technology used to produce her voice might be inconvenient or even unavailable. American Sign Language (ASL) is also probably much faster than typing in a lot of cases. I grant, this probably won't be as helpful out in the world where most people won't understand it, but in the comfort of home, being able to converse at the dinner table without a keyboard at your side might be preferable. I suggest you don't ignore this as a supplimental tool.
glenpierce
I should also point out that learning all of ASL isn't necessary for it to be helpful. "Yes", "No", and "Thank you" are pretty helpful on their own.
arethuza
What about recording messages to other people for future events (e.g. graduation of a child, birth of grandchild etc.)?

Recording a message to a yet unborn grandchild is maybe something we could all do!

arslnjmn
(off topic) Record a few things for her future self. E.g. favourite quotes, frequently used phrases.
zxter
Good advice! Maybe a few shoutouts to your future children.
happycry
We get quite a few requests for this at Resemble (https://resemble.ai). We can get her to record right on our website or you can upload an existing file (along with a video of her consent) on the platform. Feel free to shoot me a message and I'd be happy to help build a voice for her.
cdolan
I dont know how to send messages but I researched this space a few years ago. Unfortunately a family member of mine had a surgery result in loss of his speech.

We have a lot of tapes around of his voice, from voice mails to family videos to some things from his work. If you are open to reaching out that would be awesome, I’ll check out the site as well.

Edit: I’ve wanted to make some sort of soundboard + “text to talk” setup for this family member. He often can’t participate in conversations because he writes on a whiteboard, and the speed of chatter moves faster than his writing

louwhopley
Wow, this looks like a great service!

Out of interest what are the average response times to generate a clip of one or two sentences from a configured voice?

Imagining the easy text-to-speech solution the OP could build on this resemble API.

happycry
Thanks! We do have a synchronous real-time API and latency is one of the bigger issues that we're trying to improve on now. At the moment, you can expect speeds that are 10x faster than realtime.
archon810
Just FYI, your page keeps jumping on mobile as it renders and erases words. Not a good experience if I'm trying to read.
underdeserver
Consider investing in a good microphone for recording. A Blue Yeti is ~$200.
abinaya_rl
You are trying to do a beautiful thing. I don't have a knowledge of this subject, but I really wish you good luck on this project.
mbreese
You may want to look up what was done for Roger Ebert. He has lost his voice due to surgery, but because of the vast corpus of audio recordings of him, a viable text to speech engine was able to be created.

It’s a bit dated at this point, but I imagine the research has vastly improved since then.

It’s a very good question though. A decade ago this was able to be done for one man. Is it now possible to be done for anyone? Like others, I’d guess the first step is to record everything while you can.

eschaton2023
If she has time get here to read the most common english words. Then parse the text and play the audio for the known words and use traditional speech synthesis for the outliers. It will not be perfect but you can then possibly train an AI to pronounce the outliers.
jasonhn9999
When my dad lost his speech, we had Boogie Board Jot devices all over the house. It made writing short notes and simple dialogs much less tedious.

We also used the Verbally premium iPad app to help give him a voice and make transactions on easier.

Wishing you all the best.

seesawtron
Here's a recent work [0] where you can train the model with 10s audio and convert any "text to speech" (all doable in the browser). I tried with Google Colab demo [1] and its performance fluctuates with the training audio sample that you give it so might need some trial and error to get the sweet spot.

Also the model is not saved in the browser with Colab so you might also want to do it locally to save it eventualy (if it comes to that).

All the best mate!

[0] Main repo: https://github.com/CorentinJ/Real-Time-Voice-Cloning [1] Google colab repo to try it out: https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/ma...

josinalvo
IDK about the tech, but I would not worry about it right now. You dont need to play with the tech unless the bad unlikely outcome comes to pass.

The only tip I have is from a bit of amateur sound editing I did: collect many samples, and beware of big phrases: Like, ask her to say the same thing many times. And ... sometimes ... to ... stop ... at ... each ... word. And ... so ... me ... ti ... mes at each syllable.

Otherwise, if you ever need to create a sample that contains a single word/syllable, you cant. It is weird how much sound that contains clearly distinguishable syllables for the human ears still is not separable when you go to edit it.

Also, you might want to check wordlists by frequency to get a menu of common words, and ipa notation, to ensure you cover a good range of sounds

JDEW
> Otherwise, if you ever need to create a sample that contains a single word/syllable, you cant. It is weird how much sound that contains clearly distinguishable syllables for the human ears still is not separable when you go to edit it.

Don’t know why you’re being downvoted. Thought it was insightful.

thaumasiotes
Some (decades old) research on this involved a research team creating a video of JFK saying "I never met Forrest Gump". I found a writeup in Google Books: https://books.google.com/books?id=mQtGVQeQplcC&pg=PA208&lpg=...

> We evaluated our Kennedy results qualitatively along the following dimensions: ... naturalness of the composited articulation; ...

Obviously the state of the art will have advanced, but maybe this can point the way toward more current research.

While I tend to agree with everyone else that this can be a great idea, my instinct is to float the idea to your wife first and see how she responds. I can imagine someone taking this negatively.

[deleted]
(comment deleted)
audiohermit
Hey, speech ML researcher here. Make sure you have different recordings of different contexts. fifteen.ai's best TTS voices use ~90 min of utterances, some separated by emotion. If you're having her read a text, make sure it's engaging--we do a lot of unconscious voicing when reading aloud. Tbh, if she has a non-Anglophone accent, you're going to need more because the training data is biased towards UK/US speakers.

If you want to read up on the basics, check out the SV2TTS paper: https://arxiv.org/pdf/1806.04558.pdf Basically you use a speaker encoding to condition the TTS output. This paper/idea is used all over, even for speech-to-speech translation, with small changes.

There's a few open-source version implementations but mostly outdated--the better ones are either private for business or privacy reasons.

There's a lot of work on non-parallel transfer learning (aka subjects are saying different things) so TTS has progressed rapidly and most public implementations lag a bit behind the research. If you're willing to grok speech processing, I'd start with NeMo for overall simplicity--don't get distracted by Kaldi.

Edit: Important note! Utterances are usually clipped of silence before/after so take that into account when analyzing corpus lengths. The quality of each utterance is much much more important than the length--fifteen.ai's TTS is so good primarily because they got fans of each character to collect the data.

grogenaut
I came here to say this. My brother has a PhD in chemistry and no coding experience. He was able to create a voice model of himself using basic nvidia example generators in a week. My dad lost his voice and it would have been very nice to have a TTS that was much more close to him. I personally would think it would be worth it to have that database.

But obviously also attend to the human matters as well, eg spend time.

tombert
This sounds pretty cool (your brother making the voice model, not your dad losing the voice)...do you have a link to this example? I would love to play with this.
audiohermit
I work in pathological speech processing/synthesis so I'm unfortunately familiar with your father's position. It really sucks that these people didn't know that archiving their voice would've been useful. I hear snippets that people manage to glean from family videos right after listening to their current voices and it makes me really sad.

On the upside, your father can choose any celebrity he wants to voice him! Tons of celeb data is publicly available (VoxCeleb 1 & 2).

vervez
Is Morgan Freeman the most used celebrity?
[deleted]
(comment deleted)
Deinos
A ranking of used voices would be fascinating. Especially broken down by user statistics.
core-questions
I'd go for Stephen Hawking, myself.

(Not using his voice synth, reconstructed using ML, because it should sound more natural that way ;-)

shagie
I recall that the "say" program on the SGI from the mid 90's was approximately Hawking's voice. Hawking gave his speech for the Whitehouse Millennium Lecture at SGI also, and while I wasn't able to attend I found the transcript of it and fed it in there... there were some jokes that he had that only really came through with the intonation and pacing of a voice synth -- its the ultimate dead pan voice.

https://clinton.presidentiallibraries.us/items/show/16112 https://youtu.be/orPUQm1ZRSI

And his voice was his - even with the American accent.

https://www.news.com.au/technology/innovation/why-stephen-ha...

> “It is the best I have heard, although it gives me an accent that has been described variously as Scandinavian, American or Scottish.”

> ...

> “It has become my trademark and I wouldn’t change it for a more natural voice with a British accent.

> “I am told that children who need a computer voice want one like mine.”

Somewhere, I recall a NOVA(?) program from the mid 80s where it showed him using the speech synthesizer and the thing that he said with it that still sticks in my mind is the "please excuse my American accent". In later years he was given the opportunity to upgrade it to a more natural sounding voice - but that voice was his.

hapless
Hawking’s original voice synth was the default sound of the DECtalk hardware speech synth.

It would not surprise me if SGI’s software implementation were similar to the he most popular hardware of the 1980s.

clan
Are there any simple howtos anywhere which describes the process in as simple terms as possible? Without knowing the cool toolkits du jour.

Something like: - Download these texts - Record in WAV at least 48 kHz - Record each line in a separate file. - Do 3 takes of each line: flat, happy, despair

Maybe even a minimal set and a full set depending on how much effort you are willing to put in.

A plain description on how to capture a raw base which within reason and technology could be used as a baseline for the most common toolkits.

I have myself looked into this (for fun) but I felt I needed a very good understanding of the toolkits before even starting to feed in data. And for my admittedly unimportant use it seemed a huge investment to create a corpus I was not even confident would work. I ended up taking the low road and used an existing voice.

grogenaut
Unfortunately dad passed 21 years ago. But the options now are much better. Just projecting my past experiences on the obvious Delta.
dheera
Which generator works the best, qualitatively? I come from a vision/ML background but haven't played with speech at all, so it's completely new to me, and wondering what the state of the art is.

I've been wanting to create a TTS of myself so I can take phone calls using headphones and type back what I want to say so that I don't have to yell private information out loud in public locations. Would be nice if during non-COVID times I could sit in a train seat and take phone calls completely silently.

audiohermit
Much of the work in speech synthesis has been about closing the gap in vocoders, which take a generated spectrogram and output a waveform. There's a clear gap between practical online implementations and computational behemoths like WaveNet. As you implied it's hard to quantitatively judge which result is better, papers usually use surveys to judge.

Here's a recent work that has a good comparison of some vocoders: https://wavenode-example.github.io/

Edit: WaveRNN struck a good balance for me in the past but is not shown in the link. Tons of new work coming out though!

sdenton4
WaveRNN (and even slimmer versions, like LPCNet) are great, and run for a tiny fraction of the compute of the original WaveNet. Pruning is also a good way to reduce model sizes.

I'm not sure what's up with the WaveGLOW (17.1M) example in the linked wavenode comparison... The base WaveGLOW sounds reasonable, though. They're also using all female voices, which strikes me as dodgy; lower male voice pitch tracking is often harder to get right, and a bunch of comparisons without getting into harder cases or failure modes makes it seem like they're covering something up.

(I've run into a bunch of comparisons for papers in the past where they clearly just did a bad job of implementing the prior art. There should be a special circle of hell...)

audiohermit
Agreed. I didn't have a better comparison at hand.

I'm looking at you GAN papers.

oh_sigh
Make sure to get recording of true honest laughter of hers too
kevmoo1
Came here to post this. Glad someone else thought of this first!
ta1548177231
Depending on your desired level of hedging...

I would say also consider recording a variety of honest utterances of all kinds, situations, and emotions. Anger outbursts, apathetic grunts, sexual even if you so desire (hence throwaway account)... Please dont be offended by this, just thinking of all scenarios for you to decide for yourself...

tech4all
Thanks so much for the resources and the well thought out reply!
Veedrac
Also buy a (half) decent mic! They're much cheaper than you might expect.
core-questions
Seconding this, it's worth a hundred or so for at least a Yeti or something, considering it's not like you'll get another chance to do this.
gknoy
Also, in a similar vein as testing backups, make sure to test + listen to the recorded audio, if you can.
lunixbochs
I have an open source web service for rapidly recording lots of text prompts to flac: https://speech.talonvoice.com (right now the live site prompts for single words because I’m trying to build single word training data, but the prompts can be any length)

You can set it up yourself with a bit of Python knowledge from this branch: https://github.com/talonvoice/noise/tree/speech-dataset

There are keyboard shortcuts - up/down/space to move through the list and record quickly.

If you want to use it on arbitrary text prompts, you can modify this function to return each line from a text file: https://github.com/talonvoice/noise/blob/speech-dataset/serv...

If you use this, before recording too much, do some test recordings and make sure they sound ok. Web audio can be unreliable in some browsers.

The uploaded files are named after the short name, so make sure you can correspond the short name with the original text prompts, eg with string_to_shortname().

If you aren’t easily able to do this yourself, I’d be happy to spin up an instance of it for you with text prompts of your choosing.

exikyut
Somewhat OT question: after taking a quick look at this I stumbled on the eye-tracking video you made using pops to click... and I'm curious, can eye trackers not detect and report blinking?

Also, I noted the VLC demo says it doesn't use DNS! That's awesome...

lunixbochs
I can detect blinking, yes. However your eyelids have very small muscles that are not meant to be consciously controlled all day and I don’t recommend straining them. Your eyelids twitching from muscle strain is rather uncomfortable (from experience)

The VLC demo was using macos speech recognition. In the beta now I’m shipping my own engine+trained models based on Facebook’s wav2letter, which is going pretty well.

daanzu
I wrote a simple little Python GUI app to record training audio. Given a text file containing prompts, it will choose a random selection and ordering of them, display them to be dictated by the user, and record the dictation audio and metadata to a .wav file and recorder.tsv file respectively. You can select a previous recording to play it back, delete it, and/or re-record it. It comes with a few selections of sentences designed to cover a broad diverse range of English (Arctic, TIMIT). Pretty simple and no-nonsense.

https://github.com/daanzu/speech-training-recorder

Originally intended for recording data for training speech recognition models [0], it should work just as well for recording to be used for speech synthesis.

[0] https://github.com/daanzu/kaldi-active-grammar

lunixbochs
Did you figure out why half of Shervin’s audio was empty? I would hesitate to recommend this if there’s still a chance half of the data isn’t usable after recording.
ChrisGammell
I have about 500 hours of high quality, channel isolated (separate from the person I was speaking to) audio. It comes from my podcast that I have done for many years. It's probably closer to 75-100 hours audio of me actually speaking, since I am more the interviewer.

Is that something that would be useful to a researcher in any context? I am intrigued by the idea of having my voice preserved (you know, ego), but also am happy to donate the sound files if they would help researchers in any way for datasets.

If so: chris@theamphour.com

lunixbochs
Do you have transcripts, even just for some of the episodes? Unsupervised learning is possible but more difficult.

In general, yes, this is probably useful data in some way for speech recognition or TTS.

pkinnaird
get a great microphone and have her read her favorite books. Go for books with lots of dialog and emotional content.

Later, you can extract all the phonemes you want from it and you will retain the emotional expressiveness of her voice.

She should probably sing some songs -- lullabies, rock, etc. Go for emotional diversity.

mproud
Roger Ebert has some articles about his troubles he encountered that may be worth a read.
abjecton
Your approach towards the situation might determine the life quality of you and your wife. I can't imagine how it's like to think in a logic way while you're in the middle of such of an emotional event.
bigmasterofnone
Good luck with what you are doing and more importantly, I wish your wife good health.