Whatever happened to Voco, their 'photoshop for voice'?
I'm not a frequent HN poster, so I don't know how to 'properly' cite the presentation, but I remember them showing it off years ago and... nothing came of it.
To find old things like that, use the search box at the bottom of the page: https://hn.algolia.com/?q=adobe+voco. Big splash 6 years ago, then nothing.
It seems Adobe never advanced Voco past the research project stage ([1] via [2]). I'm guessing they had trouble getting it to work reliably on a wide-enough range of real-world audio.
I imagine it got shut down, either by their own executives/ethics department, or by outside pressure. It was announced in the same year that the word "fake news" took off. Not the best time to get the world excited for a "photoshop for voice"
AMD and NVIDIA have already done that. There's also Krisp which you can pay to use anywhere or just use the free Discord integration. I'm sure there are others too.
I tried this on a couple of speeches that were recorded in front of an artificial waterfall, and the output is not just bad - it’s not the English language. Nor any language on earth that I’m aware of. Haha
The tool that now comes with the latest updates to FCPX handled them without a problem. (Still some background noise, but you can clearly hear every word.) I think Adobe has a long way to go on this.
Interesting "automagic" tool for audio post targeted towards hobbyists and creators. Anyone able to compare this with the entry level[1] version of the professional standard dialog cleanup software, RX?
To push it to the limit I recorded exactly the same recording with my phone microphone and my AT875R XLR shotgun. I did this because my phone microphone is poor and picks up a lot of echo. Results are as follows:
- If the microphone quality itself is bad, the enhanced audio is still pretty horrible.
- It does clean up echo but with there's some pretty aggressive EQ that doesn't sound nice, and the noise gate is pretty severe
- Compared to my XLR shotgun, the quality of the phone was pretty horrible
What we can conclude is that if you already have a good recording but with some problems, you might be able to use this to remove those problems. However, don't expect a crappy microphone to turn into a good microphone, or a crappy recording to turn into "studio quality".
The bottom line is that there's no substitute for a decent microphone in a decent space. (At the very minimum, small room without echo.)
OP should clean both microphones on their phone, usually a sewing needle or thin toothpick can do the trick, but 99% Isopropyl on a toothbrush might be needed afterwards if the grill inside that protects the microphone is also clogged up.
Both microphones need to be cleaned of any blockages so the hardware echo and noise cancellation on a given phone works well. Otherwise you've got distorted audio getting processed as if it's not distorted...
Important question because a microphone array (which exists on some phones, and things like home voice assistants) can be steered into the equivalent of a shotgun mic's pattern, or even more focused than that. It's just that an algorithm aims it toward the strongest signal, instead of the user aiming a hypercardioid mic manually. Either way, this is what reduces the ratio of reverberant room sound ("echo").
Look at all the money that's been going into making phones compelling substitutes for professional cameras. And camera tech is still evolving. Consumer audio devices will get the same attention and investment.
We'll eventually have models for audio signals in all sorts of distorted and noisy environments. I'd bet that in ten years a cellphone microphone can duplicate a professional audio setup in 90% of circumstances.
Sound takes lesser time to travel in smaller rooms, hence the difference in time is not very large between the original sound wave and the reflected one which makes it harder to distinguish to human ears.
>In theory, would a gigantic room that was miles to the nearest wall be even better?
Yes, as effectively it's open space. In practice though, to record in high quality you would rather build an anechoic environment, as small as possible (preferably a booth).
Small spaces bring hard surfaces closer to the mic, making flutter echoes and room mode resonances louder, create murky-sounding bass pooling, and can be overly-sensitive to mic position within the space; you can sound oddly different without warning.
You need to absorb the sound of your voice so there is less echo by baffling material on the back of the mic, and absorb room tone and echoes in the area the directional microphone is pointed, generally behind your head. Small spaces have only disadvantages as studios.
To take advantage of a reach-in closet full of clothes, put some pillows on the shelf over the clothes, take the closet doors off, and back into the closet as much as you can. In this way the microphone is primarily listening to the baffled sound inside the closet, and you can avoid bass pooling by speaking into the room—ideally with baffling material (e.g. see http://PillowFortStudios.com/ ) ON the back of a LDC microphone.
The demo is pretty disappointing in that it hitches whenever you flip the switch. They should have invested a little more in making it seamlessly switch.
I tested a few recordings of an Indian Swami giving speeches in English back in the 70's. Recording had a lot of background noise, not great. You have to listen very carefully to hear what is being said. I was hoping for good results, but...
Results:
- background noise was reduced
- some previously clear words are turned into garbled non-words
- some parts are replaced by a different Indian voice, I assume AI, so it sounds like multiple people talking
All in all, the results are not anywhere near what the sample shows.
In time, these tools will gain control knobs and eventually start to focus on longer tail audio recovery tasks. I have hope for our old audio. Where there's signal, there's a way.
Honestly, it sounds like you're judging on a pretty big outlier example. The sample seems to more be aimed at background noise and even that sample is extremely easy to understand without the enhancement. There are a bunch of tools out there that are probably better aimed at your goals.
This is a 20 yr old vid shot at Pearl Harbor on the deck of USS Missouri with a low-end consumer Hi-8 camera using built-in mics. Notice flags and clothes rippling in the wind. This filter works pretty well in this case. Maybe it was trained on a New York accent? Mixing in the enhanced track as needed works well. Using only the enhanced track may sound artificial at times.
Here's another vid shot at a brewery where this filter helped clarify the brewmaster's voice over a noisy restaurant and outdoor machinery (again, built-in mic):
> Mixing in the enhanced track as needed works well. Using only the enhanced track may sound artificial at times.
Funnily, in order to use this sort of model for tasks involving speech recognition it's often recommended in the literature to mix back in some of the original noisy audio. This reduces the impact of artifacts introduced by the enhancement which would otherwise reduce ASR quality due to domain shift in the data.
Guess humans and computers have similar needs in this case. :)
These are impressive results, the audio mostly sounds like you gave the guy a lapel mic. :P
Having unlisted version on Youtube “before” the audio was filtered would make it easier to hear the true difference, but agree the quality is impressive as is.
Thanks! Honestly I liked the post filtering audio with no “natural audio” mixed back in. Get the reasoning, but still. Did you do testing with anyone to gauge viewer preference? If so, how?
This filter is not perfect so you'll have a few audio artifacts that you won't want in your final mix. Mixing with the original audio hides these artifacts somewhat. Using this filter also increases your audio mixing effort (you're mixing in a second vocal track) so it's easiest to mix the 2 audio tracks at constant levels then adjust when you hit an artifact. This is my personal preference on this style of audio mixing, and others who watched (and gave feedback) couldn't tell it was enhanced and had no difficulty understanding the speaker.
Tried to upload my favorite recorded audio file (MP3) from Japanese program given in the middle of 1980s about a ancient mound (grave) in Japan. A Japanese famous archaeologist, late Koichi Mori (a professor of Doshisya University at that time) talks about Hashihaka mound, but he talks in ... in Spanish?
> Tried to upload my favorite recorded audio file (MP3) from Japanese program given in the middle of 1980s about a ancient mound (grave) in Japan. A Japanese famous archaeologist, late Koichi Mori (a professor of Doshisya University at that time) talks about Hashihaka mound, but he talks in ... in Spanish?
I'm not sure I understand what you're saying. Do you mean that the talk was in Japanese but the output of this service somehow screwed it up in a way that it sounds like spanish?
Or are you just mentioning that the thing you uploaded was spanish and not describing the quality of the output?
I've been using this as one of several tools (noise gate in Audacity, The Levelator) to increase the audio quality in my podcast. Subjectively, I think it's been working, and I love the simplicity of the interface. No tuning, just trust the AI. It works well for standard spoken English, but will do some horrible things to music (it won't detect music and be like "hey don't do anything to this"). So you shouldn't run it on a file that includes both music and voice.
This is incredible. I took a talk recording I made for NeurIPS and passed it through this tool. The improvement in audio quality was night and day. It went from clearly being recorded in a bedroom to a studio-like experience.
The result is nowhere near studio quality. It's highly compressed and full of artifacts. I found the original to be more pleasant to listen to. It at least sounds natural.
I use Adobe Audition for podcast editing and they have a pretty nice feature built in for that already - not sure how this differs. I am also not a big fan of edit-by-transcription, since I do like to remove occasional long silences, weird sounds, etc. That being said the mic check looks pretty useful.
I seem to recall a demo where a trumpet sound was plugged into an aggressive high quality speech denoiser, making it sound like a speaker. Does anyone remember this demo, or have similar links on creative use of this tech?
I experienced something absolutely bizarre with this, making me want to try and reupload to see if the same thing occurs. I had some footage laying around that was taken on a windy day, with buses and wind and kids screaming, and for the most part it was greatly improved.
However, a few seconds into my recording there is a part where there is someone else's dialogue for a few seconds. I can't make out what they're saying but it definitely sounds like a man, with a Latin-American accent, speaking English for a second.
Could that be a hallucination or somehow they mixed audio from another recording? It only last for about a second, but it's so strange.
My guess would be it fixates on the most dominant source available and mutes the other factors. It probably favors human voices over other ambient noise, therefore singeing the man out.
It will really get freaky when there an ambient noise resembling a human voice. I'm thinking the Bear scene from the movie Annihilation.
I tried reuploading and again the exact same thing happened, which is interesting because it seems that it's producing audio fairly deterministically, which is not how I think of most AI produced results are but I'm not an expert.
a lot of AI produced results can be deterministic if you want them to be. For stable diffusion, just set the seed of the initial noise, and it's deterministic. With GPT-3, set the temperature to 0 (always choose the highest probability word).
It can be a useful technique for learning how slightly different prompts affect things
How long until there's a way to process more than one hour at a time? After trying a sample, I cannot wait to run some old audiobooks through this. The results sound like it was made twenty years newer.
Can it remove the professor's coughing? I have recordings of some interesting lectures recorded by a professor who had covid or something, coughing after every some words.
Many other professors (as well as me, occasionally) also use to involuntarely (and often unaware of that) say something like eeeeeh when they strugle to recall the right word. Would be great if this could be removed as well.
I don’t know their roadmap, but if anything I’d point you to a tool called Descript which is great at this. You can get transcriptions, and make edits based on the text (e.g. cough, um s, etc.) Descript.app (I am not affiliated with them, just a fan)
126 comments
[ 0.17 ms ] story [ 76.9 ms ] threadIt seems Adobe never advanced Voco past the research project stage ([1] via [2]). I'm guessing they had trouble getting it to work reliably on a wide-enough range of real-world audio.
[1] https://community.adobe.com/t5/audition-discussions/beta-tes...
[2] https://en.wikipedia.org/wiki/Adobe_Voco
If you'd like to run something locally, there's also https://www.nvidia.com/en-us/geforce/guides/nvidia-rtx-voice....
The tool that now comes with the latest updates to FCPX handled them without a problem. (Still some background noise, but you can clearly hear every word.) I think Adobe has a long way to go on this.
[1] https://www.izotope.com/en/shop/rx-10-elements.html
- If the microphone quality itself is bad, the enhanced audio is still pretty horrible. - It does clean up echo but with there's some pretty aggressive EQ that doesn't sound nice, and the noise gate is pretty severe - Compared to my XLR shotgun, the quality of the phone was pretty horrible
What we can conclude is that if you already have a good recording but with some problems, you might be able to use this to remove those problems. However, don't expect a crappy microphone to turn into a good microphone, or a crappy recording to turn into "studio quality".
The bottom line is that there's no substitute for a decent microphone in a decent space. (At the very minimum, small room without echo.)
Both microphones need to be cleaned of any blockages so the hardware echo and noise cancellation on a given phone works well. Otherwise you've got distorted audio getting processed as if it's not distorted...
Look at all the money that's been going into making phones compelling substitutes for professional cameras. And camera tech is still evolving. Consumer audio devices will get the same attention and investment.
We'll eventually have models for audio signals in all sorts of distorted and noisy environments. I'd bet that in ten years a cellphone microphone can duplicate a professional audio setup in 90% of circumstances.
It's the same as with photos. If your raw material is bad, no tool on earth can make it good.
Why small room? Does that reduce echo?
In theory, would a gigantic room that was miles to the nearest wall be even better?
(and of the rain too)
Yes, as effectively it's open space. In practice though, to record in high quality you would rather build an anechoic environment, as small as possible (preferably a booth).
This is where the Startup Garage analogous cliche for musicians comes from: recorded in the closet
You need to absorb the sound of your voice so there is less echo by baffling material on the back of the mic, and absorb room tone and echoes in the area the directional microphone is pointed, generally behind your head. Small spaces have only disadvantages as studios.
To take advantage of a reach-in closet full of clothes, put some pillows on the shelf over the clothes, take the closet doors off, and back into the closet as much as you can. In this way the microphone is primarily listening to the baffled sound inside the closet, and you can avoid bass pooling by speaking into the room—ideally with baffling material (e.g. see http://PillowFortStudios.com/ ) ON the back of a LDC microphone.
[0] https://en.wikipedia.org/wiki/I_Am_Sitting_in_a_Room [1] https://www.youtube.com/watch?v=fAxHlLK3Oyk
Results:
- background noise was reduced
- some previously clear words are turned into garbled non-words
- some parts are replaced by a different Indian voice, I assume AI, so it sounds like multiple people talking
All in all, the results are not anywhere near what the sample shows.
https://www.theverge.com/2013/8/6/4594482/xerox-copiers-rand...
https://www.youtube.com/watch?v=7FeqF1-Z1g0
Bad news: While the 5% was a minor inconvenience for customers, the 1% is bad enough to end your company
Just curious how the performance differs between PCMU @ 8khz compared to Opus @ 48k or IMBE and AMBE+2 (Project 25 Public Safety audio codecs) :D
My dream would be doing audio processing in real time to clean up the audio of phone calls
[0] https://fsi-languages.yojik.eu/languages/FSI/fsi-french-basi...
https://youtu.be/1LDlOmKtfeQ?t=60
This is a 20 yr old vid shot at Pearl Harbor on the deck of USS Missouri with a low-end consumer Hi-8 camera using built-in mics. Notice flags and clothes rippling in the wind. This filter works pretty well in this case. Maybe it was trained on a New York accent? Mixing in the enhanced track as needed works well. Using only the enhanced track may sound artificial at times.
Here's another vid shot at a brewery where this filter helped clarify the brewmaster's voice over a noisy restaurant and outdoor machinery (again, built-in mic):
https://youtu.be/nANSdnYj-R0
I found this filter useful.
Funnily, in order to use this sort of model for tasks involving speech recognition it's often recommended in the literature to mix back in some of the original noisy audio. This reduces the impact of artifacts introduced by the enhancement which would otherwise reduce ASR quality due to domain shift in the data.
Guess humans and computers have similar needs in this case. :)
These are impressive results, the audio mostly sounds like you gave the guy a lapel mic. :P
https://youtu.be/M5pdHVoXQHE
I'm not sure I understand what you're saying. Do you mean that the talk was in Japanese but the output of this service somehow screwed it up in a way that it sounds like spanish?
Or are you just mentioning that the thing you uploaded was spanish and not describing the quality of the output?
[0, original] https://youtu.be/gwkCIdwHRhc
[1, enhanced] https://youtu.be/RPnUqmSyZ6Q
Also, very enjoyable & clear presentation.
This would be wilder if I were in an unconstrained environment with similar enhanced audio quality. It would seem like a voice-over.
Is there any way one could easily train it on my own voice to make sure it isolates my or any other trained voice from noisy environments?
However, a few seconds into my recording there is a part where there is someone else's dialogue for a few seconds. I can't make out what they're saying but it definitely sounds like a man, with a Latin-American accent, speaking English for a second.
Could that be a hallucination or somehow they mixed audio from another recording? It only last for about a second, but it's so strange.
It will really get freaky when there an ambient noise resembling a human voice. I'm thinking the Bear scene from the movie Annihilation.
It can be a useful technique for learning how slightly different prompts affect things
Many other professors (as well as me, occasionally) also use to involuntarely (and often unaware of that) say something like eeeeeh when they strugle to recall the right word. Would be great if this could be removed as well.