Mobile support is harder since Krisp won't be able to register as a virtual mic/speaker on Android or iOS.
At least we haven't found any way around it.
How about a VST plugin? that would give you cross-platform support (Mac/Windows/Linux/iOS) and would allow it to be used as a part of many, many audio processing pipelines - from simple streamers/youtubers all the way to pros
What does it actually filter on? Is it trained to recognize my voice or just any human voice? I can see you can smartly filter out noisy noise, but can it also filter out nearby conversations like in an office environment?
Yeah, and I have to de-noise a conference I recorded a few days ago before uploading it. A Linux version of such a tool would be of great help, although I will rather try standard approaches first.
I had been wondering when a product like this would come along. Being able to remove background noise from calls sounds pretty great, but it's not a problem I run into too often - most people on my calls are in a meeting room or a quiet place.
But, if it can really fill in missing or distorted chunks then for me that's a killer feature. I'm going to give it a try on my morning check-in call, which is coming up.
I'd love to see a product like this be able to adapt to the voices of individual speakers, and fill in gaps or distortion in their natural voices. I presume that would be more difficult, though, because you'd need a model of each speaker at the output device, so both parties would need to be running Krisp, and you'd somehow have to share the model between them - which if you already have network issues causing dropouts and distoration might not be feasible. Unless it was a side-channel thing, where for regular calls with the same people the voice model for is updated after each call, ready for the next one.
Though I'm not sure I love the idea of a model of my voice being constructed and transmitted around. But I think it would be really cool :)
This call typically has decent audio quality, and as it's a stand-up / status call people tend to speak one at a time. No-one today had much in the way of background noise, so I'm not sure there was a lot of opportunity for the app to show off.
That said, I switched the Krisp speaker mode on and off repeatedly while each person who was talking.
1. When just one person was talking Krisp didn't seem to ever make the sound worse, which I guess is a start.
2. When there was a conversation going back and forth, or when one person started talking right after someone else finished, the voices got mangled or muted and I couldn't understand them. I had to turn Krisp off.
3. When there was some background echo (like in a large room) or some minor distortion I thought it maybe sounded a bit clearer with Krisp running, but it didn't really make much difference. I could understand the speaker either way.
With the audio problems I mentioned at (2) and no real gain from using Krisp I doubt I would use it regularly, though if I run into a call with bad background noise I might try it again.
I also tried the Krisp microphone, and at one point I had to repeat myself, which doesn't usually happen. But I have no way of knowing whether that was due to me speaking unclearly, or audio issues at the listeners end, or something else. So I don't really have an opinion about the microphone, but as I am in a quiet place anyway I wouldn't probably use it.
It would be nice if there was a single channel evaluation mode for the speaker. If I could hear in my right ear the normal audio was, and in the left what the Krisp-processed sound, then I would have a better chance of evaluating the performance. I guess if you have a lot of continuous background noise that mode would be redundant, and it should be an obvious improvement switching back and forth.
Such feedback from users is priceless. Thanks for it. Krisp is still in beta and we keep experimenting with different DNN models. For example we have a DNN model which maximally preserves multiple voices while removing noise. It's not shipped yet.
Re #2, one thing we noticed is that Conferencing apps themselves will distort the voice when multiple voices are overlapping. Especially when there is also noise. There is not much Krisp can do here since the stream it receives is already distorted.
Unfortunately for krisp speaker we don't control the audio stream. Imagine how many times the stream gets signal processed before krisp speaker gets it (noise cancellation in the headset, noise cancellation in RTC, codec, etc).
Re Krisp microphone, the DNN model used here is more effective since what Krisp receives is "less processed/distorted stream".
Please stay tuned, our release cycle is around 2 weeks. More quality and UX features are under way.
>When there was a conversation going back and forth, or when one person started talking right after someone else finished, the voices got mangled or muted and I couldn't understand them. I had to turn Krisp off.<
This may have nothing to do with Krisp as an app. All phones/communications device have the ability for either half or full duplex audio. Half duplex means only one caller can be heard. If someone were to speak, the audio would cut out as you experienced. Full duplex allows for conversations to happen all at once.
So this all could just be a symptom of the feature of the device or service you're using. I would try your test on different devices and/or services.
> most people on my calls are in a meeting room or a quiet place
Since I moved to NYC, I think I haven't had one phone call without some obnoxiously loud siren/horn/baby/dog/jack hammer/piano/bros in the background. People must think I listen to the NYC soundtrack on repeat at full volume and make phone calls.
Not really. The tech used in noise-cancelling headphones is called ANC (active noise cancellation). It eliminates the noise coming to your ear from your surrounding environment.
In contrast, Krisp eliminates the noise going from your environment to the call participants and vice versa.
As an amateur radio operator, I struggle very often with noise-related issues. Nice tech. I'm used to see digital processing applying in this field, but never thought about applying machine learning. Feeling curious about learning about the technology some day, because, for me, based on the solutions I currently know, this seems, indeed, magic.
I'd be a little worried about the feature that fills in missing voice chunks. It sounds ripe for accidentally replacing one lost word with a completely different word that could also make sense in context. Almost like the issue where Xerox copiers would sometimes replace one character with another. Hopefully the filling in of missing chunks is done in a way that doesn't allow it to fill in whole words, but rather just short sub-syllable chunks of audio?
It sounds ripe for accidentally replacing one lost word with a completely different word that could also make sense in context.
For awhile, it seemed like the autocorrection in iOS was deliberately trying to break up me and my then-girlfriend. It even seemed to have a penchant for doing an unfortunate autocorrect just a sliver of a second before my finger hit send on a txt. I finally turned the feature off.
All audio is processed on device, but is the goal to use the public training/learning to tailor a more robust model which they can sell commercially/integrate into apps/phones/etc?
My suspicion is that the founders/engineers will be acqui-hired for a not-insignificant sum by one of the existing big players in the teleconference space (Zoom, Google, Cisco, etc.).
Further, this is probably their goal. In fact, I wouldn't be surprised if Apple bought something like this as system-wide background noise canceling would be a fantastic OS feature.
iPhones already have this feature if you hold your phone up to your ear. There is a microphone in the back of the phone near the camera that phase cancels out background noise from the input from the front mic.
They need this for the AirPods then. Everyone I call while outside in NYC says the microphone is super sensitive and picks up all the surrounding noise.
they also recently added 4 (or 5?) mics on the new iPad Pro so that it could do facetime calls while cancelling out the feedback.
Wouldn’t be surprised if a software update makes it possible to use all of those mics for some very good ambient noise cancellation during face time calls
Just tried the Skype call testing service with and without Krisp while mashing on my keyboard. It did seem to almost eliminate the typing sounds which were rather loud otherwise.
I just tried this over a Zoom call. It worked well but not as well as the samples on their website would make you believe. Background noise was muted during silence periods but not always during speech from the active speaker. This makes it even more annoying because you hear tidbits of background noise only at the same time that the active speaker is speaking. I'm still very impressed though.
Is there a way to run an existing audio file through this to clean it up? For example, if I have a video of a presentation with lots of background noise like chairs creaking, people coughing, etc.?
Sure you can. I don't think anybody can stop you doing that. You just need 2 computers.
On one you play the clip.
On the other you record it.
The remove the audio from the video and put the new audio onto it.
Yes, of course I can run my audio file through it by actually playing it and recording the result, but I'm asking if there's a version of this tool that takes an input file and writes to an output file on its own.
Cool! I thought I had a test recording for this, but (un?)fortunately, my workplace sprung for a fancy directional mic that already did a very good job of isolating the speaker's voice.
I still plan to play with the app for voice chat, though.
Yes, that's how phone microphones for example work.
To remove static background noice would just need to get a silent sample and reverse out the polarity of the static noise from the audio signal. It can be done real-time or in post production.
To sum out random background sounds you would need an omnidirectional mic that picks up everything. Then you'd need a close range shotgun type mic from which you subtract all that noise mix picks out.
What I am thinking of is more is using signal phase differences on spatially separated similar microphones to isolate the desired source and filter the rest. There is probably a name for this I wouldn't remember. You wouldn't need a specific directional mic and could instead build a synthetic virtual microphone with whatever directional properties you wanted (including distance and not just directional selection)
My first internship was at a company that was doing this over a decade ago, but before machine learning had proliferated to adjacent fields. It's easy to separate out stationary signals like ambient noise, which are easily isolated in the frequency domain. But it's a totally different thing to remove something like a baby's cry, shuffling of papers, or other sharp transients. Blindly separating without a large corpus of inputs + machine learning techniques seems impossible.
I remember as intern spending hours in front of spectrograms manually deleting noise so that the researchers could get clean targets. Let me tell you, I started being able to identify a lot of phonemes just by visually examining waveforms.
Eventually, the company did pursue some noise cancellation, but only as one part of their offering. I don't think they ever could get the holy grail of separating non-stationary noise.
There is a new paper out from Google research that specifically addresses the use case with screaming children in the background - very exciting!
https://www.youtube.com/watch?v=zL6ltnSKf9k
108 comments
[ 5.0 ms ] story [ 170 ms ] threadEdit: I know it's a typo, but I figured it's a door to somewhat interesting line of linguistic thought.
You could have "seemful" which might be somewhat synonymous with "inauthentic" and "seemless" which might be in with "genuine"?
Most businesses, especially outside of Silicon Valley, don't use Mac; Windows will likely be a larger market.
Windows support will come in Dec. We are working hard on it.
But, if it can really fill in missing or distorted chunks then for me that's a killer feature. I'm going to give it a try on my morning check-in call, which is coming up.
I'd love to see a product like this be able to adapt to the voices of individual speakers, and fill in gaps or distortion in their natural voices. I presume that would be more difficult, though, because you'd need a model of each speaker at the output device, so both parties would need to be running Krisp, and you'd somehow have to share the model between them - which if you already have network issues causing dropouts and distoration might not be feasible. Unless it was a side-channel thing, where for regular calls with the same people the voice model for is updated after each call, ready for the next one.
Though I'm not sure I love the idea of a model of my voice being constructed and transmitted around. But I think it would be really cool :)
This call typically has decent audio quality, and as it's a stand-up / status call people tend to speak one at a time. No-one today had much in the way of background noise, so I'm not sure there was a lot of opportunity for the app to show off.
That said, I switched the Krisp speaker mode on and off repeatedly while each person who was talking.
1. When just one person was talking Krisp didn't seem to ever make the sound worse, which I guess is a start.
2. When there was a conversation going back and forth, or when one person started talking right after someone else finished, the voices got mangled or muted and I couldn't understand them. I had to turn Krisp off.
3. When there was some background echo (like in a large room) or some minor distortion I thought it maybe sounded a bit clearer with Krisp running, but it didn't really make much difference. I could understand the speaker either way.
With the audio problems I mentioned at (2) and no real gain from using Krisp I doubt I would use it regularly, though if I run into a call with bad background noise I might try it again.
I also tried the Krisp microphone, and at one point I had to repeat myself, which doesn't usually happen. But I have no way of knowing whether that was due to me speaking unclearly, or audio issues at the listeners end, or something else. So I don't really have an opinion about the microphone, but as I am in a quiet place anyway I wouldn't probably use it.
It would be nice if there was a single channel evaluation mode for the speaker. If I could hear in my right ear the normal audio was, and in the left what the Krisp-processed sound, then I would have a better chance of evaluating the performance. I guess if you have a lot of continuous background noise that mode would be redundant, and it should be an obvious improvement switching back and forth.
Re #2, one thing we noticed is that Conferencing apps themselves will distort the voice when multiple voices are overlapping. Especially when there is also noise. There is not much Krisp can do here since the stream it receives is already distorted. Unfortunately for krisp speaker we don't control the audio stream. Imagine how many times the stream gets signal processed before krisp speaker gets it (noise cancellation in the headset, noise cancellation in RTC, codec, etc).
Re Krisp microphone, the DNN model used here is more effective since what Krisp receives is "less processed/distorted stream".
Please stay tuned, our release cycle is around 2 weeks. More quality and UX features are under way.
And nowhere to "leave my email".
This may have nothing to do with Krisp as an app. All phones/communications device have the ability for either half or full duplex audio. Half duplex means only one caller can be heard. If someone were to speak, the audio would cut out as you experienced. Full duplex allows for conversations to happen all at once.
So this all could just be a symptom of the feature of the device or service you're using. I would try your test on different devices and/or services.
Since I moved to NYC, I think I haven't had one phone call without some obnoxiously loud siren/horn/baby/dog/jack hammer/piano/bros in the background. People must think I listen to the NYC soundtrack on repeat at full volume and make phone calls.
I envy you. :)
In contrast, Krisp eliminates the noise going from your environment to the call participants and vice versa.
Similar to that project to colorize old photos with a GAN, but in audio form. [1]
[1] https://news.ycombinator.com/item?id=18456527
https://devblogs.nvidia.com/nvidia-real-time-noise-suppressi...
For awhile, it seemed like the autocorrection in iOS was deliberately trying to break up me and my then-girlfriend. It even seemed to have a penchant for doing an unfortunate autocorrect just a sliver of a second before my finger hit send on a txt. I finally turned the feature off.
All audio is processed on device, but is the goal to use the public training/learning to tailor a more robust model which they can sell commercially/integrate into apps/phones/etc?
Wouldn’t be surprised if a software update makes it possible to use all of those mics for some very good ambient noise cancellation during face time calls
https://2hz.ai/api/index.html
"It's only free during beta. Although we haven't decided on the exact monetization strategy yet."
We are also building web and iOS apps to cancel the noise in the files.
I still plan to play with the app for voice chat, though.
You have to apply for it and it's apparently priced per minute. I applied and I'm waiting for pricing info.
To remove static background noice would just need to get a silent sample and reverse out the polarity of the static noise from the audio signal. It can be done real-time or in post production.
To sum out random background sounds you would need an omnidirectional mic that picks up everything. Then you'd need a close range shotgun type mic from which you subtract all that noise mix picks out.
I remember as intern spending hours in front of spectrograms manually deleting noise so that the researchers could get clean targets. Let me tell you, I started being able to identify a lot of phonemes just by visually examining waveforms.
Eventually, the company did pursue some noise cancellation, but only as one part of their offering. I don't think they ever could get the holy grail of separating non-stationary noise.
This was a sensitive use case for the team since I (CEO/krisp) had a baby 3 days ago and really needed Krisp to do my calls :)