Solid work they both sound exactly the same to me. The Faux Rogan is a slightly faster speaker, so it was pretty easy to differentiate between the two.
It was easy to figure out which are the fakes, but I am very impressed and spooked at the idea of this technology improving, then getting into the wrong hands.
I'm not too spooked. I think as a society we're just going to learn to distrust recorded+reproduced media by default. Already we don't trust photos much for their potential to be a manipulation. You could call this a loss, sure - but maybe we should just never have in the first place. Photo manipulation is a very old practice.
There's already businesses that sell camera apps (or cameras? my memory is a bit dim) that save a photo along with a cryptographic hash to prove authenticity. Their customers are for example insurance companies, which require their clients to take pictures of damaged property etc. for claim filings that way.
Yup. Media will be signed cryptographically and your belief in the media will correspond to your trust in the signer. It will destroy the usefulness and damage of anonymous publication of recorded evidence. Bad for transparency, good for privacy.
We can't even get people to look for a green lock icon in their URL bar. I really doubt the kinds of people that are most vulnerable to these attacks will suddenly develop an interest in cryptography.
And if the fake confirms someone's worldview or serves their interests, confirmation bias will take over and no "fancy math" is going to change their mind.
I don't think we'll adapt "suddenly," I expect we'll have to experience some major news stories getting redacted due to deepfaked evidence before this reaction becomes mainstream, but I think it's coming. I also don't think an interest in cryptography is necessary, it's not like I expect people to read public keys directly. There will be some system that shows something along the lines of "Confirmed by $USER_NAME" under the media. You don't need an interest in video codecs to appreciate YouTube. And I have seen people reject faked evidence, such as fake screencaps of tweets, even when it backed up their worldview. We're not perfect at it, but I think we're getting better.
Do you not see the problem in societies being unable to share trusted and accurate information about reality? I mean that is 100% required for any interpersonal relationship between two people let alone a community.
I do see the problem. I'm saying that ship sailed long before the computer age. Photos have been manipulated with analog means before. Currently we're in a state of still placing undue trust.
What I'm hoping is that we'll find means to actually build trust, e.g. with signing as we discussed here. I'm not entirely confident in that, but wouldn't it be nice if we engineers had a hand in building some useful tools for society? :)
> I think as a society we're just going to learn to distrust recorded+reproduced media by default.
There have been instances of of deceit, but in general the ability to give some credence to leaked political tapes has been a positive thing for society.
With this avenue for political exposure disappearing and totalitarianism making a comeback, it is a bit worrisome how we will continue to blow the whistle on politicians when being present with a camera isn't even enough.
> Photo manipulation is a very old practice.
But not modern audio or video. We're running out of options.
> There's already businesses that sell camera apps (or cameras? my memory is a bit dim) that save a photo along with a cryptographic hash to prove authenticity
This only works if the photo is of yourself and you want to prove its authenticity. If the photo is of someone else, it's still your word against theirs regardless of what hashes you have.
It would be slightly harder. What I'm looking out for here is voice distortion and the unnatural flow of speech, and it's present in every Faux recording. It's just unnatural compared to how JR normally sounds.
I didn't do so well. I was 60% correct. The non-word noises were really good with FR, pauses and breaths did a great job of convincing me. The one thing that I picked up on quickly though, was that FR speech was a bit more slurry and fast than JR speech. Interesting...
My reliable tells for the Faux Rogan were a certain amount of noise/distortion to the audio, and the real Rogan doing more complex things with intonation/prosody. The latter requires a direct comparison, and the former must be fixable with filtering? That makes it pretty close to good enough for an original reading.
Same here. Have only watched one or two episodes of his podcast in the past, so was basing my judgement on what sounded natural in general rather than any specific recollection of what he is supposed to sound like.
I voted on each clip without listening to other clips first. I misidentified two that were real as being fake. Wonder if those might have been clips where he was reading from a script with a flatter voice than usual like another top-level comment ITT talks about.
If audio of this quality was being presented in a context where I didn’t know it might be fake, I probably would have chalked it up to issues with the recording mainly, and would have believed that the audio was real.
An interesting question (ignoring how you need Real Rogan to train Fake one for a moment) is whether the Fake Rogan could be as popular, or if the extra personality the real voice has is vital to making him a popular speaker.
I suspect that question strikes to things like gaining the tools to transpose speaker affect independent from voice characteristics etc. Imagine a remix culture where a future popular podcast voice is a mashup of different older popular speakers and things like that ...
Reminds me of Aldous Huxley's Brave New World when the characters attend a synthetic orchestra. The book goes into vivid detail about the level of craftsmanship achievable by a machine composer and orchestra, yet somehow it all seems so shallow, like the soul of it has long gone.
I definitely agree that the Faux Rogan sound bites where much flatter. The AI has a very convincing human voice but the way it speaks is not the way Joe Rogan speaks. There was very little intonation, emotion, or intensity.
What I am curious about is how much selection was involved in the fake ones. Like is it the cream of the top of the fake ones that sound the most natural or is it a typical output?
However, I call shenanigans. Some of the real clips were him reading from a script for the advertisements at the beginning of his show. Joe has a really flat/unnatural delivery when he's reading from a piece of paper.
In addition, if they were including the advertisement reading in the training material, that would definitely mess up the final model. Advertisement Joe doesn't like Joe.
I got them all correct easily. You can hear the emotion in the real voice.
Someone should test this on criminal psychopaths and see if they are significantly better or worse at it than the general pop. I would bet there will be two major populations of psychopath in solidly each category.
> I got them all correct easily. You can hear the emotion in the real voice.
Yup, it's fairly obvious there is a difference between a somewhat more robotic intonation vs someone who puts emphasis on certain parts of each word. Still, they nailed the sound/uniqueness of his voice pretty well.
Uh oh, I got nearly all of them wrong. I hope I'm not a psychopath :(
Somewhat in my defense, although I've seen a few clips with Rogan, I'm not very familiar with him. I agree with the GP's comment, I thought the real Joe Rogan reading ads sounded very staccato and robotic. Near the end I got a little better with the "OK, robotic sounding Joe is the real one."
That said, I'm an introvert and I don't enjoy social situations with people I don't know, so while perhaps not a psychopath it's plausible the my poor ability to infer emotional intonation is at play both here and in social situations.
I only missed 1 in 8, and I attribute that to a bit too much wine on a Saturday night. I heard "farm fresh" as "farm fish" and thought I was a genius for catching an algorithm error...
However, as I mentioned in this comment[1] I don't think you have much to worry about. The correct criteria for diagnosing fake vs real is actually not being robotic. I knew immediately what was real because there was emphasis and affectation, as opposed to the monotone counterpart.
Finally, a minor quib: "psychopathy" is not about an ability to infer intonation (quite the opposite). I'm an introvert like you (Hell, I'm probably considered a hermit) and I was easily able to identify the doppelganger.
> Finally, a minor quib: "psychopathy" is not about an ability to infer intonation (quite the opposite). I'm an introvert like you (Hell, I'm probably considered a hermit) and I was easily able to identify the doppelganger.
What does introversion and being a hermit have to do with psychopathy?
I don't think the parent commenter was arguing any sort of relationship between those things. You brought up the introversion/hermit thing so asking the other guy who didnt bring it up how the two things are related doesnt make much sense.
I also was thinking that one's ability to feel another's emotion--empathize--was the key determinant in success here.
I also don't want to make people feel like they're necessarily terrible people if they can't. Sorry if that's the way it came across.
Thinkint about this: It seems that culturally we are comfortable telling people feel like they're dumb (ie SATs, grades, uni admissions) or unathletic (competitive sports is a big part of growing up)... But we don't say anything about moral character.
Explained musically: the fake one goes from C to C# instantly, and the real one sweeps through all the frequencies in between in order to switch tones.
> I would bet there will be two major populations of psychopath in solidly each category.
Those psychopaths who are "bad" at determining authenticity versus those who are "good" at it (mostly-wrong as opposed to mostly-correct, respectively)?
I'm hesitant to agree with that; from my understanding, a "psychopath" (current medical definition is Antisocial Personality Disorder, ASPD) is one who is outside of social norms and mores. Much like a precise machine, they are able to attenuate their focus to the nuances most neurotypical people take for granted.
Although far from a medical professional, I would hazard to guess an autistic person would be one of the groups you might attribute to supposed psychopathy (specifically: those unable to diagnose fake from authentic). However, even that claim should be taken with an ounce of salt. Point being, I get triggered when people use psychopath willy-nilly!
Psychopaths are actually supposed to be very good at reading other peoples' emotions. They just don't feel much empathy. Autism is associated with difficulties in reading emotions.
>I actually supposed to be very good at reading other peoples' emotions. They just don't feel much empathy.
I understand people commonly intend empathy to convey 'well intentioned' or 'harboring feelings to understand and help others,' but I thought the actual meaning is the first sentence of your comment?
No, empathy is the ability to feel what others are feeling, like one piano string resonating with another. Psychopaths may have the ability to model and emulate this trait but it does not really affect them. Psychopaths are emotional impostors who operate outside of the inborn rules of social animals.
I would expect that pretty soon, we'll start seeing a lot of fake online/dating profiles with generated pictures and text. Probably a lot of chatbots posing as attractive women trying to scam people. Seems it should also be possible to condition a GAN to generate pictures that look like someone specific. These bots could try to pass themselves off as people you know.
I'm personally somewhat worried about the way modern society enables people to have less and less physical interactions. Imagine a future where it's commonplace to attend an online university, work from home, get goods, food and groceries delivered by a robot (don't even have to talk to a driver). Now, if we had chatbots that passed the Turing test, and always basically told you what you wanted to hear without ever challenging you, then what?
I'm worried we're going the way of Japan and its 'herbivore' culture. A world where people have given up on dating because it's too complicated. Other human beings are uncomfortable to interact with, so everyone retreats into their own little bubbles and gets more and more of their needs met by machines... On an individual basis, it a seductive idea to have an idealized robotic partner that does exactly what you want when you want it and doesn't have any needs of its own, will never leave you, but what will that do to humanity as a whole?
Yea. I think think the most likely scenario our civilization exists in is that we are coming up on a bunch of Great Filters, and this is just another one of them.
There's probably virtually infinitely many (just as life has infinitely many ways to kill an individual or group in pre-modern times).
Also, just as in prehistory, it would have taken visionary and moral heroes to build for their future's survival. But today, the role of those who have those concerns is not important in civilization anymore...
Deep fake videos already exist, and will fool most people for short videos. You might start to suspect something if you deep faked a feature length movie though. Combine a deep fake with Faux Rogan and we can expect 2020 to have a video of Joe Rogan taunting and then fighting a chimpanzee.
We are very quickly reaching the point where real recorded audio and video will not be distinguishable from generated ones. I'd say it will happen well before 30 years.
I cannot say what overall effect it will have just yet, but law enforcement will definitely have a harder time collecting evidence.
People keep saying this, but we've had Photoshop in the general public's hands for decades now and I can't remember even a single fake photo causing any sort of ruckus. Everyone seemed to realize that photos can be perfectly faked at about the same time photos could be perfectly faked.
Because photos aren't the same thing as feeding text to a machine that can spit out dialogue in the voice of someone else. Same thing with "deep fake".
However - I guess the answer correctly with all but one - “You are much less likely to injure yourself if you do it correctly” - which I can now hear the different (or perhaps confirmation bias).
There’s something artificial about the timing with the pauses between sentences on the ML based ones that was the main giveaway for me, also while his voice is damn near spot on - there are some vocal inflections or perhaps “emotion” in a few words that also hinted to me.
Not likely helping with the training- unless I'm underestimating the scale of people visiting the site. Also unlikely because they had to have a well trained model to launch the site and make the YouTube video that everyone has seen. It's possible that they will do fine tuning in certain areas depending on if some generated clips get consistently answered the same way.
More likely is that you're helping them write the results section of their research paper, with a sentence like this, "In a blind trial of 5,000 internet users, over 93% of people were unable to tell the difference between generated audio and real audio at a statistically significant level (P<=0.035)."
Note: if we assume a binomial distribution, you need to get 7 or 8 correct to reach the magical P<=.05 barrier. If you assume that there are a fixed 4 generated clips and 4 real clips, then you need to get them all right (https://en.wikipedia.org/wiki/Fisher%27s_exact_test). I think it's fair to use the binomial distribution because the website does not tell you the number of real and generated clips before you take the test.
It surprised me but one thing Edge is really good at is ebook support with a read-aloud feature. The text rendering is good, better than Calibre, and the cursor tracking following the voice is very satisfying.
Some of the British voices are close to perfect. If you want to listen at 2x then it's often better than a human because at speed, there is no value in dramatic intonation, instead impeccably consistent pronunciation and pace are what helps intake which would otherwise be grating at standard speed.
What is really nice is that when you start feel "beaten up" by a high pace reading, you can switch voices and brighten up the experience. I find switching back and forth between male and female voices helps avoid ear fatigue and keep up the pace. I immediately become more receptive to the content when a new voice begins.
I got 8/8. Intonation was an easy give away for me, the fake recordings had obvious tells because they didn't pause naturally or emphasized words in a weird way.
EDIT: Still impressive tho. I imagine the typical use case for this sort of tech is not just trying to fool people who have been explicitly forewarned.
How can I do this with my voice? Any suggestions? I run a site called https://sysadmincasts.com/ with tons of voiceovers and would love to automate this (without it sounding like crap). There are always places where I need to update the audio with small tweaks. I imagine it is still pretty computerized today but I think we'll eventually get there. The google WaveNet stuff is pretty good but still not there yet [1].
I imagine eventually, you could have some type of transcript that's annotated with speech synthesis markup language (SSML). Then, you have a CI/CD pipeline that would run this text-to-speech engine and regenerate the audio. I could then pair this up with the video. I honestly wonder if we are a year or two away from this being possible.
Same here. Only got the "Healthy local ingredients, farm fresh ingredients" one wrong. And I've only ever listed to this person once (the Musk interview).
118 comments
[ 2.9 ms ] story [ 92.6 ms ] threadI wonder how long it would take until Joe brings those guys to his podcast that is something I’ll be waiting for.
There's already businesses that sell camera apps (or cameras? my memory is a bit dim) that save a photo along with a cryptographic hash to prove authenticity. Their customers are for example insurance companies, which require their clients to take pictures of damaged property etc. for claim filings that way.
And if the fake confirms someone's worldview or serves their interests, confirmation bias will take over and no "fancy math" is going to change their mind.
What I'm hoping is that we'll find means to actually build trust, e.g. with signing as we discussed here. I'm not entirely confident in that, but wouldn't it be nice if we engineers had a hand in building some useful tools for society? :)
There have been instances of of deceit, but in general the ability to give some credence to leaked political tapes has been a positive thing for society.
With this avenue for political exposure disappearing and totalitarianism making a comeback, it is a bit worrisome how we will continue to blow the whistle on politicians when being present with a camera isn't even enough.
> Photo manipulation is a very old practice.
But not modern audio or video. We're running out of options.
> There's already businesses that sell camera apps (or cameras? my memory is a bit dim) that save a photo along with a cryptographic hash to prove authenticity
This only works if the photo is of yourself and you want to prove its authenticity. If the photo is of someone else, it's still your word against theirs regardless of what hashes you have.
Yeah but would you figure it out if you just listened to it expecting Joe Rogan?
I voted on each clip without listening to other clips first. I misidentified two that were real as being fake. Wonder if those might have been clips where he was reading from a script with a flatter voice than usual like another top-level comment ITT talks about.
If audio of this quality was being presented in a context where I didn’t know it might be fake, I probably would have chalked it up to issues with the recording mainly, and would have believed that the audio was real.
I suspect that question strikes to things like gaining the tools to transpose speaker affect independent from voice characteristics etc. Imagine a remix culture where a future popular podcast voice is a mashup of different older popular speakers and things like that ...
However, I call shenanigans. Some of the real clips were him reading from a script for the advertisements at the beginning of his show. Joe has a really flat/unnatural delivery when he's reading from a piece of paper.
In addition, if they were including the advertisement reading in the training material, that would definitely mess up the final model. Advertisement Joe doesn't like Joe.
Someone should test this on criminal psychopaths and see if they are significantly better or worse at it than the general pop. I would bet there will be two major populations of psychopath in solidly each category.
Yup, it's fairly obvious there is a difference between a somewhat more robotic intonation vs someone who puts emphasis on certain parts of each word. Still, they nailed the sound/uniqueness of his voice pretty well.
Somewhat in my defense, although I've seen a few clips with Rogan, I'm not very familiar with him. I agree with the GP's comment, I thought the real Joe Rogan reading ads sounded very staccato and robotic. Near the end I got a little better with the "OK, robotic sounding Joe is the real one."
That said, I'm an introvert and I don't enjoy social situations with people I don't know, so while perhaps not a psychopath it's plausible the my poor ability to infer emotional intonation is at play both here and in social situations.
However, as I mentioned in this comment[1] I don't think you have much to worry about. The correct criteria for diagnosing fake vs real is actually not being robotic. I knew immediately what was real because there was emphasis and affectation, as opposed to the monotone counterpart.
Finally, a minor quib: "psychopathy" is not about an ability to infer intonation (quite the opposite). I'm an introvert like you (Hell, I'm probably considered a hermit) and I was easily able to identify the doppelganger.
[1] https://news.ycombinator.com/item?id=19951679
What does introversion and being a hermit have to do with psychopathy?
I also don't want to make people feel like they're necessarily terrible people if they can't. Sorry if that's the way it came across.
Thinkint about this: It seems that culturally we are comfortable telling people feel like they're dumb (ie SATs, grades, uni admissions) or unathletic (competitive sports is a big part of growing up)... But we don't say anything about moral character.
I never had a coach tell me I am not athletic or a teacher tell me I am dumb.
Explained musically: the fake one goes from C to C# instantly, and the real one sweeps through all the frequencies in between in order to switch tones.
Those psychopaths who are "bad" at determining authenticity versus those who are "good" at it (mostly-wrong as opposed to mostly-correct, respectively)?
I'm hesitant to agree with that; from my understanding, a "psychopath" (current medical definition is Antisocial Personality Disorder, ASPD) is one who is outside of social norms and mores. Much like a precise machine, they are able to attenuate their focus to the nuances most neurotypical people take for granted.
Although far from a medical professional, I would hazard to guess an autistic person would be one of the groups you might attribute to supposed psychopathy (specifically: those unable to diagnose fake from authentic). However, even that claim should be taken with an ounce of salt. Point being, I get triggered when people use psychopath willy-nilly!
I understand people commonly intend empathy to convey 'well intentioned' or 'harboring feelings to understand and help others,' but I thought the actual meaning is the first sentence of your comment?
But again, I shudder to think how technology like this could be used to malign people.
Or rather, would it make maligning people impossible because you would never be able to tell if it's fake or real?
I would expect that pretty soon, we'll start seeing a lot of fake online/dating profiles with generated pictures and text. Probably a lot of chatbots posing as attractive women trying to scam people. Seems it should also be possible to condition a GAN to generate pictures that look like someone specific. These bots could try to pass themselves off as people you know.
People also seem to be wanting to get more drugged up, plastic, and bio-engineered.
Maybe in the future the average person is the meeting somewhere in the middle.
I'm worried we're going the way of Japan and its 'herbivore' culture. A world where people have given up on dating because it's too complicated. Other human beings are uncomfortable to interact with, so everyone retreats into their own little bubbles and gets more and more of their needs met by machines... On an individual basis, it a seductive idea to have an idealized robotic partner that does exactly what you want when you want it and doesn't have any needs of its own, will never leave you, but what will that do to humanity as a whole?
There's probably virtually infinitely many (just as life has infinitely many ways to kill an individual or group in pre-modern times).
Also, just as in prehistory, it would have taken visionary and moral heroes to build for their future's survival. But today, the role of those who have those concerns is not important in civilization anymore...
I cannot say what overall effect it will have just yet, but law enforcement will definitely have a harder time collecting evidence.
Unfortunately the UI/UX just isn't there yet
https://xkcd.com/1181/
https://www.explainxkcd.com/wiki/index.php/1181:_PGP
However - I guess the answer correctly with all but one - “You are much less likely to injure yourself if you do it correctly” - which I can now hear the different (or perhaps confirmation bias).
There’s something artificial about the timing with the pauses between sentences on the ML based ones that was the main giveaway for me, also while his voice is damn near spot on - there are some vocal inflections or perhaps “emotion” in a few words that also hinted to me.
Hope that helps.
I think the real tell is that Real Rogan has a lot of variance, Faux Rogan sounds too much like the average Rogan.
More likely is that you're helping them write the results section of their research paper, with a sentence like this, "In a blind trial of 5,000 internet users, over 93% of people were unable to tell the difference between generated audio and real audio at a statistically significant level (P<=0.035)."
Note: if we assume a binomial distribution, you need to get 7 or 8 correct to reach the magical P<=.05 barrier. If you assume that there are a fixed 4 generated clips and 4 real clips, then you need to get them all right (https://en.wikipedia.org/wiki/Fisher%27s_exact_test). I think it's fair to use the binomial distribution because the website does not tell you the number of real and generated clips before you take the test.
Seems like an ideal goal for them might be to have audible.com aquire them for a boatload of money.
Some of the British voices are close to perfect. If you want to listen at 2x then it's often better than a human because at speed, there is no value in dramatic intonation, instead impeccably consistent pronunciation and pace are what helps intake which would otherwise be grating at standard speed.
What is really nice is that when you start feel "beaten up" by a high pace reading, you can switch voices and brighten up the experience. I find switching back and forth between male and female voices helps avoid ear fatigue and keep up the pace. I immediately become more receptive to the content when a new voice begins.
EDIT: Still impressive tho. I imagine the typical use case for this sort of tech is not just trying to fool people who have been explicitly forewarned.
I imagine eventually, you could have some type of transcript that's annotated with speech synthesis markup language (SSML). Then, you have a CI/CD pipeline that would run this text-to-speech engine and regenerate the audio. I could then pair this up with the video. I honestly wonder if we are a year or two away from this being possible.
[1] https://cloud.google.com/text-to-speech/
> I honestly wonder if we are a year or two away from this being possible.
We've launched a service for adding voiceovers 2 months ago... https://wellsaidlabs.com/
Techcrunch: https://techcrunch.com/2019/03/07/wellsaid-aims-to-make-natu...
GeekWire: https://www.geekwire.com/2019/ai2s-incubator-gives-birth-wel...
> How can I do this with my voice?
To prevent abuse of our technology, we need to review your use-case before creating you a custom voice.
> The google WaveNet stuff is pretty good but still not there yet [1].
Google WaveNet is not built for high-quality voice-overs but rather for cheap and fast text-to-speech.
[0] https://arxiv.org/abs/1809.10460