Nor any of the follow-up articles I posted. Given onslaught of fabricated "news" that spread around the last election, this type of ML technology is almost guaranteed to play a role in the next one.
After watching the video from that link, I'm very fearful for what people are going to do with this technology in the next decade. This combined with Adobe's new "speech photoshop" will make excellent propaganda.
Well there is such a thing as productive and useful propaganda. Apple knows how to get ppl to line up for blocks to sell a 100 buck phone for 600. Disney's current formula of starwars/marvel/animation is so effificent at getting ppl to shell out cash that last year they basically told analysts they had enough resources and the formula to produce a star wars type movie every week, there are just not enough ppl to buy it. So knowing how to manipulate ppl is now quite scientific. It just has to be towards a constructive end. Something beyond religious/celebrity worship and consumerism and consumption.
With such power, if all it takes is one douchebag to cause massive harm very quickly, then you should also believe it is possible for person to do massive good very quickly.
>With such power, if all it takes is one douchebag to cause massive harm very quickly, then you should also believe it is possible for person to do massive good very quickly.
I'm not sure that applies for this technology. What massive good could be done by making a fake video of someone talking?
Could undermine terrorist organizations and 3rd world dictatorships with it. Spreading a fake of Kim Jong Un saying something embarrassing in North Korea, for example. Unfortunately I think in the long run this is more likely going to be used against us effectively rather than something we can use to destabilize our enemies.
That last part doesn't follow at all. It's much easier to destroy than to create. The ability to quickly produce something harmful doesn't imply the ability to quickly produce something beneficial. We have the technology to level an entire city in an instant, but building one still takes decades.
What we really need is simple, preferably free, tools to do this, so kids can start playing with it at an early age. That way the technology can lead to healthy skepticism.
On the other hand, there's a difference between being doubtful something is real, and being aware that people speaking in public have an agenda, and a large portion will utilize half-truths and lies more often than not.
The difference between "the President lied about WMDs and got thousands killed over it", and "maybe the president didn't actually say that - archive footage might be false".
I'm not surprised that a forum full of people who work in the information technology industry cannot see the epistomological failure of mass media, but these problems are not new, and the idea that nothing communicated through a screen can be trusted will most definitely cause more good than harm outside of the very near future.
Democratic institutions not only functioned but were put into effect before the advent of recorded or transmitted audio and video. Mass media has already been distorting the truth for over a century.
"Trust nothing you read online" becoming commonplace is something to be celebrated!
There's also another possibility: that media literacy will take off, and children will learn to verify sources. This was the original proposal early blogospherists: that ordinary citizens would learn to do journalism so as to keep microcommunities informed.
This is in fact happening, to everyone's benefit. Only in matters of vague mass policy (do we need giant walls? can markets sell health?) has it been largely corrupted by corporate propaganda. Which is probably not terribly worse than before. That scale has always been a big enough target to make it worth the effort to corrupt.
I agree with you in that mass media has been, willingly or otherwise, distorting the truth(s), but the easier and cheaper it is to "create fake information" the higher the verification bar for the public.
Right now, faking an interview via real-time 3d rendering plus capture of facial expressions would cost a ton of money. So anyone trying to do so would need (for e.g. to damage a competitor's reputation) both deep pockets and something valuable to gain from it (and would need to factorize the cost of getting caught, etc).
If this tech goes mainstream in the future, the cost would be so low that I can imagine some nasty things happening.
Hopefully there will be ways to detect this as this technique is further explored, so I'm not saying it's a "catastrophic development", but it certainly made me think more than just "oh wow, this is very neat!", which it also did by the way. It's incredible the kind of things researchers are able to achieve.
Very simple: a video or data set that has not been verified with a private key by the issuing party on a Blockchain will not be taken for serious and be assumed to be a fabrication
That would make some things we currently rely on - like voice recordings as evidence of criminal acts nearly impossible. No one is going to sign their crimes into a blockchain.
What's the alternative? If it becomes trivial to fabricate voice recordings in general, surely you wouldn't advocate that we continue allowing them as evidence?
Wow, that's absolutely incredible -- thank you for sharing.
Aside from the negative or comedy use cases, I could see this being used in video games, and probably other similar applications for narratives; based on the methodology it sounds near trivial to sync it up to any audio. Really incredible work -- love seeing this kind of research, which I've been much more exposed to doing while gamedev than I ever had been when doing web/app dev.
Obama speaks more clearly, more calmly and more fluid than Trump, who speaks with more stuttering, filler words and with more intense facial expressions.
Nothing about either is harder in principle to duplicate. You would just take different approaches to designing the algorithms or if using deep learning be sure to include the relevant data in the training data.
I look forward to a time when journalists doing journalism is what's valued in the news.
When technology to fake things is everywhere, maybe we'll pay for accurate newspapers again. With something other than our willingness to be manipulated by advertising.
> When technology to fake things is everywhere, maybe we'll pay for accurate newspapers again.
We didn't pay for them before, how could we do so again? Advertisers have always been the main funding for newspapers and sensationalism (clickbait before clicks) and propaganda have always been major influences on the medium.
Sure, things got worse for decades leading up to the arrival of internet news, and continued to get worse after that, but the golden age of readers footing the bill for accurate newspapers never happened.
This is kind of a slap in the face to anyone who's considered themselves a journalist, as well as anyone who bought a newspaper for the news. It also makes me wonder how the news industry ever got started if it never sold "news", but only something wrapped in that veil.
I don't know anything about the history of news; do you have any good reading material to recommend?
Problem is, there are always moments journalists aren't present for. Think Romney's "48%", Hillary's "deplorable", Trump's "grab em by the pussy". In all of those situations the video evidence is what proved the story. Without that all a journalist can do is trust the accounts of the people present, which aren't trustworthy at all.
You assume that news has ever been accurate. I would argue that the "news" in the past was not more accurate. The revolution that has happened is that we are much better at detecting fake news.
"A person said something? That's terrible... let me see for myself what he actually said... oh wait he said that in completely different context and didn't mean what was implied in the news at all." - Anyone who has checked primary source after reading a news article
We are no longer at the mercy of "journalists" to tell us what to think about what happened. They can be called out on their lies and misrepresentation as the same raw videos of what happened can be seen by people first-hand. How about asking person in question to clarify what they meant? We can do that too.
The news organizations, as a defense, have been publishing the "news" as "opinion pieces", "satire" and "comedy" which they can more easily walk back on when called out.
All this because they have an old population who hasn't figured out that one can actually be their own journalist a lot of the times by following people who are much closer to different sides of situation than "journalists" will ever be... and the reported news is ALWAYS a completely different version of what actually happened. When this population dies out, the mainstream fake news as we know it, will die with them.
I wonder if it's easier to make a fake speech of someone who talks the way Trump does? Not making this a political thing, but I've noticed that he's not very eloquent and frequently starts and stops in his speech and changes what he's saying mid sentence. Might it be easier to synthesize that type of speech pattern since any awkwardness can be hid in the erratic speech the real person employs?
I've seen claims that translators struggle with Trump because of this - accurately translating what he says comes out as incoherent, which makes people think that the translator is doing a bad job.
Although I think to some extent he relies on fragmented speech. It's all about emotional pattern recognition, rather than something that people engage with rationally. Tony Blair also tended to do this, with long verbless sentences.
yeah. The Daily Show has got a pretty standard formula. I read an article about how they, and their spin off shows like John Oliver, basically use the same script pattern for most of their shows in order to hypnotize their audience.
I've written naive markov chains that appear way more coherent than Donald Trump on a bad day.
Politics aside, I think it is amazing that anyone would vote for the guy - I suppose it indicates how great a distance there is between the average voter and the average politician, if most can't seem to tell the difference between those that are able to string together coherent sentences and paragraphs, and those that don't.
Perhaps a stack-based representation of speech would work well? There was an interesting post on the English language StackExchange that talked about Trump's speaking style and how, due to its more off-the-cuff structure, ends up nesting various subjects in layers, pushing and popping from the stack:
Forget "hacking elections". A botnet will be able to hack our trust in one another (see CIA reputational attacks), AI will be used to chat up girls online better than any person (see fb AI sales bots), and so on.
Computers can already beat us at Chess, Go, etc. How much different is humor, honor and reputation once companies add one more breakthrough to deep learning to model them?
An attacker that can make 100,000 jokes a second each of which is excellent? The missing breakthrough is how to automate the "human judging" factor. This is the problem when figuring out diets or treatments etc. Clinical trials take a long time. Same with textbooks.
Once we figure out how to speed that part up, we are going to be able to make AI that knows what's probably going to be funny ahead of time.
Isn't this similar to Adobe project Voco? If not how are they different. There is also another tool called Face 2 face. I think overtime people will assume everything to be fake unless it agrees with their pre-conceived notions of truth.
Project Voco creates speech based off new text for a person, this creates video based off speech. When these technologies get better I assume you'd be able to use Adobe's technology to fake a speech from someone, than use this to create a video for it.
This is the wrong way to think about this, I think. Think of media consumers, instead, as Bayesians. We have prior beliefs, we consume media, and our posterior beliefs are a function of both of those. As the process iterates over time, our priors are updated, but more slowly over time.
This is why children believe everything they hear from authority figures, while those with memories e.g. tend to doubt the offered justifications for the new war when we consider how similar they are to the previously-offered-but-eventually-disproved justifications for the previous wars.
Perhaps...but this has been my observation recently. A persons ideological position is always used as a gauge to judge the quality/value of his opinion on a particular topic. If a person is of/ perceived to be of an ideological disposition different then mine and then whatever he or she says gets discounted. Your argument about people being Bayesian assume that they are rational. In practice, you find people sticking to their prior beliefs even more strongly even when the evidence to contrary is stronge, so called backfire effect. In fact, a lot of people reject the evidence simply based on the Ideological disposition of the source that it comes from and terming it as XX wing conspiracy.
I guess that we will start to rely on multiple recordings of an event to consider it proven. Until, of course, 3D recreations start to appear rendering that useless as well.
Seeing is believing they say but what did people do years ago when they only ever caught a glimpse from behind a large crowd of their future president talking out the back of a caboose on the literal whistle-stop tour? Some people say they saw Abe Lincoln once, some people were even right.
There's been a growing fear of how much dirt is going to get spilled when the current generation with omnipresent camera-phones goes to run for office. But this signals that we could be nearing the end of the age of embarrassing leaked video or audio!
From now on, it could be plausible to deny a video and say someone built a neural net and faked it - 'at least I have no recollection of those events.' (The excuse will work for supporters)
The Mitt Romney's '47% are takers' could go down historically as the last great leak where leaks could be believed.
This demonstrates an immediate need for a trustworthy video hosting service--maybe with a companion app that records using some combination of crypto/proprietary formats/trust networks/I don't know what else, this isn't my area of expertise. Things are going to get pretty bad here soon if video evidence loses its street cred.
But how is it considered trustworthy? That's basically the central struggle every media organisation has right now and I don't see an easy answer.
Yours is an interesting idea, though. I dread to think how much it would have to exclude though - probably have no Android client because I'm sure a rooted phone could easily provide a fake video source. Maybe even on a jailbroken iOS device too. And once that becomes possible the entire platform is ruined.
Yeah, it might need to be a custom-made camera, made for news organizations and wealthy vloggers. And each device would need to be audited regularly. It would be a nightmare but not as much of a nightmare as not knowing whether anything you see on video is real or computer-generated.
Why do you need all this? Just authenticate the video file with a digital signature. It is even possible to build it into the video encoding itself, and I am almost certain there are already research papers on this, if not implementations.
I'm truly frightened by this. We are still struggling with how to deal with the "post-truth" world, and that's with the assumption that pictures are hard to fake, video even more so, and audio nearly impossible.
Fake news is going to reach a fever pitch when "speeches" of Obama leak saying, "We have to take all the white people's guns". And conversely, when a genuine "grab 'em by the pussy" leaks again, a huge chunk of people willfully will not believe it.
Seems possible that we could come up with technological and journalistic solutions, given enough time, but it's moving too quickly.
Yes, I think this may mean the end of any kind of political centre in the US. The different sides will simply have entirely different coverage of the same events, and will entirely refuse to believe each other's claims. Only a small distance from the current situation. But I can't see how people would voluntarily move back from that brink.
From the material posted it seems that this is basically a high-tech, convincing version of those late-night sketches where they have a fake version of their mouths moving; it doesn't actually solve the audio problem.
Yes but you can also track down the original video to prove something is fake. In order to have something truly convincing you need a video of Obama (or anyone else) that no one else has.
If you've ever gone through the pain of trying to convince someone why some picture or video is an obvious fake then you know just how terrifying this technology is.
Now we just need some way to digital sign speeches (and hopefully eventually with quantum cryptography).
I'm not sure how but perhaps this could actually improve the issue of fake news with or least the assimilation if there as a broader realization how easily things can be faked... probably not though (I'm eternally an optimist).
Looking ahead 10-20 years, I don't see how anyone born from 2015 onwards has any solid concept of reality the way I feel I know it. Between things like this and AR, I feel like "real" vs "simulated" will seem to the AR natives to be a pedantic distinction... Like how my parents still distinguish between "having met someone" and "someone I have talked to on the internet."
I think it's a bit of a stretch to claim that because such an exchange is plausible there is no meaningful distinction between online and in-person communication.
I really don't think there is much of a difference anymore. I remember meeting people online and hanging out with them (talking, working on projects together, playing games, etc) and it honestly didn't feel any different than meeting someone in person and doing the same thing despite the fact that I have never seen some of these people in person.
I find it especially difficult to differentiate a meaningful distinction between the two when my interactions with people go from in person to online or vice versa. We just do the same stuff we did normally, albeit maybe those things occur in different settings such as at a computer instead of face to face at a restaurant or wherever.
I wonder if eventually the SNR of electronic content will simply be so low that any communication of nontrivial value will simply take place over physical channels (e.g. face-to-face and trust networks), as it has for all of human history but the modern era. Or we truly embrace cryptography as a people and use it to build such networks of trust in the digital realm (e.g. news videos cryptographically signed by reporters and news organizations).
Speaking of which, the notion that one can possess irrefutable audio-visual evidence of an event is also a very recent one. Until the 19th century absolutely any news item traveled word-of-mouth (or ink) through trust networks and were absolutely subject to manipulation.
If this technology merges with conversational AI improving, I think we could be in trouble. I remember that article that used machine learning on someone's chat messages who passed away and had some interesting conversations based on what the machine learning process could create that wasn't perfect but somewhat believable. Pair that with this tech and we are on the road to passing the Turing Test.
If you were able to make it work in real-time you could do some really cool stuff with characters in multi-user VR spaces. Your avatar could be shown to move its mouth in a realistic way as you spoke to other users, which would go a long way towards building a feeling of "realness".
It makes you wonder just how far fake content--and fake content involving real people--can go.
Imagine a world where a service exists to which you can upload a dozen images of someone, along with a voice clip. In response, it can generate all kinds of videos--from the benign, to the person saying horrible racist things, to the person starring in graphic pornography.
It seems technically feasible in the medium term. But how do we react to it? Strict limits on the production or storage of these pseudo-artifacts? Criminal penalties for distribution? A cultural rejection of pretty much all video and audio evidence?
Oh man, it's gonna get way crazy. Imagine the 'mean girls' in HS with this tech. Imagine trying to be a male HS teacher with this around. We're gonna have to regulate the shit out of the internet just to keep teachers in schools and verify that they did not in fact cuss out a classroom and chug vodka.
112 comments
[ 2.4 ms ] story [ 171 ms ] threadNor any of the follow-up articles I posted. Given onslaught of fabricated "news" that spread around the last election, this type of ML technology is almost guaranteed to play a role in the next one.
With such power, if all it takes is one douchebag to cause massive harm very quickly, then you should also believe it is possible for person to do massive good very quickly.
I'm not sure that applies for this technology. What massive good could be done by making a fake video of someone talking?
On the other hand, there's a difference between being doubtful something is real, and being aware that people speaking in public have an agenda, and a large portion will utilize half-truths and lies more often than not.
The difference between "the President lied about WMDs and got thousands killed over it", and "maybe the president didn't actually say that - archive footage might be false".
But now that I think about it, the most damage will come from making people even more incredulous... of everything.
It will become even harder to use evidence to prove a point. Uninformed people will just say "yeah sure, it's probably faked".
On the other hand... who could blame them?
Democratic institutions not only functioned but were put into effect before the advent of recorded or transmitted audio and video. Mass media has already been distorting the truth for over a century.
"Trust nothing you read online" becoming commonplace is something to be celebrated!
This is in fact happening, to everyone's benefit. Only in matters of vague mass policy (do we need giant walls? can markets sell health?) has it been largely corrupted by corporate propaganda. Which is probably not terribly worse than before. That scale has always been a big enough target to make it worth the effort to corrupt.
Right now, faking an interview via real-time 3d rendering plus capture of facial expressions would cost a ton of money. So anyone trying to do so would need (for e.g. to damage a competitor's reputation) both deep pockets and something valuable to gain from it (and would need to factorize the cost of getting caught, etc).
If this tech goes mainstream in the future, the cost would be so low that I can imagine some nasty things happening.
Hopefully there will be ways to detect this as this technique is further explored, so I'm not saying it's a "catastrophic development", but it certainly made me think more than just "oh wow, this is very neat!", which it also did by the way. It's incredible the kind of things researchers are able to achieve.
Just like (to pick a hopefully safely historical example) no president would record himself conspiring to perform illegal activities?
Aside from the negative or comedy use cases, I could see this being used in video games, and probably other similar applications for narratives; based on the methodology it sounds near trivial to sync it up to any audio. Really incredible work -- love seeing this kind of research, which I've been much more exposed to doing while gamedev than I ever had been when doing web/app dev.
what is old is new again
When technology to fake things is everywhere, maybe we'll pay for accurate newspapers again. With something other than our willingness to be manipulated by advertising.
We didn't pay for them before, how could we do so again? Advertisers have always been the main funding for newspapers and sensationalism (clickbait before clicks) and propaganda have always been major influences on the medium.
Sure, things got worse for decades leading up to the arrival of internet news, and continued to get worse after that, but the golden age of readers footing the bill for accurate newspapers never happened.
I don't know anything about the history of news; do you have any good reading material to recommend?
"A person said something? That's terrible... let me see for myself what he actually said... oh wait he said that in completely different context and didn't mean what was implied in the news at all." - Anyone who has checked primary source after reading a news article
We are no longer at the mercy of "journalists" to tell us what to think about what happened. They can be called out on their lies and misrepresentation as the same raw videos of what happened can be seen by people first-hand. How about asking person in question to clarify what they meant? We can do that too.
The news organizations, as a defense, have been publishing the "news" as "opinion pieces", "satire" and "comedy" which they can more easily walk back on when called out.
All this because they have an old population who hasn't figured out that one can actually be their own journalist a lot of the times by following people who are much closer to different sides of situation than "journalists" will ever be... and the reported news is ALWAYS a completely different version of what actually happened. When this population dies out, the mainstream fake news as we know it, will die with them.
Although I think to some extent he relies on fragmented speech. It's all about emotional pattern recognition, rather than something that people engage with rationally. Tony Blair also tended to do this, with long verbless sentences.
Hold on, I'm going to try to find the article.
edit: well, the only source I can find after a quick search is this post on an anonymous message board, but it covers the main points pretty well: http://img.4plebs.org/boards/pol/image/1479/14/1479145599055...
a more detailed discussion is here: http://archive.4plebs.org/pol/thread/98690960/
Really??? That's rich coming from /pol
Politics aside, I think it is amazing that anyone would vote for the guy - I suppose it indicates how great a distance there is between the average voter and the average politician, if most can't seem to tell the difference between those that are able to string together coherent sentences and paragraphs, and those that don't.
https://english.stackexchange.com/questions/311381/donald-tr...
Seems like the kind of verbal structure that the puzzle-minded among HN might actually enjoy piecing together actually!
Soon, video and audio of an event or speech be proof of anything.
The only way to prove identity will be to have a device which can do challenge-response.
Without it, you won't be able to prove you're not a robot over the internet.
https://news.ycombinator.com/item?id=14787882
Forget "hacking elections". A botnet will be able to hack our trust in one another (see CIA reputational attacks), AI will be used to chat up girls online better than any person (see fb AI sales bots), and so on.
Computers can already beat us at Chess, Go, etc. How much different is humor, honor and reputation once companies add one more breakthrough to deep learning to model them?
An attacker that can make 100,000 jokes a second each of which is excellent? The missing breakthrough is how to automate the "human judging" factor. This is the problem when figuring out diets or treatments etc. Clinical trials take a long time. Same with textbooks.
Once we figure out how to speed that part up, we are going to be able to make AI that knows what's probably going to be funny ahead of time.
This is the wrong way to think about this, I think. Think of media consumers, instead, as Bayesians. We have prior beliefs, we consume media, and our posterior beliefs are a function of both of those. As the process iterates over time, our priors are updated, but more slowly over time.
This is why children believe everything they hear from authority figures, while those with memories e.g. tend to doubt the offered justifications for the new war when we consider how similar they are to the previously-offered-but-eventually-disproved justifications for the previous wars.
More technology demos should have unscripted, sincere reactions like that.
I'd agree with you if you scratched like that.
See here for an opinion on this demonstration: https://news.ycombinator.com/item?id=13002787
More technology demo should have unscripted, sincere reactions like that.
Kill Process by William Hertling Link: http://amzn.eu/c1ZJNcv
Seems it was closer to non-fiction than I expected.
It was Darknet by Mathew Mather.
Oops
The only solution to this is a education towards knowledge masochism. Destroy what you believe in, only in failure there is truth.
I think we've gone past this point already.
From now on, it could be plausible to deny a video and say someone built a neural net and faked it - 'at least I have no recollection of those events.' (The excuse will work for supporters)
The Mitt Romney's '47% are takers' could go down historically as the last great leak where leaks could be believed.
Yours is an interesting idea, though. I dread to think how much it would have to exclude though - probably have no Android client because I'm sure a rooted phone could easily provide a fake video source. Maybe even on a jailbroken iOS device too. And once that becomes possible the entire platform is ruined.
https://lyrebird.ai/demo
Future looks grim!
Fake news is going to reach a fever pitch when "speeches" of Obama leak saying, "We have to take all the white people's guns". And conversely, when a genuine "grab 'em by the pussy" leaks again, a huge chunk of people willfully will not believe it.
Seems possible that we could come up with technological and journalistic solutions, given enough time, but it's moving too quickly.
I'm not sure how but perhaps this could actually improve the issue of fake news with or least the assimilation if there as a broader realization how easily things can be faked... probably not though (I'm eternally an optimist).
What if someone is filmed doing something wrong?
They wouldn't sign it, and anyone could just say the footage was faked.
I am not sure that that distinction is really gone.
I find it especially difficult to differentiate a meaningful distinction between the two when my interactions with people go from in person to online or vice versa. We just do the same stuff we did normally, albeit maybe those things occur in different settings such as at a computer instead of face to face at a restaurant or wherever.
Speaking of which, the notion that one can possess irrefutable audio-visual evidence of an event is also a very recent one. Until the 19th century absolutely any news item traveled word-of-mouth (or ink) through trust networks and were absolutely subject to manipulation.
E: maybe dubbing foreign films could work better
Imagine a world where a service exists to which you can upload a dozen images of someone, along with a voice clip. In response, it can generate all kinds of videos--from the benign, to the person saying horrible racist things, to the person starring in graphic pornography.
It seems technically feasible in the medium term. But how do we react to it? Strict limits on the production or storage of these pseudo-artifacts? Criminal penalties for distribution? A cultural rejection of pretty much all video and audio evidence?
We're going to need signed videos...