Launch HN: Reality Defender (YC W22) – Deepfake Detection Platform
Recent advancements in machine learning make it possible to create images, videos and audio of real people saying and doing things they never said or did. The recent spread of this technology has enabled anyone to create highly realistic deepfakes. Although some deepfakes are detectable to the eye by experienced observers who look closely, many people either don’t have experience or are not always looking closely—and of course the technology is only continuing to improve. This marks a leap in the ability of bad actors to distort reality, jeopardizing financial transactions, personal and brand reputations, public opinion, and even national security.
We are a team with PhD and Master degrees from Harvard, NYU and UCLA in data science. Between us, we have decades of experience at Goldman Sachs, Google, CIA, FDIC, Dept of Defense and Harvard University Applied Research at the intersection of machine learning and cybersecurity. But our current work began with a rather unlikely project: we tried to duplicate Deepak Chopra. We were working with him to build a realistic deepfake that would allow users to have a real-time conversation with “Digital Deepak” from their iPhones. Creating the Deepak deepfake was surprisingly simple and the result was so alarmingly realistic that we immediately began looking for models that could help users tell a synthetic version from the real thing.
We did not find a reliable solution. Frustrated that we’d already spent a week on something we thought would take our coffee break, we doubled down and set out to build our own model that could detect manipulated media.
After investigating, we learned why a consistently accurate solution didn’t exist. Companies (including Facebook and Microsoft) were trying to build their own silver-bullet, single-model detection methods—or, as we call it, "one model to rule them all." In our view, this approach will not work because adversaries and the underlying technologies are constantly evolving. For this same reason there will never be a single model to solve anti-virus, malware, etc.
We believe that any serious solution to this problem requires a “multi-model'' approach that integrates the best deepfake detection algorithms into an aggregate "model of models." So we trained an ensemble of deep-learning detection models, each of which focuses on its own feature, and then combined the scores.
We challenged ourselves to build a scalable solution that integrates the best of our deepfake detection models with models from our collaborators (Microsoft, UC Berkeley, Harvard). We began with a web app proof of concept, and quickly received hundreds of requests for access from governments, companies, and researchers.
Our first users turned to our platform for some deepfake scenarios ranging from bad to outright scary: Russian disinformation directed at Ukraine and the West; audio mimicking a bank executive requesting a wire transfer; video of Malaysia’s government leadership behaving scandalously; pornography where participants make themselves appear younger; dating profiles with AI-generated pro pics. All of these, needless to say, are completely fake!
As with computer viruses, deepfakes will continue evolving to circumvent current security measures. New deepfake detection techniques must be as iterative as the generation methods. Our solution not only accepts that, but embraces it. We quickly onboard, test, and tune third party models for integration into our model stack, where they can then be accessed via our web app and API. Our mission has attracted dozens of researchers who contribute their work for testing and tuning, a...
92 comments
[ 2.9 ms ] story [ 38.4 ms ] threadMy questions are:
1 - Do you have any research on false negatives and false positives for your platform?
2 - How do you build trust in your platform so that users will use your results (and their users will trust it)? Fake news has been so widespread and people continue to believe in it, so why is that any different than with deep fake?
3 - Why are you trying a consumption-type of pricing? Cybersecurity typically charges on a per seat, and it would be very hard for a malware provider charge by 'malware detection'
I have submitted my resume: https://cphoover.github.io/
Have you also considered either offering a browser plugin to display contextual warnings attached to video elements? Or thought of working with browser makers? The web/social media is where a ton of fake media is propagated. It would be good if large platforms (fb/twitter/yt) integrated with such service, but also individuals should be able to protect themselves.
We are working with few partners (including Microsoft) who are interested in integrating our solution. We are focused right on supporting large organizations (companies and governments) that need to scan user generated content at scale.
I think "blockchain" (or even just a signed, immutable, public database) is mostly a solution in search of a problem, but I do think it may have an application here.
If you can hash a video when it's recorded and publish the hash with a timestamp that can't be forged, you can at least prove that this video existed at least as long ago as that stamp.
That allows you to invalidate any deepfakes produced on top of that video that have later timestamps.
It's not perfect, but it might be one weapon against this stuff.
Providing api access to a signed immutable database makes sense... but I'm not sure how much sense utilizing existing popular cryptocurrencies would make (e.g. bitcoin, etherium)
You probably don't need proof of work, you're right.
First, there are multiple solutions to this problem, all need to be explored and many will have a part in the future. Bad actors will do everything they can to find a way around every solution.
Second, for the hashing approach, have a look at "Perceptual Hashing", it's a way to hash content like a an image, even if the resolution changes.
Where to store the hashes? A centralized server is probably fine, but there is always a risk of a bad actor exposing it somehow. A secure blockchain can work better. But if you go that route, might as well go with the most secure blockchain. POW is generally more secure than POS. And currently the most secure blockchain is Bitcoin. So one solution is to batch hashes together and write them with a Bitcoin transaction in some cadence.
-Ali
Once they know it's edited, they can decide for themselves whether it's enjoyable satire or an attempt to deceive them.
The issue is with the hash, esp with lossy compression algos. If the video is re-transmitted somewhere, its content remains effectively "the same" (eg, it is that particular person, saying these exact things, at some specific time and place). A regular hash will not match as the video is different bit-to-bit though.
So for this scheme to work, we would need some kind of a "domain hash", something that is very precise in aspects that matter, and fuzzy in ones that don't. In case of videos, one dimension could be the text transcript. But even then it falls apart pretty quickly.
I would also like to ask three question. Do you know how well your model generalizes to video/audio deepfakes created by models that are not within your training sets? And also have you investigated whether your model can be used in a GAN setting to improve a deepfake generator towards creating better fakes? Or how robust your detectors are against adversarial attacks?
1 - We include multiple models for GAN and non-GAN related synthetic media.
2 - Models are only as good as the training data, and most training data breaks down in the real world because hackers have access to this same open source training data. So we create our own proprietary training data which we have automated, and we continuously update it based upon emerging deepfakes that we find in the wild.
3 - We target 95% accuracy with all public and proprietary training sets. And we continuously test and iterate both the data sets and the models.
4 - Our policies require a background check on all users to filter out bad actors. We additionally have technology safeguards in place to limit improper use.
IMO, KYC needs to go back to in person verification. Everything you can do digitally can be faked or impersonated.
And realistically, since deepfake detection will inevitably be more expensive than captchas or antivirus scanning, this will be adopted by human-in-the-loop organizations for critical processes where threat scoring or moderation is already being applied.
That said - Reality Defender, please train your system on diverse human data sets, do not release models where ethnicity or gender (including gender identity) are nontrivially correlated with deepfake score, and have processes in place from day 1 to allow users to report suspected patterns of bias. The kafkaesque "prove you're not a bot" scenario envisioned by the parent poster is one thing for holistic human-in-the-loop verification processes, and another thing if it suppresses minority voices and minority access to government services.
It's like ADA Compliance lawsuits. I can't prove the AccessaBe or other "ADA Compliance" web tooling are generating these lawsuits, but their company would not exist without them. Why wouldn't they want more lawsuits?
https://paperswithcode.com/task/deepfake-detection#datasets
So back to Reality Defender, why shouldn't we doubt Reality Defenders positive or negative results? There would need to be a period of verification and testing that "proves" within a reasonable margin that it works.
The worst one so far was TruePic, who even gave me a certificate of authenticity and then kept that certification PDF online on their website until they heard about us openly mocking them, and then they blocked the photo not because someone noticed that it's fake, but because "User has violated terms of service", which was me mocking them.
https://en.wikipedia.org/wiki/Mock_object
Unfakeable and unbeatable.
So someone uploads a video, you sign it, they display video. If authenticity is in question check the signature.
Deep fake detection is intractable imo. Use cryptography instead.
Hell if you want to be thorough sign each frame and create an extension for YouTube and other providers to literally check to see if a given frame or period was altered.
We totally recognize deepfake detection is a big & constantly evolving challenge, but we don't see that as a reason to cede the truth to bad actors :)
@bpcrd what's the advantage of using a block-chain to store video fingerprints, to determine provenance, over say a highly performance-optimized immutable centralized or federated system?
Fun fact: The oldest blockchain pre-dates bitcoin....and started in the 1990's in the New York Times classified via a daily recorded hash value!
https://crypto.news/finding-the-oldest-blockchain-in-the-new...
Even if smartphone manufacturers start integrating digital signatures right into their cameras, you can use the smartphone to re-record a pre-recorded fake video. And I'm sure that with enough resources you can do something way more clever.
I really don't see how crypto can solve this problem. I don't think AI can either (for reasons already mentioned in this thread). It's something we'll have to learn to live with.
I doubt this is true. Most videos are likely disseminated from a handful of companies.
> Even if smartphone manufacturers start integrating digital signatures right into their cameras, you can use the smartphone to re-record a pre-recorded fake video. And I'm sure that with enough resources you can do something way more clever.
Even if a fake video was re recorded it would have a different signature than the “original”. Problem solved.
Literally cryptography is the only solution
I assume the intent is not to store solely a direct hash of the original video, but also the original file which can be fingerprinted, or the fingerprint itself that will be matched if a later duplicate is uploaded.
Fingerprint differing from hashing in that two non-identical, but similar files (e.g. an original video and deepfake) can have the same fingerprint, but differing checksums.
If you plan to release this, I will be your first customer.
Here's an alt solution to argue against:
1. The necessary PKI gets bootstrapped by social media companies, where deepfakes begin to seriously threaten their "all-in-on-video" strategy, and simultaneously look like an opportunity to layer on some extra blue-check-style verification.
2. Example: you upload your first video to twitter, it sees the AV streams are unsigned, generates a keypair for you, does the signing, and adds the pubkey half to your twitter account. (All of this could be done with no user input.)
3. The AV streams have signatures on short sequences of frames. Say every 24th video frame has a signature over the previous 24 frames embedded into the picture. Similarly, every second of audio has a signature baked in.
4. The signatures aren't metadata that can be easily thrown away. They're watermarked into the picture and audio themselves in a way that's ~invisible & ~inaudible, but also robust to downsampling & compression. This is already technically possible.
5. Since we're signing every past second of the AV streams, this works for live video.
6. Viewers on the platform see a green check on videos with valid signatures; maybe they even see the creator's twitter handle if this is a reshare.
7. Like all social media innovations, the other major platforms copy it within the next 6 - 12 months. POTUS uses it. People come to expect it.
8. Long run: the public comes to regard any unverified video footage with suspicion.
Why won't this deepfake solution come to pass in the long run?
This would work for videos that people want to acknowledge as their own, but they could just tweet out they own it for the same affect.
The issue this tackles is videos that people don't want to claim ownership of, e.g. if a video emerged of <insert politician here> kicking a child, the politician can't say "I haven't signed it therefore it's not mine", instead we need tools like the above to be able to say, "this is faked, do not trust it".
We hope a standard will be created and used by all digital content creation tools. But this will take time. And even then, bad actors will not be deterred. They will always find ways to create fake content and pass it as authentic. We want to be there to fight them every step of the way!
Apps like snapchat that let you apply filters could also add additional metadata, going through more hardware verified operations provided by things like Apple's secure enclaves, saying something like "X filter applied over frames N-M" and you can have an edit chain to go with the original video/photo.
With this infrastructure in place our phones could really serve as ways to certify reality itself.
- What do you use as input for the model? Does it use all the pixels in all the frames in the input video? How about the video's metadata (location, extension,...)?
- My biggest concern about fighting deepfakes is that they have a point to achieve where the line between reality and fiction is nonexistent. Namely, if a deepfake video of someone can be created to look exactly like a real one if that someone decide to record such a video, I imagine there would be no way to tell the deepfake video from the authentic one (since there is no difference between the two). Because of that, this looks like a losing battle to me, but maybe I'm just too pessimistic. Do you feel that it is a real problem? Do you believe it is such a long shot that we shouldn't be worried about, or even if things reach that point, there would still be tools in our arsenal to counter such technologies.
- The input to our models are image, video, and audio. Based on the model, we can use parts of the image (esp faces) or whole image. Yes, we also incorporate metadata for better detection.
- It's a fair concern. As quality of generative media increases, so does the sophistication of detection. Since, we fully understand how generative media is created, it gives our the leverage to reverse engg. Much like the anti-virus industry (wrt scanning), we'd need to be at the forefront of not only detection, but generation methods, re-learn models based on new generation methods, etc.
I want to comment (in ignorance because I don't know the techniques you are using) that there is more to detecting fakes or "misinformation" than just the digital attributes of the data. Specifically, confirmation from multiple sources, reputation of sources, and above all, consistency with a world model of how people behave. For example, if there's a video of Biden backing Putin, you could dismiss it as fake regardless of video attributes.
I think (and have been criticized for saying, why I don't understand), that education and emphasizing critical thinking are the biggest counters to fakes, not learning to spot them in feature space. I believe that whatever you make, sanity checks need to be a part of it, and not just blind flagging of true / false based on the digital attributes.
Thanks!
We plan to guide our users in this way, incorporating these different tools and providing general education.
Edit to include link to their website
Thank you!
Have you discussed / looked into sampling environmental radio noise at various frequencies and locations and then interpolating samples of them within the video and audio itself at recording-time?
(ideally along with some kind of near-unfalsifiable timestamp signals and/or device keys to confirm that "yes, this unique device was here at this time and the proof is within the pudding)
https://www.youtube.com/watch?v=e0elNU0iOMY
There is no way to prove 100% solution can exist. If a similar tool is used to verify the evidence of a video in legal setting, then it's just a matter of time an innocent person ends up behind the bars. The only way is for the society to adapt to the fake videos not to rely on a tool that can never be correct.
There's a guy on Reddit who restores old photographs in an interesting way. He says he finds bits and pieces in recent stock images, that he cuts and pastes and adjusts until it looks perfect. So it's a kind of deepfake HI (vs. AI).
Here's an example of his work: https://i.redd.it/eeoz0hvvwvo81.png
Maybe not your target market ATM -- although this talent could be used to make very convincing fake images.