Interesting. They train an image classifier to detect images that were generated by a GAN-trained CNN. I wonder if it could be possible to include this classifier in the training loss, such that the generated images fly under its radar as much as possible. If this makes sense, then I guess the cat-and-mouse game just gained another level. On the other hand, what the classifier is detecting could be a fingerprint of the CNN architecture itself.
(Full disclosure: I have only read the abstract so far.)
> Due to the difficulties in achieving Nash equilibria, none of the current GAN-based architectures are optimized to convergence, i.e. the generator never wins against the discriminator.
If I understand the terms used, it sounds like you're suggesting adding this classifier to the discriminator, to avoid detection. Since they are already failing to pass their existing discriminators, it seems like they could try to not be detected, but they wouldn't actually succeed.
Absolutely possible, might even be a good idea, but my expectation is that the results won't be robust: the fakes will be uncovered by a slightly differently trained classifier. Maybe even the same classifier with a different random initialization.
Sounds like overfitting the defense against classification. Would existing solutions to overfitting possibly fix this (though make such a network even more expensive to train)?
I wonder if these results hold when the CNN-generated images are converted to an analog medium and back to digital (say scanning a printout or taking a screencap).
If not, this might indicate that the fingerprints or artifacts left by the generators are not of the "perceptible" variety.
Also a discriminator trained from this experiment might be useful to train a more powerful generator.
Also, if the images are not recognizable as fakes by humans, then it's good enough. What would be interesting in going further than that? I actually see it as a feature if at the same time it's possible to prove when images are fake.
(1) We show that when the correct steps
are taken, classifiers are indeed robust to common operations
such as JPEG compression, blurring, and resizing.
(2) When using Photoshop like methods the detector performs at chance (is useless).
Off topic: Always define your abbreviations. To find out what CNN stands for here, you either have to read a comment thread on HN, or go to the paper and read the introduction. The linked page doesn't even mention neural networks. And as some other commenter here has mentioned, CNN has other more well known meanings than Convolutional Neural Networks.
This is a paper that was published in CVPR (Conference on Computer Vision and Pattern Recognition). In that context it is unambiguous that CNN means Convolutional Neural Networks.
Someone who scanned the front page and didn’t delve into these comments might infere that the American news network CNN artificially generate images for their news stories. That’s how I picked this up.
Where would you draw the line though? Would you want the same for e.g. HTML? HTTP? HN? YC?
I mean personally I'm all in favor of more usage - or even automatic insertion - of the `<abbr>` tag. Can probably be done with a browser addon as well.
"CNN generated Image" sounds like "Images generated by the Cable News Network, CNN" as if the corporation has some software/policy for editing and prepping images in a way they can be detected. It's not so absurd, photojournalism can be quite specific in rules.
The ability to classify photos by news outlet based on identifying their photojournalism rules through computer algorithms sounds like a remarkably clever idea.
I wonder if part of an issue is the generation gap? For older readers, I imagine that they're much more familiar with CNN referring to the Cable News Network. Whereas for younger readers heavily involved in tech, me included, we aren't as heavily tied to the former abbreviation, so CNN referring to neural networks comes more readily to mind.
It’s all about audience and confusion. My feel of the HN readership is that it’s a vey broad base of mostly technical backgrounds; international but US heavy. That puts the ones you list as perfectly reasonable, and CNN referring to the neural networks is usually ok.
I this case, however, there’s a conflict with the news network which could also plausibly be the subject of the headline. They have interenational recognizability, and have been using the acronym almost exclusively for years; it is effectively their name.
> Where would you draw the line though? Would you want the same for e.g. HTML?
The day someone uses “HTML” to mean “hyper-threaded machine learning” or whatever, yes definitely.
CNN was unambiguously used for the TV channel for decades now, of course some people are confused when one uses it to mean something else without warning.
I found CNN a bit confusing, even though I did guess it meant Convolutional Neural Net.
Perhaps my pre-caffeine morning brain is overly pedantic but Generative Nets use deconvolutions to generate images from latent codes, so using CNN rather than GAN (Generative Adversarial Network) is a bit confusing in this context.
CNNs are used by VAEs (Variational AutoEncoders, also generative) use convolutions to produce the latent codes and the discriminator (adversarial) part of GAN training uses convolutions.
I think Generative Networks ( or GNNs ;-) ) would perhaps have been clearer.
GANs use transposed convolutions to generate images. I haven’t seen anyone use a true deconvolution operation in deep learning. Not even sure it’s possible to invert the result of a convolutional layer, because it’s lossy.
"Transposed convolution" was called "deconvolution" for a short period of time. The authors apparently weren't aware that "deconvolution" has meant "the inverse problem of recovering a convolved signal" since at least the 70s.
Not really, I just thought that CNN (the news network) uses generated images in their articles. The topic about recognizing them, or even generating them, would make sense in CVPR.
If that were their justification, they wouldn't need to define the acronym in the paper. However, in the introduction section of the actual paper they do define CNN. But they use the acronym 9 times before defining it, which is what's kinda weird.
And the website isn't published in the CVPR, it's published on the internet.
It was drilled into us in university (engineering) that you spell out abbreviations and acronyms on first use, no matter how well known you think it is.
Some cases I've seen lately seem to forgo this not out of ignorance but as a form of eletism/knowledge gate keeping.
The same could be said for my biology education. But it never seems to stop those biologists publishing papers from invocing obscure acroynms and phrases without definition.
> Some cases I've seen lately seem to forgo this not out of ignorance but as a form of elitism/knowledge gate keeping.
It's a natural tendency for ingroups. Nearly any video game forum, or anything else that's full of hobbyists will ultimately contain posts that are absolutely full of acronyms. And they're impenetrable. Bear in mind, I'm not defending this behavior, and certainly not disagreeing with you.
Seriously? You want to be able to join a community you might not be familiar with and understand the lingo/jargon off the bat? Of course people will abbreviate things when the word/phrase is sufficiently long, commonly used, and the abbreviation mutually understood (brb, tldr, OOP, etc.). There is no elitism involved, if you don't understand something either Google it or ask. No reason to act snobbish because people cater the the most common denominator in a community rather than the lowest.
Not exactly, what I'm arguing against is writing something technical, intended to educate non-experts in the field (the experts don't need your report/paper/thing) and then not providing them with the information they need to background research the things you refer to in abbreviations or acronyms.
And I'm not saying this is _my_ solution, this was literally taught to me in engineering first year.
Oh no, I understand your point, no misunderstanding here. It's one of the reasons I keep a technical blog, as an engineer, if I learn the hardest thing I've learned this $timeperiod, I want to share it, as clearly and understandably as I can. I fully support and encourage knowledge sharing, and it hurts just that little bit more when I see it being hoarded.
I'll defend it. Not the elitism, but using jargon/abbreviations/etc. When you're writing something for a larger audience, you should of course target that audience. But when you're on the "inside" writing for people who already have the background knowledge, it's unnecessary friction to stop and think "what terms would a newbie need defined in this?" It breaks the flow of writing/discourse and is probably mostly not needed, because someone coming in not knowing the terms in play can either go look them up, or just ask in their own post. (Granted, this also depends on that being easy; either a jargon dictionary being available, or the forum members being friendly to newbie questions.) I think it's also understandable to apply a small amount of gatekeeping, insofar as that continual beginner questions in the middle of an advanced discussion are just a distraction. The answer to that should be directing them to a more beginner-friendly subforum, but FWIW I do understand why people sometimes act poorly out of frustration.
May be this is specific to I.T or Computer Science? Where there are thousands of abbreviations and acronyms which itself is often the name people use. SQL, DRAM, CPU, HTTP, SRAM, FPGA, URL, TCP/IP, UDP, NAT, DHCP, GPL, etc.
I mean if you are discussing technicals of Neural network you expect your audience to at least know CPU, GPU, and FPGA. And if you are discussing software development I hope I dont have to spell out GPL.
So I dont think it is a form of eletism/knowledge gate keeping. In the age of internet you can search those "acronyms" meant without the full name, which isn't something could be easily done 15 to 20 years ago.
In other industry such as Mobile Wireless Networking, those acronyms are often clearly spell out because there are comparatively little of it. FDD, TDD, MIMO, NR or LTE are often spelt out in full when they first use.
I agree with all of that, however, I still stand by; if you're writing a technical document/paper you should spell out all of them, it's just how I was taught.
It doesn't have to be a hard rule, but major topics of a subject should be spelled out, at least, then you're giving people something to work with in their web search.
> And if you are discussing software development I hope I dont have to spell out GPL.
I'm my university first-year CS class, a third of the students had never heard of GitHub. Now, that's easy to look up, and GPL seems to be a lucky acronym as well, but CNN certainly isn't. Expanding it the first time or adding a footnote costs you nothing, but people not right in your field or still learning tremendously. Someone who got their Master's in CS 10 years ago likely wouldn't have heard of CNNs at all, and neither would most new CS students.
A scientific publication in such a broad field with such a widely-applicable topic and one of the most clashing acronyms right in the title should most certainly at least expand their key terms.
Also, the convolution part is only a speedup thing. You can do very similar neural network operations without the convolution, except that everything will be much slower and you'd need a lot more memory.
It's not really "only a speedup thing" because the training process is different: as a CNN learns to (say) recognize dog-noses in the top left portion of the image, it's simultaneously learning to recognize dog-noses everywhere else too. A fully-connected MLP with the same layer structure doesn't have that property.
It's true that once you've trained your CNN you could make a non-convolutional NN that computes exactly the same things but less efficiently, but the point of an NN is not just what it can compute -- there are lots of systems that can, given enough parameters, approximate arbitrary functions well -- but how you train it.
Yes that's why I said "very similar". Without a convolution, you will have to replicate a lot of the network structure, and you'd have to train with shifted versions of your data. But fundamentally the convolution only gives an advantage in speed and memory, not in functionality.
Genuinely thought this was some reference to something on the topic of 'fake news' Abbreviations are great if you're using the term multiple time. Not as an intro.
I also thought it was a reference to a manner of distinguishing genuine Cable News Network (CNN) screenshots from doctored ones divulged by “fake news” outlets.
it’s a scientific paper from a computer vision conference, it would be absurd in that context to assume anyone reading it doesn’t know that it stands for convolutional neural network. they didn’t write this with Hacker News in mind.
Yes, with an abbreviation like CNN it is remarkably presumptuous to not define it in this article. I followed the headline specifically because it was in the title and I assumed it referred to the news network.
The character limit is 80. Many abbreviations on the front-page are almost certainly caused by the low character limit making it difficult to express concepts that don't have a singular word for them.
I think it depends upon the audience and the work being written. Anything on the level of a news article definitely should spell it out, but a forum post about some game can get away with the common acronyms used by the community. It is part of knowing your audience.
If this "universal detector" is now used as a discriminator and the original models are fine-tuned/re-trained then it will stop being a universal detector no?
As long as the underlying CNN math stays the same it would not matter, Uber worked on a coord2conv to make better image outputs from CNNs without the artefacts this method capitalises on.
To my knowledge, adversarial networks are actually two different networks, one “correcting” the output of the other one. On the other hand, CNN consist of just one architectural model that internally uses convolution.
Surely if you want you can train the network to produce images that are not easily detectable?
So:
1) train a network that can detect CNN generated images
2) train the CNN network to generate whatever you want, politicians in compromising positions, etc. but also add in weights against the the other network
3) Images won't be easy to spot...
People will obviously start writing CNNs that detect images that are generated obfuscated this way with CNNs, but still, it's all possible.
What you describe is exactly the way these models work!
Typically; a GAN (Generative Adversarial Network) consists of (1) the generator; a model generating images and (2) the discriminator; a model that learns whether images it is fed come from the generator or from the image dataset. The (gradient) information of how the discriminator made its decision is fed back into the generator, in order to help it learn how to generate more _real_ images.
The discriminator is what you describe in step 1, and the generator is your step 2.
My understanding from the discussion part, section 5 of the linked paper, is that the GAN could be modified so that the relative power of the discriminator and generator is fine tunned in order to generate hard to detect images, by giving more power to the discriminator in the final steps.
I have read the paper and there are plenty of useful references and points: related work, the 11 CNN based image generators models, and the discussion part.
But sadly I could not obtain a clear picture of what is the difference between their detector and a baseline one. There are some minor points and references about upsampling, downsampling, resizing, cropping and fourier spectra comparison across generators, but those seems to be just comments and comparison and not crucial points in the construction of the detector. Furthermore data augmentation doesn't play a big role, they say that it usually improves (a little) the detector.
As a math person I like to get some more meat from papers, but here it seems that little tricks allow then to win the game. Perhaps that is the way (little or no math involved) to make advances. Well, at least they say that shallow methods modify the fingerprint of the fourier spectra so that now you can't detect which is the generator of the image.
Perhaps the "universal word" was what captured my attention.
Does it matter that they are easy to spot when the damage they can do would be well underway before a trusted service invalidates the image?
I am coming at this from the angle of, who would use this type of service other than the courts? Certainly major news organizations could benefit but we have numerous recent examples where they have either run with CNN imagery but they have also purposefully run video and use images of similar events to portray the view they wanted for a current event.
Of course in the end, if the end game is to have news, image, and video, validation there will need to be more than one and in separate enough areas of the world to have some chance all would not be intimidated / infiltrated to the point they are not trust worthy
This is the premise of GANs. It seems to me that the tech already exists to make extremely hard to spot fakes using GANs, it's just that nobody has bothered to write the code to do it.
I’ve seen a number of attempts to identify deepfakes and other forms of manipulated images using AI. This seems like a fool’s errand since it becomes a never ending adversarial AI arms race.
Instead, I haven’t seen a proposal for a system I think could work well. Camera and phone manufacturers could have their devices cryptographically sign each photo or video taken. And that’s it. From that starting place, you can build a system on top of it to verify that the image on the site you’re reading is authentic. What am I missing that makes this an invalid approach?
I do understand that this would require manufacturers to implement, but it seems achievable to get them onboard. I even think you get one company like Apple to do this and it’s enough traction for the rest of the industry to have to follow suit.
101 comments
[ 682 ms ] story [ 2249 ms ] thread(Full disclosure: I have only read the abstract so far.)
If I understand the terms used, it sounds like you're suggesting adding this classifier to the discriminator, to avoid detection. Since they are already failing to pass their existing discriminators, it seems like they could try to not be detected, but they wouldn't actually succeed.
Could the classifier that they're using here be used as a discriminator in a GAN, to help train it to avoid this detection method?
If not, this might indicate that the fingerprints or artifacts left by the generators are not of the "perceptible" variety.
Also a discriminator trained from this experiment might be useful to train a more powerful generator.
Also, if the images are not recognizable as fakes by humans, then it's good enough. What would be interesting in going further than that? I actually see it as a feature if at the same time it's possible to prove when images are fake.
(1) We show that when the correct steps are taken, classifiers are indeed robust to common operations such as JPEG compression, blurring, and resizing.
(2) When using Photoshop like methods the detector performs at chance (is useless).
Also, let me quote the linked page:
> malicious use of fake imagery is likely be deployed on a social media platform
Nope, that couldn't possibly relate to any news network. Never never never gonna admit that!
I mean personally I'm all in favor of more usage - or even automatic insertion - of the `<abbr>` tag. Can probably be done with a browser addon as well.
https://trends.google.com/trends/explore?geo=US&q=%2Fm%2F0x2...
The ability to classify photos by news outlet based on identifying their photojournalism rules through computer algorithms sounds like a remarkably clever idea.
HTML doesn't have that ambiguity.
I this case, however, there’s a conflict with the news network which could also plausibly be the subject of the headline. They have interenational recognizability, and have been using the acronym almost exclusively for years; it is effectively their name.
We are not computer algorithms here. A human being can decide "yeah this sounds like cable news network" and use the long form of this CNN.
The day someone uses “HTML” to mean “hyper-threaded machine learning” or whatever, yes definitely.
CNN was unambiguously used for the TV channel for decades now, of course some people are confused when one uses it to mean something else without warning.
HN is suitable here because it can be assumed that Hacker News denizens are acquainted with a rather obvious shorthand for their own community.
YC... likely as above but might be safer explicated and explained.
Perhaps my pre-caffeine morning brain is overly pedantic but Generative Nets use deconvolutions to generate images from latent codes, so using CNN rather than GAN (Generative Adversarial Network) is a bit confusing in this context.
CNNs are used by VAEs (Variational AutoEncoders, also generative) use convolutions to produce the latent codes and the discriminator (adversarial) part of GAN training uses convolutions.
I think Generative Networks ( or GNNs ;-) ) would perhaps have been clearer.
Did they really hoped for that their paper will remain in a specific group of experts? I seriously doubt so.
And the website isn't published in the CVPR, it's published on the internet.
Some cases I've seen lately seem to forgo this not out of ignorance but as a form of eletism/knowledge gate keeping.
It's a natural tendency for ingroups. Nearly any video game forum, or anything else that's full of hobbyists will ultimately contain posts that are absolutely full of acronyms. And they're impenetrable. Bear in mind, I'm not defending this behavior, and certainly not disagreeing with you.
And I'm not saying this is _my_ solution, this was literally taught to me in engineering first year.
If you're writing a paper, define every acronym the first time you use it.
If you're in a forum with a set of acronyms known to all, define them in a sticky or the forum readme.
For example: images generated by convolutional neural networks (CNN) are easy to identify.
May be this is specific to I.T or Computer Science? Where there are thousands of abbreviations and acronyms which itself is often the name people use. SQL, DRAM, CPU, HTTP, SRAM, FPGA, URL, TCP/IP, UDP, NAT, DHCP, GPL, etc.
I mean if you are discussing technicals of Neural network you expect your audience to at least know CPU, GPU, and FPGA. And if you are discussing software development I hope I dont have to spell out GPL.
So I dont think it is a form of eletism/knowledge gate keeping. In the age of internet you can search those "acronyms" meant without the full name, which isn't something could be easily done 15 to 20 years ago.
In other industry such as Mobile Wireless Networking, those acronyms are often clearly spell out because there are comparatively little of it. FDD, TDD, MIMO, NR or LTE are often spelt out in full when they first use.
It doesn't have to be a hard rule, but major topics of a subject should be spelled out, at least, then you're giving people something to work with in their web search.
I'm my university first-year CS class, a third of the students had never heard of GitHub. Now, that's easy to look up, and GPL seems to be a lucky acronym as well, but CNN certainly isn't. Expanding it the first time or adding a footnote costs you nothing, but people not right in your field or still learning tremendously. Someone who got their Master's in CS 10 years ago likely wouldn't have heard of CNNs at all, and neither would most new CS students.
A scientific publication in such a broad field with such a widely-applicable topic and one of the most clashing acronyms right in the title should most certainly at least expand their key terms.
UK reports rampant student marijuana use before class
That headline has quite a different meaning if “UK” is abbreviating “United Kingdom” versus “University of Kentucky”.
"However, these methods represent only two instances of a broader set of techniques: image synthesis via convolutional neural networks (CNNs)."
[1] https://arxiv.org/pdf/1912.11035.pdf
It's true that once you've trained your CNN you could make a non-convolutional NN that computes exactly the same things but less efficiently, but the point of an NN is not just what it can compute -- there are lots of systems that can, given enough parameters, approximate arbitrary functions well -- but how you train it.
https://en.m.wikipedia.org/wiki/CNN_controversies
Edit: Although, I see it does in first use in the introduction, so maybe that's just conforming to whoever's style guide.
Would this be receiving as much attention if they had used "Convolutional Neural Networks" instead of just CNN?
So:
1) train a network that can detect CNN generated images
2) train the CNN network to generate whatever you want, politicians in compromising positions, etc. but also add in weights against the the other network
3) Images won't be easy to spot...
People will obviously start writing CNNs that detect images that are generated obfuscated this way with CNNs, but still, it's all possible.
Typically; a GAN (Generative Adversarial Network) consists of (1) the generator; a model generating images and (2) the discriminator; a model that learns whether images it is fed come from the generator or from the image dataset. The (gradient) information of how the discriminator made its decision is fed back into the generator, in order to help it learn how to generate more _real_ images.
The discriminator is what you describe in step 1, and the generator is your step 2.
But sadly I could not obtain a clear picture of what is the difference between their detector and a baseline one. There are some minor points and references about upsampling, downsampling, resizing, cropping and fourier spectra comparison across generators, but those seems to be just comments and comparison and not crucial points in the construction of the detector. Furthermore data augmentation doesn't play a big role, they say that it usually improves (a little) the detector.
As a math person I like to get some more meat from papers, but here it seems that little tricks allow then to win the game. Perhaps that is the way (little or no math involved) to make advances. Well, at least they say that shallow methods modify the fingerprint of the fourier spectra so that now you can't detect which is the generator of the image.
Perhaps the "universal word" was what captured my attention.
I am coming at this from the angle of, who would use this type of service other than the courts? Certainly major news organizations could benefit but we have numerous recent examples where they have either run with CNN imagery but they have also purposefully run video and use images of similar events to portray the view they wanted for a current event.
Of course in the end, if the end game is to have news, image, and video, validation there will need to be more than one and in separate enough areas of the world to have some chance all would not be intimidated / infiltrated to the point they are not trust worthy
Next, it is possible to have a test which detects those. And that test can be improved by better training.
Then, another AI learns how to synthesize images which the fake image detector AI can spot, until it learns how to fool the fake image detector.
Then the fake image detector is improved by training it against the improved fake image synthesizer.
Repeat.
Instead, I haven’t seen a proposal for a system I think could work well. Camera and phone manufacturers could have their devices cryptographically sign each photo or video taken. And that’s it. From that starting place, you can build a system on top of it to verify that the image on the site you’re reading is authentic. What am I missing that makes this an invalid approach?
I do understand that this would require manufacturers to implement, but it seems achievable to get them onboard. I even think you get one company like Apple to do this and it’s enough traction for the rest of the industry to have to follow suit.